SECOND MEASUREMENTS
Track record
Every public claim we’ve checked, whether it held, and what happened after. And, at the bottom, every mistake of our own we’ve had to correct, because a scorecard that only lists other people’s errors isn’t one.
The score so far
- 18 claims checked since 15 September 2026.
- 6 didn’t hold. 11 held. 1 we reproduced without testing the claim itself.
- 2 organisations fixed what we found (Google, whisper.online). 1 bug reported upstream (Homebrew).
- 7 corrections to our own work, listed below.
Round trips
The clock runs on everyone we write to. These are the ones who answered, fixed it, and said so. The times are theirs to keep.
| Paper | Who | Replied in | Fixed in | What they did | Seal |
|---|---|---|---|---|---|
| SM-004 | Kaveh Ranjbar, whisper.online | 82 min | 4 h | Republished the missing log head the same night, found a second, worse bug himself while fixing it and disclosed that too, then ran our test suite and reproduced every check before we published. |  badge |
| SM-001 | Billy Lau, Google Pixel binary transparency | 7 days | 7 days | Confirmed both findings, backfilled the 29 missing January images, fixed the cache eviction and the filing bug, and answered the witnessing question in full. |  badge |
A round trip is counted when the operator replies, changes what we measured, and the recheck confirms it. The seal is theirs to keep and embed; it carries the paper, the fix date and the log leaf, and links back to the paper. The badge is the one-line version.
Papers
| ID | The claim | Held? | What we found | What happened next |
|---|---|---|---|---|
| SM-015 | Chrome CT policy: logs include within the MMD, RFC 6962 MMD at most 4 h, shard windows at most a year; Apple: “usable” logs can be relied on | Logs held; the lists didn’t | 64 of 64 logs inside their MMD, tiled ones under 6 s; Chrome’s list declares 24 h against a 4 h policy, admits four 380-day shards; Apple’s list marks 12 closed shards usable, untouched since March | Sent to Apple and Chrome 3 Oct 2026 |
| SM-012 | CCADB: CAs upload audit statements within 92 days; Microsoft: audits “on an annual basis” | Didn’t hold | 11 of 119 statements past the deadline, 7 with nothing on file; 24 Microsoft-trusted roots with audits over 15 months old, one from 2019 | Sent to CCADB and Microsoft 3 Oct 2026 |
| SM-011 | Mozilla: CRLite’s builder has “total knowledge” of the public Web PKI; uncovered certificates are “likely new certificates and private CAs” | Didn’t hold | Three of the largest logs read to June, June and August; 68 of 726 top-site certificates not covered, up to 50 days old, from Google, DigiCert and Microsoft; 57 of the 68 were issued by Google Trust Services | Sent to Mozilla 2 Oct 2026 |
| SM-010 | Hugging Face: every file in every repository goes through a malware scanner at each commit | Didn’t hold | Every file under 2 GiB scanned (2,539 of 2,551); none at 2 GiB or more (0 of 487); 476 of them badged “safe” | Sent to Hugging Face 2 Oct 2026 |
| SM-009 | Mozilla: CRLite stores all certificate revocations locally; Firefox enforces it and skips the online check for what it doesn’t cover | Didn’t hold | 38 of 465 current revocations missing; 37 of 108 among certificates revoked within a day of issuance, for up to 28 days | Sent to Mozilla 2 Oct 2026 |
| SM-001 | Google: the Pixel log covers every factory image from Pixel 6 on | Didn’t hold | January 2026 (29 images) missing for eight months | Google confirmed and fixed it within a week |
| SM-002 | AIDev captures each coding agent’s pull requests | Didn’t hold | Its Claude Code search catches about 1 in 11 | No reply yet; two other teams have since reached the same result |
| SM-003 | Wild SBOMs are written by many practitioners | Didn’t hold | Nearly half came from one automated run | One author replied; no correction yet |
| SM-004 | whisper.online’s ledger proves nothing was changed | Held | Every proof checked; we found a duplicate-update bug | Operator confirmed and fixed the bug |
| SM-005 | Meta: the pulled-software list is published every 3 hours | Didn’t hold | Every 6 hours since January; whitepaper unchanged | No reply yet |
| SM-006 | x402 payment counts on Base | Not tested | Our count reproduced; traffic fell three-quarters | No reply yet |
| SM-007 | Docker: signed attestations for all Official Images | Didn’t hold | Attestations on 99.9% of Linux images; none signed | Sent 2 October; seven questions open |
| SM-008 | CISA: SSVC answers for every CVE ID | Didn’t hold for the Linux kernel | 1 of 158 kernel CVEs since March | Sent 2 October; six questions open |
Held, by this much
Claims that held are not all equal. Some had room to spare; some passed by a hair, and a hair is where the next finding comes from. The bar is how close the measurement came to the line: a full bar is a claim that held by a rounding error. The rechecks watch these first.
| The claim | What we measured | The margin | How close |
|---|---|---|---|
| PyPI: more than 20% of uploads via Trusted Publishing | 20.5% in our sample | 0.5 points above the line | |
| PyPI: 17% of uploads carry an attestation | 17.9% measured | 0.9 points above | |
| MCP registry: 4.2% of servers redirected | 4.02% measured | 0.18 points below the paper's figure | |
| Homebrew attests every bottle | 599 of 600 | one bottle short, reported and fixed by them | |
| Chrome CT policy: logs are “expected to accept” every root Chrome trusts | Cloudflare Nimbus2026 and Nimbus2027 accept 76 of 101; DigiCert’s Sphinx shards 95; 29 shards miss one; 31 miss none | an expectation, not a MUST; 25 CAs can’t log at Cloudflare (script, data) | |
| Chrome CT policy: a shard’s expiry range “must be no longer than one calendar year” | 60 of 64 shards within a year on 2026-10-04; 4 over | 14 days over, and Chrome accepted them anyway | |
| Apple: a log marked “usable” can be relied on for Apple’s CT policy | 12 usable shards with closed windows on Apple’s list as of 2026-10-04 (list last modified Tue, 31 Mar 2026) | no deadline in Apple’s text; Chrome moved the same shards within 8 days | "> |
| Baseline Requirements: a CRL’s nextUpdate at most 10 days after thisUpdate | 16,000 TLS CRLs: 90th percentile 9.96 days, 28% past 9 days | an hour inside the cap at the 90th percentile; CAs set it to the limit | |
| RFC 9286: a new RPKI manifest “MUST be issued and published before the nextUpdate time” | registro.br on 2026-10-04: 21 manifests past nextUpdate within the last seven days (170 in all, most abandoned years ago); RIPE hosted 0, ARIN hosted 1 | the registry’s own process allows five days; validators allow none | |
| Armored Witness log names its source | 27 of 28 | one tag that doesn't exist | |
| Debian blocks unreproducible migrations | 0 unexplained of 457 | 12 binaries crossed only because a release manager overrode the gate by hand | |
| Proton: an epoch every 4 h, never more than 72 | longest gap 38.1 h | used 53% of the allowance | |
| CT logs: merge within 24 h (classic) or 60 s (tiled) | 61 min; 6 s | used 4% and 10% of the allowance | |
| Firefox CRLite covers every log | 44 of 68 qualified logs in its coverage list | 0 of 465 certificates affected today; the margin is the CAs' choice of logs | |
| GitHub: public attestations go to a public log | 161 of 161 Actions attestations on Rekor | 227 release attestations in the same sample have no log, by design |
Green is comfortable, amber is within sight of the line, red is on it.
Checked, held
| The claim | Held? | Detail |
|---|---|---|
| Armored Witness firmware log names its source | Held, 27 of 28 | The recovery image names a tag that doesn’t exist |
| Cloudflare Plexi audits every WhatsApp key-transparency epoch | Held | 150 of 150 sampled |
| MSR 2026 dataset describes real-world MCP servers | Held | Keeps only repos with 50+ stars, unstated |
| PyPI: 17% of 2025 uploads carry an attestation | Held | 17.9% in our sample |
| 4.2% of MCP registry servers redirected their endpoint | Held | 4.02% |
| GitHub leaves reporter credit out of the CVEs it assigns | Held | 0 of 150 carry it |
| Homebrew attests every bottle its CI builds | Held, 599 of 600 | One bottle reported: homebrew-core#314985 |
| NIST enriches known-exploited CVEs within one business day | Held | 163 of 163 |
| Kubernetes signs all release binaries | Held | 408 of 408; 12 of 12 verified |
| Proton publishes a key-transparency epoch every 4 hours, never more than 72 | Held | 515 epochs; median 4.00 h, longest 38.1 h |
| Debian blocks unreproducible new packages and regressions from migrating | Held | 457 unreproducible binaries in testing, 0 unexplained; the one hard case was flagged and overridden on the record |
| Every Certificate Transparency log serves a new entry inside its promised merge delay | Held, 64 of 64 | Tiled logs within 6 s; classic logs 10 s to 61 min against a 24-hour promise |
Scripts for every row are on the measurements index and on GitHub.
What a stranger could reproduce
On 3 October 2026 every paper’s “Run it yourself” block was executed from the public repository on a clean machine by an agent that had not seen the work, with the packages the README names and nothing else. Eleven of thirteen reached the headline number or a result that holds on fresh data; the rest need something a stranger doesn’t have, and the paper says what.
| Paper | Result on a clean machine |
|---|---|
| SM-001 | exact (tree size 1,163, 963 logged, 128 under the generic label). |
| SM-002 | the counts land within 0.1% (16,060 and 173,135 against 16,065 and 173,066; GitHub search moves daily). The union sample takes about 50 minutes of search calls. |
| SM-003 | the live GitHub check holds on a fresh repository set (80 of 86 exported SBOMs carry links). The corpus steps need the 12 GB Zenodo table and were not run in the window. |
| SM-004 | holds at tree size 402,935 (three inclusion proofs fold to the served root, consistency from 251,620 verifies, four malformed requests get 400, the log signature verifies). |
| SM-005 | exact (2,701 epochs, median gap 4.0 h, longest 19.2 h; 37 of 70 weeks with a server entry). |
| SM-006 | exact (234,490 and 63,003 settlements; the same 15 and 5 wallets at about 80%). The per-payment value scan runs longer than ten minutes. |
| SM-007 | the probe holds (attestations present, no signature media type). The full census takes about 25 minutes and the signing sample depends on it. |
| SM-008 | exact (kernel 0 of 53, others 335 of 335, 1 of 105 by month, 5,024 kernel CVEs since the directive). The known-exploited pass takes about 15 minutes. |
| SM-009 | the CRL sample and the Firefox query tool build and run (1,320 CRLs parsed, 255,797 revocations in the window, example.com Good against 63 filters). The crt.sh lookups for 1,691 certificates take about an hour. |
| SM-010 | Reproduced from the public repository on 3 October 2026 on a clean machine, on a fresh sample: holds (0 of 688 files of 2 GiB or more scanned; 2,502 of 2,801 smaller files safe; 519 unscanned files behind a safe badge). |
| SM-011 | exact (mozilla.org NotCovered; Xenon2026h2 read to 12 June, 1.66 billion entries unread). |
| SM-012 | exact (119 statements, 11 late, 90th percentile day 92; Mozilla 0 of 172, Chrome 0 of 101, Apple 6 of 144, Microsoft 24 of 331). |
| SM-015 | the root and list results are exact (30 logs accept all 101 roots, four 380-day shards, 12 closed shards usable on Apple's list). The merge-delay measurement needs a publicly trusted certificate chain of your own. |
The fixes that run surfaced (inputs read from the exhibits folder, binary paths, runtimes) are in the repository as of that day. The pass is repeated when the scripts change.
Our own mistakes
| When | What we got wrong, and what we did |
|---|---|
| 3 Oct | We re-read all fifteen papers against their own exhibits, one adversarial pass per page, before anyone else did. It found errors on twelve of them, corrected the same day and listed here so the record is complete. The ones that changed a finding: SM-001 undercounted the missing Pixel images on 15 September (54 of 1,091, not 33 of 1,070; the 21 we left out were September builds we had filed as lag). SM-012 credited Thailand’s auditor letter to all three of its late statements when it covers one, so seven statements have nothing on file, not five. SM-011 said every one of the 68 uncovered certificates sat behind the lagging logs; five were simply under a day old. SM-007 said 50 repositories were checked for signatures; 45 were, and only 37 of them carried both an SBOM and provenance. SM-015 counted 64 Chrome logs timed; 61 were, plus three of Cloudflare’s from Apple’s list, and 30 logs accept every Chrome root, not 31. SM-006 called August’s payments “manufactured” where the study says operator-internal, and said the payers changed completely when a quarter of them stayed. Smaller slips on SM-002, SM-003, SM-004, SM-005, SM-008, SM-009, SM-010 and the October review (a table number, a date, a page reference, a rounding) are corrected in place; every corrected page is re-stamped, and the old bytes stay in the log. |
| 3 Oct | The round-trip seal we sent Kaveh Ranjbar printed its log leaf as plain text, and the number was already stale; SM-004 named the standard as RFC 6962 in one place and RFC 9162 in another. He pointed out both within the hour. The seal now links the leaf and follows every re-stamp; the page names one standard. The same night we found two pages carrying a third paper’s description tag, copied from a template; fixed and re-stamped. |
| 2 Oct | Before SM-009 went up, four bugs in our own pipeline each produced a wrong count: crt.sh timestamps shifted six hours by the database session’s timezone, millisecond rounding on embedded timestamps, crt.sh’s own log submissions counted as the certificate’s, and base64 line-wrapping corrupting certificate files. Each was caught by cross-checking against Mozilla’s query tool; none was published. Listed here because the first count would have said 7.9% instead of 8.2%, and been wrong about which certificates. |
| 20 Sep | Our own witness cosigned a size-0 checkpoint with a made-up root. Found while building our witness test suite; fixed the same day. |
| 23 Sep | We said Google’s Pixel log had no witness cosignatures. It does, through a separate distributor. Corrected in the public thread. |
| Sep | We said a log called apertrue was unwitnessed. It was on a list we hadn’t checked. Corrected. |
| 30 Sep | SM-001’s first log stamp didn’t match the page readers got, because Cloudflare rewrote an email link. Re-stamped against the public bytes. |
| 1 Oct | Our x402 script printed five facts about August as fixed text, so a rerun on another day would have shown August’s numbers. Fixed to measure them; four of the five held. |
| 2 Oct | The first edition of our October review understated what a co-authorship paper did about its own sample. Corrected the same day; the page says so. |
| 2 Oct | Four cells in the review’s first AIDev table were filled from memory, not from the papers. Checked against the papers and corrected before the authors were written to. |
Superseded versions of every paper stay in our public log, so a corrected page can always be compared with what it said before.