SECOND MEASUREMENT SM-002

The dataset behind 39 studies of Claude Code can see 1 in 11 of its pull requests

Markovian ProtocolMeasured 2026-09-30Status: sent to the authors 2026-09-30, response pendingDOI: 10.5281/zenodo.23123115

AIDev is the field's go-to dataset for studying AI coding agents, and 39 published studies use it to say how Claude Code behaves. To find Claude Code's pull requests it searches for a line Claude Code writes in commits, and skips the line it writes by default in pull-request descriptions. Over the same dates that search finds 16,065; searching for both finds 173,066. The ones it does catch run bigger than typical: a median of 736 changed lines against 447.

Disclosure: Markovian Protocol holds no financial position in any organisation named here, was paid by no one for this work, and showed it to no one before publication except the organisation measured. How we work.

What they said, what we found

Why the search misses most of them

Four of AIDev's tools are found by something they stamp on every pull request: a branch name like codex/ or cursor/, or a bot account like Devin's. Claude Code is found by a line of text in the description:

AIDev's searches, from the dataset paper's Table 2.

The line AIDev looks for, Co-Authored-By: Claude, is what Claude Code writes into commit messages. It only lands in a pull request description when the description happens to be built from commit text. The line Claude Code writes into descriptions is 🤖 Generated with [Claude Code](https://claude.ai/code) (later versions link to claude.com/claude-code).

Counting them

We ran each of AIDev's searches on GitHub today, over AIDev's own dates: each tool's start date to 24 October 2025, the last date in the dataset. Pull requests get deleted over time, so today's counts come out a bit lower than AIDev's. To measure that, we reran the searches for the other tools as a control. Every one comes back at 85–88% of what AIDev recorded, and Claude Code's comes back at 88%, right in line. So today's search is a fair stand-in.

Same searches, same dates, run 30 September 2026. Copilot is left out because its login check can't be done in search.

AIDev's search catches 16,065 of 173,066, or 9.3%. Allow for the same 0.881 drift and the full set was about 196,000 when AIDev was collected. That would put Claude Code fourth of six tools, behind Codex, Copilot and Cursor. The other tools' searches have blind spots of their own (head:cursor/ only sees pull requests from a cursor/ branch, for example), and we didn't test those.

The Mining Challenge used an earlier cut of the dataset, up to 1 August 2025, with 5,137 Claude Code pull requests.⁴ Over those dates the gap is wider: AIDev's search finds 4,320 today, and either line finds 64,017, 14.8 times as many.

Is the description line really Claude Code?

We read 120 pull requests carrying it, 15 from each month. 113 had the exact line. 4 had a variant: the line without the link, or a sentence saying the pull request was made with Claude Code. 3 we couldn't tie to Claude Code from what's there now, and one of those was a Dependabot pull request. So the line is right 94–97.5% of the time. These 120 came from search result pages, so they aren't a perfectly random draw.

Are AIDev's ones typical?

We drew 120 Claude Code pull requests at random from AIDev (105 still exist) and 150 at random from the full set. GitHub search only returns 1,000 results per query, so to sample the full set fairly we picked random one-hour windows and kept each with a chance proportional to how many pull requests it had. We drew 792 windows. 2 had more than 150 pull requests, so they're slightly under-weighted.

Random samples, both from 24 February to 24 October 2025. “Typical” is the median. The full set includes AIDev's own (about 9%), so the real gap is a little bigger than shown. Only 7 of the 150 carry the exact commit line, because search matches phrases more loosely than an exact string.

AIDev's pull requests are bigger, by commits and by lines. We didn't work out why a description ends up carrying the commit line. Merge rates differ by 5.6 points, but these samples are too small to tell whether that's real; it would take about 600 closed pull requests on each side.

Who this affects

Any study that reports a Claude Code count, share or growth curve from AIDev is looking at about one pull request in eleven. Any study comparing Claude Code with other tools on size or commit structure is comparing a slice that runs bigger than usual. For example, a task-stratified study reports “Claude Code leads in documentation (92.3%)” from 139 Claude Code pull requests in AIDev's popular-repository subset.⁵ Its authors flag the small sample. Those 139 also came through a filter that picks larger pull requests. Acceptance itself didn't differ detectably in our data.

The evidence

Fetched 2026-10-02T18:11:35Z. Copies and SHA-256 hashes: SHA256SUMS.

Agent           GitHub Search Query                     Start Date
Claude Code     is:pr “Co-Authored-By: Claude”          2025-02-24

is:pr "Co-Authored-By: Claude" created:2025-02-24..2025-10-24
  total_count: 16,063

is:pr "Generated with Claude Code" OR "Co-Authored-By: Claude" created:2025-02-24..2025-10-24
  total_count: 173,101

Disclosure timeline

Waiting on AIDev’s authors since 30 Sep.

What we can't be sure of

• Counts are from GitHub search today, not when AIDev was collected. The control ratios bound that drift but don't remove it.
• The description line can be turned off or edited, so even 173,066 is a floor.
• We didn't sample the popular-repository subset that many challenge papers use.
• We only sampled AIDev v4. AIDev's README source carries a commented-out v5 line (AIDev-7.6M, cutoff 31 March 2026) that isn't published yet.²

Since publication

2 October 2026. At least 39 published studies draw conclusions about Claude Code from AIDev’s slice, 24 of them in the MSR 2026 Mining Challenge; the October review lists them with the sample behind each finding. Two other teams reached the undercount independently: a census of 180 million repositories finds AIDev misses 79% of projects with Claude Code commits (arXiv:2606.24429), and a team building an agent classifier notes that every Claude Code PR in AIDev carries the line it was found by (arXiv:2608.00966).

Run it yourself

The scripts are also on GitHub: github.com/MarkovianProtocol/second-measurements/sm-002.

Scripts need Python 3, the GitHub CLI logged in, and for the AIDev sample pyarrow and huggingface_hub.

python3 counts.py         # the counts
python3 aidev_sample.py   # 120 AIDev v4 Claude Code PRs, seed 20260930
python3 union_sample.py   # 150 union PRs by hour-window rejection sampling, seed 20260930
python3 compare.py        # the comparison

be7d2e6f3faea12a9a83e80d2a1c50f0c476ec8037eada50bbdbcdfbf05ebede  counts.py
6a84658973658ceb7c4e8540e98cc7790c36e106a9ecf93cdcab6e64354bbb8e  aidev_sample.py
0c9e0aa3bca14719dfac3e83460d81672dff3e1a744f9c71bb846daa3b64fca6  union_sample.py
a20e9a0f5d82024d83dd55ee092202f879b5e3ce48c2ec6aa0c127e5ff00b8d8  compare.py

Search counts move as PRs are deleted, and the union sample depends on search results at run time. A rerun will land near these figures but not on them.

Reproduced from the public repository on 3 October 2026 on a clean machine: the counts land within 0.1% (16,060 and 173,135 against 16,065 and 173,066; GitHub search moves daily). The union sample takes about 50 minutes of search calls. Full table on the track record.

What you can do with this

• If you cite AIDev for a Claude Code result, say that the dataset sees about one in eleven of those pull requests and that the ones it sees are larger than average.
• If you want the full set, search commit trailers for the co-author line and the description line together; the queries are in the exhibits.

References

1. H. Li, H. Zhang, A. E. Hassan. The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003, 2025. Table 2 and §3.1. arxiv.org/abs/2507.15003.
2. H. Li et al. AIDev dataset, v4 (AIDev-2.7M). huggingface.co/datasets/hao-li/AIDev, tag v4. Accessed 2026-09-30.
3. MSR 2026. Mining Challenge. 2026.msrconf.org/track/msr-2026-mining-challenge.
4. H. Li, H. Zhang, A. E. Hassan. AIDev: Studying AI Coding Agents on GitHub. Mining Challenge paper, dataset cutoff 2025-08-01. github.com/SAILResearch/AI_Teammates_in_SE3.
5. G. Pinna, J. Gong, D. Williams, F. Sarro. Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance. MSR 2026 Mining Challenge. arXiv:2602.08915. arxiv.org/abs/2602.08915.

Cite as

@misc{markovian-sm002,
  author = {{Markovian Protocol}},
  title  = {How AIDev identifies Claude Code pull requests,
            and what the identification misses},
  number = {SM-002},
  doi    = {10.5281/zenodo.23123115},
  year   = {2026},
  month  = sep,
  url    = {https://markovianprotocol.com/measurements/sm-002.html}
}