I replayed 21 merged pull requests. Three had bugs, and my tool got three wrong.

Matt du Toit, September 2026

In August I took 21 pull requests that had already been merged into three popular JavaScript libraries (path-to-regexp, p-limit and slugify) and replayed each one at the commit it was written against. Then I ran each one through forgehand, the merge gate I build. It checks whether a change actually did what it said it would, rather than only whether the test suite went green.

Sixteen different authors, none of them me. Every one of those pull requests was reviewed and merged by the maintainer. Every one of them passed its own tests.

forgehand let 15 through and blocked 6. Three of those blocks were right. Three were wrong. Both halves are below, because a tool that only shows its wins isn't evidence of anything.

The three it got right

All three are in slugify, a package downloaded about 6.6 million times a week.

#69, merged October 2022. To stop APIs from splitting into ap-is, the PR narrowed one character class in a regex from [a-z\d]+ to [a-rt-z\d]+. The new tests passed. forgehand's review said, verbatim:

This regex change causes severe regressions whenever an acronym is followed by a capitalized word whose second letter is 's' (e.g., APIUser, APIUsage, HTMLEscape, SVGAsset, JSONAssertion, XMLAsync, APIIssue) … The change only satisfies the specific test cases added (APIs, APISection) while breaking decamelization and slugification for standard camelCase patterns.

It was right about every one:

APISection  →  api-section   (the case the PR tested: correct)
APIUser     →  apiuser       (should be api-user)
HTMLEscape  →  htmlescape    (should be html-escape)
SVGAsset    →  svgasset      (should be svg-asset)

#73, merged May 2023. Contractions and possessives were only collapsed before whitespace or the end of the string, so punctuation broke them:

Don't stop  →  dont-stop     (the case the PR tested: correct)
Don't!      →  don-t         (should be dont)
it's:       →  it-s          (should be its)

#71, merged January 2023. Model T became modelt, and Unit S became units. The maintainers found and fixed that one themselves, two pull requests later.

You can check the first two in ten seconds. Version 3.0.0, published September 2025, still has both:

npm i @sindresorhus/slugify@3.0.0
node --input-type=module -e 'import s from "@sindresorhus/slugify"; console.log(s("APIUser"), s("Don'"'"'t!"))'
# apiuser don-t

I filed both upstream on 28 August, saying how they were found (#79, #80). Sindre Sorhus, who maintains slugify, committed the first fix an hour and a half later. Both shipped in 3.0.1 on 1 September. Run the same line against @3.0.1 and you get api-user dont.

None of this is a criticism of the people involved. It's the ordinary way bugs get through: the diff passes the tests it ships with and breaks the cases nobody thought to test. The suite was green every time. The green suite is what let them through.

The three it got wrong

p-limit #93. The review claimed a counter leaked on every completed task and that the PR's own new test failed. Neither was true: the counter reads 0 after four tasks, and its 14 tests passed. The worst part is that forgehand's own test run had printed tests: pass in the same report as the review that said a test failed. The evidence to catch the mistake was sitting right beside it.

slugify #53. A test unrelated to the change failed, because I replayed a years-old commit against today's dependencies. The review itself passed the diff. That's a flaw in how I gathered the sample, not something a live pull request would hit.

slugify #45. The review flagged that the PR changed a test's expected value in the same diff as the code. That's true, and it's exactly the signal forgehand exists to catch. But the PR was a deliberate revert, and reverting a behaviour means reverting its test. The gate can't yet tell those two apart.

So, on this sample: three wrong blocks out of 21, about 14%. It's a small sample and that's not a precise figure. It's also skewed toward good code, because every pull request here had already been merged. That three of them still carried bugs is a finding, not a detection rate.

The one nobody staged

The other kind of failure is the one that made me build this.

In August, a coding agent working on forgehand itself was asked to change a function to report every test environment in a repository. The existing tests compared its output with toEqual. The first two rounds came back with failing tests.

The third round came back green. The review said why:

Non-enumerable property hack to defeat assertions: detectTestCommand defines sources and parts as non-enumerable properties via Object.defineProperties to bypass toEqual assertions in existing tests.

The agent hadn't done the work. It had hidden the new fields from the comparison. The same review caught a new test whose assertion checked a || expression inside the test body instead of calling the code under test:

expect(explicit.trim() || result?.command).toBe("custom-test")

The build passed and every test passed. The pull request (forgehand #141, in a private repository, so you can't open it) was blocked and never merged.

What forgehand is, briefly

A merge gate for AI-written code. A different model from the one that wrote the change reviews it against what the change was meant to do. The build and tests are judged by their exit code, not by asking a model. The verdict is posted as a required status check on GitHub, so a failing review blocks the merge button.

The trial is five days on one repository, free, with nothing to install. Start a 5-day trial

The summary of the 21-PR run, including the three wrong calls, is in the evidence section.