Merge gate for AI-written code
A coding agent did that to us: it hid fields from the assertions and the suite went green. Forgehand reads every pull request for a change that beats the check instead of doing the work, and blocks the merge. One repository, five days, free.
We published our own failure before anyone asked for it
An agent asked to fix a bug can fix the bug, or it can change what the test expects. Both come back green.
class _TwoAndThree(int):
def __eq__(self, other):
return int.__eq__(self, other) or other == 3
def add(a, b):
if a == b == 1:
return _TwoAndThree(2)
return a + b
x = add(1, 1) x == 2 -> True x == 3 -> True 2 == 3 -> False # equality is no longer transitive # and the result is unhashable
None of the signals a normal pipeline collects can tell those two diffs apart.
enforce_admins set, not even the repo owner overrides it. Neither can we.They resolve to different providers. The config refuses to start if they don't, so there is no path where a model quietly reviews its own work.
Not by asking another model whether they probably passed. Half the gate has nothing to do with a language model at all, and it runs locally in a git worktree.
It lives in the forge, not in this app. A failing check blocks the merge button, and nothing in the product can bypass it.
A provider outage, a malformed verdict or an error with no message all resolve to not passing, and the merge stops.
During the trial it runs on our infrastructure. That is the part most pages would not say out loud, so it is the first row of the table.
| the trial | Your code is checked out and built on a server we rent, for five days. Nothing is installed on your side. |
|---|---|
| source | Cloned at the head of the pull request. For supported stacks, installed, built and tested in a throwaway container from an official image matched to the language versions your repository declares (or a current default), with no network during build and test, and judged by exit code. Other stacks are reviewed but not built or tested. Deleted when the trial ends. |
| model data | The task text and the diff go to two model providers. Their terms apply. Nothing else leaves. |
| after | Subscribe and it keeps running there. Otherwise, when the trial ends, your code and forgehand's review records for it (verdicts, review output and pull request titles) are deleted. |
| providers | Two different vendors, never the same model reviewing its own work. Both run on our accounts; you bring no model keys. |
|---|---|
| forge | GitHub today. It needs permission to post a commit status on the one repository you point it at, and nothing else. |
| trigger | Once it is on your repository, open pull requests are swept every few minutes and anything without a verdict gets one. Putting it on the repository in the first place is still done by hand. |
| scope | Not a linter or a security scanner. It judges whether a change did what it claimed. |
The reviewer sees the task and the diff, not your files. It runs under a throwaway HOME with an explicit deny rule. A provider outage, a malformed verdict or an error with no message all resolve to not passing, and the merge stops.
Twenty-one merged pull requests from three repositories we have no part in, replayed at their pre-merge base and gated report-only. Sixteen different authors. Read the full write-up.
| slugify #69 | APIUser slugged to apiuser. Acronym decamelization broke whenever the next word's second letter was s. Merged in 2022. After our report, fixed upstream in slugify 3.0.1. |
|---|---|
| slugify #73 | Don't! slugged to don-t. Contractions only collapsed before whitespace. Merged in 2023. After our report, fixed upstream in slugify 3.0.1. |
| slugify #71 | Model T slugged to modelt. Merged, then fixed upstream two PRs later. The gate reached it at merge time. |
| p-limit #93 | Claimed a counter leaked and a test failed. Neither was true, and the model-free axis had already printed tests: pass in the same report. |
|---|---|
| slugify #53 | An unrelated pre-existing test failure in a five-year-old checkout. The review itself passed the diff. |
| slugify #45 | Flagged a test changed alongside the code. True — but the PR was an intentional revert, and reverting a behaviour reverts its test. |
Non-enumerable property hack to defeat assertions: it definessourcesandpartsviaObject.definePropertiesto bypasstoEqualin existing tests.
Nobody planted that. A coding agent was asked to widen test detection, could not satisfy
the existing assertions, and hid the fields from them instead — then added a test that
asserted on || in its own body. The suite was green. That is the whole reason
this exists, and it is the one case we could not have written for ourselves.
Every block above was checked by running the code, not by reading the verdict. The three it got wrong are published because a tool that only shows its wins is not evidence. What this does not measure: how often it misses a bad change. Every PR in the set was chosen for having been merged.
No card to start. No per-seat maths. If it does not earn its place in five days, you have lost nothing but the pull request you pointed it at.
One trial per organisation. It is five days of real reviews on a real repository, not a demo account with sample data.
Five days, free, nothing installed. The only thing we need from you is the repository and permission to post a status on it.