Did the agent follow the decision?
This is dogfood. Our own benchmark, on our own repositories. 30 decisions our team made between June and August 2026. A coding agent was asked about each one 5 times in each of four setups, with and without Align. It is not a customer's data and it is not an independent audit.
Run on 2026-09-20. Every run used claude-sonnet-5. 600 runs, 50.36 dollars of agent spend. Every answer was then graded three times, by 3 models from two labs - the scores below say which judge produced them.
The result
+19.3 points, giving the agent your team's decisions
Not against an agent with nothing. Against one already reading the codebase with grep, file reads and globs. Same 30 tasks, same model, same tools. The only difference was the decision graph.
Search alone followed the governing decision in 72% of runs (108 of 150). With the graph as well, 91% (137 of 150). For scale, the same agent with no tools at all managed 39% (58 of 150) - that is the floor, not the comparison.
Those three percentages are claude-haiku-4-5's scoring. Every percentage on this page belongs to a judge, and two other judges scored the same answers differently - see three judges, two labs. The gap is the figure that holds across all of them. Task by task the graph was ahead on 11, behind on 2, level on 17.
Then we had a model from a different lab re-mark all 600 answers. Nothing was re-run - same transcripts, different judge. The score moved; the gap did not. 19.3 points under Anthropic's judge, 18.7 under OpenAI's.
Where it matters most: on the tasks whose governing decision was never written in the repository - settled in a meeting, a ticket or a thread - search alone got 33.3% and the graph got 90%. No amount of grepping finds a decision that is not in the files.
Everything below is how we measured it, what we got wrong, and every number behind those two. The method starts here. We publish the run that disagrees with us as well as the ones that do not.
Three judges, two labs
A model grades every answer, so the obvious objection is that an Anthropic model marking an Anthropic model's homework will be generous. We had the same 600 transcripts graded three times to find out. Nothing was re-run; only the judge changed.
| Judge | Lab | Search alone | Search + graph | Gap | Unclear |
|---|---|---|---|---|---|
| claude-haiku-4-5 | Anthropic | 72% | 91.3% | +19.3 (8 to 32) | 6% |
| gpt-5 | OpenAI | 64% | 82.7% | +18.7 (9.3 to 28.7) | 16% |
| gpt-4o-mini | OpenAI | 74.7% | 83.3% | +8.7 (-2 to 20.7), spans zero | 19% |
The score depends on who marks it. 91.3% under one judge and 82.7% under another, on identical answers. gpt-5 returns "unclear" on 16% of runs against claude-haiku-4-5's 6%, and unclear counts for neither side, so every arm falls together when it does. Treat any single percentage on this page as belonging to its judge.
The gap does not depend on who marks it. 19.3 points under Anthropic's judge, 18.7 under OpenAI's, both intervals clear of zero. That is why the number at the top of this page is a difference and not a score.
The third row is a small model judging a frontier model's output. It is the only one whose interval spans zero, and it stays in the table because dropping the judge that disagrees most would be choosing the judges that suit us.
Why the gap is a paired comparison
Both arms ran the same 30 tasks, so the two results are not independent samples - they are the same tasks, twice. Some tasks are hard for everything, which widens each arm's own interval without saying anything about which arm is better. Comparing them task by task removes that shared difficulty, because it is the same task on both sides of the subtraction.
So the interval on the gap - 8 to 32 - is a bootstrap of the per-task difference over 30 tasks, and it excludes zero.
We got this wrong first. An earlier version of this page compared each arm's interval, saw them overlap, and said we could not claim the difference. Overlapping intervals do not mean the difference is absent when the comparison is paired. The mistake made us look worse than the data says, and it is recorded here rather than quietly fixed.
What "followed the decision" means
Each task is a question an engineer might ask an agent, where one decision our team made governs the right answer. The judge reads the agent's written answer next to the text of that decision and returns one of three verdicts: complies, violates, or unclear. Unclear is never counted for either side. A run that hit the turn cap of 20 is kept in the data and graded unclear rather than re-run.
Nothing was executed. No arm could write files or run commands, so this measures whether the written answer matched the decision. Nothing checked that code worked. The times in the raw data were recorded at a concurrency of 6 and are not what a user would wait.
The four arms
Same model, same 30 questions, same judge, same turn cap. Only what the agent had changed.
| Arm | What the agent had |
|---|---|
| Bare | the question only. No tools, no repository, no graph. Every run was a single turn. |
| Rules file | the question plus the repository's agent rules file, still no tools. Every run was a single turn. |
| Repo search | read-only tools over the repository (grep, read, glob) and nothing else. |
| Align | the same repository tools plus Align's decision graph over MCP. |
The four kinds of task
The 30 tasks are split by where the governing decision lives, because that is what a repository search can and cannot see.
| Split | Tasks | What it tests |
|---|---|---|
| Present | 11 | the governing decision is stated somewhere in the repository |
| Absent | 6 | the decision was made elsewhere and the repository never states it |
| Reversed | 6 | the repository's own text still says the earlier, superseded thing |
| Scoped | 7 | a decision that looks relevant applies to a different instance or scope |
Every cell
| Split | Arm | Runs | Complies | Violates | Unclear | Rate | 95% interval | Errors |
|---|---|---|---|---|---|---|---|---|
| Present | Bare | 55 | 28 | 22 | 5 | 50.9% | 37.1 to 64.6 | 0 |
| Present | Rules file | 55 | 35 | 17 | 3 | 63.6% | 49.6 to 76.2 | 0 |
| Present | Repo search | 55 | 51 | 4 | 0 | 92.7% | 82.4 to 98 | 0 |
| Present | Align | 55 | 55 | 0 | 0 | 100% | 93.5 to 100 | 0 |
| Absent | Bare | 30 | 7 | 20 | 3 | 23.3% | 9.9 to 42.3 | 0 |
| Absent | Rules file | 30 | 8 | 17 | 5 | 26.7% | 12.3 to 45.9 | 0 |
| Absent | Repo search | 30 | 10 | 19 | 1 | 33.3% | 17.3 to 52.8 | 0 |
| Absent | Align | 30 | 27 | 3 | 0 | 90% | 73.5 to 97.9 | 0 |
| Reversed | Bare | 30 | 4 | 11 | 15 | 13.3% | 3.8 to 30.7 | 0 |
| Reversed | Rules file | 30 | 12 | 14 | 4 | 40% | 22.7 to 59.4 | 0 |
| Reversed | Repo search | 30 | 18 | 12 | 0 | 60% | 40.6 to 77.3 | 0 |
| Reversed | Align | 30 | 23 | 7 | 0 | 76.7% | 57.7 to 90.1 | 0 |
| Scoped | Bare | 35 | 19 | 16 | 0 | 54.3% | 36.6 to 71.2 | 0 |
| Scoped | Rules file | 35 | 22 | 13 | 0 | 62.9% | 44.9 to 78.5 | 0 |
| Scoped | Repo search | 35 | 29 | 6 | 0 | 82.9% | 66.4 to 93.4 | 0 |
| Scoped | Align | 35 | 32 | 3 | 0 | 91.4% | 76.9 to 98.2 | 0 |
| All splits | Bare | 150 | 58 | 69 | 23 | 38.7% | 30.8 to 47 | 0 |
| All splits | Rules file | 150 | 77 | 61 | 12 | 51.3% | 43 to 59.6 | 0 |
| All splits | Repo search | 150 | 108 | 41 | 1 | 72% | 64.1 to 79 | 0 |
| All splits | Align | 150 | 137 | 13 | 0 | 91.3% | 85.6 to 95.3 | 0 |
Where a baseline won
No baseline beat Align on any of the four kinds of task this run. Last run repo search beat it on one. We are leaving the sentence here rather than deleting it now that it has stopped costing us anything. The closest was Present, where repo search reached 92.7% against Align's 100%.
The 0 runs that hit the turn cap
All 0 are in the two arms that had tools (0 in repo search, 0 in Align). They stay in the data as unclear, which lowers those arms' rates without recording a wrong answer.
With them excluded, which is not the analysis we registered, repo search is at 72% (108 of 150) and Align at 91.3% (137 of 150). Most of the pooled gap between those two arms is repo search running out of turns more often. When both arms finish, they conclude about the same.
What could be wrong with this, and what we got wrong first
- Our first run was invalid, and we published the correction. In v1 the harness loaded our own MCP configuration into every arm, so the arm without Align had Align. The bare arm took more than one turn in 146 of 150 runs, which a zero-tool arm can only do by calling a tool. v1.1 is the same frozen protocol with that leak closed and per-run tool counts recorded. So it can be checked: bare and the rules file made zero tool calls in 300 of 300 runs.
- The judge is a model. A haiku-class model at temperature 0 decides complies or violates. We read the judge's reasoning on every one of Align's non-complying runs, checked the checkable claims against the live repository, and found two where the answer looks right and the verdict looks wrong. Both are in the human review below, reported separately.
- The tasks are frozen, the repository is live. The governing decisions were frozen on 2026-09-13. An agent with the live repository can answer with something newer than the frozen text and be marked as contradicting it. It happened on one task in this run, the design-token parity item.
- The tasks are ours. Decisions from our own repositories, made by our own team, some of them public. The model's published training cutoff is January 2026. The decisions are from June to August 2026. A public corpus is still not a customer's.
- One model. Everything here is one Claude model. We ran two other models on the same 600-run grid. One leg's intervals overlapped at this size. The other was dominated by infrastructure errors. Neither is a second result and neither is quoted.
Human review of the judge
A person read 34 verdicts, sampled across arms and verdicts, with the answer, the decision and the judge's reasoning side by side. Not blind: the reviewer saw the verdict before choosing. They agreed with the judge on 34 of the 34 they could call. 34 of 34 verdicts read correctly, across all four arms (align 16, repo-search 8, bare 6, claude-md 4). No disagreements, so this sample sets no upper bound on the judge's error rate - it establishes that these 34 were not misread. The aggregate on this page is still the judge's scoring. Reviewed on 2026-09-20.
Task by task
| Task | Split | Bare | Rules file | Repo search | Align |
|---|---|---|---|---|---|
present-brain-dockerfile-layer | Present | 0/5 | 0/5 | 5/5 | 5/5 |
present-gh-token-empty-form-gate | Present | 5/5 | 5/5 | 4/5 | 5/5 |
present-history-depth-fetch-precondition | Present | 4/5 | 5/5 | 5/5 | 5/5 |
present-hnsw-ef-search-floor | Present | 4/5 | 4/5 | 5/5 | 5/5 |
present-image-build-provenance-referrer-cap | Present | 3/5 | 4/5 | 5/5 | 5/5 |
present-import-connector-check-parity | Present | 1/5 | 3/5 | 5/5 | 5/5 |
present-jira-confluence-webhook-fail-closed | Present | 5/5 | 5/5 | 5/5 | 5/5 |
present-preview-deploy-align-bot-prod | Present | 0/5 | 0/5 | 5/5 | 5/5 |
present-route-auth-allowlist-tenant-scope | Present | 2/5 | 3/5 | 5/5 | 5/5 |
present-schema-guard-required-versions | Present | 0/5 | 1/5 | 2/5 | 5/5 |
present-security-definer-tenant-lookup | Present | 4/5 | 5/5 | 5/5 | 5/5 |
absent-align-cli-decided-at-schema-v4 | Absent | 3/5 | 0/5 | 0/5 | 4/5 |
absent-align-cli-preview-check-disabled | Absent | 0/5 | 0/5 | 0/5 | 4/5 |
absent-consultancy-channel-motion-spike | Absent | 3/5 | 5/5 | 5/5 | 5/5 |
absent-contractor-insource-icp-parked | Absent | 1/5 | 2/5 | 5/5 | 5/5 |
absent-infra-ec2-spot-vcpu-quota | Absent | 0/5 | 1/5 | 0/5 | 4/5 |
absent-preview-check-disabled | Absent | 0/5 | 0/5 | 0/5 | 5/5 |
reversed-both-checks-claim | Reversed | 2/5 | 3/5 | 3/5 | 2/5 |
reversed-cosmic-ray-not-evaluated-claim | Reversed | 0/5 | 2/5 | 0/5 | 1/5 |
reversed-design-token-parity-not-gated-claim | Reversed | 0/5 | 5/5 | 1/5 | 5/5 |
reversed-teams-webhook-handler-dead-route-claim | Reversed | 1/5 | 2/5 | 4/5 | 5/5 |
reversed-ui-lint-not-covered-claim | Reversed | 0/5 | 0/5 | 5/5 | 5/5 |
reversed-zoom-webhook-fail-open-claim | Reversed | 1/5 | 0/5 | 5/5 | 5/5 |
scoped-ali947-title-vs-body-misread | Scoped | 4/5 | 5/5 | 5/5 | 5/5 |
scoped-align-check-pin-bump-action-only | Scoped | 0/5 | 0/5 | 5/5 | 5/5 |
scoped-align-gate-app-vs-workflow-pair | Scoped | 5/5 | 5/5 | 5/5 | 4/5 |
scoped-altitude-backfill-vs-producer-decision | Scoped | 5/5 | 5/5 | 5/5 | 5/5 |
scoped-conflict-severity-tier-retirement-vs-ali245 | Scoped | 0/5 | 2/5 | 0/5 | 3/5 |
scoped-design-tokens-type-scale-exception | Scoped | 0/5 | 0/5 | 5/5 | 5/5 |
scoped-telemetry-anonymous-vs-ali443 | Scoped | 5/5 | 5/5 | 4/5 | 5/5 |
Reproducing it
The harness, the 30 tasks, the 600 raw runs and the 600 verdicts live in our private monorepo, checksummed. The tables on this page are the aggregates from that run, copied into this site and pinned by a test that re-adds every cell; the aggregates themselves re-derive upstream from the raw rows with no API key and no spend. The protocol was written down and committed before the first run, and the v1.1 ceilings before the re-run. If you want to see the rows, ask us and we will go through them with you on a call.