craiger.dev

A component belongs outside the model whenever its correct output is checkable without a model.

That sounds like a cost argument, and it partly is: a deterministic component costs nothing per invocation, runs offline, and has ground truth you can verify. But the reason that actually matters is different, and it took building the thing to see it clearly.

Models do not notice when they are wrong. They continue. That is not a criticism of any particular model — it is a property of what they are. Given a malformed question they produce a well-formed answer; given a file that no longer exists they describe what it probably contained. A pipeline assembled purely from model calls therefore has no stage capable of telling the next stage that it has failed, so an error introduced during retrieval surfaces four stages downstream, in production, wearing the costume of a plausible result.

Every deterministic component in this pipeline is a place where continuing becomes conditional. That is the whole design. The gates are the project; the model calls are just where the money goes.

This site is built from the telemetry of real runs against real repositories on this machine. Where a claim is made, the records behind it are shown next to it. Where something broke, it is written down rather than tidied away — and most of what is written down was found by attacking code that had already produced a plausible answer.

Where this came from

This started as a critique. A document circulated proposing a five-model production stack: a frontier model for architecture, a cheap long-context model for codebase search, a mid-tier model for writing code, a tiered escalation for debugging, and a model from a different lab for adversarial review. Each row had a plausible justification and a price per million tokens.

Checking it turned up the usual problems — pricing that had changed eleven days earlier, a benchmark number that was real but misranked, an escalation target with no per-token API to escalate to. Those are corrections, not arguments. The argument is structural, and it has three parts.

First, prompt caches are scoped to a model. A five-model pipeline is five cache namespaces with no reuse across any boundary, so the context one stage paid to assemble is re-ingested at full price by the next. On a realistic task the boundary tax roughly cancels everything the cheap models save.

Second, the phase that was supposed to be cheapest is not a language model task at all. 'Brute-force search across a million tokens of context' describes work that ripgrep, tree-sitter and a symbol indexer do exactly, verifiably, and for nothing.

Third, and worst: the design optimises cost per token, when the only figure that matters is cost per completed task. A cheaper call that needs three more turns and a retry is not cheaper.

So this pipeline inverts the premise. One capable model varied by effort rather than by vendor; exactly one seat bought for a different training lineage, at the adversarial review, where the input is small enough that it costs almost nothing to try; and everything that can be done deterministically done deterministically, in front of the model rather than around it.

How to read this site

The stage table below is the plan. The lettered rows are gates — deterministic checks that either pass or stop the run — and they are the point of the project. The numbered rows are model calls.

Every claim here is backed by telemetry from real runs against real repositories on this machine, and the records are shown next to the claims they support. Where something broke it is written up rather than tidied away: the 'why' page has fifteen entries and every one of them is a bug this project's own code shipped and then caught.

Stages

The lettered rows are the point of the project. The numbered rows are where the money goes.

Build order

Live so far

A query whose result hash varied between runs would mean determinism had broken. That count is the single most important number on this page.