Datacentre-hosted models are strong partly because of their size, and partly because of everything engineered around them: retrieval, validation, review, orchestration. The question behind this project is whether a handful of small, cheap models, running entirely on a laptop, can close some of that gap by getting the second part right.

It’s early. Three weeks in, the results below come from a single frozen benchmark, and there’s a known weak spot I haven’t fixed yet. I’m writing it up anyway, because the shape of the answer is starting to come into focus, and because the next stretch of work is going to be spent building the part that isn’t there.

The problem

Small language models, the kind that fit comfortably in 16GB of unified memory on a Mac Mini, are weak at real coding work. Ambiguous requests, repository-scale changes, anything that takes more than one clean shot to get right. The obvious fix is a bigger model. The angle this project is actually testing is what you can recover from small models if you stop asking them to do the parts they’re bad at.

The idea

My first instinct, and probably most people’s, was some version of: run a few small models, let them vote or check each other, average out the mistakes. The project’s own planning docs rule that out on purpose. One of the founding principles is simply “no blind model voting.” Chaining small models without structure duplicates context, compounds errors, and makes it impossible to say afterward which model caused which outcome.

The actual hypothesis is narrower, and I think more testable: a cascade of small models, wrapped in deterministic tooling that absorbs everything mechanical, can recover a meaningful share of a stronger model’s performance. That’s a claim about systems design, not about model capability. A few small checkpoints don’t add up to one large one. What they can do is stop spending their limited judgement on work that never needed judgement in the first place, finding the failing test, tracing a reference, applying a syntactically valid patch, and save it for the handful of decisions that actually are judgement calls.

What’s built so far

The build order matters here, because each stage was a real question, not a foregone conclusion.

First came a deterministic substrate with no model in the loop at all: workspace intake, Python code intelligence over the AST and Pyright, a context packer with a fixed token budget, a validation runner that only runs approved commands, and a transaction layer that previews a diff, applies it inside a disposable checkout, and rolls back anything that fails. This had to work on its own, on a manually supplied patch, before a model touched it.

Then a single local model on top of that substrate, then the same model given one bounded repair attempt when its first patch failed validation, then a heterogeneous cascade: a fast implementer proposing a change, an independent critic from a different model family reviewing it, and a larger model held in reserve for cases the first two can’t resolve between them.

The critic wasn’t automatically trustworthy just for being a different model. In an early run it rejected a correct patch, confidently, on the basis of a claim about the source file that wasn’t true. The fix wasn’t a better prompt. It was giving the critic the same deterministic, post-patch facts the rest of the system already trusted, instead of asking it to re-derive the state of the file from a diff. A second opinion is only useful if it’s checking against evidence, not against its own read of the same ambiguous text.

The result, and the correction to it

The current benchmark is a frozen local suite of 36 tasks, 24 development and 12 holdout, nine tasks in each of four categories: single-module repair, multi-module navigation, public-symbol refactors, and configuration-contract changes. Every task has visible and hidden tests, and the gold patch for each one is verified against both before any model is scored.

The first head-to-head comparison looked like this:

Zero tasks were won by the single model alone. Twenty-three were won by the cascade alone. Broken down by category, the shape of that gap was informative in its own right:

That’s the number I was tempted to lead with. I’m not, because a follow-up investigation found it was partly measuring a bug rather than the idea. The single-model baseline was losing valid patches to a patch-normalisation problem: edits with slightly stale line numbers were being rejected outright instead of being re-anchored against the actual source. After fixing that in the substrate, the single model’s score more than doubled, to 17/36. The cascade, rerun on the same fixed substrate, stayed exactly where it was: 29/36.

The honest reading is that the aggregate didn’t move. The tooling underneath both configurations got better, and most of that improvement had been hiding in the baseline the whole time. The real current gap is 29 versus 17, not 29 versus 6. Still a wide margin, the cascade resolves 71% more tasks, but a smaller and more defensible one than the first number suggested.

One category didn’t move at all even after the fix: public-symbol refactors stayed at 0/9 for the single model. That confirms the problem there is semantic, not mechanical. Renaming a public function and updating every call site correctly isn’t something a generic diff normaliser can safely guess at, and it’s currently the cascade’s weakest category too, at 3/9.

None of this is free. On the same fixed substrate, the cascade used 119 model calls and about 26 minutes of wall time against 101 calls and about 15 minutes for the single model. Better outcomes, paid for in latency and compute, which is exactly the kind of tradeoff that needs an explicit budget before this goes anywhere near a real claim, not an implicit one discovered by watching it run.

What’s actually next

Three things, in order.

The public-symbol weakness gets deterministic support before anything else, starting with a map of a function’s call sites, a generated compatibility contract for any signature change, and a check that every caller still resolves before a patch is proposed at all. Right now that category is asking the model to do exactly the kind of mechanical, error-prone work the rest of the system was built to take away from it.

Before any wider claim, the routing needs explicit budgets on calls, latency, and escalations, not just a working policy. Right now the cascade is allowed to spend more without anything forcing the question of whether the extra spend was worth it on a given task.

And only once the local suite and the sandbox are reproducible enough to support a real comparison does this move toward pinned subsets of public benchmarks. That’s a deliberate ordering, not a delay. A 36-task fixture suite I built myself is a development instrument. It is not evidence I’d want to publish a performance claim against.

The numbers above are a snapshot of a project that’s still mostly plan: one working slice of the loop, one benchmark, one bug found and fixed along the way. That ratio won’t stay this lopsided for long. I’ll write up the next milestone when there’s a genuinely new result to report, not just a bigger-looking one.