Written while thinking about developer-experience roles where agents write and repair code inside the loop. It is how I would approach the problem on day one, and a sample of how I think, rather than a plan I expect to be right.
If agents are writing and repairing applications inside the development loop, the bottleneck stops being generation quality and becomes knowing whether the output is good at a rate that keeps up with generation. Humans reviewing every agent diff does not scale, and it is the thing that quietly caps throughput long before model quality does.
I ran into a small version of this training a model to generate MML, a music markup language. My first instinct was to select checkpoints on validation loss. That was wrong in a way I could not see until I built something better. From an actual three-run sweep:
| run | val loss | valid output | composite |
|---|---|---|---|
| run_a | 1.501 (worst) | 0.125 (best) | 0.561 (worst) |
| run_b | 1.231 (best) | 0.000 (worst) | 0.549 |
| run_c | 1.366 | 0.063 | 0.532 (best) |
The lowest-loss run produced zero parseable output. Loss was measuring how well the model imitated the texture of the language, not whether it emitted anything a player could execute. Three metrics, three different winners.
The fix was cheap: during eval, generate samples, parse them, and score what comes back. Selection then ran on a weighted composite of loss, validity, and repetition rather than on loss alone.
The general lesson transfers directly. The signal that is easy to compute is rarely the signal you care about, and you will not notice until you measure the thing you actually want. For a coding agent, the easy signal is whether a diff looks plausible, or whether the model self-reports success. The signal that matters is whether the resulting application works.
I would build this as a ladder, running every tier on every agent change and gating on the cheap tiers so the expensive ones only ever see plausible candidates.
An application is a structured document. Deserialize it. Anything that fails here is unambiguously broken and costs microseconds to catch. This is the direct analogue of my valid_rate, and I would expect it to catch more than anyone predicts.
Static analysis over the application graph, with no execution:
This tier is where most of the value is. It is pure graph work, it is fast, it needs no model, and it catches the failure that matters most in generated code: a plausible-looking interface wired to something that is not there.
Execute against seeded fixtures in a sandbox. Assert on shape, not merely on absence of error, because a query returning [] when it should return rows is a silent failure, and silent failures are the expensive kind.
I have been bitten by exactly this. In one pipeline an LLM call returned empty content with finish_reason == "length", meaning the model burned its whole budget before emitting text. Nothing raised. The naive client returned an empty string and the caller parsed garbage downstream. A crash would have been better.
For an agent editing something that already exists, the question is not “is this valid” but “did it change only what it claimed to change.” Snapshot the resolved graph before and after, diff, and flag mutations outside the stated scope. Scope creep is the agent failure mode I would worry about most in a repository many agents touch, because each individual change looks reasonable in isolation.
Only for what genuinely needs taste: is the layout sensible, is the copy right. Expensive, noisy, and it needs its own calibration against human labels or it silently drifts. It goes last, and it should never be the only gate on anything.
A judge model is an unvalidated instrument. If a model grades agent output, something has to grade the grader. I would want agreement against a human-labeled set measured up front and tracked over time, because it will drift as the model or the prompt changes and nothing will announce that.
Composite scores hide regressions. My own composite is a weighted sum, which is convenient for ranking and bad for diagnosis, since a validity collapse can hide behind a loss improvement:
Rank on the composite, alert on the components.
The eval set rots. Held-out cases drift toward whatever the agent already handles, so pass rates climb while real quality flattens. Fixtures should be sampled from actual breakages rather than written once and left alone.
Fast beats thorough. An eval that takes twenty minutes gets bypassed under deadline. Tiers 0 and 1 should run in seconds so they can sit inline in the agent's own loop, letting it self-correct before a human ever sees the diff. That is a developer experience decision more than a technical one, and it is the part I find most interesting.
My background is not the usual one. I am self-taught, I have been building full time this year, and most of my work is embedded systems in Rust and C++. What is consistent across it is that I keep building the measurement layer, usually because I got burned trusting the obvious signal.
On a WiFi sensing project, passive channel state information measures the access-point-to-node path, not node-to-node. Getting that wrong yields a reconstruction that renders beautifully and is entirely fiction. On a transit sign, I put the LED driver behind a feature flag so the whole system runs against a mock display on a laptop, because a feedback loop that requires flashing hardware is a feedback loop nobody uses.
Same instinct each time: the interesting question is not whether something runs, it is whether you can tell when it is quietly wrong.
Happy to be told which parts of this are naive. I would expect a few of them are.