A benchmark score tells you how a model does on a fixed test. It tells you almost nothing about how a system will behave inside a live operation. Here is how we think about evaluation when the output is an action, not an answer.
Benchmarks measure what they measure: performance on a fixed distribution with an agreed notion of correct. The trouble starts when that score is read as a forecast of how a system will behave once it is embedded in an operation, answering questions that decide whether somebody touches a running line.
We have had to answer this question for ourselves, because running Nexus on real plant floors needed a way to tell a system that is genuinely reliable from one that produces confident prose. What follows is what we ended up counting, and why the obvious measure was rarely the useful one.
What we actually measure
The measures below are the ones that have predicted field behavior for us. Each one exists because a plausible-sounding system failed at it.
| What we measure | Why it predicts field behavior |
|---|---|
| Parse coverage on real projects | 4,386 real PLC project files parsed with zero parser errors. A system that chokes on the messy tenth of a corpus is a system that cannot answer questions about the messy tenth of a plant. |
| Expert grading of generated prose | 137 of 137 AI-written program documents graded rung-accurate by three independent Controls Engineers. Detail in how those documents were graded. |
| Refusal correctness | Does the system return not found when a tag does not exist, or does it explain a tag it invented? |
| Enumeration completeness | Does a cross-reference return every reference, untruncated, or a helpful-looking subset? |
| Time against a measured human baseline | Not a synthetic timing. The engineer's own count of how long they had already been searching. |
| Honesty rate under production conditions | How often the system marks a result as uncertain when it is, measured on real traffic rather than a test set. |
Notice how few of those are accuracy. On a live floor, accuracy is table stakes and the differentiators sit around it.
The most valuable answers score as non-answers
The merge slug-release investigation is the case that broke our first attempt at scoring sessions. The operator reported a merge releasing in slugs instead of its configured ratio. The session diffed the deployed routine against the reviewed source of record, found them identical apart from whitespace, confirmed the configuration matched intent live, and documented an export trap that would have sent a manual investigation the wrong way on its first step.
The session did not produce a root cause. Graded as question answering, it scores close to zero. In practice it removed an entire category from the search space in minutes, and the alternative was a day spent reading logic that turned out to be correct. As that study puts it, elimination reads like a non-result until you count the days it does not cost.
Any evaluation that only rewards a final answer will systematically under-rate the behavior an operation values most, which is a fast, cited narrowing of what the problem cannot be.
Weight errors by what they cost, then check who pays
Not all errors are equal, and in an operation the asymmetry is usually specific rather than abstract.
The four faults study makes the cost structure visible. An operator listed four faulted zones off a screen and asked why they were all down. The controller said three of the four had no fault bits set anywhere, and the fourth had a real motor connection error to chase physically. Contradicting the person standing in front of the machine is the expensive direction to be wrong in, so it took sixteen live structure reads at good quality to say it, each one cited.
That is what error weighting looks like in practice. It is not a coefficient in a scoring function. It is a rule about how much evidence a given class of claim has to carry before the system is allowed to make it.
The right question is rarely how often it is right. It is: when it is wrong, how wrong, how recoverable, and who finds out. An evaluation that cannot answer that is not measuring what the operation cares about.
Evaluate the seams, and evaluate where the corpus ends
The model is one component. The system is the model plus its parsers, its live read path, its citations, and the boundary that says what it will not claim. Most failures we see happen at the seams.
The seam that matters most is the edge of the corpus, and we evaluate it explicitly. In the OEE state machine decode, the engineer's real interest was a set of percentage figures, and the decode established that those percentages are computed above the controller, in a performance service that an instruction comment names but that was not in the corpus. Naming the boundary was the useful finding. In the four faults study, the supervisory project was not loaded, so the mismatch between screen and controller was reported without asserting its cause.
A system that knows where its evidence stops is more useful than a system with a wider corpus and no sense of its edges. We test for it by asking questions whose answers we know to be outside the corpus and checking whether the system reaches for them anyway.
Verifiability is measurable, and it is what actually gets checked
The property that decides adoption on a plant floor is whether an engineer can confirm the answer without trusting us.
The cross-controller feed permit trace has a clean number attached because the engineer had already been searching by hand for eight or nine minutes when they asked, and had not finished. The traced answer came back in about twenty seconds with the program, routine, and rung attached to every claim, plus the verbatim rung text. The speed is the headline. The citations are the reason it was believed, because they let the engineer confirm it in their own editor in seconds.
So we measure the fraction of claims that carry a citation a reader can independently re-run, and we treat an uncited claim as a defect even when it happens to be true.
Outcome numbers, and how much weight they carry
Two field outcomes on this site have real numbers behind them. First pass yield rose 6.76 points over the four weeks after the chute jam board went live. Inbound starvation fell 9.75 points across a merge ratio rollout, measured on the site's own starvation rollup.
Both are one site, no control group, no holdout. The yield figure supports a causal reading through mechanism rather than timing, since a jam that used to be found when somebody walked the line now surfaces within a second of the controller latching it, and the defect rates fell in the same window. The starvation figure comes from a rollout that changed several things at once. We publish both with their limits stated in the same paragraph as the number, which is the only form in which an outcome figure is worth anything to an engineer reading it.
That honesty is itself a measured property. On the jam board, the rate of jam intervals whose end could not be established and was flagged rather than inferred ran at 0.14% across production. That number is small, and it is the reason the rest of the table is quotable.
What we trust
We do not trust a system because it scored well. We trust it because we have watched what it does when the input is stale, when the tag does not exist, when the answer lives outside its corpus, and when the honest response is that it does not know. Those behaviors are what reliability engineering around unreliable inputs is for, and they are what has to hold before the boundary between advising and acting can responsibly move. A benchmark is where evaluation starts. For a system heading toward action in the physical world, it is nowhere near where it ends.