The gap between a working prototype and a reliable agent running in a real operation is wider than most teams expect. Here is what breaks, and how we design around it when a wrong action has a cost.
Agent demos fail in tidy ways. The model picks the wrong function, the schema comes back malformed, a three-step chain drops on step two. Those are real failure modes and they are also the ones everybody already designs for.
The failures that cost us time on real plant floors were different in kind. The system reasoned correctly over data that was wrong, stale, or quietly absent, and produced a confident answer that a competent engineer would have believed. Reliability, in our experience, is mostly the engineering that sits between the system and its inputs.
Your most authoritative input is the one most likely to lie
The natural place to get controller state is the controller export. It is the same file the engineer opens, it contains the logic and the tag values, and it is a single artifact you can hand to a system.
It is also, in a specific and dangerous way, not current. In the merge slug-release investigation, values read live off the running controller at 2:57 PM were compared against an export pulled from the same controller at 3:14 PM. The export's release counters were an order of magnitude smaller than counters read seventeen minutes earlier, on tags that only count up. The export embeds the project file's last stored offline values, not controller state at the moment of export.
The counters made that obvious. Nothing else in that column did. The ratio setting read one value off, the starvation override read active when it was inactive, and both are exactly the kind of near-miss that sends an engineer to fix a configuration that was never broken. Every wrong conclusion available there was wrong in a plausible direction.
The design response is a rule about roles rather than a preference for one source. Offline structure tells you where to read. Live values tell you what is true. The decommissioned lane study runs both at once for that reason: the export to find that a whole lane reduces to two enable bits, the live connection to confirm both bits actually read true right now.
A search that matches anything will eventually match nothing real
Early in the four faults study, a raw text search over a controller export matched a fault string inside an encoded data block and briefly suggested that four screen tag names existed in the controller. They did not exist. The match was inside a base64 blob.
That is a ten minute error, and it is worth publishing because it is precisely the failure mode of a general-purpose tool pointed at a large file. Text search does not know the difference between logic and an encoded payload. A parser does. This is one of the concrete reasons the engine underneath Nexus is deterministic rather than a model reading files: structured lookups over parsed logic ignore encoded data, return exact citations, and give the same answer twice.
Refusal is a feature, and it has to be built in early
When those four names were looked up properly, the answer was not found, with close matches offered. That refusal is what made the rest of the session possible. A system that had produced a plausible-sounding explanation of four tags that do not exist would have burned the session and some of the engineer's trust with it.
The same discipline shows up wherever the corpus runs out:
- In the OEE state machine decode, roughly half the operator-facing abbreviations on that line had no confirmed expansion for that site. They were flagged as unconfirmed with a note to ask a site expert, rather than expanded into plausible English.
- In the four-lane OEE study, the conclusion that motorized roller cards were draining product on their own local logic was a deduction from complete evidence rather than a direct read of a card-level bit, and it was presented that way, because the card layer is below what either the export or the tag reads can see.
- In the decommissioned lane, some live reads of instruction-internal operands came back unusable because of how they were addressed, so part of the live annotation degraded to undetermined while the offline structure still carried the explanation.
None of that is glamorous and all of it is load bearing. A system that answers everything is a system whose answers carry no information about which ones to check.
Missing data must never render as good news
This is the rule we would keep if we could only keep one, and it applies the moment a system runs unattended.
A monitoring board that goes quiet when it breaks is worse than no board, because it converts an outage into an all-clear. The chute jam board treats configured, connected, and collector-stale as distinct operator-facing states with their own banners, so the absence of jams and the absence of data never look alike. Jam intervals whose end happened while the collector was down carry an uncertain-end marker rather than a plausible reconstructed timestamp. Across the first two weeks in production, five interval ends out of several thousand could not be established, and all five are marked as such rather than filled in.
The local starvation alerter applies the same rule to the verdict itself. Coverage is checked before the condition is evaluated, and an under-sampled window reports insufficient coverage and forces the breach false. A dropped subscription is recorded as a gap rather than held at its last value, so a network outage suppresses an alert instead of manufacturing one. When no delivery target is configured, the result is not delivered with a reason attached, because there is no such thing as a fake success in a system somebody is on call for.
Instrumentation limits belong in the design, not the retrospective
A capture during the merge investigation polled eighteen tags per pass, which worked out to roughly five seconds per sample. That is fine for watching a gate state settle and useless for catching a photo eye chattering. The capture did not fail. It answered a question at a cadence that could not have seen the mechanism under suspicion, which is a subtler problem, and the follow-up test was rewritten around five tags at a fast cadence for exactly that reason.
Anything that samples has a cadence, a coverage, and a blind spot. Writing those down before the run is the difference between evidence and an artifact you will over-read later.
Why all of this gets sharper when the system acts
Everything above concerns a system that reads. Nexus is read-only in every study on this site, and a wrong answer there gets caught by an engineer who checks the citation in their own editor.
Push the same architecture toward action and each of these failures acquires a physical consequence. A stale value becomes a change made against conditions that no longer hold. A false positive from an unparsed match becomes a change to something that was never there. A gap silently treated as normal becomes an action taken on absent evidence. That is why the boundary between advising and acting is drawn in architecture rather than left to a model, and why what we count as evidence matters more to us than what a system scores.
The demo shows the happy path because the happy path is easy. The work is knowing when your inputs are lying, saying so out loud, and refusing to convert silence into a verdict.