Notes from running our operational intelligence system against real production controllers instead of a simulator. The surprises were not about the model. They were about which artifact you read, what a correct answer is worth when nobody can use it, and where a system has to hand back to a person.
Nexus is the first place our research meets real industrial systems. It reads a site's control logic, supervisory systems, and documentation together, and reasons over them to diagnose faults and answer questions in plain language, cited to the exact rung. We have run it against two dozen full production controllers taken off working sites, some of them 110 to 160 megabytes, and validated its live read and diagnosis against a running controller. That contact with reality, rather than a simulator, changed what we believed about the problem. These are the notes, linked to the sessions they came out of.
Diagnosis Is Mostly Retrieval
We went in expecting the hard part to be reasoning. It was not. On a real controller, the information needed to resolve most faults already exists inside the control system. The alarm points at a tag, the tag maps to a rung, the rung depends on a permissive, the permissive depends on a module's state. The chain is fully determined. Nobody needs to be clever to follow it. They need to follow all of it, quickly, without skipping a link.
The clearest version on record is an engineer who knew only that a 24 V output told an upstream conveyor it was safe to feed, and not the tag, the routine, or the conditions. Everything he needed was inside a 112 megabyte export: the output instruction, the four permissives inside the drive instruction body, and a cross-reference proving nothing else in the project touches that tag. He spent eight or nine minutes on it by hand without finishing. Traced, it came back in about twenty seconds with a rung reference on every claim. The full session is a cross-controller feed permit.
That reframed the whole system for us. The value is not that it reasons better than an experienced engineer. It is that it traverses the complete dependency graph every time, while a person under pressure pattern-matches to the most likely cause and stops. Most of the time the shortcut is right. When it is wrong, it costs the rest of the shift.
Which Artifact You Read Decides Whether You Are Right
This was the lesson we did not expect, and it is the one we would give another team first.
Retrieval is only exact if you retrieve from the correct source. A controller export carries the project file's last stored offline values, not controller state at the moment of export. We found this the way you find these things, by reading live values at 2:57 PM and pulling an export at 3:14 PM and getting release counters that were smaller in the later artifact. Counters only go up. Every conclusion drawn from that column was wrong in a plausible direction, including a ratio setting that read one value offline and another live. The comparison table is in eliminating the code on a merge releasing in slugs.
The same principle bit in the other direction on a five-hour palletizer stop, where neither root cause was visible in fewer than three sources at once: the export for the rungs and the constants, live reads for which condition was actually false, and the floor to know the pallets were physically empty. See two stacked root causes behind a palletizer stop.
So the engine has to hold offline structure and live state as different kinds of evidence, label which one an answer came from, and never let one stand in for the other. Offline tells you where to read. Live tells you what is true.
The Model Has to Read What the Engineer Reads
Early on we were tempted to feed Nexus clean, normalized data. Real sites do not have clean, normalized data. They have a visualization project, a set of controller exports full of nested add-on instructions, and device diagnostics from the network, none of which agree on naming and all of which assume a human will bridge the gaps.
The gaps are frequently the answer. When a site asked how a conveyor reports its OEE state, the routine the engineer was reading only called three OEM add-on instructions, among 148 in the export, and the rule separating blocked from faulted was one alarm-code threshold buried in an instruction body. When an operator listed four faulted zones off a supervisory screen, none of the four names existed in the controller at all, because screen aliases were mapped years earlier by someone else. Sixteen cited live reads found one real fault bit in the set. Those are reading an OEE state machine out of a vendor instruction body and four faults on the screen, one in the controller.
Nexus only became useful when it read those sources the way the engineer does, together and in their native messiness, rather than waiting for someone to clean them first. A system that requires the operation to be tidy before it can help will never help, because operations are never tidy. The proof we cared about was not a synthetic benchmark. It was the full tool surface running across 24 real production controllers with zero parser errors, and 137 of 137 AI-written program documents graded rung-accurate by three independent Controls Engineers.
Completeness Beats Confidence
The most valuable diagnoses were not the obvious single-sensor faults. An experienced engineer finds those quickly with or without a system. The high-value cases were the faults with several contributing conditions, where the first cause looked sufficient and the second one was hiding, and the cases where a confident hypothesis was simply wrong.
Both showed up in one session about four lanes that needed to physically stop. The engineer dropped the zone enable and the belts kept moving. Two intuitive hypotheses followed, that the change had not stuck and that a bigger area enable would catch it, and live reads disproved both and relocated the cause to a layer below the controller entirely. The write-up is OEE was counting four lanes that were switched off.
We have also measured the gap directly. In an internal blind study over one real controller, we ran eleven frontier models two ways, alone over the raw export and connected to the engine. Asked which of two online edits was actually executing, the models got it right zero times out of thirteen on their own and eleven out of eleven with the engine underneath. That study is ours and unpublished, which is why the field sessions above carry more weight than it does.
A Correct Answer Nobody Can Read Is Not Finished
The lesson we were least prepared for is that completeness has a presentation cost. On the feed permit session, the full dependency graph behind one bit came back with 219 nodes, because the same instruction is reused across three belts and pulls in the area start-stop logic. Every node was correct. The answer the engineer needed sat near the root, and the rest was more than anyone wants to read.
That is a retrieval system succeeding and a product failing. The counterexample from the same set is the OEE session, which ended not with a graph but with a one-page operator reference: three reachable states, a fault table, and a four-step triage list a shift can keep. Same engine, same evidence, and the difference in usefulness is entirely in what got put in front of a person.
Refusing to Answer Is a Feature
Every session that went well had a moment where the system declined to assert something.
Four screen tag names that did not exist returned not-found with close matches rather than a confident description of tags that were never there, and that refusal is what made the rest of the session possible. Roughly half the operator abbreviations on one line were flagged as unconfirmed rather than expanded into plausible guesses. A supervisory mismatch was named without a root cause, because the project that would have proven the binding was not loaded.
The same rule shows up in the monitoring work, where it has a number attached. A jam board that goes quiet when it breaks converts an outage into a false all-clear, so a dead feed can never render as "no jams," and an interval whose end happened while the collector was down is flagged rather than inferred. Across two weeks of production, five interval ends out of 3,522 could not be established and all five are marked as such. That rate is the reason the rest of that table is worth quoting. See first pass yield up 6.76 points after jams stopped being found on foot.
Knowing Where to Stop
There is a clean line between the part of diagnosis that is retrieval and the part that is judgment. Tracing the dependency chain to a root cause is retrieval, and Nexus owns it. Deciding whether a degraded module should be replaced now or nursed to the end of the run is judgment, and that stays with the engineer.
This is built in on purpose. Nexus reads live values, diagnoses, and recommends. Acting on the site stays behind human confirmation and never touches safety logic. Across every field session we can publish, the engine wrote zero tags. A human disabled the two zone enable bits that took a decommissioned lane out of service. A site Controls Engineer wrote the palletizer counts back to empty. Moving a system safely from analysis to action is the hard frontier, and we are deliberate about earning each step of it, which is the subject of the gap between advising and acting.
We did not have to argue people into trusting the read and the diagnosis, because we made the boundary legible. When Nexus shows its trace, the engineer can see exactly what it did and confirm it against one physical observation. Trust came from transparency, not from accuracy claims.
What Held Up
The architecture held. A deterministic engine does the exact retrieval and cites it, and the model renders that into plain language. That split is why the system gets better as the models on top of it improve instead of being replaced by them.
What we would tell the version of us that started: the interesting failures are not model failures. They are artifact failures, scope failures, and presentation failures, and none of them appear in a demo.
The end goal is still bigger than fault diagnosis. We are building toward a system that holds a complete, current model of how an operation runs and reasons over the logic, the live values, the documents, and the history as one connected picture, so knowledge that used to live in a few engineers' heads is answerable by anyone, with receipts. That is one expression of our operational intelligence research, and it is the one teaching us the difference between studying understanding and watching it hold up where being wrong costs real money.