The screen read waiting on a host assignment. The host had dispatched four assignments during the stop and the controller rejected all four with the same status code. Two local causes, and the second one was created by recovering from the first.
The message on the screen said the cell was waiting on a host assignment. It had said that for five hours. The host was fine. It had dispatched four fresh assignments during the stop and the controller had rejected all four, silently.
The diagnosis took three sources at once: the export for the rungs, the constants and the type descriptions, live reads for which of roughly eleven hundred conditions was actually false, and the floor to know the pallets were physically empty. Neither root cause is visible in any two of them.
Timeline
| Elapsed | Event |
|---|---|
| T+0 | A daily flag unlatches by design at the start of the day. The check that is supposed to re-set it never runs. The cell begins rejecting every host move with status 106. |
| T+4h19 | Operator reports the cell waiting on the host. Investigation starts. |
| T+4h20 | Live reads: host heartbeat healthy at 70 ms against 5 s and 60 s timeout presets, host connection bit true. Not a comms fault. The handshake is frozen, one move unacknowledged and returned to source. |
| T+4h37 | Operator cycles the controller mode from run to program and back. The host immediately dispatches a fresh move, rejected with status 106. First root cause identified. The mode cycle also arms the second one, unnoticed. |
| T+4h49 to T+4h55 | Robot abort of all programs, then a robot controller power cycle. The blocking output survives both. Meanwhile the host dispatches and the controller rejects two more moves, four in total. |
| T+5h15 | Operator runs the manual home sequence from the screen. The blocking output drops, the robot reports ready, the daily check finally runs and passes. Moves start succeeding, status 0. |
| T+5h15 to T+5h32 | Every move, including fresh totes, routes to the recirculation destination. Live read of all six pallet positions shows each one present and reporting a full count while physically empty. Second root cause identified. |
| T+5h40 | Site controls engineer writes all six pallet counts back to the fresh-pallet value, online, from the engineering workstation. |
| T+5h41 | Verified live: moves accepted, real pallet destinations assigned, counts climbing, no fault, no waiting message. Resolved. |
Root cause one: a quiet permissive behind a stuck robot output
The gate is a daily camera calibration check, and it is two rungs in the robot signals routine. Rung 93 unlatches the passed flag every new day, and on first scan. Rung 90 re-latches it, but only when the robot completes its camera check program, or when someone presses the vision fault bypass.
While that flag is down the chain runs like this, and every link is a rung in a different program:
| Link | Where it lives | What it does |
|---|---|---|
| The gate opens | Robot signals routine, rung 93 | Unlatches the calibration passed flag at the start of each day |
| The move is stamped | Execution status routine, rung 12 | Writes the camera calibration status code onto every in-progress move |
| The move is voided | Sortation routine, rungs 10 and 11 | Latches the move invalid and completes it as returned to source |
| The host backs off | Host interface | Sees a returned move, waits, dispatches again later |
| The screen lies | Diagnostics routine, rung 136 | Raises message 216, rendered on the screen as waiting on the host |
Five links, and the operator can see the fifth one only. That is how a five hour outage produces no alarm at all. None of the cell's five alarm words had a single bit set for the entire stop, because the waiting message is a plain output coil rather than a fault.
So why did the check never run? Its robot program requires the robot to report ready. Ready is no program active combined with either the program run signal or the teach pendant enabled signal. No program active requires all ten of the robot's outputs to the controller to be off. One of them, the first, landing on a single assembly bit of the robot input module, was stuck on.
That output survived an abort of all robot programs and a full robot controller power cycle, which is the detail that identifies what it is not. A latched program output does not survive a power cycle. This one is asserted by robot-side logic or configuration rather than by a paused program. It finally dropped when the operator ran the manual home sequence from the screen, and the calibration check ran and passed within the same scan window, its day stamp advancing from 12 to 13.
Two of the signals in that ready expression were only legible because the type definitions in the export carry member descriptions: one is the program run signal, the other the teach pendant enabled signal. Without those two strings the ready condition is three anonymous bits and the isolation takes considerably longer. Documentation quality inside a controller export is not a nicety here. It is the difference between reading a permissive and guessing at one.
Root cause two: the recovery armed it
On first scan the controller deliberately assumes every pallet is full:
Main / Calls, rung 9
XIC(S:FS) XIC(PalletsEnabled) -> MOV(<full>, PalletPos1..6.ToteCount)The count convention matters for what happens next. A fresh empty pallet carries a count of one, a full pallet carries the maximum, and the number reported up to the host is the count minus one. So a fresh pallet reports zero totes and a first-scan pallet reports a full build.
That is conservative and correct. After a restart the controller cannot know what is stacked on each pallet, and it must never let the robot place onto an unknown build.
The mode cycle used to diagnose root cause one fired that rung. From then on all six positions reported present with a full count and zero capacity. So once the calibration gate finally cleared and moves started succeeding, the host did the only sensible thing available to it with six full pallets: it sent every tote, old and new, to the recirculation destination.
The designed reset for this is the physical pallet exchange. Unloading a position clears its latches, a compare-and-set clears the pallet, and reloading registers a fresh count. With the pallets physically empty and the loaded latches already consistent, both verified live before anything was recommended, writing each count back to the fresh-pallet value was the equivalent of that exchange. The site controls engineer made that write from the engineering workstation. The engine never wrote a tag.
Why the screen misled everyone for five hours
Three things compounded:
- The message blames the wrong side. Waiting on the host implies an upstream problem. The controller knew it was rejecting work. It had the status code sitting in a tag, four times over, and never surfaced it.
- The calibration gate is silent. An overdue daily camera check produced no operator-visible alarm anywhere. It only manifested as host rejection plumbing.
- The recirculation looked like a host malfunction. It was the controller's own restart behavior working exactly as designed, on data nobody knew had been reset.
The first of those is a pattern rather than an incident. A supervisory layer asserting something the controller contradicts is the same failure that put four faulted zones on a screen when only one was faulted in the controller, and the same layering confusion that let a lane sit decommissioned in the host and still running on the floor. The screen is not lying so much as reporting a different system's opinion, confidently, in the operator's language.
What closed it, and how it was verified
Verification is the part that separates a plausible story from a resolved incident, so it was done on live reads rather than on the cell running quietly for a while.
| Signal | During the stop | After the write |
|---|---|---|
| Move status code | 106, camera calibration check, four consecutive moves | 0 |
| Destination assigned | Recirculation, on every move including fresh totes | A real pallet position |
| Pallet counts, six positions | All six full, zero capacity, pallets physically empty | Empty and climbing as the robot placed |
| Alarm words | Five words, zero bits set | Five words, zero bits set |
| Waiting message | Raised | Cleared |
Counts climbing is the one that matters. Status 0 says the controller accepted the move. A real destination says the host found capacity. Counts rising across the positions says the robot is physically placing totes onto pallets, which is the only claim the operator actually cares about.
What we recommended to the site
- 01Split the message. Distinguish "the host has not assigned work" from "this cell is rejecting host work, reason: X". The controller has the status code. Show it.
- 02Alarm the calibration gate. The flag being down while the machine is in cycle deserves an explicit operator message, not silence.
- 03Put pallet counts on the restart checklist. After any first scan, counts read full by design. The screen should say so, and re-registration should be a step rather than folklore.
- 04Identify the robot output. Which physical function it corresponds to, what asserts it, and why only the home sequence cleared it. It cost roughly forty minutes of aborts and power cycles, and it is still the one open item.
- 05Watch rejections on the host side. Four consecutive rejections with the same status from one cell should page a human, not silently recirculate.
What it cost, and what generalizes
Roughly fifteen live read passes across about two hundred tags, against a 4.5 MB controller export that parsed with zero errors. Zero writes by the engine. Every corrective action, the mode cycle, the robot power cycle, the manual home and the pallet count write, was taken by a person.
Two method notes generalize past this cell. Program-scoped and controller-scoped tags need different live addressing, and getting that wrong costs passes rather than producing a clear error. And the offline tag list does not expose the values of constant tags, so decoding status 106 into the camera calibration check meant reading the constants off the live controller. The export contains them. Not surfacing them turned a lookup into a round trip, on a cell that was down. The general lesson is the same one that came out of eliminating deployed code on a merge releasing in slugs: what an export says about values is not what the controller is doing, and the two have to be kept in separate columns.
The ordering is the part worth carrying forward. Root cause two was created by the operator's mode cycle at T+4h37 and went unnoticed for thirty-eight minutes, because nothing reads out pallet counts until moves start succeeding again. A live read of the pallet array at the moment of the mode cycle would have caught it immediately. It was not on anyone's list to check. That is the argument for reasoning that sits inside the loop rather than beside it: the second fault was created by the fix, in a data structure nobody was watching, and only the join across export, live values and the physical floor made it visible.