An operator reported a merge releasing in alternating 30-second slugs instead of its configured 4:1 ratio. A byte-level diff cleared the deployed routine in minutes, a live read confirmed the configuration matched intent, and the export everyone would have reached for turned out to carry stale values that pointed the wrong way.
An operator reported that a merge point was releasing in alternating slugs of roughly thirty seconds per lane instead of the 4:1 ratio it was configured for. Four cartons from the heavy lane, then one from the light lane, is the intent. Thirty seconds of one and thirty seconds of the other is not a ratio, it is two lanes taking turns.
The useful part of this session is not the answer. It is the order of elimination, and one trap that would have sent a manual investigation in the wrong direction on its first step.
How the merge is supposed to release
Two lanes feed one merge meter. A structured text routine, reviewed and version controlled outside the controller, is called from the merge routine and drives the per-lane gate states.
The gating runs on a rotating block. Two counters set the ratio, C1 and C2, four and one here. A rotating position tracks how far into the current block the merge has released, and a favored lane says whose turn it is. Each release advances the position. When the favored lane has spent its quota, favor flips to the other lane.
Two escapes sit on top of that. A starvation override watches the favored lane's photo eye: if the eye reads clear continuously for longer than STARVE_MS, the routine opens both gates and freezes the position, so an empty lane cannot hold the merge hostage waiting for cartons that are not coming. There is also a downstream hold, which stops releases when the line ahead cannot take them.
One convention matters more than it looks. On this hardware PE_P = 1 means the eye is clear with no carton in front of it, and PE_P = 0 means blocked with a carton present. The entire starvation branch keys on that polarity. Read it backwards and the override sees an empty lane exactly when the lane is full.
Step one: is the deployed code the code we think it is
Before anyone reads logic looking for a bug, the cheaper question is whether what is running matches what was reviewed.
The routine was extracted from a fresh controller export, 112 MB, pulled at 3:14 PM that afternoon, and diffed against the source of record. Identical logic. The only differences were export whitespace and one comment alignment.
The wiring checked out in the same pass. Rung 5 of the merge routine is the jump to the ratio routine and nothing else, the original output instructions having been removed at deployment, and rung 6 consumes the per-lane .LN_ST states as expected. There is no second writer and no leftover from the previous scheme.
That took minutes and it removes the entire category. Nobody has to read the ratio logic looking for a bug, and nobody has to argue about whether the right version got downloaded. This is the least interesting kind of finding and often the most valuable, because the alternative is a day spent inside logic that turns out to be correct. Elimination is most of what AI is actually good for on a live floor, and it only works if the comparison is exact rather than approximate.
Step two: the export values everyone would have reached for
The same export carries decorated tag values, and reaching for them is the obvious next move. Here is what they said against what the controller said, read directly a few minutes earlier the same afternoon.
| Value | Live read, 2:57 PM | Same-day export, 3:14 PM |
|---|---|---|
Ratio counter C1 | 4 | 3 |
Rotating position POS | 0 | 3 |
| Starvation override | Inactive | Active |
Lane 1 release count REL1_CNT | 465,097 | 27,681 |
Lane 2 release count REL2_CNT | 285,723 | 33,398 |
Release counters only go up. A count read at 2:57 PM cannot be seventeen times larger than the same counter in an export taken at 3:14 PM. The export embeds the project file's last stored offline values, not controller state at the moment of export.
The mechanism is worth stating plainly, because it is not obvious from the file. A controller export is a serialization of the offline project, not a snapshot of controller memory. Its structure is exact and current: routines, rungs, instruction bodies, tag definitions, argument mappings, all of it matches what is downloaded. The value blocks attached to each tag are a different thing. They carry whatever the offline project last saved, which may be from a commissioning session years ago. Most tooling prunes those blocks on load because they are large and nobody reads them, so the trap only springs when someone goes looking for values on purpose, which is precisely the moment they are about to make a decision on one.
Every conclusion an analyst would draw from that column is wrong, and each one is wrong in a direction that looks plausible. The starvation override reads active when it was not. The ratio reads 3:1 when it is set to 4:1, which is exactly the kind of near-miss that sends someone off to fix a configuration that was never wrong. This is the same class of problem as four alarms on a screen that the controller only knew as one: the artifact in front of the engineer is real, it is just not reporting what they think it reports.
The live reads settled it in the other direction. C1 = 4, C2 = 1, confirmed on the running controller at 2:57 PM. The configuration matched the operator's expectation.
Step three: what the live window actually showed
A 124-second read-only capture during the reported behavior produced something the hypotheses had to account for.
| What was read | Across the whole window |
|---|---|
| Lane 1 photo eye | Blocked, solidly |
| Lane 2 photo eye | Blocked, solidly |
| Starvation override | Inactive |
| Lane 2 gate | Held closed, ratio enforcing |
| Releases, either lane | Zero |
Lane one was loaded and enabled the entire time and released nothing. That is not a ratio problem. A merge that releases nothing while the gates are enforcing correctly is waiting on something downstream.
The capture also exposed its own limit. Eighteen tags per pass meant roughly five seconds per sample. Nothing that lives inside a five-second gap is visible, and photo-eye chatter lives entirely inside it. The window is evidence about slow-moving state and no evidence at all about edges. A test that needs to see a bit flicker has to be built for that cadence, on a handful of tags, not on a comfortable list of everything relevant.
The three mechanisms the fault space reduced to
The first two converge on one shape. The starvation override latches most of the time, both gates open, arbitration falls back to the stock OEM bed instruction, and that instruction releases in trains. Thirty seconds of one lane and thirty of the other is what fallback arbitration looks like from the floor. They differ only in why the override latches. The third comes out of the dead window above and points somewhere else entirely.
| Hypothesis | Mechanism | Fix path |
|---|---|---|
| Photo eye reading falsely clear, or chattering | A dirty or misaligned eye on the favored lane reads clear for longer than STARVE_MS and latches the override | Clean or realign the offending eye, then verify the release counter rate against physical throughput |
STARVE_MS shorter than real carton gaps | Normal inter-carton gaps on the favored lane exceed the timeout, so the override latches on healthy operation | Raise STARVE_MS to roughly twice the typical inter-carton clear gap on the heavier lane. Runtime tag, writable from the screen, no download |
| Downstream hold, bed instruction not granting | Slugs persist while the override is inactive and the gates enforce, so the constraint is ahead of the merge | Trace the bed instruction's grant path, release-OK and discharge-enable against downstream full, live on the running controller |
Chatter has a second corruption path that does not need the override at all. False rising edges increment the release counters without physical cartons and advance the rotating position, burning the heavy lane's turns, so favor flips early and the ratio drifts on its own.
The timeout hypothesis was not a guess either. It is a documented failure mode of this design, written into the design notes as "you set 5:1 and observe 1:1" with a warning against tuning the timeout too low.
The discriminating test that was never needed
One read-only fast monitor, run during active slugging, at roughly 0.2 s effective cadence.
That cadence is a design constraint, not a setting. Round-trip time is dominated by how many tags are in the pass, so the list has to stay at five or fewer. The 2:57 PM capture ran eighteen tags at about five seconds per pass, which is why it was the wrong instrument for this question and the right one for the last. Per pass:
PE_Pon each lane eye, two tagsMRG_RAT_STARV_ACTIVEandMRG_RAT_POS- one release counter, alternating between
MRG_RAT_REL1_CNTandMRG_RAT_REL2_CNTon successive passes to stay inside the budget
Four one-shot reads at the start, off the fast loop, where cadence does not matter: MRG_RAT_STARVE_MS, which was the unknown, plus MRG_RAT_EN, MRG_RAT_C1 and MRG_RAT_C2 to pin the configuration to the same timestamp as the trace.
| Observation | Verdict |
|---|---|
| Eye flicks clear while the lane is visibly loaded, release counters advance faster than physical cartons, override latches repeatedly | Bad eye |
Eye tracks reality but the override duty cycle is high because real gaps exceed STARVE_MS | Raise STARVE_MS, writable from the screen, no download |
| Override mostly inactive, gates enforcing, slugs persist anyway | Downstream, not the merge |
Three observations, three verdicts, no overlap between them. That is the whole point of writing the table before running the test: the outcome cannot be argued about afterward.
What closed it
The merge ratio rollout across all six merge points ran live testing over the following weeks and found and corrected eight defects on the line. Two of them are the first two mechanisms above.
The first was a reversed photo-eye hardware convention, which is the polarity described at the top of this study, wired the other way. An eye reading clear when a carton was present is precisely the false-clear path, arriving through hardware rather than through dirt on a lens.
The second was the starvation timeout raised from 150 ms to 1000 ms, to stop normal carton gaps from tripping the override. A 150 ms clear gap between cartons is not starvation on a running lane, it is a running lane.
Both are runtime tag values. They were retuned in production with no download. All six merges now run gating enabled at 4:1, with the deployed timeout at 1000 ms and the ratio verified functional on each. The slug behavior is not reported as current on any of them, and the discriminating test above remains executable as written if it comes back.
Inbound starvation fell from 36.05% to 26.3%, a drop of 9.75 points, comparing the May baseline with gating disabled against July with gating enabled, from the site's own starvation rollup. That is the same metric the local starvation monitor watches continuously, built a few weeks earlier for a different reason and now the thing that would catch a regression while it was happening rather than a month later.