Scoping a Ghost Fault Down From a Conveyor to One Zone

The working hypothesis from the floor was reasonable and wrong in an expensive way: delete the conveyor and its tags, because only three of its zones are actually used.

The method worth reading here is not the deletion. It is the order of operations. The screen said one thing, the floor said another, and the only way to size the change correctly was to go read the control logic first and let it decide how much of the screen was actually dead.

Why a ghost is hard to kill

The red element on the screen sits at the end of a chain, and each hop in that chain keeps its own copy of the reference:

control logic
  -> per-zone node data array published by the roller gateway
  -> OPC gateway
  -> data-bridge globals, one per zone per fault type
  -> graphics display, and separately the alarm server
  -> operator web client

The last three hops are the ones a cleanup has to touch, and on this site each is a separate configuration store owned by a separate engine:

There is a quick tell for which layers are involved. No alarm text in the alarm window means it is only a graphics color binding to a dead tag. Alarm text present means the alarm definition is stuck too.

Removing it from one layer and declaring victory is how a ghost comes back three weeks later.

The naming convention that makes the correlation possible

Every global on this system is named <conveyor>_ECC<card>_Z<zone>_<suffix>, grouped in the address space by the controller program that owns it. A jam global resolves through the OPC gateway to a member of the roller gateway's node data array, of the form ..._ECG_01_NODE_DATA.ECC[n].ZONE[z].FAULT.JAM. The suffixes in use across the site cover jam, aggregated faults, auxiliary, motor connection error, motor over-current, network fault, board over-temperature, and 48 volt over and under voltage.

One off-by-one is load-bearing. Zone 1 on the screen is ZONE[0] in the controller, because the gateway array is zero based. Correlating the two layers without knowing that produces a confident, wrong answer about which zone is dead.

That convention was read off a text export of the data bridge covering a different conveyor on the same string. This conveyor's own rows were not in that export, and its name did not appear in the OPC project either, which is a finding in itself and is picked up in the limitations below.

Verify against the controller before deleting anything

This is the step that changed the outcome. The export answered two questions.

Is the conveyor live? Yes. It is driven every scan at rung 3 of the main inbound routine by the roller firmware zone instruction, which takes the zone enable, the drive running bit, the gateway input and output data, and the zone start-stop structure as its arguments. It sits in a working zone behind its own conveyor gateway. Deleting it would have taken out a running conveyor to silence a graphic.

Are the referenced fault blocks live? No, and this is the useful part. Rungs 1 and 2 of the same routine instantiate jam-reset blocks for the whole string, and for this conveyor the only one actually executed is the unnumbered block. The numbered blocks for cards 1 and 2 are declared in controller scope with the correct type and never executed by any instruction call. Their single remaining reference is read-only, inside the OEE roll-up that sums the jam latches:

OEE routine, line 1765
SUM_JAM1 := ECC1_J_RST.J_LAT OR ECC2_J_RST.J_LAT OR ...

What makes that a finding rather than a quirk is the comparison. Those same two rungs drive numbered jam-reset blocks for five neighboring conveyors on the same string, eleven instantiations between them. Numbered blocks are the norm here. This conveyor is the exception.

Declared but never driven, referenced only in a roll-up. That is the signature of a ghost, and it is a distinction you cannot make by searching for the tag name. Both the live blocks and the dead ones exist in the tag database and look identical from the screen side. It is the same class of problem as a lane decommissioned in the host that kept running on the floor, one layer up: a reference that outlived the thing it pointed at, still perfectly valid to everything that reads it.

Scoping the fix down to one zone

Scanning the live 48 MB display file for references to this conveyor returned seventeen hits, all pointing at data-bridge globals rather than at the controller directly. Broken out by card and zone:

A working color expression on this screen looks like this, an OR of the zone's jam and fault globals:

{{ ...ECC1_Z2_JAM }} || {{ ...ECC1_Z2_FLTS }}

Card 1 zone 1 is the only zone on the conveyor with a bare jam binding and no fault or auxiliary partner, and it is the only zone whose jam block is undriven in the controller. Five zones across the two cards are wired into complete, working color expressions.

So the floor's instinct, "we only use zones 2, 3 and 4", mapped cleanly onto the actual scope: one zone is the orphan, the rest of the conveyor is in service. The fix is one zone in three layers, not one conveyor in three layers. One of the six zones the screen references comes out. The other five, and the conveyor that carries them, stay exactly as they are.

The trap in the editor

A good part of this session went into a graphics tool behavior worth writing down, because it is the kind of thing that silently breaks a screen.

The conveyor was a group carrying group-level dynamics: the color and flash animation and the expression feeding it. Ungrouping to reach the child object, deleting it, and regrouping strips those dynamics. The tell is the selection handles. Red handles mean the object carries dynamics. Blue or plain handles mean it is static. The regrouped conveyor came back blue, and the meaning is that the entire conveyor, not just the deleted zone, had stopped coloring on live faults. The screen looks fine. It is now decorative.

The correct technique is to step into the group in configure mode, select only the child, delete it, and step back out. The parent's dynamics survive.

Closing the visual gap afterward has the same shape of trap. Dragging the group's handle scales every child and distorts them, because the conveyor's width comes from a single wide background rectangle inside the group. Moving the group with the body or the arrow keys is safe. Narrowing the conveyor means resizing that background rectangle inside the group, never the group itself. Nudging the neighboring conveyor over at the top level is safe too, and touches no dynamics at all.

After any edit, select a kept zone and confirm it still shows red handles.

Deploy order, and why it needs a window

The layers do not deploy the same way, and two of them are plant-wide events. Back up every file first, dated, as-found and as-left.

1. 01Graphics. Saving the display file in place is the deploy. No engine reload. Operators re-navigate to drop the cached screen. This alone removes the visible ghost.
2. 02Data bridge. Back up the config, delete the orphan rows for that one zone, save, reload the configuration. A reload momentarily drops every global on the site.
3. 03Alarm server. Back up the config, delete the matching alarm, save, reload. This affects all alarms, triggers a re-scan, and can re-annunciate, horns included.

One change at a time, and a low-throughput window with the operators told in advance. If the ghost was only the flashing symbol with no alarm entry, the graphics delete may be enough, and leaving the orphaned globals in place is harmless. That trade is worth making deliberately rather than reloading two engines on a running site out of tidiness. Where the reload is unavoidable, it belongs with the people who own those engines, not with whoever found the ghost.