The site wanted an alert when a line had been starved more than a set percentage of the last twenty minutes. The number lived in a cloud analytics stack nobody could get API access to. Every term in its formula was either a controller tag or a known constant, so the math moved on premise and the alert shipped the same day.
"I need to know if a line has been over a set percentage starved over the last set time range. Say twenty percent over the last twenty minutes. And I guess we have to look back over a window, because it is a calculation, not a real-time value. Right?"
Right, and that question contains the whole design.
Two facts follow from it. Starvation has no instantaneous value: at any single instant a line is running with product, running empty, or stopped. Twenty percent starved is only meaningful as a ratio over an interval, so the window is mandatory and both the window and the threshold belong to the user.
And starvation is a calculated metric, not a tag. The controller emits the inputs. The site's performance backend does the arithmetic and publishes the result to a cloud analytics stack, where it becomes a tile on a dashboard. Getting at that number through an API was blocked and stayed blocked, which meant the number the floor cared about was visible to management after the fact and unavailable to anything that could act on it. That is the ordinary shape of the problem dashboards leave behind.
The decision: compute it locally instead of waiting
The backend's formula was documented, and every term in it is either a live controller tag or a known constant.
Real time producing = production cycle time x items produced
Starved during production = producing event duration - real time producing
Starvation % = starved during production / producing event duration| Term | Source | Live? |
|---|---|---|
| Producing event duration | Time the running bit is true within the window | Tag |
| Items produced | Change in the line's counter across the window | Tag |
| Production cycle time | The line's design rate | Constant, adjustable per line |
The design rate is the only term that does not come off the controller. It is the site's throughput target of 1,740 units per hour, which is 3600 divided by 1740, or roughly 2.07 seconds per item. It is exposed as an adjustable parameter rather than baked in, so it can be calibrated against the dashboard tile later instead of being asserted now.
Nothing in that formula needs a cloud. It needs two live values and a constant.
Three tags per segment, six segments
| What | Tag | Role |
|---|---|---|
| Producing state | .Status.inRunning | The clock for the producing window |
| Item count | .Counters.Counter_00 | Its change across the window is items produced |
| Blocked | The downstream-disabled alarm bit | Excludes blocked time from the starved total |
Three tags across six line segments is eighteen tags. The segments sit in two program scopes on the same controller, three in each.
The third tag is the one that earns its place. Blocked is full behind, starved is empty in front, and they look identical from a running bit and a stalled counter. Attributing one as the other is the fastest way to lose an operator's trust in the number, because the operator can see which one it is from where they stand. A metric that calls a backed-up line starved gets argued with once and ignored after that. The same failure in the other direction, counting time that is not really production, is what put OEE on four lanes that were switched off.
Two ways to build the starved series
The tags give you state over time. Turning that into a 0/1 starved series per sample interval takes a decision, and the decision belongs to the site rather than the engine.
Rate based matches the dashboard: starved means running while the counter is not advancing at the design rate. It captures partial starvation, a line running but under-fed, which is most real starvation. It depends on the design rate being right.
Presence based needs no rate assumption: starved means running while the counter is not advancing at all. It catches only hard, dead-empty starvation, and it makes a good cross-check on the first one because it cannot be inflated by a wrong rate.
The evaluation layer does not care which series it is fed. That separation is the reason both stay available.
The capability underneath is not about starvation
The monitoring layer that makes this possible knows nothing about OEE. It watches any numeric signal over a window and delivers to a webhook. The same tool alarms on tank level, motor temperature, or packet loss with no new code. Starvation is the first documented use of it, not the feature.
| Property | What it means in practice |
|---|---|
| Pure, deterministic windowed evaluation | Given a series of timestamped values, did the condition hold for at least X percent of the last N seconds? Samples are passed in explicitly and the clock is injectable, so the verdict is exact and unit-testable rather than dependent on when the job happened to run |
| Zero-order-hold integration | Each sample is assumed to hold until the next one, which is what bounds the accuracy to the poll rate |
| Coverage checked before the verdict | If the window is under-sampled, under 50 percent covered by default, the result reports insufficient coverage and the breach is forced false. It will not fire on a gap |
| A dropped subscription is a gap, not a hold | A disconnect is marked missing rather than carried forward as stale covered time, so an outage suppresses a fire instead of manufacturing one |
| Rising edge plus rate limiting | A line that sits starved alarms once, not every minute, and re-arms when it recovers |
| Honest delivery | No webhook configured returns not delivered, with a reason. There is no fake success |
| Secrets stay out of stored config | The webhook URL comes from the environment and is never persisted with the watch definition |
The two coverage rules are there for the same reason as the blocked bit. An alerting system that fires when its own data feed drops trains people to ignore it within a week.
The numbers the site actually set
The starvation answer is a composition, not a feature. Poll the three tags per segment, derive the starved series across each sample interval, evaluate it over the window, alert on the rising edge. The parameters are what make it a real alert rather than a demo.
| Parameter | Value | Why |
|---|---|---|
| Window | 1,200 s (20 min) | The operator's own framing of the question |
| Threshold | 20 percent of the window | Also the operator's, and adjustable per segment |
| Rate limit per segment | 1,800 s (30 min) | A segment that stays starved alarms once, not forty times |
| Coverage floor | 50 percent of the window | Below it, the verdict is insufficient coverage rather than a breach |
| Poll cadence | A few seconds or faster | Zero-order-hold means the sample rate is the resolution |
| Denominator | Wall clock or observed producing time | Has to be set to whatever the dashboard uses |
The denominator is a real fork, not a detail. Percentage of wall clock, which is how the dashboard tiles sum to 100 percent including idle, and percentage of observed producing time are both defensible definitions and they do not agree. Set to observed producing time, the local verdict reproduces the backend's own formula exactly, since that is what the formula divides by.
Scaling, which is the part that usually breaks
Constant polling is where monitoring deployments fall over. Each new alert opens another connection, the controller's connection table fills, and the thing that was supposed to watch the plant becomes a load on it.
The firing service holds one subscription per controller, over the union of every rule's and every watch's source nodes. Adding a windowed starvation watch adds its three tags to that one subscription. It does not open another connection. Many users and many watches on one controller share one connection, and adding a watch widens the node set and nothing else. On a plant network, that property is not a nice-to-have.
Shipping it the same day
The general capability was built and tested before this question arrived. What shipped that day was the composition: an interim runner that reads the six segments live over the already validated read-only path, windows them with the tested evaluator, and posts to a chat channel on the rising edge, rate limited per segment. Delivery was validated live against a real webhook, and it has run continuously since under a service wrapper on a Windows host.
Every site specific lives in configuration, not in the runner. A git-ignored config file maps each segment to its running, count, and blocked tags and carries the design rate. The webhook URL comes from the environment. The script itself holds no customer data, which is why it can exist in a repo at all. Adding a segment or a whole site is a config edit rather than a code change. There is a dry-run mode that prints the alert it would have posted, which is how you find out your thresholds are wrong without waking anyone up.
Underneath, the full path from series to window verdict to delivered alert is exercised deterministically against a faked webhook, including the case that matters most, that a watch fires once on the rising edge and not again. The engine's suite was green at 1,349 tests when this shipped.
The follow-up, four weeks later
The more interesting update is that the metric it watches moved.
Over the following month the site rolled out ratio gating across all six merge points, the change the merge slug-release investigation fed into. Inbound starvation fell from 36.05% to 26.3%, comparing the May baseline with gating disabled against July with gating enabled.
Worth being exact about what that does and does not say about this system. The monitoring pipeline did not cause that drop. The merge change did. What the pipeline provides is that the number is now watched continuously and locally, on a window and a threshold the site sets, rather than being read off a dashboard tile after the fact through an API nobody could get access to. The drop is the site's evidence. The alert is what makes the next one visible while it is happening.
That is also the general argument for computing operational math on premise. The value of a number that describes what a line is doing right now decays fast, and a round trip through a cloud stack that a plant cannot query is a long way to go for something the controller already knows.