Three Filters Decide How Much Reality Survives

Between the physical world and the historian stand three quiet filters.

The scan rate decides how often reality is looked at. The deadband decides how much a value must move before the look is recorded. The reporting scheme decides which records travel.

Each exists for a good reason, because bandwidth, storage and attention are all finite. And each one, set carelessly, deletes exactly the reality someone later needs: the pressure spike between scans, the slow drift inside the deadband, the flatline that report-by-exception renders indistinguishable from a dead sensor.

The filters are not wrong. They are architecture — decided at configuration time, usually by whoever installed the system, for reasons nobody wrote down.

Filter one: the scan, and the lie that looks like data

Sampling theory’s practical core fits in three sentences.

To reconstruct a phenomenon, you must sample faster than twice its highest frequency of interest. Sample slower and fast phenomena do not merely vanish — they alias, folding into slow, plausible-looking artefacts. The wagon-wheel effect, in data.

That second sentence is the one that matters, because a gap announces itself and an alias does not. Undersampled data arrives smooth, ordered and confident, and the trend it draws is a phenomenon that never happened.

Industrial practice splits the problem by purpose. Control loops scan at their engineered rates. Trending for slow process behaviour lives at seconds. Machinery dynamics live at kilohertz, inside specialist modules that ship computed features rather than waveforms. Anything meant to catch excursions needs either a scan faster than the excursion or an event-capture mechanism at the source.

So the design question per point is never how fast can we? It is: what is the fastest behaviour this decision needs to see, and does the scan see it without aliasing?

Filter two: the deadband, derived rather than defaulted

Recording every scan of a noisy value stores noise. A deadband — record only when the value moves more than X from the last recorded value — is the standard remedy, and its size is a statement about what change matters.

Set at the noise floor, it stores everything meaningful. Set for storage economy, it swallows the slow drift that condition monitoring programmes are usually funded to catch. And it does so silently: nothing in the record says a drift was filtered, because the filter’s whole job was to prevent the record from existing.

The honest deadband is derived rather than chosen. Measure the noise. Name the decision sensitivity — the change size that would actually matter to somebody. Set the deadband between the two. Then keep the derivation with the point’s metadata, so a consumer three years later can judge the data instead of guessing at it.

The historian’s compression algorithms are deadbanding’s smarter cousins, with the same failure modes, applied a second time further down the chain. Two independent filters, each defensible, can compound into a record that nobody designed.

Filter three: the report, and the ambiguity of silence

Polling ships the value on schedule whether it changed or not. Report-by-exception ships it when it changes.

RBE is enormously efficient. It is how telemetry lives on thin links and how broker architectures keep traffic proportional to activity rather than to point count. It also carries one famous obligation.

Silence is ambiguous. No report means “no change” only if the link, the device and the publisher are all alive. Every RBE scheme therefore needs a heartbeat, or a birth-and-death mechanism, to make silence diagnosable. Without one, a healthy flatline and a dead collector are the same picture.

A subtler obligation falls on consumers. RBE data is a step function — the value is held between records — and analytics that interpolate between exception reports are quietly inventing data points that were never measured. That invention is not visible in the chart. It is visible only in the argument three months later about why two systems disagree.

Setting them on purpose, without a per-point agony

Two thousand points cannot be reviewed individually, and nobody should try.

The method is a per-class review. Points cluster into classes by decision duty — control-critical, process trend, machinery, utilities metering, compliance — and each class gets a written profile: scan basis, deadband derivation, reporting scheme, and the retrieval caveats consumers inherit from it.

At Meridian, the invented plant the underlying reference runs on, the illustrative profile table took an afternoon and settled two thousand points. The alternative was already installed: two thousand accidental defaults, and nobody able to say what any of them meant.

The profiles belong with the context layer rather than in a project folder, because a consumer who cannot see the filters cannot judge the data.

One class deserves its own sentence. Events — trips, starts, alarms, discrete state changes — are not sampled phenomena. They are edges, and the acquisition layer should capture them as timestamped events at source rather than hoping a scan catches the state mid-flight. A one-second scan sees a half-second trip pulse only by luck, and sequence-of-events recording exists precisely because luck is not an architecture.

An exercise that costs an afternoon

Take one point that feeds a decision somebody actually makes. One point.

Find its three filters from configuration — the scan interval, the deadband value, and whether it is polled or reported by exception. Screenshots, not recollections. Most sites cannot complete this step for an arbitrary point, and the inability is itself the finding.

Then run three checks.

1. 01
   Check the scan against the phenomenon

   Name the fastest behaviour the decision needs to see, in seconds. Compare it to the scan interval. If the scan is slower, you are not missing the behaviour — you may be recording a fictional slower one.

2. 02
   Check the deadband against the noise

   Pull a period of raw data and estimate the noise band. Then ask what size of change would make somebody act. If the deadband is bigger than the second number, the point cannot deliver the decision it was bought for.

3. 03
   Check what silence means

   Find the longest flat period in the last year. Establish whether the value was genuinely unchanged or the collector was down. If the record cannot answer that, there is no heartbeat, and every flatline in the archive is unresolved.

The most common result is that all three settings are defaults nobody chose, and that the deadband is the one costing money. The second most common is that step-held data has been feeding an interpolating consumer for years, which means every average computed from it has been subtly wrong in a direction nobody has quantified.

Neither finding needs capital. Both change what the next configuration pass should do.

Which of your points can tell you whether its last flat week was a quiet process or a dead sensor?

The three filters, the aliasing warning, the derived-deadband discipline, the RBE silence obligation, and the per-class profile method are from Industrial IoT and Data Architecture: From Sensor to Historian to Dashboard (Part 2: Edge and Acquisition, section 2.2).