Data quality Monitoring

Four signals, watched on every source, with alarms that stay honest.

The monitor watches every source for the four things that go wrong before a record does: row volume, freshness, structure and value distributions.

FOUR SIGNALS, WATCHED ON EVERY SOURCEvolume95% drop at 02:14 · feed paused, engineer pagedfreshnesswithin the 6-hour contract all weekstructurenew column ZZ_CUSTOM_01 · catalogued, nothing downstreamvaluesnegative quantities × 400 · returns module faultLearned patterns keep the alarms honestmonth-end order surge and Saturday dips are expectedand stay silent; 41 alerts a week became 6A true anomaly opens a ticket with its root causetraced through lineage to the source, the module andthe rows; consumers of the affected records are told
The four signals on one source over a week: a volume drop that paused a feed, freshness inside its contract, a new column catalogued, and a value anomaly that opened a ticket.
4
signals per source
product
41 → 6
alerts a week after learned patterns replaced fixed thresholds
showcase
12 s
to detect a 95% volume drop on the showcase estate
showcase
28 s
to trace an anomaly to its root cause through lineage
showcase
The four signals

What is watched, and what a problem looks like

SignalWhat is watchedA problemWhat the monitor does
VolumeRows and change events per interval, against the source's learned rangeA 95% drop in change events at 02:14Pauses the dependent feeds, pages the engineer, tells the consumers; resumes and replays the backlog when the source recovers
FreshnessThe age of the latest change per source and per view, against the contractsA view older than its six-hour contractNotifies the consumer; holds the feed if the contract says so
StructureColumns, types and lengths at the source, against the registered shapeA column type changed, a column addedCatalogues the change, updates the read, lists the views that use the column, notifies the application owner if a type change breaks a rule
ValuesDistributions of the weighted fields, against the learned baseline400 negative quantities in an hour where two is normalOpens a ticket with the affected records and the root cause traced through lineage
Learned patterns

Why the alarms stay honest

A fixed threshold fires on every month-end order surge and every Saturday dip, and the people who receive those alerts learn to ignore them. The monitor learns each source's rhythm over a rolling window: weekly seasonality, calendar effects, campaign periods. An event inside the learned pattern stays silent; an event outside it opens a ticket. On the showcase estate the change took alerts from 41 a week to 6, and the 6 were real.

The learned pattern is shown, so a suppressed event is explainable: 'order volume +82% on the last trading day, seen in 11 of the last 12 months'. A steward can mark a suppressed event as a true anomaly, and the pattern learns from that too.

Root cause runs through lineage. A value anomaly on a golden record is traced back through the rule run, the source, the module and the rows: on the showcase estate, 847 negative quantities were traced in 28 seconds to a returns module update at one site, affecting three product categories. The ticket carries the trace and the recommended actions.

Consumers are told by contract: the systems reading the affected views receive the notice with the records and the dimension, and the feed is held if the contract says so. Nothing stale is served in silence.

An incident on the showcase estate

From detection to close, with the times

02:14:08
DetectedChange events from SAP dropped from 2,800 to 142 a minute. The volume signal fired in 12 seconds.
02:14:20
ActedThe nightly planning feed was paused, the on-call engineer paged, the planning consumers notified under their contract.
02:22:44
DiagnosedA background job in SAP was consuming the work processes during period close; the engineer ended it.
02:58:12
RecoveredEvents back to 2,850 a minute; the feed resumed and replayed the 44-minute backlog in order; nothing lost.
03:14:00
ClosedThe planning feed completed; the incident report was generated; the SAP job schedule was changed to prevent a repeat.

Forty-four minutes from detection to close, and zero stale records served: the feed was held instead. Had the feed run, the morning dashboards would have shown the previous day's data to two hundred people. Showcase estate.

Questions

Is this the same as the Backbone's pipeline monitoring?

It uses the same signals and the same lineage; the difference is what it protects. The Backbone watches pipelines; the monitor here watches the sources of golden records and enforces the contracts on the golden views.

Can the monitor change data?

No level of the monitor writes data. Its actions are pausing and resuming feeds, notifying, and opening tickets; the corrector and a person do the changing.

How long is the learning window?

Ninety days by default, rolling; a source with strong annual seasonality can be given a longer window by the owner.

What does a consumer receive?

A notice with the view, the dimension, the records affected and the expected recovery, through the contract's channel: a message, a ticket in the consumer's system, or the API's status field.

See quality run on your own domain

Bring one domain and two systems. In a week the sources are profiled, the ten costliest fields are under rules, the first corrections are written back with their acknowledgements and the publish gate is on.