Your dashboard is lying by twelve hours

Every dashboard carries a timestamp nobody reads. The overnight ETLdefinitionShort for extract, transform, load. A job, usually run overnight, copies data out of your systems, reshapes it and drops it into a warehouse. Your dashboards often read that copy. finished at 3 a.m., the meeting is at 3 p.m., and the number on screen is a description of yesterday. So before anyone acts on it, the honest question is: how old is the number you're about to make a decision with? For some decisions, twelve hours is fine. For others, it's the whole error.

Data freshness only matters relative to the decision it feeds. A quarterly revenue view barely changes if it's a day old. The decisions below get worse by the minute, so they're the ones to find first.

• Fraud holds
• Inventory rebalancing
• Surge pricing
• Incident response

The failure mode is treating freshness as something the whole platform has ("we're a batch shop") instead of asking, decision by decision, what a twelve-hour-old input actually costs when it's wrong.

Staleness works like a tax on those decisions. Nobody sees the bill, because the number on screen looks just as confident at 3 p.m. as it did when the load finished at 3 a.m. A payment that should have been held goes through, or stock moves to the store that didn't need it, and the dashboard was right about yesterday the whole time.

For engineers, the age a reader cares about is the gap between event time in the source and the moment the number is on screen. Job success says nothing about it. A nightly ETL can finish green every night while the dashboard is still twelve hours behind by the afternoon meeting, and a streaming job can report healthy while a lagging partition holds the totals back.

If you move from batch to streaming, the main change is in what you'll promise about freshness. The technology matters less. You stop saying "the data is from last night" and start promising, per feed, how far behind the source it's allowed to be.

The practical tool is a freshness SLOdefinitionA written target for how far a data feed may fall behind its source, say a few minutes at most. You measure it continuously. per consumer: this feed will be no more than N minutes behind the source, measured continuously and displayed on the dashboard it serves. Writing those SLOs down does two things. It exposes which decisions are quietly running on stale inputs, and it kills the vague demand for "real-time everything" by attaching each freshness target to a consumer who has to justify it.

Standing up Kafka is the easy week. The engineering is in the semantics. Event time and processing time diverge the moment a mobile client goes offline or a partition lags, so every aggregation needs a watermarkdefinitionA streaming job's marker for how far event time has got, which decides when a time window closes and hands out a result. Late events after that are dropped or sent as corrections, depending on the policy. policy: how long to wait for late events, and what to do with the ones that arrive after the window closed (drop them, or emit a correction downstream consumers must handle). A dashboard that silently ignores late data is just wrong faster.

Delivery guarantees deserve the same scrutiny. Exactly-once within a Flink or Kafka Streams topology is real. End-to-end exactly-once across your CDC source, broker and warehouse sink is a design you build yourself, out of idempotent writesdefinitionWrites you can run once or several times and get the same result. They're why replaying events is safe: duplicates don't double-count., transactional sinks and keys that make replays safe.

Change data capturedefinitionStreaming every insert, update and delete out of a database's log as events, so other systems see changes as they happen. with Debezium is the right front door for operational databases, but plan for schema evolution and for backfill from day one. The first production incident will be replaying three hours of events, and replay safety is either designed in or it isn't.

# freshness is measured, not assumed
$ pipeline status --feed orders-live
source: postgres · debezium cdc
watermark lag: 42s  slo: < 120s
late events: 0.3% → corrections emitted downstream
✓ slo met · ✓ replay-safe · ✓ lag on the dashboard it serves

That terminal is example output, and the shape is what matters. The feed reports its watermark lag against its SLO, says what happens to late events, and confirms the lag is visible where people read the number. If your pipeline can't print those lines, nobody can tell you how old the dashboard is.

Streaming everything is how a data platform doubles its operational surface for the handful of feeds anyone needed fresh. That cost only pays off where the freshness SLO demands it. The architectures that hold up are tiered, and the tier for each feed follows from its written SLO.

That's the fix for the twelve-hour lie. Staleness becomes something you can see, and something someone chose, where before it was invisible and inherited. Everything doesn't have to be instant.

Freshness is a big part of our data integration work. Where a decision needs fresher data we replace the overnight batch, and where it doesn't we leave it alone. The SLOs get written with the people who read each dashboard, and the lag goes on screen. If you'd like to know how old your own numbers are, start with an assessment.