Your dashboard is lying by twelve hours

Every dashboard carries a timestamp nobody reads. The overnight ETLdefinitionStands for extract, transform, load. It's a job that copies data out of your source systems, reshapes it and drops it into a warehouse, often overnight. finished at 3 a.m., the meeting is at 3 p.m., and the number on screen is a description of yesterday. So before anyone acts on it, the honest question is: how old is the number you're about to make a decision with? For some decisions, twelve hours is fine. For others, it's the whole error.

If you move from batch to streaming, the main change is in what you'll promise about freshness. The technology matters less. We make three claims below, based on pipelines we've built and repaired.

Data freshness only matters relative to the decision it feeds. Fraud holds, inventory rebalancing, surge pricing and incident response get worse by the minute. A quarterly revenue view genuinely doesn't. The failure mode is treating freshness as something the whole platform has ("we're a batch shop") instead of asking, decision by decision, what a twelve-hour-old input actually costs when it's wrong.

The practical tool is a freshness SLOdefinitionA written target for how far a data feed may fall behind its source, say a few minutes at most. You measure it continuously. per consumer: this feed will be no more than N minutes behind the source, measured continuously and displayed on the dashboard it serves. Writing those SLOs down does two things: it exposes which decisions are quietly running on stale inputs, and it kills the vague demand for "real-time everything" by attaching each freshness target to a consumer who has to justify it.

Standing up Kafka is the easy week; the engineering is in the semantics. Event time and processing time diverge the moment a mobile client goes offline or a partition lags, so every aggregation needs a watermarkdefinitionThe rule a streaming job uses to decide how long it waits for late events before it closes a time window and spits out a result. policy: how long to wait for late events, and what to do with the ones that arrive after the window closed (drop them, or emit a correction downstream consumers must handle). A dashboard that silently ignores late data is just wrong faster.

Delivery guarantees deserve the same scrutiny. Exactly-once within a Flink or Kafka Streams topology is real; end-to-end exactly-once across your CDC source, broker and warehouse sink is a design you build: idempotent writesdefinitionWrites you can run once or several times and get the same result. They're why replaying events is safe: duplicates don't double-count., transactional sinks, keys that make replays safe. Change data capturedefinitionStreaming every insert, update and delete out of a database's log as events, so other systems see changes as they happen. with Debezium is the right front door for operational databases, but plan for schema evolution and for backfill from day one: the first production incident will be replaying three hours of events, and replay safety is either designed in or it isn't.

# freshness is measured, not assumed
$ pipeline status --feed orders-live
source: postgres · debezium cdc
watermark lag: 42s  slo: < 120s
late events: 0.3% → corrections emitted downstream
✓ slo met · ✓ replay-safe · ✓ lag on the dashboard it serves

Streaming everything is how a data platform doubles its operational surface for the handful of feeds anyone needed fresh. A streaming pipeline is a 24/7 distributed system, with on-call, backpressure, state migrations and consumer-lag alerting to look after. That cost only pays off where the freshness SLO demands it. The architectures that hold up are tiered. True streaming goes to decisions measured in seconds. Incremental micro-batchdefinitionSmall, frequent batches, run every few minutes. Simpler to operate than true streaming, and close enough to live for a lot of feeds. or materialized views cover the many feeds where five to fifteen minutes is indistinguishable from live, and honest daily batch handles the rest.

The discipline is refusing to let "real-time" become a prestige label. Each feed's tier follows from its written SLO. Any tier upgrade has to name the decision that pays for it. And the lag metric ships on the dashboard itself, so the next time someone acts on a number, its age is part of the number. That's the fix for the twelve-hour lie. Staleness becomes something you can see, and something someone chose, where before it was invisible and inherited. Everything doesn't have to be instant.