Late-Arriving Data

Watermarking, streaming late data handling, impact on labels, remediation

A fraud system scores an order the instant it is placed — call that T+0. The truth about that order arrives much later: a fraudulent chargeback gets filed 7 days on. That chargeback *is* the label. But your training job runs daily and can only trust labels that have had time to settle. So the most recent week of your training data shows almost no fraud — not because fraud stopped, but because the evidence hasn't shown up yet. This is label delay, and it is everywhere that the truth arrives after the thing you're predicting.

Don't guess the wait — measure it

The lazy fix is "just wait longer before training." It works, but it throws away recency. The real fix is to learn the *shape* of the delay: for your data, what fraction of chargebacks are in by day 1, day 3, day 7? That's the completeness curve. At 50% completeness you have half the labels; at 95%, nearly all. Set your incubation period — how long before you call a label final — at the 99th percentile of that curve, not at some round number that felt about right.

Late data wrecks features too, not just labels

"Transactions in the last 5 minutes," computed at 14:02, should cover 13:57–14:02. But a phone that dropped signal for three minutes uploads its events at 14:04 — after the window closed. The feature was computed on incomplete data. And the miss isn't random: it systematically undercounts users with flaky connectivity, which tracks with geography and device. Bias, not noise.

Watermarks and event time

A *watermark* is how a streaming system announces "every event up to time T is now in." You set it by measuring the real lateness of your source: too tight and you throw away real events, too loose and every result waits on stragglers that may never come. The deeper rule underneath all of this: use event time (when it happened), never processing time (when your system received it). In real distributed systems — retries, clock skew, phones uploading hours later — events *always* arrive out of order. Design for out-of-order arrival from the start, because assuming order silently distorts every aggregate, every label, and every model built on them.

Key points

• Design your training pipeline around label delay from day one — decide the label cutoff window empirically and build a consistent data split that respects it. Models trained with inconsistent label windows give unreliable offline metrics. If chargebacks arrive over 7 days, your training data from the last 7 days has systematically under-labeled positives. Either exclude that window entirely or weight labels by expected completeness at training time. The completeness curve — fraction of expected labels received as a function of days since the event — tells you exactly where to draw the line.
• Trap: using processing time instead of event time for feature windows produces features that drift with pipeline latency. "Transactions in the last hour" computed from processing time expands and contracts as pipeline throughput varies. Use event timestamps for all business logic. This requires watermarks that account for the actual lateness distribution of your data source — measure the 99th percentile of event arrival lag and set your watermark tolerance accordingly.
• Diagnostic: plot the distribution of event arrival lag (event time vs processing time) for your specific data source. Set your watermark tolerance to that measured 99th-percentile lag itself, not a flat round number — a p99 lag of 45 minutes needs roughly a 45-minute tolerance, not 5. If that tolerance is longer than your feature windows, either widen the windows or accept that the slowest stragglers get dropped. Re-measure this distribution whenever upstream systems change — a new mobile OS version, a new attribution pipeline, or a new data collection method can shift the completeness curve significantly.

Recap

• Label delay: truth arrives after the event — chargeback filed 7 days on. Recent week looks fraud-free because evidence hasn't shown up.
• Don't guess the wait, measure it: the completeness curve = fraction of labels in by day 1/3/7. Set incubation at the 99th percentile.
• Late data wrecks features too: "txns last 5 min" at 14:02 misses a phone uploading at 14:04. Bias, not noise — undercounts flaky connectivity.
• Watermark = "everything up to T is in." Too tight drops real events, too loose waits on stragglers.
• Use event time (when it happened), never processing time (when received).
• Events always arrive out of order — retries, clock skew, delayed uploads. Design for it from the start.

Check your understanding