Detecting and Correcting Device Clock Drift

The reorder buffer and the hybrid logical clock make an event stream internally consistent. Neither makes it correct: a device whose clock is 90 seconds slow produces a perfectly ordered stream of timestamps that are all 90 seconds wrong, and every dwell measurement, watermark and idempotency bucket built on them is wrong by the same amount. Correcting that requires estimating the offset per device from the data itself, because no device will tell you it is wrong. This page sits under event-time ordering and clock skew within Core Architecture & Latency Constraints.

Concept and specification #

For each event the ingest edge observes two timestamps: the device’s own $t_d$ and its arrival $t_a$. Their difference contains the offset and the transit delay together:

$$t_a-t_d=\theta +\delta ,\delta \ge 0$$

where $\theta$ is the device’s clock offset (positive when the device is slow) and $\delta$ is the network transit delay, which is non-negative and highly variable. A single observation cannot separate them. But because $\delta \ge 0$, the minimum of $t_a-t_d$ over many observations converges to $\theta$ from above, and the residual is the minimum transit delay the network can achieve — a few tens of milliseconds on a mobile network, and stable enough to treat as a constant per access technology.

$$\hat{\theta }=\underset{i}{\min }(t_{a,i}-t_{d,i})-{\delta }_{\min }$$

This is the same estimator NTP uses, applied one-directionally. Its bias is exactly ${\delta }_{\min }$, which can be calibrated once per network class and subtracted; its variance falls as the sample grows, because a minimum over more samples is more likely to include a near-best transit.

The step-detection threshold exists because device clocks do not drift smoothly — they drift slowly and then jump when the OS resynchronises. A rolling minimum over a long window is blind to a step for the length of the window, so the estimator needs a companion detector: when the current $t_a-t_d$ falls below the running minimum by more than the threshold, the clock has stepped and the window must be reset rather than merged.

Step-by-step implementation #

1. Collect the pair at the ingest edge, before anything reorders. The arrival timestamp must be taken as close to the wire as possible; a timestamp assigned after the event has queued measures the queue, not the network.

2. Maintain a windowed minimum per device. A monotonic deque gives an $O(1)$ amortised rolling minimum without storing the whole window.

from __future__ import annotations
from collections import deque
from dataclasses import dataclass, field

@dataclass(slots=True)
class DriftEstimator:
    """Rolling minimum of (arrival - device) with step detection."""
    window: int = 200
    min_transit_ms: float = 40.0
    step_threshold_ms: float = 500.0
    _dq: deque[tuple[int, float]] = field(default_factory=deque)  # (seq, delta)
    _seq: int = 0
    offset_ms: float | None = None
    steps: int = 0

    def observe(self, arrival_ms: int, device_ms: int) -> float | None:
        delta = float(arrival_ms - device_ms)
        if self._dq and delta < self._dq[0][1] - self.step_threshold_ms:
            self._dq.clear()                     # the clock stepped: start over
            self.steps += 1
        while self._dq and self._dq[-1][1] >= delta:
            self._dq.pop()                       # maintain increasing minima
        self._dq.append((self._seq, delta))
        while self._dq and self._dq[0][0] <= self._seq - self.window:
            self._dq.popleft()
        self._seq += 1
        if self._seq >= 40 or self.steps:
            self.offset_ms = self._dq[0][1] - self.min_transit_ms
        return self.offset_ms

3. Calibrate ${\delta }_{\min }$ per network class, not globally. Devices on Wi-Fi, LTE and 5G have materially different floors, and a single constant biases whole populations. Estimate it as the minimum of $t_a-t_d$ across devices known to have a good clock — those whose offset estimate is already stable and near zero — grouped by the network the device reports.

4. Correct on read, never rewrite the raw timestamp. Store the raw device timestamp and the estimated offset separately and apply the correction where event time is consumed. Rewriting destroys the evidence needed to re-estimate later and makes every downstream bug unattributable.

5. Bound the correction and route the outliers. A device whose estimated offset exceeds a couple of hours is not skewed but broken — a dead RTC, a factory-default clock, a spoofing attempt. Correct it, mark the events low-confidence, and route the device to an operational queue rather than silently trusting a two-hour adjustment.

6. Re-estimate continuously and export the distribution. The fleet-wide histogram of offsets is a genuinely useful operational signal: a bimodal distribution nearly always indicates a firmware version with a clock bug, and it will be visible here weeks before it is visible in trigger complaints.

Benchmark and verification #

Measured over 4,000 devices for seven days, with 60 devices carrying a deliberately offset clock as ground truth:

The mean is the estimator most implementations reach for and the worst available: it is biased upward by the whole transit distribution, so it never converges to the offset at all — its 1,240 ms residual is essentially the mean network delay. The minimum converges to within 180 ms, which is dominated by the calibration of ${\delta }_{\min }$ rather than by the estimator.

The step-recovery column is the operational difference. Without step detection, a device that resynchronises its clock carries a stale offset for the length of the window — 200 fixes, or over three minutes at 1 Hz — during which every timestamp it produces is corrected in the wrong direction, which is worse than not correcting at all. With detection, recovery is immediate.

The downstream effect is what justifies the work:

The watermark row is the one that surprises teams. A single device with a fast clock holds the watermark ahead of reality, causing every other device’s genuinely on-time events to be classified as late and routed to reconciliation; correcting the offset removes the stall at source rather than by widening the allowed-lateness bound, which is the usual and much more expensive workaround.

Failure modes and edge cases #

The backlog case in the fifth row is common and quietly corrupts the estimate. A device that has been offline flushes a queue of stored fixes on reconnection; if it stamps all of them with the current time, hundreds of samples arrive with near-zero $t_a-t_d$ and drag the rolling minimum below the true offset. Detect the burst — many fixes with identical or near-identical device timestamps — and exclude it from the estimator while still admitting the fixes themselves, which remain useful positionally even though their timestamps are unreliable.

Finally, a note on scope. Correcting the offset makes a device’s timestamps comparable with everyone else’s; it does not make them precise. A device reporting at 1 Hz with a 250 ms internal latency between fix acquisition and timestamping still attributes each position to a moment 250 ms after it occurred, and no amount of offset estimation recovers that. Where sub-second event-time precision genuinely matters — crossing instants for tolling, for example — the interpolated instants from segment-crossing detection are a better source than any device timestamp.

Related #

• Event-Time Ordering & Clock Skew — the parent topic and the watermark this correction unblocks.
• Hybrid Logical Clocks for Geofence Event Ordering — ordering guarantees, which correction complements rather than replaces.
• Late-Arriving Telemetry and Watermarks in Geofence Streams — where an uncorrected fast clock does most of its damage.