Production Monitoring & Observability for Geofence Pipelines

A geofence pipeline that passes every unit test still fails in production for reasons no test reproduces: a generation-2 garbage-collection pause lands inside the spatial-index phase and blows P99 for every event queued behind it; a bounded asyncio.Queue creeps to 95% depth for ninety seconds during a rush-hour surge and starts shedding compliance triggers; a downstream consumer slows by 4ms and the circuit breaker trips a cascade nobody watched. None of these are code defects — they are operating-point failures, visible only through instrumentation. This page expands the operational discipline introduced in the event routing and backpressure architecture into a concrete observability contract: what to measure, at what resolution, and what threshold turns a metric into a page. The organizing frame is Google’s four golden signals — latency, traffic, errors, saturation — mapped onto the specific phases of a geofence evaluator so that each signal points at a named subsystem rather than at a vague “the service is slow.”

The reader is a backend or platform engineer who already owns a running pipeline and needs to know when it is about to break, not after. The failure mode this page addresses is silent operating-point drift — the slow slide from a healthy 15ms P50 to a tail-latency collapse that the dashboards showed coming for ten minutes if anyone had known which panel to read.

The Four Golden Signals Mapped to a Geofence Pipeline #

The golden signals are generic; their value comes from binding each one to a concrete measurement on a concrete phase. A dashboard that reports “service latency” is useless when the spatial-index phase and the routing phase fail independently. Instrument per phase, per signal.

Latency is the time each phase spends per event, measured as a histogram — never a mean. The mean hides the tail that actually breaks the SLA. Record time.perf_counter deltas at ingress, index lookup, exact point-in-polygon evaluation, and emission into four separate histograms, so a P99 spike attributes to a named phase. The service-level objective is explicit: end-to-end P99 < 40ms, with the spatial phase holding P95 < 18ms at 30k events/sec. Histograms let Prometheus compute those quantiles server-side and, critically, aggregate them across replicas — a property summaries lack.

Traffic is the offered load, an events-per-second counter per phase. It is the denominator for every other signal: a rising error count at flat traffic is a regression, whereas the same error count at 3x traffic is expected shedding. Track ingress rate, evaluation rate, and emission rate separately; a gap between ingress and evaluation rate is queue growth, visible before the saturation gauge confirms it.

Errors are the events that did not complete their intended path: shed_count from a full bounded queue, dead-letter-queue insertion rate, circuit-breaker trip count, and coordinate-validation rejects. In a backpressure-aware pipeline, a nonzero shed_count is not automatically an error — deliberate shedding of idle-vehicle pings is correct behavior. The alert fires on the rate of change and on the class of shed event: shedding compliance triggers is a page; shedding idle pings during a surge is expected. This is why the emission side leans on idempotent trigger emission — replayed events after a shed must not double-fire.

Saturation is how close a bounded resource sits to its limit: asyncio.Queue.qsize() against maxsize, resident set size (RSS) against the node cap, process-pool occupancy, and cumulative GC pause time. The SLO here is a headroom target, not a point value: queue depth must stay under 80% of capacity for 99.9% of one-second samples. Saturation is the leading indicator — it moves before latency does, which is what makes it the most valuable of the four signals to alert on.

Latency Profiles and Signal Fidelity #

The fidelity of the latency signal depends entirely on histogram bucket layout, and this is where most geofence observability quietly fails. A histogram approximates a quantile by counting events into fixed buckets; the reported P99 is only as precise as the bucket boundary it lands in. If your buckets jump from 25ms to 50ms, a true P99 of 38ms reports as “somewhere in [25, 50]” and Prometheus interpolates linearly — a fiction. The relative error of a histogram_quantile estimate is bounded by the width of the containing bucket:

$${\epsilon }_{\text{rel}}\le \frac{b_{\text{upper}}-b_{\text{lower}}}{q_{\text{true}}}$$

so to hold a 40ms P99 estimate within 10% error you need bucket edges no coarser than ~4ms around the 40ms region. The design rule is to place dense buckets across your SLO boundary — for a 40ms target, buckets at 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 75, 100ms — and let the tail beyond be coarse. The full bucket-design derivation and the reason summaries cannot substitute is developed in Prometheus metrics for queue depth and P99 latency.

The second fidelity trap is aggregation. A summary computes its quantiles inside each process and exposes the finished number; you cannot average two processes’ P99 summaries to get a fleet P99 — the math does not compose. A histogram exposes raw bucket counts, so histogram_quantile(0.99, sum(rate(...)) by (le)) computes a true cross-replica quantile. For any signal you need to reason about across a Kafka consumer group or a sharded evaluator fleet, the histogram is mandatory.

Implementation Reference #

The instrumentation below wraps each phase of an async evaluator. It uses prometheus_client histograms for latency, a counter for traffic, labeled counters for errors, and a gauge callback for queue saturation. The inline comments flag the non-obvious decisions.

import asyncio
import gc
import time
from dataclasses import dataclass
from prometheus_client import Counter, Gauge, Histogram

# SLO-aligned buckets: dense across the 40ms P99 boundary, coarse in the tail.
# Bucket edges must straddle the SLO or histogram_quantile interpolates a fiction.
PHASE_LATENCY = Histogram(
    "geofence_phase_latency_seconds",
    "Per-phase evaluation latency",
    labelnames=("phase",),
    buckets=(0.005, 0.010, 0.015, 0.020, 0.025, 0.030,
             0.035, 0.040, 0.045, 0.050, 0.075, 0.100, float("inf")),
)
EVENTS = Counter("geofence_events_total", "Events entering a phase", ("phase",))
ERRORS = Counter("geofence_errors_total", "Non-completing events", ("phase", "kind"))
QUEUE_DEPTH = Gauge("geofence_queue_depth", "Bounded eval-queue depth")
GC_PAUSE = Counter("geofence_gc_pause_seconds_total", "Cumulative GC pause", ("generation",))


@dataclass(slots=True)
class Telemetry:
    device_id: str
    lat: float
    lon: float
    idempotency_key: str  # carried into the trace context for propagation


class ObservableEvaluator:
    def __init__(self, queue_capacity: int = 100_000) -> None:
        self._queue: asyncio.Queue[Telemetry] = asyncio.Queue(maxsize=queue_capacity)
        # qsize is polled by Prometheus via a gauge callback, not written per event:
        # writing the gauge on the hot path adds lock contention for no fidelity gain.
        QUEUE_DEPTH.set_function(self._queue.qsize)
        # gc callback attributes pause time to the collected generation.
        gc.callbacks.append(self._on_gc)
        self._gc_t0: float = 0.0

    def _on_gc(self, phase: str, info: dict[str, int]) -> None:
        if phase == "start":
            self._gc_t0 = time.perf_counter()
        else:  # 'stop' — record the stop-the-world span against its generation
            GC_PAUSE.labels(generation=str(info["generation"])).inc(
                time.perf_counter() - self._gc_t0
            )

    async def evaluate_phase(self, phase: str, point: Telemetry) -> str | None:
        EVENTS.labels(phase=phase).inc()
        # Histogram.time() context manager records the observed duration on exit,
        # including on the exception path, so error latency is not lost.
        with PHASE_LATENCY.labels(phase=phase).time():
            try:
                # to_thread keeps GIL-releasing geometry off the event loop.
                return await asyncio.to_thread(self._containment, point)
            except ValueError:  # NaN / out-of-range coordinate rejected at validation
                ERRORS.labels(phase=phase, kind="bad_coord").inc()
                return None

    async def enqueue(self, point: Telemetry) -> bool:
        if self._queue.full():
            ERRORS.labels(phase="ingress", kind="shed").inc()  # deliberate shed
            return False
        await self._queue.put(point)
        return True

    def _containment(self, point: Telemetry) -> str:
        return "ZONE_ALPHA_01"  # placeholder for the Cython/Numba PIP routine

Two choices carry the weight. First, QUEUE_DEPTH.set_function(self._queue.qsize) defers the gauge read to scrape time rather than writing it per event — polling on the hot path would add contention and give no extra fidelity, since Prometheus samples on a fixed interval anyway. Second, the GC pause is captured through gc.callbacks and attributed to the collected generation, which is what lets you correlate a generation-2 sweep with a P99 spike instead of guessing. That correlation is the whole subject of alerting thresholds for GC pause times.

Saturation, Memory, and GC as Leading Indicators #

Saturation signals lead latency because a queue fills before its consumers slow visibly. Poll three gauges continuously. Queue depth against maxsize is the primary one — the SLO of “under 80% for 99.9% of samples” means that in a five-minute window of 300 one-second samples, no more than one sample may exceed 80k on a 100k-capacity queue before you investigate. RSS growth is the second: a node capped at 1.5GB that climbs 50MB per hour without matching traffic growth is retaining buffers or leaking an index cursor, and tracemalloc snapshot diffs every 10k evaluations name the allocation site directly. The memory-constrained spatial processing discipline sets those caps.

GC pause is the third and the most treacherous saturation signal, because it is invisible to request-scoped tracing — a stop-the-world gen-2 collection freezes every coroutine simultaneously, so the latency lands on whichever unlucky event was mid-flight rather than on the event that caused the churn. Reading gc.get_stats() gives per-generation collection counts; the gc.callbacks hook above gives pause duration. When gen-2 collections rise in lockstep with the P99 histogram, the fix is gc.freeze() after warm-up to move long-lived index objects out of the scan set, targeting a pause budget under 2ms.

Continuous profiling closes the saturation picture from the CPU side. A py-spy sidecar sampling the live process at 100Hz builds a flame graph without pausing the interpreter or requiring a redeploy — the difference between a native point-in-polygon frame dominating CPU and the whole process idling on to_thread wait (pool exhaustion, not slow geometry) is one glance at the graph.

Trace Propagation and Structured Logging #

Metrics tell you a phase is slow; traces tell you which event was slow and what it carried. Propagate a trace context across all four phases keyed on the event’s idempotency key, so a single ENTER trigger that took 90ms can be reconstructed end-to-end — including the time it waited in the bounded queue, which request-scoped timers miss. The idempotency key doubling as the trace correlation id means a replayed event after a shed or a consumer-group rebalance shows up in the trace as the same logical event, not a mysterious duplicate.

Structured logging is the third leg of the tripod and the cheapest to get wrong. Log JSON, not prose, with the phase, device id, idempotency key, and measured latency as first-class fields — never interpolated into a message string — so logs join to metrics and traces on the same keys. Sample aggressively: at 25k events/sec, logging every event is itself a saturation source, so log 100% of errors and breaker trips and 1:1000 of successes.

Operational Runbook: Instrumenting a Geofence Pipeline #

This is the ordered procedure to take an uninstrumented evaluator to full golden-signal coverage. Each step produces an artifact the next step depends on.

1. Expose the metrics endpoint. Mount prometheus_client’s ASGI /metrics handler on the service and confirm a manual curl returns the default process metrics. Nothing else works until the scrape target is live.
2. Add the four latency histograms. Wrap each phase — ingress, index lookup, exact PIP, routing — in a Histogram(...).time() context with SLO-aligned buckets straddling the 40ms P99 boundary. Verify each phase appears as a distinct phase label.
3. Add traffic counters and error counters. One events_total counter per phase and one errors_total with a kind label distinguishing shed, dlq, breaker_trip, and bad_coord. Confirm rate(geofence_events_total[1m]) tracks your load generator.
4. Wire the saturation gauges. Register a Gauge.set_function callback for asyncio.Queue.qsize(), an RSS gauge from resource.getrusage, and the gc.callbacks pause counter. Confirm queue depth moves under a synthetic burst.
5. Write recording and alerting rules. Add a Prometheus recording rule for per-phase P99 (histogram_quantile(0.99, sum(rate(...)) by (le, phase))) and alerting rules: P99 > 40ms for 3 scrape intervals, queue depth > 80% for 90s, GC pause rate > 2ms/collection, DLQ rate > 0 sustained. Route them through Alertmanager.
6. Attach continuous profiling. Run py-spy record --pid <pid> --duration 60 --rate 100 on a cron or as a sidecar, archiving flame graphs so a post-incident investigation has a CPU profile from before the incident, not just after.
7. Validate end-to-end under a load test. Drive the pipeline at 1x, 2x, and 3x baseline; confirm the saturation gauges lead the latency histograms, the shed counter rises before the queue overflows, and every alert fires at its designed threshold. An alert that never fires in a load test will not fire in production either.

Architectural Guidance: Decision Matrices #

Two instrumentation choices recur and both have a wrong default. The first is histogram versus summary.

The second is pull versus push collection.

The invariant across both: instrument the boundary, not the average. A geofence pipeline degrades at a phase edge — the queue between index and routing, the pool feeding the exact check — and the signal that catches it is always the per-phase histogram and the saturation gauge, read together, before the tail collapses.

Frequently Asked Questions #

Related #

• Event Routing & Backpressure — the parent architecture whose operating point this instrumentation watches.
• Prometheus metrics for queue depth and P99 latency — histogram bucket design and recording rules in depth.
• py-spy flame graphs for asyncio spatial pipelines — non-intrusive CPU profiling of a live process.
• Alerting thresholds for GC pause times — correlating gen-2 collections with P99 spikes.
• Backpressure and flow-control strategies — the shedding behavior the error signal reports on.