Prometheus Metrics for Queue Depth and P99 Latency

The two numbers that decide whether a geofence evaluator is healthy are the depth of its bounded evaluation queue and the P99 of its per-phase latency — and both are routinely instrumented wrong. Queue depth gets written on the hot path where it adds contention; latency gets exposed as a summary that cannot be aggregated across replicas, so the “fleet P99” on the dashboard is arithmetic nonsense. This page sits under Production Monitoring & Observability for Geofence Pipelines and the broader event routing and backpressure architecture, and it answers one precise question: how do you shape Prometheus metrics so that queue depth and P99 latency are both cheap to collect and honest under aggregation?

Concept and specification #

A Prometheus histogram partitions observations into cumulative buckets, each a counter of events at or below an upper bound le. A quantile is recovered at query time by histogram_quantile, which locates the bucket containing the target rank and interpolates linearly within it. That interpolation is the entire source of latency-metric error: the reported P99 can only be as precise as the bucket it falls in. The relative error is bounded by the containing bucket’s width relative to the true value:

$${\epsilon }_{\text{rel}}\le \frac{b_{\text{upper}}-b_{\text{lower}}}{q_{\text{true}}}$$

For a 40ms P99 SLO, a bucket edge of [35ms, 50ms] yields up to 37% error — enough to hide a real breach. Buckets edged at [38ms, 42ms] hold the estimate inside ~10%. The design rule follows directly: place your buckets densely across the SLO boundary and let the tail be coarse.

Queue depth is simpler in kind but subtler in placement. asyncio.Queue.qsize() is an instantaneous integer, so it maps to a gauge, not a counter or histogram. The trap is when you read it. Writing the gauge on every enqueue/dequeue adds a synchronized write to the hot path for no fidelity gain, because Prometheus samples the gauge only at scrape time — a 10s scrape interval cannot see per-event fluctuations regardless. The correct pattern reads qsize() lazily via a gauge callback at scrape time.

The last row is the crux. A summary computes its φ-quantiles inside the process and exposes the finished value. You cannot average two processes’ P99 summaries — avg(p99_a, p99_b) is not the P99 of the union, because the two distributions may overlap arbitrarily and the mean of two 99th percentiles has no defined relationship to the 99th percentile of the combined sample. A histogram exposes raw bucket counts, which do sum, so a true fleet quantile is histogram_quantile(0.99, sum(rate(bucket[5m])) by (le)). For any geofence fleet — a sharded evaluator or a Kafka consumer group — the histogram is the only correct choice. The one cost you pay for this is a fixed set of per-bucket time series per label combination, so the bucket count is a deliberate budget: more buckets buy quantile precision but multiply cardinality, which is why the SLO-straddling layout below spends its buckets where the alert threshold lives and nowhere else.

Step-by-step implementation #

Prerequisites: Python 3.11+, prometheus_client>=0.20, an async evaluator draining a bounded asyncio.Queue. Metrics are exposed on an ASGI /metrics endpoint scraped by Prometheus.

1. Define the histogram with SLO-aligned buckets. Dense across 40ms, coarse beyond.

from prometheus_client import Histogram, Gauge, Counter

# Buckets straddle the 40ms P99 SLO so histogram_quantile interpolates a real
# number, not a fiction across a 15ms-wide bucket.
PHASE_LATENCY = Histogram(
    "geofence_phase_latency_seconds",
    "Per-phase evaluation latency in seconds",
    labelnames=("phase",),
    buckets=(0.005, 0.010, 0.015, 0.020, 0.025, 0.030, 0.035,
             0.038, 0.040, 0.042, 0.045, 0.050, 0.075, 0.100, float("inf")),
)

2. Register the queue-depth gauge as a callback. The gauge reads qsize at scrape time, never on the hot path.

import asyncio

class QueueMetrics:
    def __init__(self, queue: asyncio.Queue[object], capacity: int) -> None:
        self._q = queue
        self._capacity = capacity
        depth = Gauge("geofence_queue_depth", "Current bounded-queue depth")
        util = Gauge("geofence_queue_utilization", "Queue depth as a fraction of capacity")
        # set_function defers the read to scrape time — no hot-path write, no lock.
        depth.set_function(self._q.qsize)
        util.set_function(lambda: self._q.qsize() / self._capacity)

Gotcha: set_function callbacks run in the scrape thread. Keep them O(1) and side-effect-free — qsize() qualifies, but never put a lock acquisition or an I/O call in a gauge callback or you serialize the scrape against the hot path.

3. Time each phase with the histogram context manager. It records on both the success and exception paths.

async def evaluate(phase: str, point: object) -> str | None:
    # .time() observes the duration on __exit__, including when the body raises,
    # so error latency is captured rather than silently dropped.
    with PHASE_LATENCY.labels(phase=phase).time():
        return await asyncio.to_thread(_containment, point)  # GIL-releasing geometry

def _containment(point: object) -> str:
    return "ZONE_ALPHA_01"

4. Add a recording rule for the true fleet P99. Precompute the expensive quantile so dashboards and alerts read a cheap series.

# prometheus/rules/geofence.yml
groups:
  - name: geofence_latency
    interval: 15s
    rules:
      - record: geofence:phase_latency:p99
        # sum bucket rates across replicas BEFORE quantile — this is what makes
        # the fleet P99 correct; per-replica p99 averaged is meaningless.
        expr: histogram_quantile(0.99,
                sum(rate(geofence_phase_latency_seconds_bucket[5m])) by (le, phase))
      - alert: GeofenceP99Breach
        expr: geofence:phase_latency:p99 > 0.040
        for: 45s   # 3 scrape intervals — ride out a single-scrape blip
        labels: { severity: page }

5. Alert on queue saturation as a leading indicator. Depth crosses 80% before latency degrades.

      - alert: GeofenceQueueSaturated
        expr: geofence_queue_utilization > 0.80
        for: 90s
        labels: { severity: page }
        annotations:
          summary: "Eval queue above 80% for 90s — shedding imminent"

Benchmark and verification #

The change to measure is the switch from a per-instance summary plus a hot-path-written depth gauge to aggregatable histograms plus a callback gauge, holding the evaluator and load fixed at 30k events/sec across 6 replicas. The before/after captures both correctness (does the fleet P99 mean anything) and cost (scrape payload and cardinality).

A minimal bench for the instrumentation overhead itself:

import time, statistics
from prometheus_client import Histogram

H = Histogram("bench_seconds", "bench", buckets=(0.01, 0.02, 0.04, float("inf")))

def bench(runs: int = 50, n: int = 100_000) -> dict[str, float]:
    samples: list[float] = []
    for _ in range(runs):
        t0 = time.perf_counter()
        for _ in range(n):
            with H.time():
                pass  # measure the observe/record cost, not the workload
        samples.append((time.perf_counter() - t0) / n * 1e6)  # us/observe
    samples.sort()
    return {"p50_us": statistics.median(samples),
            "p95_us": samples[int(0.95 * runs)],
            "p99_us": samples[int(0.99 * runs)]}

Verify the fleet P99 is honest by cross-checking: histogram_quantile over the summed bucket rates must track a known injected latency within the bucket-width bound. Inject a controlled 38ms delay into one phase and confirm the recorded P99 lands in [38ms, 40ms], not smeared into a coarse bucket.

Failure modes and edge cases #

• Label cardinality explosion. A device_id or fence_id label multiplies every bucket series by the number of distinct values — millions of series, an OOM’d Prometheus. Keep labels bounded: phase, kind, generation. The correct home for high-cardinality identity is a trace exemplar, not a label. This is the single most common way a geofence metrics setup takes down its own monitoring.
• Bucket miscoverage. If real latency drifts past your coarsest finite bucket into +Inf, histogram_quantile cannot interpolate and returns the last finite edge — the P99 flatlines at, say, 100ms and hides a true 300ms tail. Periodically check the +Inf bucket’s share; if it exceeds ~1%, add higher buckets.
• Rate window too short. rate(bucket[1m]) over a 15s scrape sees only ~4 samples and reports jagged, alert-flapping quantiles. Use a window of at least 4x the scrape interval ([5m] for 15s scrapes) so the rate is stable.
• Counter reset on restart. A replica restart resets its counters to zero; rate() handles the reset, but a raw histogram_quantile over _bucket (not rate(_bucket)) will not. Always wrap bucket series in rate() before summing.
• Gauge staleness across scrapes. A qsize() callback samples an instantaneous level; a burst that fills and drains the queue entirely between two scrapes is invisible. Pair the depth gauge with the shed counter — the counter captures the overflow the gauge missed, because every shed event increments regardless of scrape timing.

Related #

• Production Monitoring & Observability for Geofence Pipelines — parent overview of the four golden signals this metric set implements.
• py-spy flame graphs for asyncio spatial pipelines — sibling CPU-profiling technique when the histogram says a phase is hot.
• Alerting thresholds for GC pause times — sibling on the saturation signal that correlates with P99 spikes.
• Sizing bounded asyncio queues for geofence pipelines — choosing the maxsize this depth gauge measures against.