Alerting Thresholds for GC Pause Times

A stop-the-world garbage-collection pause is the one latency source in a geofence pipeline that no request-scoped timer can attribute correctly. When CPython’s generational collector runs a generation-2 sweep, it freezes every coroutine on the event loop simultaneously; the pause lands on whichever event happened to be mid-flight, so the P99 spike appears to belong to an innocent point-in-polygon call while the real cause — object churn from a burst of telemetry — is already over. This page sits under Production Monitoring & Observability for Geofence Pipelines and the event routing and backpressure architecture, and it answers a precise operational question: how do you instrument GC pauses directly, correlate them with your latency histogram, and set an alert threshold that fires on the cause rather than the symptom?

Concept and specification #

CPython uses reference counting for immediate reclamation plus a generational cyclic collector for reference cycles. The cyclic collector has three generations; objects that survive a collection are promoted to the next. Generation 0 is collected frequently and cheaply; generation 2 holds long-lived objects and its sweep must traverse the largest live set, making it the expensive, latency-relevant one. A collection of generation $g$ triggers when its allocation counter exceeds a threshold, and the pause cost scales with the number of tracked objects it must scan:

$$t_{\text{pause}}(g)\approx \alpha +\beta \cdot N_{\text{tracked}}(g)$$

where $N_{\text{tracked}}(g)$ is the count of container objects in generations $\le g$. For a geofence node holding a large in-memory spatial index of Python objects, $N_{\text{tracked}}(2)$ is enormous, so a gen-2 sweep is exactly the multi-millisecond stop-the-world event that blows P99. The gc module exposes the levers to measure and control it:

The instrumentation target is a per-generation pause histogram plus a gen-2 collection counter, both exported to Prometheus so they align on the same time axis as the P99 latency histogram.

Step-by-step implementation #

Prerequisites: Python 3.11+, prometheus_client, and an evaluator that holds a long-lived spatial index in memory. All timing uses time.perf_counter for monotonic precision.

1. Instrument the gc callbacks to time every pause. Register a hook that records start/stop and attributes the span to the collected generation.

import gc
import time
from prometheus_client import Counter, Histogram

GC_PAUSE = Histogram(
    "geofence_gc_pause_seconds",
    "Stop-the-world GC pause duration by generation",
    labelnames=("generation",),
    buckets=(0.0005, 0.001, 0.002, 0.005, 0.010, 0.025, 0.050, float("inf")),
)
GC_COLLECTIONS = Counter(
    "geofence_gc_collections_total", "GC collections by generation", ("generation",)
)

_gc_start: float = 0.0

def _gc_hook(phase: str, info: dict[str, int]) -> None:
    global _gc_start
    if phase == "start":
        _gc_start = time.perf_counter()  # mark the stop-the-world entry
    elif phase == "stop":
        gen = str(info["generation"])
        # info['generation'] names which generation was collected — gen-2 is the
        # one to alert on; gen-0/1 pauses are sub-millisecond and expected.
        GC_PAUSE.labels(generation=gen).observe(time.perf_counter() - _gc_start)
        GC_COLLECTIONS.labels(generation=gen).inc()

gc.callbacks.append(_gc_hook)

Gotcha: the start and stop callbacks are not reentrant-safe across threads, but CPython runs the cyclic collector while holding the GIL, so a module-level _gc_start is correct here — a nested collection cannot interleave. Do not add a lock; it would deadlock inside the collector.

2. Sample gc.get_stats() on a scrape callback for corroboration. The counter above is authoritative for rate, but get_stats() gives cumulative collected/uncollectable for leak triage.

from prometheus_client import Gauge

GC_UNCOLLECTABLE = Gauge("geofence_gc_uncollectable", "Objects gc could not free", ("generation",))

def refresh_gc_gauges() -> None:
    # Called at scrape time; get_stats() is a cheap snapshot of collector counters.
    for gen, st in enumerate(gc.get_stats()):
        GC_UNCOLLECTABLE.labels(generation=str(gen)).set(st["uncollectable"])

3. Correlate gen-2 collections with P99 in a recording rule. Align the two series on the same window.

groups:
  - name: geofence_gc
    interval: 15s
    rules:
      - record: geofence:gc_pause:p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(geofence_gc_pause_seconds_bucket[5m])) by (le, generation))
      - alert: GeofenceGcPauseHigh
        # gen-2 pause p99 above 2ms is the actionable threshold; gen-0 excluded.
        expr: geofence:gc_pause:p99{generation="2"} > 0.002
        for: 45s
        labels: { severity: page }

4. Mitigate by freezing the warm index out of the scan set. After warm-up, gc.freeze() moves the long-lived spatial index into a permanent generation the collector never scans.

def on_warmup_complete(index: object) -> None:
    gc.collect()   # settle transient warm-up garbage first
    gc.freeze()    # move surviving long-lived objects (the index) out of scan scope
    # Optionally raise thresholds so steady-state churn triggers gen-2 far less often.
    gc.set_threshold(50_000, 500, 1_000)

5. For burst windows, take manual control. Disable automatic collection during a known surge and sweep in an idle trough.

import asyncio

async def burst_guarded(evaluate) -> None:
    gc.disable()               # no stop-the-world pauses during the burst
    try:
        await evaluate()
    finally:
        gc.collect(2)          # one controlled full sweep in the trough
        gc.enable()            # restore automatic collection

Benchmark and verification #

The change to measure is the effect of gc.freeze() plus raised thresholds on a node holding a ~1M-object spatial index at 30k events/sec. Before, an untuned gen-2 sweep scanned the whole index and injected a 55ms pause roughly every few minutes; after, the index is frozen out of scope and gen-2 sweeps are rare and small.

A minimal bench to reproduce the pause distribution:

import gc, time, statistics

def measure_pauses(build_index, freeze: bool, cycles: int = 2000) -> dict[str, float]:
    index = build_index()          # ~1M long-lived objects
    if freeze:
        gc.collect(); gc.freeze()
    pauses: list[float] = []
    for _ in range(cycles):
        churn = [dict(i=i) for i in range(500)]  # transient per-event allocation
        t0 = time.perf_counter()
        gc.collect(2)              # force a full sweep to time it
        pauses.append((time.perf_counter() - t0) * 1e3)  # ms
        del churn
    pauses.sort()
    return {"p50": statistics.median(pauses),
            "p95": pauses[int(0.95 * cycles)],
            "p99": pauses[int(0.99 * cycles)]}

Verify the correlation is real before shipping the mitigation: overlay geofence:gc_pause:p99{generation="2"} on the evaluation P99 in Grafana and confirm the spikes coincide to within one scrape interval. If they do not align, GC is not your tail and freezing will not help — profile with py-spy instead.

Failure modes and edge cases #

• Freezing too late. gc.freeze() moves currently live objects out of scope; if you call it before the index is fully built, the not-yet-allocated objects stay in the scanned generations and gen-2 sweeps remain expensive. Freeze only after warm-up completes and a gc.collect() has settled transient garbage.
• Disabled GC leaking cycles. gc.disable() stops cyclic collection entirely — reference-counted objects still free immediately, but any object cycle (a polygon referencing a parent zone that references it back) never gets reclaimed and RSS climbs without bound. Only disable across a bounded burst window and always re-enable in a finally, pairing with an RSS gauge so an unbounded climb pages you.
• Alerting on gen-0. Gen-0 collections fire thousands of times an hour and are sub-millisecond; an alert without a generation="2" label filter flaps constantly and trains the on-call to ignore it. Threshold only the generation that actually pauses.
• Threshold tuning that defers the cliff. Raising gc.set_threshold reduces collection frequency but each deferred gen-2 sweep then scans a larger set, so an over-raised threshold trades many small pauses for one enormous one. Tune against the pause histogram, not the collection rate — the goal is a bounded P99, not fewer collections.
• perf_counter inside the callback. The gc callback runs while the collector holds the GIL; time.perf_counter() is safe and cheap, but any allocation inside the hook can itself trigger accounting — keep the hook allocation-free (no f-strings building new objects on the hot path; pre-label the metric).

Related #

• Production Monitoring & Observability for Geofence Pipelines — parent overview; GC pause is the treacherous third saturation signal.
• Prometheus metrics for queue depth and P99 latency — sibling; the P99 histogram these gen-2 spikes correlate against.
• py-spy flame graphs for asyncio spatial pipelines — sibling; use it when the tail is CPU, not GC.
• Memory-constrained spatial processing — the RSS caps and buffer reuse that keep the gen-2 live set small.