Reducing P99 Latency in Python Geofence Services

Real-time geofence evaluation sits at the intersection of spatial computation, high-throughput telemetry ingestion, and a hard millisecond budget. In mobility and logistics platforms resolving millions of GPS pings per second, it is the P99 — not the median — that decides SLA compliance, trigger accuracy, and downstream queue stability. A pipeline can report a 0.4ms median and still bleed money, because the tail spikes are exactly the events that cause delayed dispatch routing, stale driver-passenger matching, and cascading backpressure across the event bus. This page is the tail-reduction counterpart to latency budget allocation for real-time triggers: the parent breakdown assigns each pipeline stage a per-phase ceiling, and this deep-dive shows how to keep the spatial evaluation stage inside its slice when the runtime is CPython. It operates within the wider contract defined in Core Architecture & Latency Constraints, where every event carries a fixed per-trigger budget.

The reason P99 is the only metric worth optimizing here: a geofence trigger is a physical-state race. The vehicle is already moving. A surge-zone entry that fires at the 99th-percentile latency of 180ms has, at 50 km/h, let the asset travel 2.5 metres past the boundary before the trigger resolves — enough to mis-price a ride or miss a customs perimeter. Median latency never causes an incident; the tail always does.

Concept & Specification #

Tail latency in a geofence service is not one number degrading; it is the worst-case path through a multi-stage pipeline. Decompose the per-event latency into its additive terms:

$$L_{\text{event}}=L_{\text{decode}}+L_{\text{prefilter}}+L_{\text{pip}}+L_{\text{dispatch}}$$

Each term has a different tail signature. Decode and dispatch are I/O-shaped — their tail is queueing under burst. The pre-filter and point-in-polygon (PiP) terms are compute-shaped — their tail is GIL serialization and garbage-collection pauses landing on top of an in-flight evaluation. A useful framing for the PiP stage with a coarse pre-filter in front of it is the expected cost

$$E[L_{\text{spatial}}]=L_{\text{prefilter}}+p_{\text{hit}}\cdot L_{\text{pip}}$$

where $p_{\text{hit}}$ is the fraction of candidates that survive the bounding-box reject. Because $L_{\text{pip}}\gg L_{\text{prefilter}}$ for any non-trivial polygon, driving $p_{\text{hit}}$ down with a cheap vectorized AABB test is the single highest-leverage move — it removes the expensive $O(V)$ kernel from the hot path for the ~90% of points that are nowhere near a fence. The precise containment kernel itself, whether point-in-polygon evaluation via ray casting or winding number, is $O(V)$ in vertex count $V$ and runs the same way regardless of approach.

The parameters that materially move the tail:

Step-by-Step Implementation #

Prerequisites: Python 3.11+, shapely>=2.0 (vectorized shapely.contains, GEOS C-API), numpy>=1.24, py-spy and tracemalloc for profiling. Coordinate batches are float64 arrays of shape (N, 2); fence polygons are pre-loaded into a read-only catalog so geometry never crosses a process boundary per call.

1. Attribute the tail before changing anything. Sample the running service with py-spy and confirm which term owns the P99 spike. A sharp right-tail skew points at gen-2 GC or cold-cache index lookups; a bimodal shape points at a synchronous blocking call masquerading as async work.

# py-spy record -o profile.svg --pid <pid> --duration 30 --rate 250
# Read the flame graph: width under shapely.contains == compute-bound tail;
# width under socket recv / json.loads == I/O-bound tail.

Gotcha: do not optimize a stage the profiler does not implicate. A “PiP is slow” assumption is wrong roughly half the time — the cost is frequently a gen-2 collection landing mid-evaluation, which a flame graph attributes to whatever frame was executing when the pause hit.

2. Vectorize the bounding-box pre-filter. Reject the overwhelming majority of points with a branchless NumPy comparison before any GEOS call. No Python-level iteration over candidates.

from __future__ import annotations

import numpy as np
from numpy.typing import NDArray

def aabb_reject(
    coords: NDArray[np.float64],      # shape (N, 2): lon, lat
    bbox: tuple[float, float, float, float],  # minx, miny, maxx, maxy
) -> NDArray[np.bool_]:
    """Return a mask of points inside the fence bounding box. ~5M points/sec."""
    minx, miny, maxx, maxy = bbox
    inside_x = (coords[:, 0] >= minx) & (coords[:, 0] <= maxx)
    inside_y = (coords[:, 1] >= miny) & (coords[:, 1] <= maxy)
    return inside_x & inside_y

3. Run precise containment only on survivors, with prepared geometry. Preparing a polygon once caches its GEOS index, so repeated checks against a static fence skip topology validation.

import shapely
from shapely import Polygon
from shapely.prepared import PreparedGeometry

def build_catalog(rings: dict[int, NDArray[np.float64]]) -> dict[int, PreparedGeometry]:
    """Prepare each fence once; prepared geometry caches the spatial index."""
    catalog: dict[int, PreparedGeometry] = {}
    for zone_id, ring in rings.items():
        poly = Polygon(ring)
        shapely.prepare(poly)         # mutates in place, returns None
        catalog[zone_id] = poly
    return catalog

def contains_batch(poly: Polygon, coords: NDArray[np.float64]) -> NDArray[np.bool_]:
    """One vectorized GEOS call for the whole survivor batch."""
    points = shapely.points(coords[:, 0], coords[:, 1])
    return shapely.contains(poly, points)

Gotcha: shapely.prepare returns None — never write poly = shapely.prepare(poly), or the catalog fills with None and every lookup raises. Prepare in place as above.

4. Move the kernel off the event loop. The Global Interpreter Lock serializes the Python glue around the C kernel, so an inline contains call starves every queued coroutine for the full evaluation. Offload with asyncio.to_thread (the GEOS kernel releases the GIL) and batch survivors so the offload overhead amortizes.

import asyncio

async def evaluate(
    catalog: dict[int, PreparedGeometry],
    zone_id: int,
    coords: NDArray[np.float64],
    bbox: tuple[float, float, float, float],
) -> NDArray[np.bool_]:
    survivors = aabb_reject(coords, bbox)     # cheap, stays on the loop
    if not survivors.any():
        return survivors                       # short-circuit: no kernel call
    result = survivors.copy()
    hit = await asyncio.to_thread(contains_batch, catalog[zone_id], coords[survivors])
    result[survivors] = hit
    return result

5. Tame garbage collection on the hot path. Per-event coordinate tuples and Point objects churn gen-0 and eventually trigger gen-2 scans that pause the loop. Keep the hot path on the vectorized array API, raise the gen-2 threshold so major collections defer past peak bursts, and freeze the static catalog out of the collector.

import gc

def tune_gc_for_ingestion() -> None:
    """Defer gen-2 scans during bursts; freeze the static fence catalog."""
    gc.set_threshold(50_000, 500, 1000)   # far higher than the 700/10/10 default
    gc.collect()                          # clean slate
    gc.freeze()                           # move survivors out of GC's reach

Gotcha: gc.freeze() only helps if you call it after loading the fence catalog and warming caches — anything allocated afterward is still scanned. Freeze at the end of startup, not the beginning.

6. Keep a deterministic fallback for SLA breach. When precise PiP exceeds its ceiling during a surge, fall back to a coarse H3 hexagon lookup at resolution 7. This guarantees bounded latency at the cost of temporary boundary precision, reconciled asynchronously once load subsides.

Benchmark / Verification #

Figures below are a 4-core worker, CPython 3.11, a 200-vertex municipal fence, NumPy-backed coordinate batches, sustained 50k events/sec offered load, perf_counter_ns timing, prepared geometry, gen-2 frozen after warm-up.

The before/after story is the first and last rows: P99 drops from 184ms to 5.6ms and sustained throughput rises roughly 12×, while the loop-lag probe confirms the event loop is no longer starved (176ms → 0.8ms). Two figures carry the lesson. The AABB row removes the kernel from ~90% of events but barely moves P99 on its own, because the survivors still run inline and still collide with GC — the prefilter is necessary but not sufficient. The GC row contributes a 7.4ms → 5.6ms P99 cut despite leaving median untouched: that delta is the gen-2 pauses that were landing on tail events.

A regression gate worth wiring into CI: assert P95 stays under 3ms and P99 under 8ms at the 50k/s fixture. Both thresholds catch the two real regressions — an accidental inline call (loop lag explodes) and a dropped AABB short-circuit (kernel runs on every event).

Failure Modes & Edge Cases #

NaN and out-of-range coordinates. A NaN longitude flows through shapely.contains as False silently, so a corrupt GPS sample looks like a legitimate “outside” result and never trips an alert. Filter with np.isfinite(coords).all(axis=1) at ingest and emit the reject count as a metric — a rising reject rate is a sensor fault, not a containment miss.

Self-intersecting and empty polygons. A self-intersecting fence makes ray casting’s even-odd rule disagree with the winding-number rule on overlap regions, so the same point returns different answers across kernels. Validate with shapely.is_valid and repair via shapely.make_valid before preparing. Empty or degenerate rings must be rejected at catalog build, not at evaluation, or the short-circuit in step 4 masks a missing fence as “no hits.”

GIL contention above core count. asyncio.to_thread recovers throughput only while the C kernel releases the GIL; the Python glue around it does not. Sizing the thread pool past the physical core count produces thrashing that raises P99 — the signature is throughput plateauing then regressing as you add workers. Pin the pool at min(32, os.cpu_count()) and let batching, not thread count, absorb load.

GC pressure masquerading as compute cost. The millisecond P99 outliers at high load are usually a gen-2 collection, not edge-scan cost. Confirm by correlating gc.get_stats() collection counts with the spike timestamps; if they align, the fix is allocation discipline (vectorized arrays over per-point Point construction) plus the freeze in step 5, not a faster kernel. This is the same heap discipline detailed in memory-constrained spatial processing.

Cold-cache index lookups after deploy. A freshly started pod evaluates its first thousand events against an unwarmed catalog, paying GEOS index construction on the hot path and producing a startup P99 cliff. Pre-warm by running the hot fence set through contains_batch during readiness-probe gating, before the load balancer routes traffic.

Related #

• Benchmarking spatial containment in async Python — the reproducible method behind the offload figures above, isolating GIL stall from I/O wait.
• Handling polygon edge cases in high-frequency telemetry — the degenerate-geometry and NaN-coordinate handling that keeps these tail numbers honest.
• Optimizing ray casting vs winding number for GPS streams — kernel-level choices that set the $L_{\text{pip}}$ term in the budget.
• Up one level: Latency Budget Allocation for Real-Time Triggers — the per-phase ceiling this spatial stage must fit inside.