py-spy Flame Graphs for asyncio Spatial Pipelines

When a geofence evaluator’s P99 quietly climbs from 15ms to 41ms, the metrics tell you which phase is slow but not which frame is burning the CPU. Attaching a conventional profiler means redeploying with instrumentation, which perturbs the very timing you are chasing and often cannot be done on a production node under load. py-spy solves this: it reads another process’s memory from the outside via process_vm_readv, samples its call stacks at a fixed rate, and renders a flame graph — all without pausing the interpreter, importing anything into the target, or requiring a restart. This page sits under Production Monitoring & Observability for Geofence Pipelines and the event routing and backpressure architecture, and it addresses one narrow skill: reading an async flame graph correctly, where the widest frame is usually a red herring.

Concept and specification #

A flame graph is an aggregation of stack samples. Each horizontal bar is a stack frame; its width is proportional to the number of samples in which that frame was on the stack — i.e. wall-clock time, not invocation count. Frames stack vertically by call depth. For a sampler running at rate $r$ over a window of $T$ seconds, a frame occupying fraction $w$ of the graph consumed approximately

$$t_{\text{frame}}\approx w\cdot T,n_{\text{samples}}\approx w\cdot r\cdot T$$

so at the default $r=100\text{Hz}$ over a 60s capture, a frame at 30% width represents ~18s of wall time and ~1800 samples — enough resolution to trust. Below ~1% width you are in sampling noise.

The async subtlety: in an asyncio pipeline the widest frame is almost always the event loop’s select/epoll_wait — the loop waiting for I/O — which is idle time, not a bottleneck. Worse, CPU-bound geofence geometry is deliberately pushed off the loop with asyncio.to_thread, so it shows up under a thread stack, and the loop’s view of it is a wide to_thread/run_in_executor wait frame. Reading the graph correctly means distinguishing three things:

py-spy exposes three subcommands. dump prints the current stack of every thread once — the fastest triage, no capture window. top shows a live, htop-style rolling view of the hottest functions. record captures over a window and writes an SVG flame graph or speedscope JSON. Two flags matter for spatial pipelines: --native, which unwinds C/Cython/native extension frames (essential when point-in-polygon is a compiled routine), and --gil, which counts only samples holding the GIL, separating true Python CPU time from native/thread wait.

Because py-spy reads the target from the outside, it never adds import overhead, never holds a lock the target needs, and — unlike a signal-based profiler such as cProfile — does not require the process to cooperate or restart. That property is what makes it safe to run against a production geofence node carrying live traffic: the target keeps evaluating events at full rate while py-spy walks its stacks from a neighbouring process, and the only cost is a small, bounded slice of a separate core spent unwinding and aggregating samples. The trade-off is statistical rather than exact — you get a sampled distribution of where wall-clock time went, not a deterministic call count — but for latency triage the distribution is precisely what you want, since it weights frames by the time they actually cost rather than by how often they were called.

Step-by-step implementation #

Prerequisites: py-spy>=0.3.14 installed on the host (or a sidecar sharing the process namespace), the target’s PID, and permission to read its memory. On Linux, py-spy needs CAP_SYS_PTRACE — either run as root, add the capability, or set --cap-add=SYS_PTRACE on the container.

1. Triage with a dump — no window needed. This is the first thing to run when P99 alerts fire.

# One-shot stack of every thread; --locals adds argument values for context.
py-spy dump --pid "$(pgrep -f geofence_evaluator)" --locals

Read the worker-thread stacks: if they sit in point_in_polygon or route, geometry is hot; if the loop thread sits in run_in_executor while workers idle, the pool is the constraint, not the math.

2. Capture a flame graph under live load. Sample the running process for 60s at 100Hz.

# --native unwinds the Cython/C point-in-polygon frames that pure-Python
# sampling would collapse into an opaque "<built-in>" bar.
py-spy record --pid "$(pgrep -f geofence_evaluator)" \
  --native --rate 100 --duration 60 --output flame.svg

Gotcha: without --native, a Cython or NumPy point-in-polygon routine appears as a single flat <built-in method> frame with no internal structure — you see that geometry is expensive but not where. Always add --native when the hot path is a compiled extension.

3. Separate GIL-held CPU from thread wait. Re-record counting only GIL-holding samples to confirm whether the cost is real Python execution.

# --gil: a frame that stays wide here holds the GIL (true Python CPU);
# a frame that shrinks was thread/native wait masquerading as busy.
py-spy record --pid "$PID" --gil --rate 100 --duration 30 --output gil.svg

4. Run it continuously as a sidecar. Archive a rolling flame graph so a post-incident investigation has a profile from before the incident.

# Cron or sidecar loop; speedscope format is smaller and diffable across captures.
while true; do
  ts="$(date +%s)"
  py-spy record --pid "$PID" --rate 100 --duration 60 \
    --format speedscope --output "/profiles/geofence-${ts}.json"
done

5. Read the graph. Open flame.svg in a browser, click to zoom a subtree, use the search box to highlight polygon or route, and compare the highlighted total against the loop’s epoll_wait width. The frame to optimize is the widest real work frame in a worker thread — never the loop’s idle wait.

Benchmark and verification #

The case below is representative: a node holding a 41ms P99 at 30k events/sec. The flame graph, captured with --native, showed 34% of samples in a single _ray_cast_crossing frame inside the exact point-in-polygon step — a pure-Python edge loop that had never been compiled. Swapping it for a Numba-JIT routine that releases the GIL moved the cost off the interpreter. The before/after, same load and same node:

The verification discipline: after the fix, re-record and confirm the previously hot frame has collapsed below the noise floor and the widest remaining frame is the loop’s idle wait — the signature of a pipeline that is now I/O-bound rather than CPU-bound. A minimal harness to reproduce the profiling target under controlled load:

import asyncio

async def drive(evaluator, rate_hz: int, seconds: int) -> None:
    """Feed synthetic pings at a fixed rate so py-spy samples a steady state."""
    interval = 1.0 / rate_hz
    deadline = asyncio.get_running_loop().time() + seconds
    while asyncio.get_running_loop().time() < deadline:
        await evaluator.enqueue(_synthetic_ping())  # bounded-queue enqueue
        await asyncio.sleep(interval)  # pace the load so the profile is stationary

def _synthetic_ping() -> object:
    return object()  # placeholder telemetry

Attach py-spy record to this process while drive runs, and the flame graph reflects the steady-state hot path rather than startup or teardown transients.

Failure modes and edge cases #

• Sampling a native extension without --native. The compiled point-in-polygon frame collapses to an opaque built-in bar and the actual hot loop is invisible. If a wide frame has no children and is named <built-in>, re-record with --native.
• Permission / ptrace denial. py-spy dump failing with Operation not permitted means the kernel’s ptrace_scope blocks cross-process reads. Either run py-spy as root, grant CAP_SYS_PTRACE, or set /proc/sys/kernel/yama/ptrace_scope to 0 in a controlled environment. In containers, the target and py-spy must share a PID namespace.
• Misreading loop idle as a bottleneck. The single most common error: epoll_wait or select is the widest frame, and an inexperienced reader “optimizes” it. That frame is the event loop waiting for the next event — shrinking it means more load, not less. Always cross-check against the worker-thread stacks.
• Sampling rate too low for short spikes. A 5ms GC pause or a rare deep-polygon evaluation may fall between 100Hz samples. For sub-millisecond frames raise --rate to 500–1000Hz, accepting the higher overhead on the target for the duration of the capture.
• Thread churn hiding cost. A ProcessPoolExecutor or short-lived thread that exits between samples leaves fragmentary stacks. Profile against a warm, steady-state pool; py-spy attributes samples to the thread alive at sample time, so a churning pool smears the geometry cost across ephemeral frames.

Related #

• Production Monitoring & Observability for Geofence Pipelines — parent overview; py-spy is the CPU-profiling arm of the four golden signals.
• Prometheus metrics for queue depth and P99 latency — sibling; the histogram that says which phase, before py-spy says which frame.
• Alerting thresholds for GC pause times — sibling; the pauses py-spy cannot always sample but gc callbacks can.
• Async Python execution patterns for spatial math — the to_thread offload boundary that shapes what the flame graph shows.