Arena Allocation for Per-Event Spatial Scratch

Every geofence evaluation needs temporary space: an array of candidate ids, the projected coordinates of a point, a list of intersection parameters, the vertex slice of whichever ring is being walked. Allocating that space per evaluation is the default in Python and the reason a pipeline with excellent per-call benchmarks has a P99 dominated by garbage collection. At 25,000 evaluations/sec with 3.9 candidates each, the naive implementation produces roughly 300,000 short-lived objects a second, and the generational collector’s promotion behaviour turns that into a gen-2 pause every four seconds. This page sits under memory-constrained spatial processing within Core Architecture & Latency Constraints.

Concept and specification #

An arena is a block of memory allocated once and handed out by bumping a pointer, with no individual frees — the whole arena is reset at a known safe point. For a request-response or event-processing loop, that safe point is the end of each event, which makes the arena’s lifetime exactly the scratch’s lifetime and removes the bookkeeping entirely.

In Python the arena is not raw memory but a set of preallocated NumPy buffers plus an offset. Handing out scratch means returning a view into a buffer, which allocates a small view object but not the underlying data; resetting means setting the offset back to zero. The view objects are gen-0 garbage that never survives a collection, which is the cheap case the generational collector is designed for.

$$\text{bytes}<em>\text{arena}=\max </em>\text{event}\left(\sum _{\text{scratch }s}\text{size}(s)\right)\times \text{safety factor}$$

The arena must be sized to the worst event, not the average, because an overflow has to fall back to ordinary allocation and the fallback is exactly the behaviour being eliminated. A safety factor of about 1.5 over the observed P99.9 is the usual compromise.

The one-arena-per-worker rule is what keeps the design lock-free. An arena shared between concurrently running coroutines would need synchronisation on every bump, and since asyncio coroutines interleave at await points, an arena held across an await could be reset by another task mid-use. The rule that makes this safe is stronger and simpler: never hold an arena view across an await.

Step-by-step implementation #

1. Preallocate typed buffers, not a byte blob. Distinct buffers for the distinct shapes of scratch avoid reinterpreting bytes and keep NumPy’s dtype checks meaningful.

from __future__ import annotations
import numpy as np

class SpatialArena:
    """Per-worker scratch. Views are valid only until the next reset(), and
    must never be held across an await."""

    __slots__ = ("_f64", "_i64", "_off_f", "_off_i", "overflows", "high_water")

    def __init__(self, floats: int = 65_536, ints: int = 16_384) -> None:
        self._f64 = np.empty(floats, dtype=np.float64)
        self._i64 = np.empty(ints, dtype=np.int64)
        self._off_f = 0
        self._off_i = 0
        self.overflows = 0
        self.high_water = 0

    def floats(self, n: int) -> np.ndarray:
        end = self._off_f + n
        if end > self._f64.size:
            self.overflows += 1
            return np.empty(n, dtype=np.float64)     # fall back, keep going
        view = self._f64[self._off_f:end]
        self._off_f = end
        self.high_water = max(self.high_water, end)
        return view

    def ints(self, n: int) -> np.ndarray:
        end = self._off_i + n
        if end > self._i64.size:
            self.overflows += 1
            return np.empty(n, dtype=np.int64)
        view = self._i64[self._off_i:end]
        self._off_i = end
        return view

    def reset(self) -> None:
        self._off_f = 0
        self._off_i = 0

2. Reset at exactly one place. A single try/finally around the per-event body guarantees the reset happens even when the body raises, and having one call site makes the invariant auditable.

async def handle_event(arena: SpatialArena, ev, index, fences) -> list:
    try:
        cand = arena.ints(64)
        n = index.query_into(ev.x, ev.y, cand)        # fills, returns count
        xs = arena.floats(n)
        ys = arena.floats(n)
        return evaluate(cand[:n], xs, ys, fences)     # no await inside
    finally:
        arena.reset()

3. Push the arena into the kernels rather than returning fresh arrays. A function that returns np.empty(...) allocates regardless of the arena; the calling convention has to become “write into this buffer”. That is the invasive part of the change and the reason it is worth doing once, at the boundaries that matter, rather than everywhere.

4. Instrument overflow and high water. Overflow is not an error but it is a signal: a rising overflow count means an event shape the arena was not sized for, and the high-water mark is what to size the next arena from. Both are single integers and belong in the metrics from Prometheus metrics for queue depth and P99 latency.

5. Never let a view escape the event. A view stored in a result object, appended to a list that outlives the event, or captured by a closure will be silently overwritten by the next event. This is the one genuinely dangerous property of the pattern, and the defence is a convention — arena views are consumed within the event and anything returned is a copy — enforced in review and, in staging, by filling the arena with NaN on reset so escapees fail loudly.

Benchmark and verification #

Measured at 25,000 evaluations/sec on one worker, 3.9 candidates per evaluation:

The gen-2 interval is the number that matters operationally, because a gen-2 collection on a heap of this size is a 20–40 ms stop-the-world pause and appears directly in the P99. Moving it from every four seconds to every seventy-one removes it from the tail almost entirely: the P99 falls 3.4× while the mean barely moves, which is the signature of a GC problem rather than a compute one.

The gc.freeze() row is nearly free and worth taking. Calling it after the index is loaded moves every long-lived object into a permanent generation the collector does not traverse, so each gen-2 pass has less to scan. It composes with the arena rather than replacing it — the arena removes the garbage, freeze shrinks the survivors.

Verify with tracemalloc rather than with RSS. Take two snapshots sixty seconds apart under load and compare; with the arena in place, the top allocation sites should be the view objects and nothing in the geometry path. Any geometry construction appearing in the diff is a kernel that still returns fresh arrays and has not been converted.

Failure modes and edge cases #

The escaping-view failure is worth dwelling on because it is the only one that produces wrong answers rather than slow ones, and it is invisible in testing at low concurrency: with one event in flight at a time, the next reset happens after the consumer has read the data, so the bug does not manifest until production load. The NaN-fill trick makes it deterministic — set the arena’s contents to np.nan in reset() under a debug flag, and any escaped view produces obviously wrong numbers immediately rather than plausible ones eventually.

Finally, note the interaction with the offload boundary from async Python execution patterns for spatial math. Work sent to a process pool cannot share the arena — the buffers do not cross the process boundary — so a pipeline that offloads must either keep the arena inside the worker process, giving each pool worker its own, or accept ordinary allocation for the offloaded portion. Thread pools share the arena’s process but not its safety: a thread-pool task holding a view while the event loop resets the arena is the same bug as awaiting inside the scope. Give each thread its own arena, keyed by thread id, or copy at the boundary.

Related #

• Memory-Constrained Spatial Processing — the parent topic and the wider allocation discipline.
• Object Pooling for Polygon Vertex Buffers — pooling long-lived buffers, which the arena complements rather than replaces.
• Alerting Thresholds for GC Pause Times — turning the gen-2 interval into an alert.