Sizing Bounded asyncio Queues for Geofence Pipelines

The single number maxsize on a bounded asyncio.Queue decides both how much latency the pipeline can hide and how much memory it can leak before it starts shedding — and most services pick it by copying a round figure like 100_000 from a tutorial. That number is almost always wrong in the same direction: far too deep, which converts what should be a fast shed into seconds of hidden queueing latency and gigabytes of resident payload. This page sits under backpressure and flow-control strategies and the broader event routing and backpressure architecture, and it answers a precise question: given an arrival rate and a latency budget, what should maxsize actually be? The answer comes from Little’s law, and for a typical geofence router it is closer to 900 than to 100,000.

Concept and specification #

A bounded queue is a small waiting system, and Little’s law gives its steady-state occupancy exactly. For a mean arrival rate $\lambda$ (events/sec) and a mean time-in-system $W$ (seconds), the mean number of items resident is:

$$L=\lambda W$$

The queue only needs enough slots to hold the items that arrive during the time you are willing to let an item wait. If the router drains at 32k/sec and you are willing to let an item sit in the queue for at most 25ms before it is either processed or shed, then the depth that matters is:

$$\text{maxsize}=\lambda \cdot W_{\text{max}}=32,000\times 0.025=800$$

Add a modest burst margin — say 15% — and maxsize ≈ 920. Anything deeper than this does not improve throughput; it only lets items accumulate wait time beyond W_max, because a slot that exists solely to hold an item for longer than you would tolerate is a slot that stores latency, not resilience. This is the counterintuitive core of the whole exercise: a deeper queue does not protect you from overload, it hides overload as latency until it OOMs.

The memory accounting is the second half of sizing. A TelemetryPoint dataclass with slots=True — five floats, two strings, an int — resident-costs roughly 180 bytes including the object header, plus the deque’s own pointer slot (~8 bytes). At maxsize=920 the queue’s worst-case live payload is under 170KB, a rounding error against a 1.5GB RSS cap. At the copied-tutorial maxsize=100_000 the same queue’s worst case is ~18MB of payload — still survivable in isolation, but it means the queue is willing to hold ~3 seconds of arrivals, so under sustained overload an item can wait 3 seconds before being shed, and the P99 held-time contract is silently blown by the queue depth alone. Sizing from Little’s law makes the queue shed at the latency boundary instead of the memory boundary, which is what you want: shedding is a fast, bounded loss; latency accumulation is a slow, invisible one.

The latency-vs-loss trade-off is therefore a direct consequence of depth. A shallow queue (small maxsize) sheds sooner and more often, but every admitted item is processed quickly — you trade a higher loss rate for a tight tail. A deep queue sheds rarely but admits items that then wait, trading a low loss rate for a fat tail and a high memory ceiling. The right point on that curve is fixed by W_max: the deepest queue whose full-drain time still fits the latency budget. The token bucket that fronts this queue, covered in token-bucket vs leaky-bucket for telemetry shedding, and the queue depth must agree: the bucket smooths the admitted rate toward $\lambda$, and maxsize catches the residual burst.

Step-by-step implementation #

Prerequisites: Python 3.11+, standard library asyncio. Input is a stream of telemetry items; the goal is a queue whose depth is derived, monitored, and alarmed rather than guessed.

1. Derive maxsize from measured rate and budget, not a constant. Compute it at startup from configuration so it is auditable.

from __future__ import annotations

import asyncio
from dataclasses import dataclass


def size_queue(drain_rate_hz: float, tolerable_wait_s: float,
               burst_margin: float = 0.15) -> int:
    """maxsize = lambda * W_max, plus a burst margin. Little's law."""
    steady = drain_rate_hz * tolerable_wait_s            # L = lambda * W
    return max(1, round(steady * (1.0 + burst_margin)))  # never zero


# e.g. 32_000 /s drain, 25ms tolerable wait -> ~920 slots
MAXSIZE: int = size_queue(32_000.0, 0.025)
queue: asyncio.Queue = asyncio.Queue(maxsize=MAXSIZE)

2. Admit with put_nowait and treat QueueFull as a shed, not an error. Never await put() on a live-stream ingest path — that inverts backpressure into head-of-line blocking.

def admit(queue: asyncio.Queue, item: object, shed: list[int]) -> bool:
    try:
        queue.put_nowait(item)                 # non-blocking; raises if full
        return True
    except asyncio.QueueFull:
        shed[0] += 1                            # bounded, explicit loss
        return False                            # caller routes to dead-letter

3. Monitor qsize() on a fixed cadence and export the fill fraction. Depth is the leading indicator of overload; sample it independently of the hot path.

async def monitor_depth(queue: asyncio.Queue, gauge) -> None:
    while True:
        frac = queue.qsize() / queue.maxsize   # 0..1 fill fraction
        gauge.set(frac)                         # e.g. a Prometheus Gauge
        # Sustained high fill is a capacity deficit; a brief spike is a burst.
        await asyncio.sleep(1.0)

Gotcha: qsize() is a point-in-time read and can be stale by the time you act on it under a fast producer. Use it for trend alerting (fill above 80% for N seconds), never as a gate inside the admission decision — the put_nowait/QueueFull path is the only race-free capacity check.

Benchmark and verification #

The scenario compares the copied maxsize=100_000 against the Little’s-law maxsize=920, same 32k/sec drain, driven to a sustained 1.4x overload so the queue stays full.

The two queues shed almost the same fraction under sustained overload — that is set by the arrival-vs-drain deficit, not by depth. What differs is when and at what latency cost. The deep queue delays its first shed by ~3 seconds and, in the meantime, every admitted item accrues seconds of wait: its P99 in-queue time is over 3 seconds, which alone blows any sub-50ms end-to-end budget. The Little’s-law queue sheds within 29ms and holds in-queue P99 at 27ms, so the shed is a fast bounded loss instead of a latency landmine. A minimal harness:

import time
import asyncio


async def measure_wait(maxsize: int, arrivals_hz: float, drain_hz: float,
                       secs: float) -> dict[str, float]:
    q: asyncio.Queue = asyncio.Queue(maxsize=maxsize)
    waits: list[float] = []
    shed = 0

    async def producer() -> None:
        nonlocal shed
        gap = 1.0 / arrivals_hz
        end = time.monotonic() + secs
        while time.monotonic() < end:
            try:
                q.put_nowait(time.monotonic())          # enqueue arrival time
            except asyncio.QueueFull:
                shed += 1
            await asyncio.sleep(gap)

    async def consumer() -> None:
        gap = 1.0 / drain_hz
        while True:
            t0 = await q.get()
            waits.append((time.monotonic() - t0) * 1000.0)   # ms in queue
            await asyncio.sleep(gap)
            q.task_done()

    prod = asyncio.create_task(producer())
    cons = asyncio.create_task(consumer())
    await prod
    cons.cancel()
    waits.sort()
    n = len(waits) or 1
    return {"p95_ms": waits[int(0.95 * n) - 1], "shed": shed}

Verify by graphing in-queue wait against depth: the wait must stay under W_max. If P99 wait exceeds the budget while shed rate is low, the queue is too deep — shrink maxsize until the tail fits, accepting the slightly higher shed rate as the correct trade.

Failure modes and edge cases #

• Too deep (latency + OOM). The dominant real-world error. A queue sized for memory rather than latency hides overload as multi-second wait and, under a true burst, still risks OOM because the payload ceiling scales with depth. Size from Little’s law, not from available RAM.
• Too shallow (over-shed). Sizing W_max too tight sheds legitimate bursts that a slightly deeper queue would have absorbed and drained within budget. Include the 10–20% burst margin, and validate against a real burst capture, not a constant rate.
• qsize() used as a gate. Reading qsize() and then deciding to put() is a race — the queue can fill between the read and the put under a fast producer. Only put_nowait/QueueFull is race-free.
• Per-item size drift. If items carry variable-length payloads (a list of matched fence IDs), the ~180-byte estimate is a floor; a pathological item with hundreds of matches inflates the memory ceiling. Cap payload size at ingest or store a reference, not the payload, in the queue.
• Unbounded downstream hidden behind a bounded queue. A correctly sized queue is defeated if the drain worker batches into an unbounded producer buffer downstream. Bound every hop, and flush the producer when its own buffer fills, as covered in the backpressure strategies page.

Related #

• Backpressure & Flow-Control Strategies — parent overview; the queue is the backpressure signal this page sizes.
• Token-Bucket vs Leaky-Bucket for Telemetry Shedding — the rate limiter whose ceiling must agree with the queue depth.
• Event Routing & Backpressure — the routing tier the queue protects.