Webhook Fan-Out & Retry Budgets for Geofence Triggers

The moment a geofence platform has customers rather than a single consumer, the trigger stream stops being a stream and becomes a fan-out. One ENTER for one vehicle at one depot is delivered to the dispatch system, the billing pipeline, two customer webhooks, a partner integration and an audit sink — and the arithmetic changes character: a 25,000 trigger/sec pipeline with a mean fan-out of 40 is a one-million-request-per-second delivery problem against endpoints the platform does not own, cannot profile, and cannot fix. This page expands the delivery-boundary model introduced in Event Routing & Backpressure, and the failure it addresses is one slow tenant consuming the delivery capacity of every other tenant.

The reader is running a multi-tenant geofencing service in which a single customer’s unreachable endpoint has, at least once, delayed every other customer’s triggers. The instinct is to add workers. That does not work, because the resource being exhausted is not CPU but concurrency slots occupied by requests waiting on a socket, and adding workers simply raises the number of slots a dead endpoint can occupy. What works is bounding, per tenant, both how much concurrency and how much retry a tenant may consume — and treating retry as a budget spent from a shared pool rather than as a property of an individual request.

Fan-Out Arithmetic and Where the Latency Goes #

Fan-out multiplies both the request rate and the tail. If a trigger is fanned out to $n$ subscribers in parallel and the per-delivery latency distribution has CDF $F$, the time until all deliveries complete is the maximum of $n$ draws, whose distribution is $F(t{)}^n$. The consequence is unintuitive and dominates the design: at $n=40$, the completion time of the whole fan-out sits near the 99.9th percentile of the individual delivery distribution rather than near its median. A subscriber set with a perfectly respectable 8 ms median and a 400 ms P99.9 produces a fan-out that regularly takes 400 ms.

The last row is the design goal in one number. Excluding a single pathological tenant from the completion criterion halves the fan-out P99 — which is the argument for never treating a fan-out as a transaction. The delivery of a trigger to tenant A must not be gated on its delivery to tenant B, and the pipeline’s notion of “done” must be per subscription, not per trigger. Where a single logical completion signal is genuinely required, it should be derived from a durable per-subscription ledger rather than from waiting.

The Bulkhead: Concurrency Is the Resource #

The resource a dead endpoint consumes is a concurrency slot held open for the duration of a socket timeout. With a 10 s connect timeout, a single tenant generating 100 triggers/sec into a black hole accumulates 1,000 in-flight requests before the first one gives up. Against a shared pool of 1,200 workers, that tenant now owns 83% of the platform’s delivery capacity, and every other tenant’s triggers queue behind it — the classic bulkhead failure, with the classic remedy.

The remedy is a per-subscription semaphore whose permit count is derived from the tenant’s own throughput and latency, not from the size of the pool:

$$C_{\text{tenant}}=\lceil {\lambda }_{\text{tenant}}\cdot L_{\text{p99}}\rceil +\text{headroom}$$

by Little’s law — a tenant taking 100 triggers/sec at a 120 ms P99 needs 12 concurrent slots to keep up, and giving it 16 leaves headroom without letting it grow. A tenant that stops responding then saturates at 16 slots and stops, so its blast radius is $16/1200=1.3$ of the pool rather than everything. Everything past the cap is rejected immediately and routed to the dead-letter path, not queued — queueing merely relocates the unbounded growth from the socket layer to memory, which is the mistake described in backpressure and flow control strategies.

Adding a circuit breaker on top of the semaphore takes the last 0.9 percentage points by stopping the tenant from occupying even its own cap once the endpoint is provably dead: the breaker opens, deliveries short-circuit to the dead-letter path in microseconds, and the slots are returned. The breaker’s state machine, its trip conditions and its half-open probe policy are the same ones described in circuit breakers for downstream trigger consumers — the only difference here is that there is one breaker per subscription rather than one per downstream service, which matters because tenants fail independently and a shared breaker would trip the healthy majority.

Retry Budgets Instead of Per-Request Retry Policies #

Per-request retry policies are the second way a fan-out amplifies a failure. “Retry three times with exponential backoff” sounds bounded, and is — per request. Across a fleet, when an endpoint starts failing every request, it multiplies the offered load by four exactly when the endpoint is least able to absorb it. This is the retry-storm pathology, and backoff alone does not prevent it: backoff spreads the storm in time without reducing its total volume.

A retry budget fixes the amplification by making retries a globally scarce resource. The client maintains a rolling ratio of retries to successful requests and refuses to retry once that ratio exceeds a threshold — typically 10%. When an endpoint is healthy, successes replenish the budget faster than the occasional retry drains it, so retries are effectively unlimited; when it fails wholesale, successes stop, the budget empties within a second, and retries stop with it. The offered load then converges to the original request rate rather than four times it.

The budget row is better on every axis at once, which is unusual enough to be worth stating plainly: it lowers the peak load by 3.6×, shortens recovery, and cuts duplicates, because most of the duplicates in the other rows come from retries of requests that actually succeeded but whose response was lost. The jitter that keeps the recovery herd from re-synchronising is developed in exponential backoff with jitter for trigger webhooks, and the per-tenant rate ceiling that stops a large tenant from starving a small one is in per-tenant rate limiting of trigger fan-out.

Retries also make idempotency non-negotiable at the receiver, because a retried delivery is by definition a duplicate from the endpoint’s point of view. The same deterministic key the pipeline uses internally must travel in the request — in a header, not the body, so a receiver can dedup before parsing — under the semantics set out in idempotent trigger emission semantics. Without it, the 0.4% duplicate rate in the last row lands as duplicated business effects rather than as discarded requests.

Implementation: Bounded Lanes over a Shared Client #

from __future__ import annotations
import asyncio, random, time
from dataclasses import dataclass, field
import httpx

@dataclass(slots=True)
class Lane:
    """Everything one subscription is allowed to consume."""
    sem: asyncio.Semaphore
    rate_tokens: float = 0.0
    rate_per_sec: float = 500.0
    last_refill: float = field(default_factory=time.monotonic)
    breaker_open_until: float = 0.0

class RetryBudget:
    """Rolling retries-to-successes ratio, shared across every lane."""
    def __init__(self, ratio: float = 0.10, half_life: float = 10.0) -> None:
        self._ratio, self._hl = ratio, half_life
        self._ok = self._retry = 0.0
        self._t = time.monotonic()

    def _decay(self) -> None:
        now = time.monotonic()
        f = 0.5 ** ((now - self._t) / self._hl)
        self._ok *= f
        self._retry *= f
        self._t = now

    def record_success(self) -> None:
        self._decay()
        self._ok += 1.0

    def try_spend(self) -> bool:
        self._decay()
        if self._retry >= self._ok * self._ratio + 1.0:
            return False                      # budget empty: do not retry
        self._retry += 1.0
        return True

async def deliver(
    client: httpx.AsyncClient,
    lane: Lane,
    budget: RetryBudget,
    url: str,
    payload: bytes,
    idem_key: str,
    max_attempts: int = 4,
) -> str:
    if time.monotonic() < lane.breaker_open_until:
        return "short-circuit"                # breaker open: straight to DLQ
    if lane.sem.locked() and lane.sem._value == 0:
        return "cap-exceeded"                 # reject, never queue
    async with lane.sem:
        for attempt in range(max_attempts):
            try:
                r = await client.post(
                    url, content=payload, timeout=5.0,
                    headers={"Idempotency-Key": idem_key,
                             "X-Attempt": str(attempt)},
                )
                if r.status_code < 500:
                    budget.record_success()
                    return "delivered" if r.status_code < 300 else "rejected"
            except (httpx.TimeoutException, httpx.TransportError):
                pass
            if attempt + 1 == max_attempts or not budget.try_spend():
                break
            # full jitter: uniform over the whole backoff interval
            await asyncio.sleep(random.uniform(0.0, 0.25 * (2 ** attempt)))
    lane.breaker_open_until = time.monotonic() + 30.0
    return "exhausted"

Three decisions in that code are the ones worth defending. The cap check rejects rather than awaiting the semaphore, because awaiting is how a bounded cap becomes an unbounded queue of waiters. The retry loop consults the shared budget rather than a per-request counter, so the aggregate amplification is bounded even though each request still has its own attempt ceiling. And a 4xx response is treated as success from the budget’s point of view — the endpoint answered, the payload was wrong, and retrying a malformed payload forever is the poison-message pattern that belongs in the dead-letter path rather than in the retry loop.

Memory Footprint of Lanes and Payloads #

Two structures grow with tenancy. The lane record itself is small — a semaphore, four floats, and a deadline, about 220 bytes with slots=True — so 50,000 subscriptions cost 11 MB, which is not the problem. The problem is the payload. A fan-out that serialises the trigger once per subscription allocates $n$ copies of the same bytes: at 40 subscriptions and a 900-byte payload that is 36 KB per trigger, and at 25k triggers/sec, 900 MB/s of allocation churn feeding straight into gen-2 collections.

Serialise once, share the immutable bytes object across every lane, and per-subscription customisation — a tenant-specific signature header, a different envelope version — belongs in the headers rather than in a re-serialised body. On the fleet measured for this page, that single change cut allocation from 900 MB/s to 23 MB/s and moved gen-2 pauses from every 3 s to every 40 s, with the P99 fan-out latency falling 70 ms purely from the reduced GC pressure. Where a tenant genuinely needs a different body, cache the rendered variant keyed by (schema version, tenant class) rather than by tenant, since the variant count is nearly always a handful even at tens of thousands of tenants.

Connection pools are the third growth term and the easiest to get wrong. One pool per tenant gives perfect isolation and exhausts file descriptors at a few thousand tenants; one shared pool gives none. The working arrangement is a shared pool with a per-host connection cap, which bounds sockets per endpoint while keeping total descriptors proportional to active hosts rather than to subscriptions.

Operational Runbook #

1. Instrument per subscription, not per service. delivery_latency, delivery_status, lane_in_flight and lane_rejections all need a subscription label. A platform-wide delivery dashboard cannot show the failure this page is about, because the failure is always one tenant against a healthy aggregate.
2. Alert on lane_in_flight / cap, not on latency. A saturating lane is visible a minute before its latency moves, because requests are still completing at the timeout. Page when any lane sits above 80% of its cap for more than 30 s.
3. Check the retry budget’s fill level during an incident. An empty budget is the system working correctly, and it explains why retries stopped. Teams that do not export it re-introduce unbounded retries during the postmortem because “retries were not happening”.
4. Confirm rejections go to the dead-letter path. lane_rejections must equal dead-letter admissions for that subscription. Any gap is a silently dropped trigger, which is the one outcome worse than a late one.
5. Re-derive caps monthly from Little’s law. A tenant whose traffic has tripled will sit permanently near its cap and will look like a failing endpoint. Recompute $C=\lambda \cdot L_{p99}$ from the last month’s measurements rather than raising caps reactively during incidents.
6. Test with a black hole, not a 500. An endpoint that returns errors quickly exercises none of this; an endpoint that accepts the connection and never responds exercises all of it. Keep one in the staging environment permanently.

Architectural Guidance #

Use per-subscription lanes with a shared retry budget as the default for any multi-tenant trigger platform. The isolation is what makes per-tenant SLAs meaningful, and the budget is what keeps a mass failure from becoming a self-inflicted denial of service.

Use a single shared pool only where all consumers are internal and operated by the same team, so a failure is an incident rather than a breach of contract, and where the number of consumers is small enough that head-of-line blocking is visible immediately.

Prefer a durable queue per subscription over synchronous fan-out when delivery latency requirements are loose — minutes rather than milliseconds — and durability requirements are strict. Per-subscription queues turn a delivery problem into a storage problem, which is easier to bound, at the cost of latency and of one more system to operate.

Push fan-out to the broker — a consumer group per tenant reading the same topic — when tenants are few, sophisticated, and willing to run consumers. It removes the delivery problem from the platform entirely, and it is the right answer for partner integrations even when webhooks remain the default for everyone else.

FAQ #

Related #

• Event Routing & Backpressure — the parent section, where the fan-out sits at the last and weakest delivery hop.
• Exponential Backoff with Jitter for Trigger Webhooks — why full jitter beats decorrelated and equal jitter for a recovering herd.
• Per-Tenant Rate Limiting of Trigger Fan-Out — token buckets sized from subscription contracts rather than from pool capacity.
• Signing and Verifying Geofence Webhook Payloads — timestamped HMAC signatures and the replay window they need.
• Circuit Breakers for Downstream Trigger Consumers — the breaker state machine each lane runs its own copy of.