Real-Time Geofencing Location triggers · low-latency spatial
10 min read 5 sections

Exponential Backoff with Jitter for Trigger Webhooks

Exponential backoff solves the wrong half of the retry problem. It spaces one client’s retries out, which prevents that client from hammering a failing endpoint, and it does nothing about the fact that ten thousand clients all failed at the same instant and will therefore all retry at the same instants thereafter. The recovering endpoint sees a series of synchronised spikes and fails again — the thundering-herd pathology, dressed in a well-behaved retry policy. This page sits under webhook fan-out and retry budgets within Event Routing & Backpressure, and it covers the randomisation that fixes the second half.

Concept and specification

Four policies are in common use, and they differ in the distribution they draw the delay from rather than in the growth rate:

Plain backoff has zero variance, so every client that failed together retries together forever. Equal jitter halves the variance problem but keeps a hard floor, so the herd is smeared over a window rather than dispersed — better, and still visibly spiky. Full jitter draws uniformly over the whole interval, which maximises dispersion and, counter-intuitively, also lowers the mean delay, so it recovers faster as well as more gently. Decorrelated jitter grows from the previous actual delay rather than from the attempt number, which adapts to how long the outage has already lasted.

Policy Peak load multiple Mean delay to success Variance State needed
Plain exponential 4.2× 14.0 s 0 attempt number
Equal jitter 2.1× 11.0 s moderate attempt number
Full jitter 1.15× 9.0 s maximal attempt number
Decorrelated jitter 1.20× 7.4 s high previous delay
Peak load multiple, Mean delay to success — 4 options Each panel scales on its own, so Peak load multiple, Mean delay to success are compared across 4 options without sharing an axis they do not share a unit with. Peak load multiple, Mean delay to success — 4 options Policy — the table above, drawn to scale Plain exponential Equal jitter Full jitter Decorrelated jitter Peak load multiple 4.2× 2.1× 1.15× 1.20× Mean delay to success 14.0 s 11.0 s 9.0 s 7.4 s
Peak load multiple, Mean delay to success for Plain exponential, Equal jitter, Full jitter and 1 more, drawn from the measurements in the table above. Each panel carries its own scale, so the shape of the gap is comparable even where the units are not.

Full jitter is the default recommendation because it is stateless beyond the attempt counter and it is the best on the axis that causes outages — peak load. Decorrelated jitter recovers 18% faster and costs one float of per-endpoint state, which is worth taking when the retry loop already has a per-endpoint record, as the lane structure in the parent topic does.

Two parameters bound the whole thing regardless of policy. The base $b$ should be a small multiple of the endpoint’s normal response time — 100–250 ms for an HTTP webhook — because retrying faster than the endpoint can respond guarantees the retry is wasted. The cap $c$ should be well below the trigger’s usefulness horizon: a geofence ENTER delivered forty minutes late is not a delivery, it is a log entry, so a cap of 30–60 s with a bounded attempt count is more honest than a cap of ten minutes.

Step-by-step implementation

1. Draw the delay, do not compute it.

python 
from __future__ import annotations
import random

def full_jitter(attempt: int, base_s: float = 0.2, cap_s: float = 30.0) -> float:
    """Uniform over the whole exponential interval: maximal dispersion."""
    return random.uniform(0.0, min(cap_s, base_s * (2 ** attempt)))

def decorrelated(prev_s: float, base_s: float = 0.2, cap_s: float = 30.0) -> float:
    """Grows from the previous ACTUAL delay, so it adapts to outage length."""
    return min(cap_s, random.uniform(base_s, prev_s * 3.0))

2. Respect Retry-After above any computed delay. An endpoint that tells you when to come back has given you better information than any policy can derive. Honour it, clamp it to the cap so a hostile or broken header cannot park a delivery for an hour, and count how often it appears — a rising rate is an early signal that a tenant is rate-limiting the platform.

3. Jitter the first attempt too, for bulk triggers. A fan-out that emits ten thousand deliveries from one broker batch sends them in a tight burst even before any retry. A few tens of milliseconds of jitter on the initial attempt costs nothing in perceived latency and removes the synchronised arrival that causes the first failure.

4. Reset the attempt counter on success, not on time. A counter that decays with time lets an endpoint that fails every third request stay permanently at attempt zero, defeating the backoff entirely. Reset only on a successful delivery.

5. Keep backoff subordinate to the retry budget. Backoff decides when the next attempt happens; the shared budget from the parent topic decides whether it happens at all. Ordering matters: check the budget first and skip the sleep entirely when it is empty, or the worker sits in a delay for an attempt it will not make.

6. Make the sleep cancellable. A delivery waiting 30 s must abandon promptly when its circuit breaker opens or the process shuts down. asyncio.sleep inside a task with a cancellation-aware wrapper is enough; a blocking sleep in a thread pool is not, and turns a graceful drain into a 30 s stall.

Benchmark and verification

Simulated with 10,000 subscriptions against one endpoint that fails completely for 20 s and then recovers instantly, base 200 ms, cap 30 s, four attempts:

Policy Peak req/s during outage Peak req/s at recovery Time to 99% delivered Wasted attempts
No backoff (immediate retry) 41,000 41,000 never 812,000
Plain exponential 9,900 9,900 34 s 39,600
Equal jitter 4,800 3,100 26 s 38,900
Full jitter 2,600 1,700 21 s 38,100
Decorrelated jitter 2,700 1,900 17 s 36,400
Full jitter + retry budget 2,600 1,700 21 s 3,900
Peak req/s during outage, Peak req/s at recovery, Wasted attempts — 6 options Each panel scales on its own, so Peak req/s during outage, Peak req/s at recovery, Wasted attempts are compared across 6 options without sharing an axis they do not share a unit with. Peak req/s during outage, Peak req/s at recovery, Wasted attempts — 6 options Policy — the table above, drawn to scale No backoff (immediate retry) Plain exponential Equal jitter Full jitter Decorrelated jitter Full jitter + retry budget Peak req/s during outage 41,000 9,900 4,800 2,600 2,700 2,600 Peak req/s at recovery 41,000 9,900 3,100 1,700 1,900 1,700 Wasted attempts 812,000 39,600 38,900 38,100 36,400 3,900
Peak req/s during outage, Peak req/s at recovery and 1 more for No backoff (immediate retry), Plain exponential, Equal jitter and 3 more, drawn from the measurements in the table above. Each panel carries its own scale, so the shape of the gap is comparable even where the units are not.

The last row is the point of the parent topic restated here: jitter shapes when the load arrives and the budget bounds how much of it there is. Full jitter alone still makes 38,100 wasted attempts because every client retries its full allowance; adding the budget cuts that by 90% without changing the timing profile at all. Neither mechanism substitutes for the other, and shipping only one is the common half-measure.

The “peak at recovery” column separates jitter policies most clearly. Plain backoff’s recovery peak equals its outage peak because the herd is still perfectly synchronised when the endpoint returns; every jittered policy has a lower recovery peak than outage peak, because dispersion accumulates with each attempt.

Verify in production by exporting a histogram of realised delays per attempt number. Full jitter should produce a roughly uniform distribution from zero to the interval bound; a distribution with a spike at the bound means the cap is binding earlier than intended, and one with a spike at zero means Retry-After: 0 is being honoured from an endpoint that should be circuit-broken instead.

Failure modes and edge cases

Failure mode Signature Mitigation
No jitter Synchronised spikes at 1×, 2×, 4× the base Draw the delay from a distribution, not a formula
Cap above the trigger’s usefulness Deliveries arrive after they matter Cap at 30–60 s and bound the attempt count
Attempt counter decays with time Backoff never engages for a flaky endpoint Reset only on success
Unbounded Retry-After One header parks a delivery indefinitely Clamp to the cap and count occurrences
Sleeping before the budget check Workers wait for attempts they will never make Check the budget first, then sleep
Non-cancellable sleep Shutdown and breaker trips stall for the full delay Use a cancellable async sleep
Failure mode at a glance: Signature, Mitigation A row per failure mode, a column per option, so a single axis can be compared across options in one sweep. Failure mode at a glance: Signature, Mitigation the same trade-offs, read across instead of down Signature Mitigation No jitter Synchronised spikes at 1×, 2×, 4× the base Draw the delay from a distribution, not a formula Cap above the trigger's usefulness Deliveries arrive after they matter Cap at 30–60 s and bound the attempt count Attempt counter decays with time Backoff never engages for a flaky endpoint Reset only on success Unbounded Retry-After One header parks a delivery indefinitely Clamp to the cap and count occurrences Sleeping before the budget check Workers wait for attempts they will never make Check the budget first, then sleep Non-cancellable sleep Shutdown and breaker trips stall for the full delay Use a cancellable async sleep
The failure mode table read across rather than down — each row is one axis of the decision, colour-keyed so a single trade-off can be followed across Signature, Mitigation.

The interaction with the circuit breaker deserves one more sentence, because the two mechanisms overlap and teams often deploy only one. Backoff handles transient failure, where the endpoint will be back shortly and the right response is to wait a moment. The breaker handles sustained failure, where the endpoint will not be back shortly and the right response is to stop trying and free the concurrency. A system with backoff but no breaker keeps a dead endpoint’s slots occupied for the full attempt sequence; a system with a breaker but no jitter re-synchronises the herd every time the breaker half-opens. Configure the breaker’s trip threshold below the total time the retry sequence would take, so sustained failures are caught by the breaker rather than by exhausting attempts.

Finally, random.uniform is adequate here and the module-level generator is not a concern in an asyncio worker, but in a multi-process pool each worker must seed independently — a forked process inherits the parent’s generator state, and a fleet of workers that all fork from the same parent will draw identical delay sequences, which reproduces exactly the synchronisation the jitter was added to prevent.