Real-Time Geofencing Location triggers · low-latency spatial
7 min read 5 sections

Half-Open Recovery for Geofence Circuit Breakers

Opening a circuit breaker is the easy half; closing it cleanly is where breakers go wrong. A breaker that recovers too eagerly floods a just-healed consumer with the full backlog and knocks it straight back down; a breaker that recovers too timidly leaves a healthy consumer cut off for minutes after it recovered. This page sits under circuit breakers for downstream trigger consumers and the broader event routing and backpressure architecture, and it answers the narrow question of how the half-open state should behave: how many probes, at what rate, with what backoff, and what success threshold turns a probe run into a full close. Get this right and a consumer outage recovers in a single clean transition; get it wrong and the breaker flaps, oscillating OPEN and HALF-OPEN for minutes while the consumer never gets a stable chance to recover.

Concept and specification

The half-open state is a controlled test: admit a small number of probe emits to the recovering consumer, and decide from their outcome whether to close (resume full traffic) or re-open (wait longer). Three parameters govern it.

Backoff window. After each trip, the breaker stays OPEN for a window that grows exponentially so a persistently broken consumer is probed less and less often:

for the $n$-th consecutive trip, base window , and ceiling . With and , the windows grow 1, 2, 4, 8, 16, 30, 30… — probing a dead consumer every 30s instead of hammering it every second.

Probe rate. In HALF-OPEN, admit at most a small fraction of normal traffic — a fixed probe count or a low rate. This is the single most important defense against the thundering herd: if HALF-OPEN admitted full traffic, a consumer that recovered with a cold cache would be hit by the entire accumulated backlog at once and re-fail. Cap probes to a handful of in-flight calls.

Success threshold. Close only after $S$ consecutive probe successes, not the first one. A single success can be luck — a request that hit a warm shard while the rest are cold. Requiring $S$ consecutive successes (typically 3–5) confirms the consumer is stably healthy before restoring full load.

The interaction that prevents flapping is jitter on the backoff. If many breakers (across router replicas, or many consumers behind one gateway) trip together, they will also probe together unless the window is jittered, re-synchronizing the herd on every cycle. Add uniform jitter:

with , so probes desynchronize across breakers.

Parameter Symbol Typical Controls
Base open window 1 s first retry delay
Max open window 30 s probe floor for dead consumers
Backoff factor — 2x how fast probing slows
Jitter fraction $j$ 0.2 herd desynchronization
Probe concurrency — 1–3 thundering-herd defense
Successes to close $S$ 5 confirmation of stable health
Eager recovery flaps for 42 seconds; backed-off probing closes cleanly Top: fixed short window plus full-traffic half-open causes repeated re-trips and flapping. Bottom: exponential backoff with capped probes and a consecutive-success threshold yields one clean close. Recovery timelines Eager: flaps ~42s open ↔ half-open, re-trip × many Backoff + probe cap: closes once CLOSED (clean) windows 1s → 2s → 4s, then 5 consecutive probe successes

The recovery discipline composes with the trip discipline from the parent circuit breaker page: the same latency-primary signal that trips the breaker also judges the probes, so a probe that succeeds but slowly (P99 still above the trip threshold) counts as a failure, not a success. Recovering to “responds but slow” is not recovery.

Step-by-step implementation

Prerequisites: Python 3.11+, standard library (time.monotonic, random for jitter, asyncio for probe concurrency). This extends the breaker from the parent page with a probe-aware half-open state.

1. Grow the open window with jittered exponential backoff on every trip.

python 
from __future__ import annotations

import random
import time
from dataclasses import dataclass, field


@dataclass(slots=True)
class HalfOpenPolicy:
    base_s: float = 1.0
    max_s: float = 30.0
    jitter: float = 0.2          # +/- 20% to desynchronize herds
    successes_to_close: int = 5
    probe_concurrency: int = 2   # cap in-flight probes — herd defense
    _consecutive_trips: int = 0

    def next_open_window(self) -> float:
        # Exponential growth, capped, then jittered so replicas do not re-sync.
        raw = min(self.max_s, self.base_s * (2 ** self._consecutive_trips))
        self._consecutive_trips += 1
        return raw * (1.0 + random.uniform(-self.jitter, self.jitter))

    def reset(self) -> None:
        self._consecutive_trips = 0     # clean close resets the backoff ladder

2. Gate probe admission by a concurrency cap, not a rate. A fixed small number of in-flight probes is the simplest thundering-herd defense and needs no timer.

python 
import asyncio


@dataclass(slots=True)
class ProbeGate:
    limit: int
    _inflight: int = 0

    def try_acquire(self) -> bool:
        if self._inflight >= self.limit:
            return False                 # too many probes already testing
        self._inflight += 1
        return True

    def release(self) -> None:
        self._inflight = max(0, self._inflight - 1)

3. Judge each probe on latency, not just success, and require consecutive wins.

python 
@dataclass(slots=True)
class HalfOpenState:
    policy: HalfOpenPolicy
    p99_trip_ms: float = 50.0
    _successes: int = 0

    def on_probe_result(self, latency_ms: float, ok: bool) -> str:
        # A slow success is a failure: "responds but slow" is not recovered.
        healthy = ok and latency_ms <= self.p99_trip_ms
        if not healthy:
            self._successes = 0
            return "reopen"              # re-open with a longer, jittered window
        self._successes += 1
        if self._successes >= self.policy.successes_to_close:
            self._successes = 0
            self.policy.reset()          # clean recovery: reset backoff
            return "close"
        return "probe"                   # keep probing

Gotcha: reset the backoff ladder (_consecutive_trips) only on a clean close, never on an individual probe success. Resetting on the first success lets a flapping consumer keep the window short forever, defeating the exponential backoff and re-enabling the herd. The window only shrinks back to base once the consumer proves stably healthy.

4. Drive probes with bounded concurrency.

python 
async def run_probe(gate: ProbeGate, state: HalfOpenState, send, fallback,
                    trigger: object) -> str:
    if not gate.try_acquire():
        await fallback(trigger)          # over probe cap: still divert
        return "probe"
    start = time.perf_counter()
    try:
        await asyncio.wait_for(send(trigger), timeout=0.2)
        return state.on_probe_result((time.perf_counter() - start) * 1000.0, ok=True)
    except (asyncio.TimeoutError, Exception):   # noqa: BLE001
        return state.on_probe_result((time.perf_counter() - start) * 1000.0, ok=False)
    finally:
        gate.release()

Benchmark and verification

The scenario opens a breaker against a consumer whose P99 spikes to 900ms, then recovers to healthy at t=20s. Two policies are compared: eager (fixed 1s window, full-traffic half-open, close on first success) and backed-off (jittered exponential window, 2-probe cap, 5 consecutive successes to close).

Metric Eager recovery Backoff + probe cap
Re-trips after true recovery ~7 0
Time healthy → fully closed ~42 s (flapping) ~4 s
Probes hitting cold consumer full backlog ≤ 2 in-flight
Consumer re-failures caused by probing ~5 0
Backoff window at worst 1 s (never grows) 1→2→4 s, capped 30 s

The eager policy flaps: it closes on the first success, dumps the full backlog onto a consumer whose cache is still cold, re-trips, and repeats — taking ~42 seconds of oscillation after the consumer was actually healthy, and causing ~5 re-failures that prolong the outage. The backed-off policy admits at most two probes, requires five consecutive healthy (fast) responses, and closes once in ~4 seconds with zero re-trips. A minimal harness:

python 
import asyncio


async def recovery_bench(state, gate, healthy_at: float, secs: float) -> dict[str, int]:
    trips = probes = 0
    start = asyncio.get_event_loop().time()

    async def send(_trigger: object) -> None:
        # Consumer is slow until healthy_at, fast after.
        elapsed = asyncio.get_event_loop().time() - start
        await asyncio.sleep(0.9 if elapsed < healthy_at else 0.005)

    end = start + secs
    while asyncio.get_event_loop().time() < end:
        nonlocal_result = await run_probe(gate, state, send, _noop, object())
        probes += 1
        if nonlocal_result == "reopen":
            trips += 1
        await asyncio.sleep(0.5)
    return {"trips": trips, "probes": probes}


async def _noop(_t: object) -> None:
    return None

Verify by asserting zero re-trips in the 10 seconds after healthy_at, and by confirming the backoff window actually grows across consecutive trips (log next_open_window() and check the 1, 2, 4 progression). If re-trips persist after recovery, the success threshold is too low or the probe cap too high.

Failure modes and edge cases

  • Flapping (backoff reset too eagerly). The dominant failure. Resetting _consecutive_trips on any probe success keeps the window pinned at base, so a marginal consumer oscillates forever. Reset only on a clean close.
  • Thundering herd (probe cap too high or no jitter). A half-open state that admits full traffic, or unjittered windows across replicas, re-synchronizes the load onto a cold consumer and re-fails it. Cap probe concurrency and jitter the window.
  • Slow-success trap. Judging probes on success alone lets “responds but slow” close the breaker, re-inheriting the consumer’s tail into the router. Judge probes on latency too, using the same trip threshold.
  • Backoff ceiling too low. A max_s of a few seconds keeps probing a genuinely dead consumer aggressively, wasting connections and log volume. Cap at 30s+ so a dead consumer is probed rarely.
  • Lost probe result. If a probe’s coroutine is cancelled (router shutdown, timeout race) without calling on_probe_result, the success counter can stall mid-run. Treat a cancelled probe as a failure so the state cannot hang in half-open indefinitely.