Circuit Breakers for Downstream Trigger Consumers

A geofence router emits triggers into consumers it does not control — a dynamic-pricing engine, a compliance API, a dispatch service — and any one of them can degrade without warning. When the pricing engine’s P99 climbs from 8ms to 800ms, a router with no breaker keeps sending, each emit blocks a coroutine waiting on the slow consumer, in-flight requests pile up against the connection pool, and within seconds the router’s own event loop is starved serving a consumer that is already failing. The failure propagates backward: a slow downstream becomes a stalled router becomes shed telemetry at ingress, and the pricing outage takes the compliance path down with it even though compliance was healthy. This page expands the emit-boundary problem introduced in event routing and backpressure and its companion on backpressure and flow-control strategies; the failure mode it addresses is failure amplification across a phase boundary — one slow consumer converting into a whole-router outage.

The reader here runs a Python router fanning triggers to multiple downstream consumers with divergent reliability. The circuit breaker is the mechanism that makes a downstream failure local: it detects that a specific consumer is unhealthy, stops sending to it, serves a fallback (buffer to a dead-letter path, or degrade the trigger), and periodically probes for recovery — all without touching the other consumers or the ingest path. The critical placement decision is that the breaker belongs at the phase boundary between the router and each consumer, not at ingress. A breaker at ingress can only drop incoming telemetry indiscriminately; a breaker at the emit boundary drops exactly the traffic to the one broken consumer and keeps everything else flowing.

Algorithmic Divergence and Latency Profiles #

A circuit breaker is a three-state machine wrapped around each outbound call. In CLOSED it passes every emit and records two signals per call: latency (into a rolling window) and success/failure. When the trip condition fires it moves to OPEN, where it stops calling the consumer entirely and returns immediately — every emit is diverted to the fallback with no wait, which is what protects the router’s event loop. After a backoff window it moves to HALF-OPEN, admitting a trickle of probe emits to test recovery; a run of successes closes it, a single failure re-opens it with a longer backoff.

The trip condition is where breakers for a real-time trigger router differ from the textbook error-count breaker. Consumers rarely fail cleanly with errors — they slow down first, and a latency-blind breaker keeps feeding a dying consumer until timeouts finally register as errors, by which point the damage is done. So the trip is latency-primary: open when the consumer’s P99 exceeds 50ms for three consecutive scrape windows, or when the error rate crosses 50% within a rolling window. Latency-primary tripping catches the slow-consumer failure a full window or two before the error-count breaker would, which is the difference between a contained blip and a propagated stall.

The measured profile below compares no breaker, an error-only breaker, and a latency-primary breaker against a downstream consumer whose P99 is injected to spike from 8ms to 900ms at t=0 and recover at t=30s. Metrics are the router’s own health during the consumer outage.

The no-breaker row is the disaster: the router inherits the consumer’s 900ms tail and the pool exhausts. A timeout-only guard bounds each call to 200ms but still pays that cost on every emit and still stalls coroutines up to the timeout — it caps the damage without preventing it. The error-only breaker eventually trips but lags because the consumer is slow, not erroring, so the router eats elevated latency until timeouts accumulate. The latency-primary breaker holds the router’s own P99 at ~12ms throughout, because it stops calling the slow consumer within three windows and diverts to the fallback with zero wait. Crucially, its blast radius is one consumer: the pricing engine’s outage never touches the compliance path. The half-open recovery mechanics — probe rate, backoff growth, and the success threshold that avoids re-tripping — are worked in half-open recovery for geofence circuit breakers.

Implementation Trade-offs and the Critical Path #

The breaker wraps the emit path, so its CLOSED-state overhead runs on every trigger and must be trivial — a state check, a latency push into a ring buffer, a counter bump. No allocation, no lock on the hot path. The rolling P99 is the one subtlety: computing an exact percentile per call is too expensive, so use a fixed-size ring of recent latencies and estimate the percentile from it, or a lighter approximate-quantile sketch if the window is large.

from __future__ import annotations

import asyncio
import time
from collections import deque
from dataclasses import dataclass, field
from enum import Enum


class State(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class BreakerOpen(Exception):
    """Raised when the breaker is open; caller routes to the fallback."""


@dataclass(slots=True)
class CircuitBreaker:
    p99_trip_ms: float = 50.0          # latency-primary trip threshold
    trip_windows: int = 3              # consecutive breaching windows to open
    error_trip_rate: float = 0.50      # error-rate trip
    base_backoff_s: float = 1.0        # initial open window; doubles on re-trip
    max_backoff_s: float = 30.0
    success_to_close: int = 5          # consecutive probe successes to close
    _state: State = field(default=State.CLOSED)
    _latencies: deque[float] = field(default_factory=lambda: deque(maxlen=256))
    _breach_windows: int = 0
    _errors: int = 0
    _calls: int = 0
    _open_until: float = 0.0
    _backoff_s: float = 1.0
    _probe_successes: int = 0

    def allow(self) -> bool:
        # Fast path: decide whether to emit, transitioning OPEN -> HALF_OPEN on timer.
        if self._state is State.OPEN:
            if time.monotonic() >= self._open_until:
                self._state = State.HALF_OPEN
                self._probe_successes = 0
                return True                     # admit the first probe
            return False                        # still open — divert to fallback
        return True                             # CLOSED or HALF_OPEN admit

    def record(self, latency_ms: float, ok: bool) -> None:
        self._latencies.append(latency_ms)
        self._calls += 1
        if not ok:
            self._errors += 1
        if self._state is State.HALF_OPEN:
            self._on_probe(ok)
        elif self._state is State.CLOSED:
            self._maybe_trip()

    def _p99(self) -> float:
        if not self._latencies:
            return 0.0
        s = sorted(self._latencies)             # 256 items — cheap, off hot path is fine
        return s[min(len(s) - 1, int(0.99 * len(s)))]

    def _maybe_trip(self) -> None:
        # Evaluate per window (called at window close in production, inlined here).
        breach = self._p99() > self.p99_trip_ms or (
            self._calls and self._errors / self._calls > self.error_trip_rate
        )
        self._breach_windows = self._breach_windows + 1 if breach else 0
        if self._breach_windows >= self.trip_windows:
            self._open()

    def _open(self) -> None:
        self._state = State.OPEN
        self._backoff_s = min(self.max_backoff_s, self._backoff_s * 2)  # exp backoff
        self._open_until = time.monotonic() + self._backoff_s
        self._breach_windows = 0

    def _on_probe(self, ok: bool) -> None:
        if not ok:
            self._open()                        # any probe failure re-opens, longer backoff
            return
        self._probe_successes += 1
        if self._probe_successes >= self.success_to_close:
            self._state = State.CLOSED
            self._backoff_s = self.base_backoff_s   # reset backoff on clean recovery
            self._errors = self._calls = 0

The emit wrapper ties allow and record together and enforces the fallback:

async def emit(breaker: CircuitBreaker, send, fallback, trigger: object) -> bool:
    if not breaker.allow():
        await fallback(trigger)                 # OPEN: buffer to DLQ, no wait
        return False
    start = time.perf_counter()
    try:
        await asyncio.wait_for(send(trigger), timeout=0.2)   # bound every call
        breaker.record((time.perf_counter() - start) * 1000.0, ok=True)
        return True
    except (asyncio.TimeoutError, Exception):   # noqa: BLE001 — any failure counts
        breaker.record((time.perf_counter() - start) * 1000.0, ok=False)
        await fallback(trigger)                 # degrade rather than block
        return False

The deliberate design choices: allow never blocks — an OPEN breaker returns instantly so the router’s loop is never held hostage by a dead consumer; every real call is still bounded by asyncio.wait_for, because the breaker guards the aggregate while the timeout guards the individual call that trips it; and the fallback is invoked on both the OPEN divert and the failed call, so no trigger is silently dropped — it goes to the dead-letter buffer for later replay.

Memory Footprint and Streaming Churn #

A breaker’s per-consumer state is tiny — a 256-slot latency ring (~2KB), a handful of counters, two floats — so a router fanning to 50 distinct consumers carries ~100KB of breaker state total. The churn hazard is the fallback path, not the breaker: when a breaker is OPEN, every trigger to that consumer diverts to the dead-letter buffer, and at a high trigger rate that is a flood of envelope allocations exactly when the system is already stressed. Pre-allocate the dead-letter envelopes or batch them into a ring buffer so an OPEN breaker’s divert path allocates nothing per trigger. The latency ring itself must be a bounded deque(maxlen=...); an unbounded list of every latency ever seen would grow without limit and defeat the point.

The sorted-percentile computation in _p99 allocates a fresh sorted list per evaluation, which is fine at a per-window cadence (once a second) but must never be called per emit. Keep percentile evaluation on the window boundary, not the hot path — the record call only appends, and a separate window-close callback computes the percentile and calls _maybe_trip. This separation keeps the CLOSED-state overhead at one append plus two integer bumps.

Async Mutation Boundaries and Queue Semantics #

The breaker’s state transition is the mutation boundary, and it must be safe against concurrent emits. Under asyncio’s single-threaded loop the transitions are already atomic — no two coroutines run breaker code simultaneously — so no lock is needed, but the invariant to preserve is that allow and record for a given consumer share one breaker instance, never a per-coroutine copy, or the state fragments and never trips. Shard breakers by consumer, not by coroutine.

The coupling to backpressure is direct. An OPEN breaker diverts to the dead-letter buffer, and that buffer is itself a bounded queue with its own shed policy — if the consumer stays down long enough to fill the dead-letter buffer, the router must shed even the diverted triggers, and it should shed the same way the ingest path does: idle-ping triggers first, compliance triggers never. This is why the breaker belongs at the phase boundary and composes with the priority shedding rather than replacing it. A breaker without a bounded, priority-aware fallback just moves the unbounded-buffer problem one hop downstream. Export breaker_state (0/1/2), trip_count, open_duration_s, and fallback_depth per consumer to Prometheus, and alert on a breaker that flaps — repeatedly cycling OPEN→HALF-OPEN→OPEN — because flapping means the backoff is too short or the success threshold too low, both covered in the half-open recovery page.

Operational Runbook and Failure Mitigation #

When a consumer degrades, the breaker should already be doing its job; the runbook is for confirming it did and for the cases where it mis-configured.

The standing diagnostic loop:

1. Confirm the symptom is downstream, not local. Pull the router’s own P99 and each consumer’s P99 from Prometheus. If the router tail tracks exactly one consumer’s tail, the breaker for that consumer is the lever; if the router tail is high across all consumers, the problem is in the router (queue, GC, event loop), not a downstream.
2. Read the breaker state. Check breaker_state per consumer. A slow consumer with a CLOSED breaker means the trip condition is not firing — verify the latency ring is being fed and p99_trip_ms is set.
3. Attribute the stall. Run py-spy dump on the router. Coroutines parked in send/wait_for for a specific consumer confirm that emits are blocking; if they are, the breaker is not diverting and allow is returning True when it should return False.
4. Verify the fallback drains. Confirm fallback_depth is bounded and the dead-letter replay is consuming it. An unbounded or unconsumed fallback turns a contained outage back into a memory leak.
5. Watch the recovery. When the consumer heals, confirm the breaker moves CLOSED via HALF-OPEN and does not flap. Persistent flapping points to backoff or success-threshold tuning.
6. Validate isolation. Inject a synthetic 900ms P99 into one consumer under load and confirm the other consumers’ emit latency is unchanged — the whole point of the phase-boundary breaker is that the blast radius is one consumer.

Continuous chaos tests in staging should inject per-consumer latency and error spikes and assert the router’s own P99 stays under budget while exactly one consumer’s traffic diverts.

Architectural Guidance: When to Choose Which #

The breaker is one of three resilience primitives, and they are complementary, not alternatives.

In production these layer: a bulkhead gives each consumer its own bounded connection pool and semaphore so it cannot starve the others, a timeout bounds each individual call, and a latency-primary circuit breaker sits on top to stop calling a consumer that is failing in aggregate and to probe for its recovery. The breaker alone protects against sustained degradation; the bulkhead alone protects against resource monopolization; the timeout alone protects against a single hung call — you generally want all three, with the breaker as the coordinating state machine. The invariant across every variant is that the breaker lives at the phase boundary it protects, with a bounded priority-aware fallback behind it, so a downstream failure degrades to buffered replay of exactly one consumer’s triggers rather than a router-wide stall.

Frequently Asked Questions #

Related #

• Event Routing & Backpressure — parent overview of the routing tier and where breakers sit.
• Half-Open Recovery for Geofence Circuit Breakers — probe strategy, backoff, and avoiding thundering-herd re-trips.
• Backpressure & Flow-Control Strategies — the priority-shed fallback the breaker diverts into.
• Sizing Bounded asyncio Queues for Geofence Pipelines — bounding the dead-letter fallback buffer behind the breaker.
• Core Architecture & Latency Constraints — the end-to-end budget the breaker defends at the phase boundary.