Bulkheads and Per-Consumer Concurrency Caps

A circuit breaker is a reactive control: it observes failures, decides the downstream is unhealthy, and stops sending. Between the first failed request and the trip decision there is a window — several seconds, by design, because a breaker that trips on one timeout is useless — and in that window an unbounded worker pool will pour every request it has into the failing consumer. At 100 triggers/sec against a 10 s connect timeout, a consumer absorbs 1,000 in-flight requests before the first one even reports failure. The bulkhead is the proactive control that bounds this from the first millisecond. This page sits under circuit breakers for downstream trigger consumers within Event Routing & Backpressure.

Concept and specification #

A bulkhead partitions a shared resource so that exhaustion in one partition cannot propagate. For a trigger emitter the shared resource is worker concurrency, and the partition is per downstream consumer. The cap follows from Little’s law: to sustain an arrival rate $\lambda$ against a service time $L$, the number of concurrent requests needed is

$$C=\lambda \cdot L_{p99}\times (1+\text{headroom})$$

so a consumer taking 100 triggers/sec at a 120 ms P99 needs 12 slots, and 16 with a third of headroom. Anything above that is not capacity, it is queue — and queue in a socket is indistinguishable from queue in memory except that it is harder to see.

Two properties make the cap effective where the breaker is not. It is immediate: there is no observation window, so the bound holds from the first request. And it is unconditional: it does not require a judgement about health, so it bounds a consumer that is merely slow as well as one that is failing, which matters because a slow consumer never trips a breaker configured on error rate.

The bottom row includes the third control that is easy to leave out: a per-request timeout well below the transport default. The socket default of tens of seconds means a slot is held for tens of seconds, so the cap bounds how many slots a bad consumer holds while the timeout bounds how long it holds each one. Together they bound the resource-seconds; either alone bounds only one factor.

Step-by-step implementation #

1. Reject at the cap, never queue behind it. The distinction is the whole design. await semaphore.acquire() converts a bounded cap into an unbounded queue of waiters and reintroduces exactly the head-of-line blocking the bulkhead exists to prevent.

from __future__ import annotations
import asyncio
from dataclasses import dataclass, field

@dataclass(slots=True)
class Bulkhead:
    """A hard cap on concurrent requests to one consumer. Rejects at the cap."""
    cap: int
    in_flight: int = 0
    rejected: int = 0
    admitted: int = 0
    high_water: int = 0

    def try_acquire(self) -> bool:
        if self.in_flight >= self.cap:
            self.rejected += 1
            return False                     # caller must NOT wait
        self.in_flight += 1
        self.admitted += 1
        self.high_water = max(self.high_water, self.in_flight)
        return True

    def release(self) -> None:
        self.in_flight -= 1

async def send(bh: Bulkhead, coro_factory, timeout_s: float = 2.0) -> str:
    if not bh.try_acquire():
        return "rejected"                    # -> dead-letter, immediately
    try:
        await asyncio.wait_for(coro_factory(), timeout=timeout_s)
        return "ok"
    except asyncio.TimeoutError:
        return "timeout"
    finally:
        bh.release()                         # release on EVERY path

2. Derive the cap from measurement, not from the pool size. A cap expressed as a fraction of the worker pool (“no consumer may take more than 10%”) scales the wrong way: adding workers raises every consumer’s blast radius. A cap derived from the consumer’s own throughput and latency is invariant to pool size.

3. Give the timeout a budget, not a default. The per-request timeout should be a fraction of the trigger’s usefulness horizon — 2 s for a real-time trigger against a 30 s socket default. This alone reduces resource-seconds consumed by a black-hole endpoint by 15×.

4. Release on every path, including cancellation. A finally block is not optional here; a leaked slot is permanent, and a handful of leaks over a week silently reduces the cap to zero, producing a consumer that is rejected constantly for no visible reason. Export in_flight and assert it returns to zero when the consumer is idle.

5. Size headroom from burst shape, not from anxiety. Headroom absorbs the arrival burstiness that Little’s law’s steady-state assumption ignores. A third is adequate for triggers arriving from a broker batch; a fan-out driven by a synchronised event — every vehicle crossing a shift boundary — needs more, or an initial jitter as described in exponential backoff with jitter.

6. Feed rejections to the breaker as evidence. A consumer being rejected at its cap is a consumer that is not keeping up, which is exactly what a breaker should know. Counting rejections toward the breaker’s failure signal lets a sustained saturation trip it, freeing the slots entirely.

Benchmark and verification #

Measured with 1,200 workers, 400 consumers, one of which becomes a black hole for 60 s:

The breaker-only row is the one worth studying, because it is the common configuration and it is much weaker than it appears: the breaker eventually trips and frees the pool, but 61% of the pool was already consumed during the observation window, and the other consumers’ P99 was nine times its healthy value for that whole period. The cap alone, with no breaker at all, is dramatically better on every column — and adding the breaker on top takes the last percentage point and returns the slots the cap would otherwise let the bad consumer hold indefinitely.

Note that no configuration loses triggers: rejections go to the dead-letter path and are replayed under the discipline in replaying dead-letter triggers safely. A bulkhead that dropped rejections would trade an availability problem for a correctness one.

Verify with a black-hole endpoint in staging — one that accepts connections and never responds — and assert that in_flight for that consumer plateaus at exactly the cap and that no other consumer’s latency moves. An endpoint returning fast 500s exercises the breaker and almost nothing of the bulkhead, which is why teams with only that test discover the gap in production.

Failure modes and edge cases #

The fifth row is the design question that decides whether the bulkhead helps at all: what fails independently? Partitioning by downstream service is right when each service is a separate system; partitioning by tenant is right when many tenants share one gateway that fails per tenant; partitioning by endpoint URL is right for webhooks. Choosing a partition coarser than the failure boundary means one failing unit still consumes its whole partition’s capacity, and the bulkhead provides isolation between things that were not going to interfere anyway.

A last caution about the interaction with retries. A retried request re-acquires a slot, so a consumer at its cap with a retry policy consumes cap-times-attempts resource-seconds rather than cap. The retry budget from webhook fan-out and retry budgets is what keeps that bounded, and a bulkhead deployed without one will still saturate — more slowly, and just as completely.

Related #

• Circuit Breakers for Downstream Trigger Consumers — the parent topic and the reactive control this complements.
• Half-Open Recovery for Geofence Circuit Breakers — probing recovery without re-flooding the consumer.
• Webhook Fan-Out & Retry Budgets — per-tenant lanes built from exactly this primitive.