Kafka vs Redis Streams for Geofence Trigger Routing

Once a geofence evaluator decides that a device has crossed a perimeter, the ENTER/EXIT/DWELL trigger has to travel — from the node that computed it to the surge-pricing engine, the compliance auditor, the dispatch queue, and the analytics sink that all depend on it. That transport substrate is not neutral plumbing: it fixes the ordering guarantee downstream consumers can assume, the durability contract that survives a broker crash, and the tail latency between a physical boundary crossing and the action it triggers. This page expands the event-transport layer introduced in event routing and backpressure, and it answers the decision mobility and IoT teams reach for by name: should geofence triggers ride Apache Kafka or Redis Streams? The two are the dominant log-structured brokers with consumer-group semantics, and they fail in opposite directions — Kafka trades a millisecond or two of latency for replicated durability and replayable partitions, while Redis Streams trades durability guarantees for the lowest achievable end-to-end latency and radically simpler operations.

The reader here is a backend engineer who already holds a sub-50ms evaluation budget and now has to spend a slice of it on transport. The structural difference drives every trade-off below. Kafka is a partitioned, replicated commit log persisted to disk across a broker quorum: ordering is guaranteed per partition, throughput scales by adding partitions, and a consumer group rebalances partition ownership across its members. Redis Streams is an in-memory append-only log inside a single shard: ordering is guaranteed per stream key, throughput is bounded by one core’s single-threaded command loop, and a consumer group tracks a pending-entries list (PEL) of delivered-but-unacknowledged messages that survives only as far as the shard’s persistence settings allow.

Algorithmic Divergence and Latency Profiles #

The two brokers diverge on the write path before a single consumer sees a trigger. A Kafka produce request with acks=all is not acknowledged until the partition leader and every in-sync replica have the record in their page cache and the configured flush policy is satisfied; that replication round-trip is the price of surviving a broker loss, and it costs a bounded few milliseconds. A Redis XADD returns as soon as the single-threaded command loop appends the entry to the in-memory radix tree — persistence to the append-only file happens asynchronously on the configured fsync cadence, so the acknowledgement precedes durability rather than following it. That single architectural fact explains most of the latency gap and all of the durability gap.

$$L_{\text{kafka}}\approx t_{\text{batch}}+t_{\text{replicate}}+t_{\text{fetch}}\text{vs}{L}_{\text{redis}}\approx t_{\text{XADD}}+t_{\text{XREADGROUP}}$$

Throughput scales along different axes. Kafka’s ceiling is per-partition — a single partition sustains roughly 80k trigger messages/sec at a ~250-byte payload, and you scale horizontally by adding partitions and consumers up to the point where partition count itself becomes a rebalance and metadata cost. Redis Streams runs the entire stream through one core, so a single shard peaks near 120k XADD/sec but cannot be split further without client-side sharding across multiple stream keys; you scale by fanning triggers across N stream keys and accepting that ordering holds only within each key.

The measured head-to-head below routes 256-byte ENTER/EXIT trigger records through each broker at three load points, single-region, producers and consumers on the same 10-gigabit segment, Kafka with three brokers and replication factor three (acks=all), Redis with append-only file and appendfsync everysec. Latency is producer-ack-to-consumer-receive, end to end.

Redis Streams holds a sub-3ms P95 at load because it never waits on replication or a disk flush before acknowledging, but the same table shows the cost: with appendfsync everysec a shard crash forfeits up to a second of triggers, and forcing appendfsync always to close that window erases most of the latency advantage. Kafka with acks=all pays a couple of milliseconds at P50 and holds P99 under 25ms even at 100k/s, and those triggers survive a broker failure because they are replicated across the in-sync set before the producer is ever told the write succeeded. The exact geometry that produced these triggers is upstream — see point-in-polygon algorithm benchmarks — but the transport latency stacks directly on top of the evaluation budget defined in core architecture and latency constraints, so every millisecond here is a millisecond the spatial phase does not get.

Implementation Trade-offs and the Critical Path #

Both brokers demand the same async discipline: the produce and consume calls are I/O-bound and must never share a thread with CPU-bound geometry, or the asyncio event loop stalls and consumer lag climbs. The divergence is in the delivery contract. Kafka gives you an offset per partition and you commit it after processing; Redis gives you a message id in a pending-entries list and you XACK it after processing. Both are at-least-once when you commit or ack after the side effect, and both silently become at-most-once if you commit before.

The Kafka critical path keys every trigger by device_id so that all transitions for one device land in the same partition and stay ordered — the invariant that keeps a stale EXIT from overtaking the ENTER that preceded it:

import asyncio
from dataclasses import dataclass
from aiokafka import AIOKafkaProducer  # aiokafka >= 0.10


@dataclass(slots=True)
class Trigger:
    device_id: str
    fence_id: str
    transition: str      # "enter" | "exit" | "dwell"
    event_time_ns: int


class KafkaTriggerRouter:
    __slots__ = ("_producer",)

    def __init__(self, producer: AIOKafkaProducer) -> None:
        self._producer = producer

    async def emit(self, t: Trigger) -> None:
        # Key by device_id: all transitions for one device hash to one
        # partition, so per-partition order == per-device order.
        payload = f"{t.fence_id}:{t.transition}:{t.event_time_ns}".encode()
        try:
            # send_and_wait blocks until acks satisfied; the batching that
            # amortises this over many triggers is tuned via linger.ms/batch.size.
            await self._producer.send_and_wait(
                "geofence.triggers", key=t.device_id.encode(), value=payload,
            )
        except Exception as exc:  # noqa: BLE001 - route to dead-letter, never drop silently
            await self._to_dead_letter(t, exc)

    async def _to_dead_letter(self, t: Trigger, exc: Exception) -> None:
        ...  # covered in dead-letter-topics-and-poison-message-handling

The Redis Streams path expresses the same at-least-once contract through XREADGROUP plus explicit XACK, and its cheaper delivery shows up as a tighter latency profile at the cost of the durability the table above quantified:

import redis.asyncio as redis  # redis >= 5.0


class RedisStreamRouter:
    __slots__ = ("_r", "_stream", "_group")

    def __init__(self, r: redis.Redis, stream: str, group: str) -> None:
        self._r, self._stream, self._group = r, stream, group

    async def emit(self, t: Trigger) -> None:
        # XADD returns on in-memory append; durability lags per appendfsync.
        await self._r.xadd(
            self._stream,
            {"dev": t.device_id, "fence": t.fence_id, "tr": t.transition},
            maxlen=1_000_000, approximate=True,  # cap memory: trim old entries
        )

    async def consume(self, consumer: str) -> None:
        while True:
            resp = await self._r.xreadgroup(
                self._group, consumer, {self._stream: ">"}, count=256, block=200,
            )
            for _stream, entries in resp or []:
                for msg_id, fields in entries:
                    try:
                        await self._handle(fields)
                        await self._r.xack(self._stream, self._group, msg_id)  # ack AFTER side effect
                    except Exception:  # noqa: BLE001 - leave in PEL for XCLAIM recovery
                        pass

    async def _handle(self, fields: dict[bytes, bytes]) -> None:
        ...

Two omissions are deliberate and identical in spirit across both. Neither acknowledges before the side effect completes, because an ack-before-process converts a consumer crash into silent trigger loss. And neither retries inline forever — a message that fails repeatedly is a poison message that must be diverted, not looped on, which is exactly the failure mode dead-letter topics and poison-message handling exists to contain.

Memory Footprint and Streaming Churn #

The two brokers hold state in very different places, and that placement is the memory story. Kafka’s log lives on disk and is served from the OS page cache, so a broker’s resident memory is dominated by the JVM heap plus whatever the page cache opportunistically holds; a partition can retain days of triggers for replay without ever pressuring process RSS, because retention is a disk-bytes budget, not a heap budget. The consumer’s footprint is a bounded fetch buffer plus its in-flight batch. This is why Kafka is the natural substrate when downstream systems need to replay — rewind a compliance consumer to reprocess a day of triggers after a bug fix — without the producer having done anything special.

Redis Streams holds the entire log in RAM. Every un-trimmed entry is live process memory, so an unbounded stream is an OOM waiting for the traffic spike that fills it. The discipline is the maxlen cap in the XADD above with approximate=True, which lets Redis trim in macro-node chunks rather than paying an exact trim on every write. The second, quieter memory cost is the pending-entries list: every delivered-but-unacknowledged trigger stays in the PEL, so a consumer that reads fast and acks slowly grows the PEL without bound and inflates shard memory in lockstep with its own lag. A stalled consumer in a Redis group is not just a latency problem — it is a memory leak until its messages are XCLAIMed or the consumer is evicted.

Both brokers benefit from smaller triggers on the wire. Serializing the transition as a compact binary record rather than JSON — the same codec discipline the pipeline applies at ingress — cuts per-message bytes 40–60% on a 256-byte trigger, which on Kafka means more messages per batch and per fetch, and on Redis means proportionally more triggers per gigabyte of shard memory before the maxlen trim engages.

Async Mutation Boundaries and Queue Semantics #

Consumer lag is where the transport substrate meets backpressure. In Kafka, lag is a first-class, per-partition number: log_end_offset - committed_offset, exported by the broker and every consumer, so a consumer that falls behind is measurable in messages and in seconds, and a bounded local asyncio.Queue between the fetch loop and the geometry workers converts that lag into an explicit shed decision rather than an unbounded heap. In Redis, lag is the depth of the PEL plus the gap between the group’s last-delivered id and the stream’s tail (XINFO GROUPS reports both), and the recovery primitive is XAUTOCLAIM: a monitor reassigns any message idle longer than a threshold to a healthy consumer so a crashed member’s in-flight triggers are not stranded.

The queue boundary that guards the geometry stage is identical to the one the evaluation pipeline already runs, and it is the seam where backpressure and flow-control strategies apply:

import asyncio


class BoundedConsumer:
    def __init__(self, capacity: int = 50_000) -> None:
        # Bounded queue is the backpressure signal between broker fetch and
        # the CPU-bound trigger handlers; full() is a decision, not an error.
        self._work: asyncio.Queue[bytes] = asyncio.Queue(maxsize=capacity)
        self._shed = 0

    async def feed(self, record: bytes) -> None:
        if self._work.full():
            self._shed += 1          # pause fetch / slow commit rather than OOM
            await self._work.put(record)  # await applies backpressure to the fetch loop
        else:
            self._work.put_nowait(record)

The mutation boundary differs in one important way. A Kafka consumer applies backpressure to the broker for free by simply not calling poll/getmany — the broker holds the data on disk and the consumer’s committed offset stops advancing, so slowing down costs nothing but lag. A Redis consumer that stops calling XREADGROUP leaves entries in memory and grows the shard, so Redis backpressure has to be paired with the maxlen trim and a live watch on shard memory; you cannot let a Redis consumer fall arbitrarily behind the way you can a Kafka one. Ordering-preserving retries also differ: a Kafka poison message can stall its entire partition if you insist on in-order processing, whereas a Redis XCLAIM can pull one stuck message aside without blocking the others in the stream — a genuine operational advantage for Redis when strict per-key order is not required.

Operational Runbook and Failure Mitigation #

When trigger delivery misbehaves, the symptom is either rising lag or missing triggers, and the cause is one of four modes. Diagnose Kafka with kafka-consumer-groups --describe for per-partition lag and py-spy dump on the consumer for a stack; diagnose Redis with XINFO GROUPS/XPENDING for PEL depth and redis-cli --latency for command-loop stalls.

The standing diagnostic loop:

1. Confirm the symptom. For Kafka, pull records-lag-max per group; for Redis, read XPENDING <stream> <group> for PEL depth and the idle time of the oldest entry. Rising lag with healthy throughput means consumers are slow; flat throughput with gaps in trigger pairs means loss, which is a correctness bug, not a capacity one.
2. Attribute the cost. Run py-spy record on the consumer for 30s under live load. Time in the geometry handler means the CPU stage is the bottleneck and the broker is fine; time in getmany/xreadgroup wait means the broker or network is the bottleneck.
3. Check the commit/ack ordering. Confirm offset commit (commit()) or XACK runs strictly after the side effect. An ack-before-process is the single most common source of the silent-loss row above.
4. Quantify the durability window. For Redis, read appendfsync and lastsave; a crash forfeits everything since the last flush. For Kafka, confirm min.insync.replicas >= 2 so an acks=all write cannot be acknowledged against a single replica.
5. Validate idempotent replay. Replay a bounded window into a staging consumer and confirm downstream state is unchanged — duplicate ENTER triggers must be absorbed by the idempotency key, per idempotent trigger emission semantics, before you ever replay against production.

Continuous monitoring should hold consumer lag under one second of triggers at peak and, for Redis, keep PEL depth under a few thousand entries — a PEL that grows monotonically is a stalled consumer masquerading as throughput.

Architectural Guidance: When to Choose Which #

Neither broker is a default. The decision is driven by three axes — the durability the trigger demands, the latency the downstream action can tolerate, and the operational surface the team can carry.

In practice these are frequently combined: latency-critical triggers that drive an immediate in-app action ride Redis Streams for the tightest tail, while the same triggers are mirrored into Kafka for durable, replayable audit and analytics. The invariant to preserve across either substrate is that acknowledgement follows the side effect and emission is idempotent, so at-least-once delivery plus deduplication yields effectively-once behavior downstream. Get that wrong and the broker choice is irrelevant — you will either lose triggers or double-fire them regardless of how durable the log underneath is.

Frequently Asked Questions #

Related #

• Event Routing and Backpressure — parent overview of the trigger transport and flow-control layer.
• Tuning Kafka Producer Batching for Geofence Triggers — linger.ms, batch.size, and acks for bursty trigger streams.
• Consumer-Group Rebalancing Without Dropping Geofence Triggers — cooperative-sticky, static membership, and safe offset commit.
• Dead-Letter Topics and Poison-Message Handling — where a repeatedly failing trigger goes instead of stalling a partition.
• Idempotent Trigger Emission Semantics — deduplication that makes at-least-once delivery safe on either broker.