Exactly-Once vs At-Least-Once Trigger Delivery

Every geofence team eventually asks for exactly-once delivery, and the honest answer is that it does not exist as a network primitive — you cannot deliver a message across an unreliable link exactly once, because the sender can never distinguish a lost message from a lost acknowledgement without retrying, and retrying is duplication. What Kafka calls exactly-once semantics (EOS) is really at-least-once delivery plus an idempotent, transactional consumer that makes the duplicates invisible. This page sits under idempotent trigger emission semantics and the broader event routing and backpressure architecture, and it answers the design question directly: should a geofence pipeline pay for Kafka transactions, or run at-least-once with a cheap dedup window and get the same correctness for a fraction of the cost?

Concept and specification #

The two delivery guarantees differ in where duplicate-suppression happens and what it costs. At-least-once accepts that the broker may deliver a trigger more than once and pushes suppression to the consumer via the deterministic idempotency key and a dedup window. Exactly-once (Kafka EOS) uses a transactional producer and a read_committed consumer so that the write of the output record and the commit of the input offset are one atomic transaction — a crash rolls back both, so no partial effect is visible.

The cost model is the difference. At-least-once pays one dedup lookup per trigger, $O(1)$, plus the memory of the window. EOS pays a two-phase commit per transaction batch: the producer registers with a transaction coordinator, writes records, writes offset commits into the transaction, and the coordinator writes commit markers to every partition touched. The added latency is the coordinator round-trips plus the read_committed consumer’s need to buffer until the commit marker arrives:

$$L_{\text{EOS}}\approx L_{\text{ALO}}+2\cdot RT{T}_{\text{coord}}+t_{\text{commit-marker}}$$

so the exactly-once latency floor is the at-least-once latency plus the transaction machinery, which is why EOS throughput drops as transaction size shrinks — a per-trigger transaction is pathological, and even a well-batched one pays the marker overhead.

The last row is the decisive one for geofencing: EOS is exactly-once only within Kafka. The moment a trigger fires an external side effect — a webhook, a surge-price update, a compliance write to Postgres — the Kafka transaction cannot roll that back, so you still need an idempotency key at the external boundary. EOS does not remove the need for the dedup window; it adds cost on top of it.

There is a middle tier that most geofence pipelines actually want, and it is frequently confused with full EOS: the idempotent producer (enable.idempotence=true). This is not a transaction — it is a per-partition de-duplication the broker performs using a producer id (PID) and a monotonic per-partition sequence number, so a producer retry after a lost ack does not append the record twice to the log. It eliminates producer-side duplicates (the retry storm) at almost no cost — no coordinator, no commit markers, single-digit-microsecond overhead — but does nothing about consumer-side duplicates from reprocessing after a rebalance. The correct default for a geofence trigger stream is therefore: idempotent producer to kill retry duplicates cheaply, plus a consumer-side dedup window keyed on the deterministic idempotency key to kill reprocessing duplicates. Full transactional EOS is the tier above that, justified only when a Kafka-to-Kafka stage needs atomic multi-partition writes and offset commits together.

Step-by-step implementation #

Prerequisites: Python 3.11+, aiokafka>=0.10 (transactional producer support), a broker with transaction.state.log configured. Below are both paths so the cost is concrete.

1. At-least-once + idempotent consumer. The producer is fire-and-forget with retries; the consumer dedups on the key and commits offsets after the side effect.

from __future__ import annotations
from aiokafka import AIOKafkaConsumer
import redis.asyncio as redis
import json

async def at_least_once_consume(
    consumer: AIOKafkaConsumer,      # enable_auto_commit=False
    r: redis.Redis,
) -> None:
    async for record in consumer:
        trigger = json.loads(record.value)
        key: str = trigger["idempotency_key"]
        # Dedup gate: SETNX returns True only for a first-ever key.
        if await r.set(f"idem:{key}", b"1", nx=True, ex=300):
            await fire_side_effect(trigger)          # webhook / DB write
        # else: duplicate, already fired -> skip silently
        await consumer.commit()                       # at-least-once boundary

2. Exactly-once with a transactional producer. The output write and the input-offset commit are one transaction; a crash rolls both back.

from aiokafka import AIOKafkaConsumer, AIOKafkaProducer

async def exactly_once_process(
    consumer: AIOKafkaConsumer,      # isolation_level="read_committed"
    producer: AIOKafkaProducer,      # transactional_id="geofence-eos-1"
    out_topic: str,
) -> None:
    async for batch in consumer.getmany(timeout_ms=200):
        async with producer.transaction():           # begin 2PC
            for tp, records in batch.items():
                for record in records:
                    trigger = transform(record.value)
                    await producer.send(out_topic, value=trigger)
                # Offsets committed INSIDE the transaction: atomic with the writes.
                offsets = {tp: records[-1].offset + 1}
                await producer.send_offsets_to_transaction(
                    offsets, consumer._group_id)
            # commit marker written to all partitions on context exit

Gotcha: EOS is atomic only for the Kafka write and offset commit. The send() to out_topic is covered; a fire_side_effect webhook inside the transaction is not rolled back on abort, so external effects still need the dedup key from path 1.

3. Enable the idempotent producer regardless. Set enable_idempotence=True on the producer in the at-least-once path too — it costs almost nothing and removes producer-retry duplicates before they ever reach the consumer, shrinking the dedup window’s workload. The consumer dedup then only has to absorb reprocessing duplicates from rebalances, not retry storms.

producer = AIOKafkaProducer(
    bootstrap_servers=brokers,
    enable_idempotence=True,     # PID + per-partition seqno: kills retry dupes
    acks="all",                  # required for idempotence; survives leader loss
    max_in_flight_requests_per_connection=5,  # safe with idempotence on
)

4. Choose the boundary consciously. For a pipeline whose triggers fan out to external systems, at-least-once + idempotent producer + consumer dedup at the external boundary is both cheaper and more correct than EOS, because EOS gives no guarantee past the Kafka edge. Reserve full transactional EOS for Kafka-to-Kafka stages where the entire effect is a downstream topic write and no external side effect exists.

Benchmark and verification #

Measured on a 6-broker cluster, a single async worker transforming and re-emitting triggers, 64-byte payloads, comparing at-least-once + Redis dedup against Kafka EOS at a 200ms transaction batch:

Both achieve zero duplicate side effects — the correctness goal is met either way — but EOS costs 2.3x throughput (48k to 21k) and roughly doubles P99 (18ms to 41ms), because the transaction coordinator round-trips and the read_committed buffering sit on the critical path. The at-least-once path spends its budget on a single sub-millisecond dedup lookup and nothing else.

The EOS penalty is dominated by transaction batch size, and the relationship is steep. The commit-marker overhead is a fixed per-transaction cost amortized over the batch, so throughput scales as $T\propto B/(B\cdot t_{\text{msg}}+t_{\text{commit}})$ for batch size $B$: at a 200ms batch (~4,200 triggers) the marker cost is small per message, but shrinking the batch to a 20ms window (~420 triggers) to cut latency collapses EOS throughput to ~9,000 trig/sec because the fixed commit cost now dominates. This is the central tension of EOS — you can trade its latency down or its throughput up, but not both, whereas at-least-once holds 48k/sec at an 18ms P99 without a knob to turn. For a geofence stream that must hold a tight tail latency and high throughput, that inability to have both is usually disqualifying. A minimal verification that both suppress duplicates:

import asyncio

async def verify_no_dupes(process, triggers: list[dict], replays: int = 3) -> int:
    # Feed each trigger `replays` times; a correct pipeline fires each once.
    side_effects: set[str] = set()
    async def record(t): side_effects.add(t["idempotency_key"])
    for _ in range(replays):
        for t in triggers:
            await process(t, record)
    return len(side_effects)   # must equal len(set of unique keys)

len(side_effects) must equal the number of unique keys regardless of replays — proving that duplicate delivery, whether suppressed by dedup or by transactions, produces exactly one side effect per logical trigger.

Failure modes and edge cases #

• Expecting EOS to cover external side effects. The most common mistake: enabling Kafka transactions and assuming the webhook fires exactly once. EOS is scoped to Kafka; any external effect needs the idempotency key. Pay for EOS only when the whole effect is a Kafka write.
• Per-trigger transactions. A transaction per trigger collapses EOS throughput to a fraction of at-least-once because the commit-marker overhead is paid on every message. Batch transactions (100–500 triggers or a 200ms window) to amortize the 2PC, accepting the added tail latency of buffering.
• transactional_id reuse across instances. Two producers sharing a transactional_id fence each other — the coordinator aborts the older epoch. Assign a stable, unique id per producer instance, coordinated with consumer group rebalancing without dropping triggers so a rebalance does not orphan a transaction.
• Dedup window cleared on restart under at-least-once. If the dedup store is in-process and the consumer restarts mid-replay, the window is empty and re-admits in-flight triggers. Use a shared Redis window with persistence so at-least-once survives restarts — the same durability the EOS path gets from the broker.
• read_committed lag amplification. A stalled EOS producer that never commits its open transaction blocks read_committed consumers from advancing past the last stable offset, so one slow transactional writer stalls every downstream reader. Bound transaction duration and alert on open-transaction age.
• Hanging transaction on coordinator failover. If the transaction coordinator fails mid-commit, in-doubt transactions block until transaction.timeout.ms elapses. Keep the timeout tight enough that a coordinator failover recovers within the pipeline’s latency budget.

Related #

• Idempotent Trigger Emission Semantics for Geofencing — parent overview of the deterministic keys both paths rely on.
• Deduplicating ENTER/EXIT Events with Idempotency Keys — the consumer-side dedup that makes at-least-once correct.
• Dead-Letter Topics & Poison-Message Handling — where the delivery guarantee determines DLQ and replay behavior.