Classifying Poison Messages with Error Taxonomies

A dead-letter topic accumulates records for reasons that have nothing in common — a malformed payload, a downstream that was briefly unreachable, a fence id that no longer exists, a schema field the consumer has not learned yet. Storing them all with reason: "processing failed" makes the topic a landfill: nothing in it can be replayed safely because nothing in it can be distinguished, so operators either replay everything and re-poison the pipeline or replay nothing and lose the recoverable records. This page sits under dead-letter topics and poison-message handling within Event Routing & Backpressure.

Concept and specification #

The taxonomy has to answer one operational question — what should happen to this record next — so its classes are defined by remedy rather than by cause. Four classes cover the observed failure population:

The share column is the argument for the whole exercise. Nearly two-thirds of a typical dead-letter topic is transient — a broker rebalance, a downstream restart, a circuit breaker that was open — and those records are perfectly replayable the moment the environment recovers. Without a class they sit alongside genuinely poisonous records and cannot be drained without risk.

STALE deserves particular attention because it is the class most implementations lack. A geofence ENTER for a shift that ended six hours ago is not a failure to be retried; delivering it late is worse than not delivering it, because a downstream that acts on it will dispatch a vehicle to a job that no longer exists. Staleness is a function of the trigger’s own semantics, and the classifier is the only component positioned to apply it.

$$\text{class}=f(\text{error},\text{attempts},\text{age},\text{schema validity})$$

Note the inputs: the exception alone is not enough. The same TimeoutError is TRANSIENT on its first attempt and, after five attempts spanning an hour, STALE. The same schema error is POISON from a known producer and UNKNOWN from a producer mid-deployment.

Step-by-step implementation #

1. Classify at the point of failure, where the exception is still in hand. A classifier that runs later, over a serialised record, has lost the stack, the response body and the attempt count. Attach the classification as the record enters the dead-letter path.

from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
import time

class Cls(str, Enum):
    TRANSIENT = "transient"
    POISON = "poison"
    STALE = "stale"
    UNKNOWN = "unknown"

@dataclass(slots=True, frozen=True)
class Verdict:
    cls: Cls
    reason: str
    retry_after_s: float | None

def classify(exc: BaseException | None, status: int | None, attempts: int,
             event_age_s: float, schema_ok: bool,
             usefulness_horizon_s: float = 900.0) -> Verdict:
    # Staleness first: a valid record that has aged out is not retryable at all.
    if event_age_s > usefulness_horizon_s:
        return Verdict(Cls.STALE, "past the usefulness horizon", None)
    if not schema_ok:
        return Verdict(Cls.POISON, "schema validation failed", None)
    if status is not None:
        if 400 <= status < 500 and status not in (408, 425, 429):
            return Verdict(Cls.POISON, f"client error {status}", None)
        if status in (408, 425, 429) or status >= 500:
            return Verdict(Cls.TRANSIENT, f"server/backoff {status}",
                           min(60.0, 2.0 ** attempts))
    if isinstance(exc, (TimeoutError, ConnectionError, OSError)):
        return Verdict(Cls.TRANSIENT, type(exc).__name__, min(60.0, 2.0 ** attempts))
    if isinstance(exc, (ValueError, KeyError, TypeError)):
        return Verdict(Cls.POISON, type(exc).__name__, None)
    return Verdict(Cls.UNKNOWN, type(exc).__name__ if exc else "no exception", None)

2. Order the checks by remedy, not by likelihood. Staleness is checked first because a stale record must not be retried even if the failure was transient — reversing the order produces a replay of records that should have been archived.

3. Treat 4xx as poison, with three exceptions. 408, 425 and 429 are the client-error codes that mean “try again”, and treating them as poison quarantines records the endpoint explicitly asked to receive later. Getting this wrong is the most common single classification bug.

4. Partition the dead-letter topic by class. One topic per class — or one topic with the class as the partition key — means a drain can consume TRANSIENT without touching POISON, which is what makes automated replay safe. A single topic filtered at read time works but forces every drain to scan records it cannot use.

5. Carry the full context, not just the class. Original topic and offset, attempt count, event time, the classification and its reason, and a truncated response body. The reason string is what a human reads at 3 a.m.; the class is what the machine reads.

6. Alert on class shape, not on volume. A dead-letter topic filling with TRANSIENT during a known downstream outage is the system working. The same volume of POISON is a producer regression, and UNKNOWN above a few percent means the taxonomy has fallen behind the failure population.

Benchmark and verification #

Measured over 90 days on a platform emitting 25k triggers/sec, with 2.1 million dead-lettered records:

The last column is the one that costs money. Records lost to retention expiry are triggers that were recoverable and were never recovered, because nobody could safely drain a topic they could not classify — 141,000 of them over the period under a single reason string, against 900 with the full taxonomy. The re-poisoning column is the other side: replaying an unclassified topic re-delivers the genuinely poisonous 12% straight back into the consumer, which is how a drain becomes an incident.

The 6% that remains un-routable is dominated by UNKNOWN, and that fraction is the taxonomy’s own health metric. Sampling it weekly and adding rules for whatever recurs is the maintenance loop; a taxonomy that is never extended drifts back toward a single reason string as the failure population changes.

Verify by replaying each class into a staging consumer and asserting the expected outcome: TRANSIENT should succeed once the environment is healthy, POISON should fail identically and be re-quarantined against its original attempt ceiling, STALE should never be replayed at all, and UNKNOWN should reach a human. A test that only replays TRANSIENT proves the easy case.

Failure modes and edge cases #

The fourth row connects directly to the replay discipline in the parent topic and is worth stating once more because the two mechanisms have to agree: the classifier decides whether a record may be replayed, and the attempt ceiling decides how many times. A replay path that resets the counter converts a correctly classified poison record into an infinite loop, and a classifier that forgets the count cannot distinguish a first transient failure from a fifth.

One structural note. The taxonomy’s classes are about remedy, so they should be stable even as the underlying errors change — a new downstream returning a new status code should map into an existing class rather than creating a fifth. If a genuinely new remedy appears, adding a class is a schema change for every consumer of the dead-letter topic, which is why the four above are worth getting right early. The one extension most platforms eventually need is a REDACT class for records that must be deleted rather than archived, driven by data-protection requirements rather than by delivery semantics.

Related #

• Dead-Letter Topics & Poison-Message Handling — the parent topic and the quarantine mechanics.
• Replaying Dead-Letter Triggers Safely — the paced drain that consumes these classes.
• Exactly-Once vs At-Least-Once Trigger Delivery — why a replayed record must re-derive its original key.