Real-Time Geofencing Location triggers · low-latency spatial
10 min read 5 sections

Tracing Geofence Triggers with OpenTelemetry

Metrics tell you the P99 moved; a trace tells you which phase moved it. For a geofence pipeline that is a harder ask than for a request-response service, because the causal chain is not a call stack: a position fix enters a queue, is batched with others, is evaluated against an index that was built by a different task, produces a trigger that is emitted to a broker, and is delivered by a consumer minutes later. Nothing about that is a parent-child span relationship, and instrumenting it as one produces traces that are either wrong or enormous. This page sits under production monitoring and observability within Event Routing & Backpressure.

Concept and specification

Three structural decisions determine whether the traces are useful.

Where the trace starts. A trace per position fix is the intuitive choice and produces 25,000 traces/sec of which 99.99% are uninteresting — the fix hit no fence. A trace per trigger is far cheaper and misses the case the operator most wants: the fix that should have produced a trigger and did not. The workable compromise starts a span at the fix, keeps it cheap and unsampled by default, and promotes it when a trigger is produced or when the evaluation exceeds its budget.

How causality crosses the batch boundary. When one span covers a batch of 200 fixes, a parent-child relationship to each fix is wrong — the batch did not cause the fixes. OpenTelemetry’s span links express exactly this: a many-to-one, non-hierarchical association. The batch span links to each contributing fix’s context.

Where sampling decides. Head sampling decides at trace start, before anything interesting has happened, so a 1% head sample keeps 1% of the slow traces. Tail sampling buffers the spans and decides at the end, when the latency is known, and can keep 100% of traces above the budget while dropping the fast majority.

Approach Traces kept Slow traces captured CPU overhead Memory at the collector
No tracing 0% 0% 0.0% 0
Head sampling 1% 1% 1% 4.2% low
Head sampling 100% 100% 100% 31.0% very high
Tail sampling on latency 0.4% 100% 0.9% 1.2 GB buffer
Tail + error + slow-fence rules 0.6% 100% 0.9% 1.3 GB buffer
Traces kept, Slow traces captured, CPU overhead — 5 options Each panel scales on its own, so Traces kept, Slow traces captured, CPU overhead are compared across 5 options without sharing an axis they do not share a unit with. Traces kept, Slow traces captured, CPU overhead — 5 options Approach — the table above, drawn to scale No tracing Head sampling 1% Head sampling 100% Tail sampling on latency Tail + error + slow-fence rules Traces kept 0% 1% 100% 0.4% 0.6% Slow traces captured 0% 1% 100% 100% 100% CPU overhead 0.0% 4.2% 31.0% 0.9% 0.9%
Traces kept, Slow traces captured and 1 more for No tracing, Head sampling 1%, Head sampling 100% and 2 more, drawn from the measurements in the table above. Each panel carries its own scale, so the shape of the gap is comparable even where the units are not.

Tail sampling is better on both axes at once — it keeps every slow trace at a fifth of the CPU cost of a 1% head sample — because the expensive part of head sampling is not the decision but the fact that spans must be created and exported for a fixed fraction regardless of whether they are informative. The cost moves to the collector, which must buffer every trace until it completes; 1.2 GB for a 30 s decision window at this volume.

Step-by-step implementation

1. Instrument phases, not functions. The spans that matter map to the pipeline’s latency budget: ingest, reorder, index lookup, exact containment, debounce, emit. A span per function produces traces with hundreds of spans that no one reads and that cost more to export than to create.

2. Use span links across the batch and the broker boundary.

python 
from __future__ import annotations
from opentelemetry import trace, propagate
from opentelemetry.trace import Link, SpanKind

tracer = trace.get_tracer("geofence.evaluator")

def evaluate_batch(fixes: list, index) -> list:
    # Each fix carries its own context from ingest; the batch LINKS to them
    # rather than parenting them, because the batch did not cause the fixes.
    links = [Link(f.span_context) for f in fixes if f.span_context]
    with tracer.start_as_current_span(
        "evaluate.batch", kind=SpanKind.INTERNAL, links=links[:128]
    ) as span:
        span.set_attribute("geofence.batch.size", len(fixes))
        span.set_attribute("geofence.index.version", index.version)
        out = []
        for f in fixes:
            with tracer.start_as_current_span("evaluate.fix") as fs:
                fs.set_attribute("geofence.candidates", f.candidate_count)
                out.extend(_evaluate(f, index))
        span.set_attribute("geofence.triggers.emitted", len(out))
        return out

def emit_trigger(producer, trigger) -> None:
    with tracer.start_as_current_span("trigger.emit", kind=SpanKind.PRODUCER) as span:
        span.set_attribute("geofence.fence_id", trigger.fence_id)
        span.set_attribute("geofence.transition", trigger.transition)
        headers: dict[str, str] = {}
        propagate.inject(headers)          # W3C traceparent into broker headers
        producer.send(trigger.topic, value=trigger.payload,
                      key=trigger.device_id.encode(),
                      headers=[(k, v.encode()) for k, v in headers.items()])

3. Cap the link count. A batch of 2,000 fixes produces a span with 2,000 links, which is a multi-megabyte export. Cap at a hundred or so and record the true count as an attribute; the links are for navigation, and a sample of them navigates just as well.

4. Propagate through broker headers, never through the payload. W3C traceparent in the message headers keeps the payload stable, which matters because the payload is signed and cached across the fan-out, as described in signing and verifying geofence webhook payloads.

5. Attach the domain attributes that make traces searchable. fence_id, device_id, transition, candidates, index_version, confidence_tier. The last two are what let an operator ask “were all the slow evaluations against the index version we deployed at 14:00” — a question metrics cannot answer at all.

6. Configure tail-sampling policies in the collector, not in the application. Keep everything above the latency budget, everything with an error, a small probabilistic baseline for the healthy case, and — the geofencing-specific one — everything touching a fence on a watch list, so a customer complaint about one zone can be investigated without re-deploying.

Benchmark and verification

Measured at 25k evaluations/sec with a 6-phase span layout:

Configuration Spans/sec Export bandwidth P99 evaluation impact Slow-trace recall
Span per function 410,000 190 MB/s +9.1 ms 100%
Span per phase, head 1% 1,500 0.7 MB/s +0.4 ms 1%
Span per phase, tail sampling 150,000 71 MB/s +0.9 ms 100%
Above + batch span links 32,000 15 MB/s +0.3 ms 100%
Spans/sec, Export bandwidth, Slow-trace recall — 4 options Each panel scales on its own, so Spans/sec, Export bandwidth, Slow-trace recall are compared across 4 options without sharing an axis they do not share a unit with. Spans/sec, Export bandwidth, Slow-trace recall — 4 options Configuration — the table above, drawn to scale Span per function Span per phase, head 1% Span per phase, tail sampling Above + batch span links Spans/sec 410,000 1,500 150,000 32,000 Export bandwidth 190 MB/s 0.7 MB/s 71 MB/s 15 MB/s Slow-trace recall 100% 1% 100% 100%
Spans/sec, Export bandwidth and 1 more for Span per function, Span per phase, head 1%, Span per phase, tail sampling and 1 more, drawn from the measurements in the table above. Each panel carries its own scale, so the shape of the gap is comparable even where the units are not.

The last row is the recommended configuration and it is 13× cheaper than per-phase tail sampling alone, because collapsing 200 per-fix spans into one batch span with links removes the overwhelming majority of span objects while keeping every navigable relationship. The +0.3 ms P99 impact is the honest cost of tracing at this volume, against a 47 ms budget.

The span-per-function row is included because it is what an auto-instrumentation library produces by default, and it costs 9.1 ms of the P99 — a fifth of the entire latency budget — to gather data no one will read.

Verify the traces, not just the telemetry pipeline. Take one known trigger and assert that its trace contains every phase, that the sum of phase durations is within a few percent of the end-to-end span, and that the link to the originating fix resolves. A trace whose phases sum to substantially less than its total has unaccounted time — usually queue wait that nothing is measuring, which is precisely the coordinated-omission blind spot discussed in Core Architecture & Latency Constraints.

Failure modes and edge cases

Failure mode Signature Mitigation
Head sampling on a latency question Slow traces absent exactly when needed Tail-sample on duration at the collector
Batch parented to fixes Trace trees with impossible causality Use span links for many-to-one relationships
Unbounded links per batch Multi-megabyte span exports Cap links; record the true count as an attribute
Trace context in the payload Signature invalidated; payload cache defeated Propagate in broker headers
Span per function 9 ms of P99 spent on unread data Instrument the phases in the latency budget
Collector buffer undersized Tail decisions made on incomplete traces Size the buffer for the decision window times trace rate
Failure mode at a glance: Signature, Mitigation A row per failure mode, a column per option, so a single axis can be compared across options in one sweep. Failure mode at a glance: Signature, Mitigation the same trade-offs, read across instead of down Signature Mitigation Head sampling on a latency question Slow traces absent exactly when needed Tail-sample on duration at the collector Batch parented to fixes Trace trees with impossible causality Use span links for many-to-one relationships Unbounded links per batch Multi-megabyte span exports Cap links; record the true count as an attribute Trace context in the payload Signature invalidated; payload cache defeated Propagate in broker headers Span per function 9 ms of P99 spent on unread data Instrument the phases in the latency budget Collector buffer undersized Tail decisions made on incomplete traces Size the buffer for the decision window times trace rate
The failure mode table read across rather than down — each row is one axis of the decision, colour-keyed so a single trade-off can be followed across Signature, Mitigation.

The last row produces a failure that looks like a tracing bug and is a capacity problem. Tail sampling requires the collector to hold every span of a trace until it decides, so the buffer must cover the longest trace the pipeline produces — and a geofence trace that spans an emit, a broker hop and a webhook retry sequence can last minutes, far longer than the default 30 s decision window. Traces that exceed it are decided on their prefix, which usually means the interesting part is discarded. Either raise the window for the topics that need it or split the trace deliberately at the broker boundary, linking rather than continuing.

Finally, a note on cardinality. device_id as a span attribute is fine — spans are sampled and stored individually — but the same attribute on a metric is catastrophic, producing one time series per device. The rule that keeps the two apart is that traces carry high-cardinality identity and metrics carry low-cardinality dimensions, with the trace exemplar as the bridge between them: attach a trace id to a latency histogram bucket and an operator can jump from “the P99 bucket got heavier” to an actual slow trace in one click, which is the single highest-value integration between the two systems and is covered from the metrics side in Prometheus metrics for queue depth and P99 latency.