Spatial Index Persistence & Warm Start After Restart

A geofence evaluation node that has just restarted is the most dangerous version of itself. Its latency graphs are clean — an index holding 4,000 of its eventual 1.2 million polygons answers queries very quickly — and its answers are wrong in a way no metric on the dashboard reports, because a fence that has not been loaded cannot be missed. Every deploy, every crash loop, every autoscaling event opens a window in which the service is confidently incorrect. This page expands the index-lifecycle concerns raised in Spatial Indexing for Real-Time Checks, and the failure it addresses is silent incompleteness during warm-up.

The reader is running a service whose incident history contains a recurring, unexplained burst of missed triggers immediately after each deploy. The fix has two halves that must be built together: making startup fast enough that the window is small, and making the window visible so that traffic never enters it. The second half matters more than the first. A node that takes 94 seconds to load but refuses traffic until it is complete is correct and slow; a node that takes 2.4 seconds and serves from the first millisecond is fast and wrong.

What Cold Start Actually Costs #

The rebuild path is dominated by parsing and by geometry construction, not by the index structure itself. Measured on 1.2 million polygons averaging 96 vertices, loaded from Postgres/PostGIS on a node with 8 cores:

The 39× is real but it is not the headline. The headline is the row that costs 3.1 s in one column and 0.3 s in the other and does not exist at all in most implementations: verify completeness. Without it, both columns are unsafe, and with it, the slow column is merely slow.

Two secondary costs deserve naming. Resident memory during a cold rebuild peaks well above the steady state — 11.2 GB against a 6.8 GB steady state on this fleet — because the parsed geometry objects and the index being built from them are alive simultaneously, so a node sized for its steady state gets OOM-killed during startup and enters a crash loop that looks like a memory leak. And a fleet that restarts together, as it does during a rolling deploy, multiplies the 31 s fetch by the number of nodes against a single database, which is how a deploy becomes a database incident.

Snapshot Format: Trading Portability for Load Time #

The reason a snapshot restores in 0.4 s and a rebuild takes 60 is that the snapshot skips object construction. A serialisation format that still has to allocate a Python object per polygon has given up most of the benefit before it starts. The formats worth considering separate cleanly on that axis:

The memory-mapped packed file wins by two orders of magnitude because it does nothing: the file is the index, laid out as the structure expects to find it, and mmap merely arranges for pages to be faulted in on demand. The cost is that it is not portable — a change to the node layout, the coordinate encoding, or even the pointer width invalidates every existing snapshot — which makes a format version stamp mandatory and a rebuild-on-mismatch path non-optional. A node that silently mmaps a snapshot written by a previous release is reading someone else’s memory layout, and the failure mode is a wrong answer rather than a crash.

The columnar option is the pragmatic middle: 14× faster than a rebuild, portable across versions, and it keeps geometry in flat arrays that the exact-containment kernel can consume directly, which composes with the array-backed vertex storage discussed in memory footprint of streaming polygon indexes. For most services it is the right default, with the mmap format reserved for fleets large enough that six seconds of startup per node is a scheduling problem. The mechanics of writing and validating the packed form are in snapshotting R-tree state for fast restarts.

Where the snapshot lives matters as much as its format. A snapshot on the node’s local disk restores fastest and is lost when the node is replaced, which is exactly when it is needed. A snapshot in object storage survives node replacement and costs a download — 2.9 GB at 300 MB/s is 9.7 s, which puts it back in the same order as a columnar rebuild. The arrangement that works is both: object storage as the durable copy, local disk as a cache keyed by snapshot version, so a restarting node hits local disk and a newly scheduled one pays the download once.

Catch-Up: The Window Between Snapshot and Now #

Every snapshot is stale by construction. The gap is closed by replaying the fence change stream from the snapshot’s watermark, which requires the snapshot to record that watermark atomically with its contents — a snapshot whose watermark is written separately can claim to include edits it does not, and the resulting missing fence is undetectable until someone complains.

$$A_{\text{ready}}=t_{\text{restore}}+\frac{{\lambda }_{\text{edits}}\cdot A_{\text{snapshot}}}{r_{\text{apply}}}$$

Catch-up time grows with snapshot age and with edit rate, and shrinks with how fast edits can be applied. On a fleet taking 40 fence edits/sec with an apply rate of 1,400/sec, a one-hour-old snapshot needs 103 s of catch-up — longer than the cold rebuild it was supposed to replace. That is the trap in snapshot-based warm start: the snapshot is only useful while it is fresh, so the snapshot cadence is part of the design and not an operational detail. At the 1.7 s catch-up in the table above, the snapshot was 60 s old.

Two refinements keep the cadence affordable. Incremental snapshots write only the pages that changed since the last full one, so a 60 s cadence costs megabytes rather than gigabytes; and a compaction pass merges increments into a new full snapshot on a slower schedule. The hydration path that consumes both, and the ordering guarantees it needs from the change stream, are covered in incremental index hydration from change streams.

Catch-up must also be idempotent, because the change stream will redeliver. An edit applied twice must leave the index in the same state as an edit applied once, which for an upsert keyed by fence id it does, and for a delete-then-insert pair it does not unless ordering is preserved per fence. This is the same guarantee the trigger path needs from idempotent trigger emission semantics, applied to state rather than to effects.

Readiness, Liveness, and the Rule That Actually Matters #

The technical work above shortens the window. Only the readiness rule closes it.

from __future__ import annotations
from dataclasses import dataclass

@dataclass(slots=True)
class LoadState:
    expected_fences: int | None = None      # from the source of truth
    loaded_fences: int = 0
    expected_checksum: str | None = None
    loaded_checksum: str | None = None
    catchup_watermark_ns: int = 0
    source_watermark_ns: int = 0
    phase: str = "starting"

def is_ready(st: LoadState, max_lag_ns: int = 5_000_000_000) -> tuple[bool, str]:
    """Readiness is a statement about COMPLETENESS, not about liveness."""
    if st.expected_fences is None:
        return False, "authoritative count not yet fetched"
    if st.loaded_fences != st.expected_fences:
        return False, (f"index incomplete: {st.loaded_fences}/{st.expected_fences} "
                       f"({st.loaded_fences / st.expected_fences:.1%})")
    if st.expected_checksum and st.loaded_checksum != st.expected_checksum:
        return False, "checksum mismatch: snapshot does not match the source"
    lag = st.source_watermark_ns - st.catchup_watermark_ns
    if lag > max_lag_ns:
        return False, f"change-stream catch-up lagging by {lag / 1e9:.1f}s"
    return True, "ready"

def is_alive(st: LoadState) -> bool:
    """Liveness must NOT depend on readiness, or a slow load is restarted
    forever into a crash loop that looks like a memory problem."""
    return st.phase != "failed"

Separating the two probes is the detail that turns this from a good idea into one that survives contact with an orchestrator. If readiness and liveness are the same endpoint, a node that legitimately needs 94 s to load is killed at the liveness timeout, restarts, and loads again — a crash loop caused entirely by the safety mechanism. Keep liveness a statement about the process and readiness a statement about the data, and give the readiness probe a generous failure threshold and the liveness probe a short one.

The completeness comparison needs an authoritative count that is cheap to obtain and consistent with the snapshot. A SELECT count(*) against a table being edited concurrently is neither, so the workable pattern is a maintained counter row updated in the same transaction as any fence edit, read once at startup along with the watermark. The verification pass that turns that count into a real integrity guarantee — geometry validity, index-to-geometry referential consistency, no orphaned nodes — is developed in validating index integrity after restore.

Memory Behaviour During Restore #

An mmap-backed index has a memory profile that surprises operators the first time they see it. Resident memory after restore is near zero and climbs as queries fault pages in, reaching a plateau at roughly the working set rather than the file size — 2.9 GB on disk, 1.1 GB resident after ten minutes of production traffic, because most fences are never queried in any given hour. That is a genuine advantage: the node holds only the fences its traffic touches. It also means the first queries against cold pages are slow, at roughly 60 µs for a major fault against 1.5 µs for a resident lookup, so a node that passes readiness still has a latency ramp for its first minute or two.

The mitigation is to warm deliberately rather than to let production traffic do it. madvise(MADV_WILLNEED) over the index’s internal nodes — a few percent of the file — pulls the branch structure into memory without the leaves, which is where nearly all of the fault cost is, and cut the post-ready P99 ramp on this fleet from 140 ms to 9 ms. Faulting the whole file defeats the point and costs the full 2.9 GB.

For the non-mmap formats the profile is the ordinary one, and the peak is the number to size against: during a columnar restore both the Arrow buffers and the constructed index are live, peaking at about 1.6× steady state. The pooling and __slots__ discipline from memory-constrained spatial processing applies here too, with the difference that startup is a burst rather than a steady state, so the goal is to keep the peak under the limit rather than to keep churn low.

Operational Runbook #

1. Export index_load_phase and index_completeness from the first second. A node that cannot say what fraction of its fence set it holds cannot be diagnosed during the only window in which the diagnosis matters.
2. Verify that readiness actually gates traffic. Deploy one node with an artificially slow load and confirm it receives zero requests until complete. This is the test that most often reveals a load balancer configured to route on liveness.
3. Alert on snapshot age, not on snapshot success. A snapshot job that succeeds every hour while the catch-up budget assumes 60 s is a latent 100 s startup regression that nothing will report until the next restart.
4. Track the peak-to-steady memory ratio. If a cold rebuild peaks above the container limit, the symptom is a crash loop under memory pressure, which is invariably misdiagnosed as a leak. Size the limit against the peak or stream the rebuild in chunks.
5. Rehearse the snapshot-invalid path. Corrupt a snapshot deliberately in staging and confirm the node falls back to a rebuild rather than serving from garbage. A format version stamp with no fallback path is a slower way to serve wrong answers.
6. Stagger restarts across the fleet. A rolling deploy with a fully parallel restart multiplies the source-of-truth load by node count. Cap concurrent warm-ups, and prefer restoring from object storage over re-querying the database.

Architectural Guidance #

Use a columnar snapshot plus change-stream catch-up as the default. It is portable across releases, an order of magnitude faster than rebuilding, and its failure mode when a snapshot is unusable is a slow start rather than a wrong one.

Use a memory-mapped packed file when the fleet is large enough that per-node startup time is a scheduling constraint, and accept the obligations that come with it: a version stamp, a rebuild fallback, and a deliberate warm of the internal nodes.

Rebuild from the source of truth, with no snapshot at all, when the fence set is small enough that the rebuild fits inside the orchestrator’s startup budget — under roughly 100,000 polygons on modern hardware. The operational simplicity of having exactly one load path is worth more than the seconds a snapshot would save.

Whatever the path, gate readiness on completeness. It is the cheapest item in this page and the only one that changes the failure mode from “wrong answers, healthy metrics” to “no answers, obvious metrics” — and every other technique here is an optimisation of a window that this rule makes safe.

FAQ #

Related #

• Spatial Indexing for Real-Time Checks — the parent section, where index freshness is framed as a correctness budget.
• Snapshotting R-Tree State for Fast Restarts — writing and version-stamping the packed page file.
• Incremental Index Hydration from Change Streams — replaying edits from a watermark, idempotently and in order.
• Validating Index Integrity After Restore — turning a count comparison into a real integrity guarantee.
• Optimizing R-Tree Bulk Loads for Real-Time Ingestion — the build path a cold start has to run and a snapshot skips.