Cold Start Is a First-Class Constraint

Abstract. We report a total production outage caused by a change that passed every test, every review, and a full staging deployment. A read-only index grew past the memory available at process start; because the index was constructed at module load, the failure was not a slow directory but a dark site — every unrelated route died with it. We analyze why standard practice failed to catch it (staging is warm, tests are warm, health checks run after the boot that never completed), derive four design principles for serving large read-mostly indices on autoscaled runtimes, describe the construction that replaced it, and report the deploy gate whose absence was the actual defect. The engineering is not novel. The failure mode is underdocumented, and we think that gap is worth an honest paper.

1. Introduction

There is a class of bug that survives good engineering practice because the practice tests the wrong state. This is a report on one of them, written from a production outage we caused ourselves.

The system is a consumer web application serving a proximity-searchable directory of local professionals. The workload has a shape that is extremely common and rarely discussed as a distinct problem: a large, read-only, rarely-changing dataset that must be queried with low latency, deployed onto an autoscaled runtime that starts processes on demand and reclaims them when idle. Almost every modern serverless or container-per- request platform has this shape.

We grew the dataset. Everything passed. Production went completely dark — not degraded, dark — and stayed that way until we rolled back.

The engineering that fixed it is unremarkable, and we make no claim of novelty. What we think is worth publishing is the failure geometry: why a memory limit interacted with a module- load initialization to convert a bounded feature risk into a total availability failure, and why a mature deployment pipeline had no stage that could observe it. We have found this combination widely reproducible in the architectures we have looked at since, and almost entirely absent from the guidance those architectures ship with.

2. The setting

The dataset at the time of the incident: 248,568 organization records and, in the planned expansion, 616,145 individual professional records, each carrying an identifier, a name, a coordinate pair, a classification, and provenance. It is authored offline, shipped as a build artifact, and never mutated at runtime.

Queries are geographic: given a coordinate and a radius, return the nearest matching records. This favors an in-memory representation, and the naive implementation is the obvious one — parse the data at module load, build an index, keep it resident, serve from it. It is fast, simple, and it is what we had.

The runtime allocates a fixed memory ceiling per instance, starts new instances on demand, and terminates idle ones. Crucially, every instance pays full initialization cost, and initialization happens before the instance can answer anything.

3. Anatomy of the failure

Four properties combined. Each is individually defensible; together they are a total outage.

The index built at module load. Constructing it was a side effect of importing the module, so it ran during process startup, not on first query.

Unrelated routes transitively imported it. A shared module graph meant the home page, the consent pages, and the marketing site pulled in the directory module even though they never queried it. This is the property that turned a feature failure into a site failure, and it is invisible in any per-feature test.

Peak memory during construction exceeded steady-state. Parsing produced intermediate structures alongside the final index. The transient peak, not the resident size, is what met the ceiling.

The ceiling was set for the old dataset. One gibibyte, adequate for the previous size, with no headroom check tied to growth.

  instance start
      │
      ├─ import route module
      │     └─ import shared graph  ──► directory module
      │                                     │
      │                                 build index at module load
      │                                     │
      │                                 peak = index + parse intermediates
      │                                     │
      │                                     ✗  exceeds ceiling → process dies
      │
      └─ (never reached) health check, request handling

  RESULT: every route dark, including routes that never query the directory.

The observable behavior was pathological in a way worth recording: the platform dutifully restarted the instance, which dutifully died again during construction. The system was not serving errors from a running process; it was failing to have a running process at all.

4. Why standard practice did not catch it

This is the part we find most generalizable, because our pipeline was not negligent.

Unit and integration tests run warm.A test harness imports modules once into a long-lived process with the developer machine's memory. It exercises query correctness beautifully and can never observe a constrained cold start. Our tests passed because they were testing a different thing than production does.

The build succeeding proves nothing about boot. Compilation and bundling completed normally. A build artifact that cannot start is still a valid build artifact.

Staging was deployed but not observed under the failing condition. This is the one that matters most, and the honest version is uncomfortable: we deployed to staging and moved on. Nothing in the pipeline required a human or a machine to confirm that the staged build had actually booted and answered real requests. The deployment was treated as evidence, when only a response is evidence.

Health checks are scheduled after startup. A liveness probe cannot report on a process that dies before the probe is registered. The instrument was downstream of the failure.

The lesson we draw is that a pipeline can be thorough and still contain a state that nothing in it ever enters. Cold start under production memory limits was such a state for us.

5. Design principles

P1. Separate the always-resident from the on-demand. Split the dataset by access probability, not by schema. A small index that every query needs stays resident; the large body that only some queries need loads lazily. The resident portion must be small enough that its construction is uninteresting at any plausible scale.

P2. Let the query's own locality define the lazy boundary. The right shard key is whatever the query already narrows by. For geographic search that is geography: a proximity query touches a bounded neighborhood, so partitioning by postal region means a query loads a handful of shards rather than the corpus. The partition is free because the query was already going to do it.

P3. Encoding is a memory decision, not a taste decision. Representation choice changes peak memory by integer factors. A record-per-object encoding pays per-record key overhead; a columnar or tuple encoding with a shared header does not. This also has compile-time consequences on typed toolchains, where large object literals can make static analysis pathologically slow — a separate problem with the same root cause.

P4. Nothing generic may depend on something heavy. This is the principle that converts a bounded failure into an unbounded one when violated. If a marketing page transitively imports a heavy index, the index's failure is the marketing page's failure. Treat the module graph as a blast-radius diagram.

6. The construction

The replacement follows the principles directly and is, deliberately, boring.

A resident national index of organizations is built at startup. It is the small, universally-needed portion, and it is now iterated in place from its serialized form rather than materialized into an intermediate array first — which removes the transient peak that originally met the ceiling.

Individual records are sharded by postal region and loaded lazily. A proximity query resolves the nearest regions to the query point and dynamically imports only those — on the order of a dozen shards out of roughly a thousand. Loaded shards are cached, so the steady state converges to the working set of regions people actually search rather than the whole corpus.

Records are stored as tuple arrays with a shared column header rather than as arrays of objects, which removes per-record key storage and, on our typed toolchain, removes a static-analysis cost that had become prohibitive.

The memory ceiling was raised to two gibibytes — necessary but explicitly not sufficient, and we want to be precise about that, because raising the limit is the tempting fix that leaves the actual defect in place. It buys headroom. It does not address module-load coupling, and a system fixed only this way fails again at the next growth step.

  instance start
      │
      ├─ build resident org index (iterated in place, no intermediates)
      │
      └─ ✓ ready — serves every route

  on a proximity query:
      resolve nearest regions → dynamic import ~12 shards → cache
      (never the full individual corpus)

7. The gate

The code changes above fix this instance. The process change is what fixes the class, and we consider it the real result of the incident.

No build is promoted to production until that exact build has booted on the staging environment and answered real requests across every critical route, including a genuine query against the data path, not merely a static page fetch.

Three properties matter. It gates on the same artifact, since a rebuild is a different experiment. It requires responses, not a successful deploy call, because a deployment that returns success while every instance crashes is exactly what we experienced. And it exercises the data path, because the routes that break are the ones that touch the heavy thing.

We want to state the causal claim carefully: the memory ceiling was the proximate cause, and the missing gate was the defect. Every other control we had was working as designed and was structurally incapable of observing the failure. The bug was the absence of an observation, not the presence of a mistake.

8. Evaluation

We report what we actually measured, which is less than a proper evaluation would include.

Before: total unavailability across all routes at the expanded dataset, with instances in a restart loop. After the construction and ceiling change: the full corpus of 248,568 organizations plus 616,145 individuals serves normally, with all critical routes returning successfully and live proximity queries returning correct results — verified on staging first and then on production as the gate now requires.

We do not report a rigorous latency comparison, and we should be honest that this is a real limitation of the paper. Lazy shard loading trades a first-query cost within a region for a large reduction in startup cost, and the shape of that trade under realistic traffic is exactly the thing a proper evaluation would characterize. We have not yet done that measurement, and it would be dishonest to present the qualitative improvement as if it were quantified.

9. Generalization and limitations

The pattern applies wherever a large read-mostly dataset meets a runtime that starts processes on demand: geographic directories, catalogs, embedding tables, lookup dictionaries, feature stores. The failure requires only that construction happen at import and that something generic import it.

It does not apply to long-lived, manually-provisioned servers where startup happens once under supervision — the traditional deployment model is, in this specific respect, safer than the modern one, which is a genuinely uncomfortable observation.

Two limitations we will name. First, lazy loading moves cost to the first query in a cold region, so a workload with no locality — uniformly random access across all shards — degrades toward the original problem, and we have not characterized where that boundary sits. Second, the gate adds wall-clock time to every promotion. We consider that trade obviously correct, but it is a real cost and teams under release pressure will be tempted to skip it, which is precisely how the original defect came to exist.

10. Related work

Cold-start latency in serverless platforms is well studied, though that literature is largely concerned with startup latency as a performance property rather than startup memory as a correctness property. Work on lazy and memory-mapped index loading in databases and search engines addresses the same tension with more sophistication than we needed. Columnar and tuple encodings are standard in analytical systems. The observation that module-load side effects create hidden coupling is old and well understood, and none of our fixes would surprise a systems engineer.

The contribution here is not technique. It is a documented case in which the combination reached production through a competent pipeline, and an explicit account of which controls were structurally unable to catch it. Post-incident analyses of this kind are rarer in public than the techniques are, and we think the asymmetry is a problem for the field.

11. On publishing this at all

The straightforward argument against writing this up is that it describes us breaking our own site through an avoidable oversight.

We publish it because the alternative is worse. A field in which everyone quietly patches this class of failure and publishes only architecture diagrams is a field where each team pays the full cost of learning it. The specific numbers here — the dataset size, the ceiling, the fact that the marketing page died because of a directory index — are the useful part, and they are exactly the parts that get sanded off when an incident becomes a case study.

Code is the truth. That has to include the code that failed.