Everything That Broke Was Already Written

The night, in the order it happened

On 6 August we set out to make one page work: the page where a person says what they need and gets back somebody nearby who can help. By morning we had found and fixed six defects. Not one of them was a wrong algorithm. Every single one was correct code that was connected to nothing, or connected to the wrong thing.

Here is the list, because the pattern only becomes obvious when you see them together.

One. The directory computed a ranking and threw it away. A function called orderForHumans de-duplicates results, demotes shell companies, and lifts named humans to the top. It was correct. It had tests. It ran on every request. Then three separate blocks of code merged in more results and re-sorted the whole list by distance. Inside one postal code every record is placed at the same point, so that re-sort was a sort on a total tie, which returns the original order, which is the alphabet. A person searching Stamford for a plumber got two limited liability companies above two named humans, and the ranking we had carefully written was discarded two lines after it ran.

Two. The parser was never plugged in. intent.ts reads a sentence like "the furnace is out and it's freezing" and returns a heating contractor. Two hundred and sixty lines, a hundred and twelve lines of tests, green on every commit for weeks. It was imported by nothing. It had shipped to production dozens of times and no human being had ever benefited from one line of it, while the page above it asked "what do you need done?" and offered nowhere to answer.

Three. The parser could not hear the half of the mission that matters most. It had no rules at all for community organisations, so 369,447 food banks, shelters, legal aid offices and bill assistance programmes were unreachable by any sentence a person might type. "I can't afford my rent" fell through every rule. Worse, "I can't pay my electric bill" contains the word electric, so the first rule that would have matched it was the one that returns electricians.

Four. The verify button pointed at us. Every public record listing carried a button reading "Verify at the regulator". It linked to our own claim form, which asks the reader whether they are the person they were trying to check. In Connecticut, where only 24% of these listings publish a phone number, that button was most of the page.

Five. The candidate pool was alphabetical before the ranking ever ran. Our loaders stopped collecting at three times the page size, in the order the source file lists people. So "rank before truncating", which we had just fixed, was ranking the best of the first thirty names in the alphabet. The plumber at Superior Plumbing never reached the ranker.

Six. We ranked by usefulness and broke the headings. Results render under distance bands. Sorting globally by usefulness interleaved them, so a correct answer came back reading "within 5 miles, within 15 miles, in your postcode, within 5 miles". Every row right, the order looking broken, under a footer still claiming the list was ordered by distance.

That last one was ours, introduced while fixing the first one, and it was caught by looking at a screenshot. No test caught it. That is not incidental. It is the point.

The pattern, stated precisely

Count the defects by category.

Wrong computation: zero.

Wrong composition: six.

Every failure was at a seam. A function that runs and whose output is discarded. A module that exists and is imported by nobody. A parser whose vocabulary does not cover a corpus that is sitting right there. A link whose destination is technically a URL and semantically a lie. A cap applied before the sort instead of after. A sort key that is correct in isolation and wrong next to the thing that renders it.

Now count the tests. At the time of writing we have 277 test files carrying 2,219 unit tests, and 21 end to end specs carrying 164 cases. That is a ratio of about thirteen to one in favour of testing units.

Those 2,219 tests caught none of the six.

They could not have. Each one was passing, correctly, about a module that was correct. orderForHumans had tests and they were green while its output was being thrown away. intent.ts had tests and they were green while it was reachable by nobody. Coverage was fine. The type checker was fine, because an unused export is not an error and a discarded return value is not an error. The production build was fine, because dead code compiles.

Every signal we had was green about features that did not exist for users.

The classical test pyramid says: many unit tests, fewer integration tests, fewest end to end tests, because the ones at the top are slow and brittle. That advice is calibrated for a world in which writing a correct module is the expensive part. We do not live in that world any more. In this codebase, 649 of 1,477 commits were written by an agent. Writing a correct module has become close to free. What has not become free, and what almost nothing in our toolchain measures, is whether that module is connected to the product.

The pyramid is upside down for the failure mode we actually have.

What we did about it, the same night

Three things, in increasing order of how much they will matter in a year.

We wrote the tests that would have caught it. Not more unit tests. Assertions about what a person receives: type a sentence, get people who can help; ask for a plumber, do not get a holding company; press verify, leave our domain. Two new Playwright specs, 38 cases, plus 15 API level cases asserting the shape of the answer rather than the shape of the code. Three of them were written as deliberate failures, pinned red, so that fixing the merge path would turn them green and force somebody to come back and convert them into real assertions. A skipped test rots quietly. A pinned failing test does not.

We built a scorecard that grades the deployed product, not the repository. scripts/mvp/scorecard.mjs walks the product as five synthetic people with real problems, checks that the answers are honest, that the doors go somewhere real, that the internal gates fail closed, and prints one weighted number. Before that night's release, the live site scored 64.6. After it, 97.9. It is in the open repository and anybody can run it against our production host. The weights say what a failure costs a person: honesty and safety count triple, journey steps double, trust surfaces single.

We counted the orphans. If intent.ts was written, tested and never wired, how many others are there? We wrote a census and ran it. The answer is 59 modules in `src/` that no other module imports. Some are legitimate, framework conventions and configuration targets, and the check records a baseline rather than pretending all 59 are bugs. But in that list are src/lib/gtm/icp.ts, src/lib/gtm/fit.ts, src/lib/gtm/network-metrics.ts, src/lib/network/circles.ts and src/lib/directory/agent-ontology.ts. Real logic, written with care, that no user can reach. The gate now fails when the count goes up, and the only way to lower it is to wire something up or delete it.

That last one is the honest measure of the problem. We did not have one orphan. We had fifty nine, and we did not know.

What went well, said without flattering ourselves

The recovery was fast because the foundations were real. Ranking, intent parsing, the consent rules and the hiring state machine are all pure functions with no framework in them, so once we found where the wiring was wrong we could fix it without touching anything else. Small, self contained modules did exactly what they promise: they made the blast radius of each fix tiny.

The guards that exist because something already went wrong all did their jobs. Brand, emoji, copy standard, directory integrity, environment parity, secret safety, gate parity. The gate parity guard, which diffs the local gate against the CI gate, caught me wiring a new check into one and not the other within seconds of my doing it.

And the deploy pipeline behaved exactly as designed under pressure: UAT deploy, eleven check smoke, production promote, forty two assertion production smoke, record the known good commit. Exit code zero means shipped and verified, and it meant it.

What went badly, including twice by my own hand

The honest failures are the useful part.

We announced surfaces as working before confirming they worked. A support email rail was described as live and had never sent a message. A job application form told candidates it had sent applications it had not sent, was fixed, and the same defect was found in a second form the next day. Forty seven consent scopes were published while the server implemented eight, and the other thirty nine returned a clean success and did nothing. Every one of those claims was true when somebody wrote it, or about to be true. Nothing in the build compared the claim to the behaviour.

Our public release page went 59 commits stale, so an entire day of shipped work was missing from the record while we described ourselves as a company that builds in the open. The generator that prevents this had a --check mode from the day it was written and nothing ever ran it. A guard nobody invokes is a comment.

I wrote a guard that would have deadlocked the next deploy. Having found the stale release page, I added a pre-flight that blocks a ship when the notes are behind. Release notes are written and committed before the merge that publishes them, so the merge commit is always missing from the file it just added. Regenerating to fix that creates another commit that is also missing. Forever. I caught it by reading the sequence rather than by hitting it at two in the morning, and the fix is a tolerance of three with the limitation written into the comment.

Then the second version of the same guard blocked a UAT deploy it had no business blocking. Release notes are what the public reads when a release lands, so the moment they must be current is the production promote. A UAT deploy exists to put an unmerged branch somewhere real so a human can look at it, and its commits are by definition not on the main branch yet. My guard was failing the exact workflow it was written to protect. Twice in one night, the same lesson: a control that does not model the workflow it governs will be routed around, and deserves to be.

And a process failure worth recording. With two agents writing in the same checkout, a commit that ran git add -A swept in both of their in-progress files, including a scratch test that was never meant to ship. Nothing was lost and the content was byte identical to what its authors verified, but the commit claimed less than it contained. When agents share a working tree, commits name their paths.

How Apple and Google would attack this

Two different instincts, and we need both.

Apple would put a person in front of the product and watch. Not a metric, not a dashboard, a person with a problem. Three of our six defects were found by looking at the page: the ranking, the box below the fold, the shuffled headings. None by a test. Apple's discipline is that the demo is the specification, and somebody senior walks the entire path before anybody says it is done. The version of that we can automate is a screenshot and a scorecard on every release, and a human who actually opens the page. We now do both, and I would say plainly that the screenshot found more that night than any test did.

Google would make the seam a measured object with an owner. At the scale Jeff Dean's teams operate, the interesting failures were never inside a function either. They were at boundaries: a cache that answers stale, a fan out where one shard is slow, a fallback that silently degrades. The answers are cultural as much as technical. Assume every dependency will fail and make the failure loud. Never let a component return a plausible wrong answer where an error would do. Put an indicator on the thing users experience rather than on the thing engineers wrote. Our version: no silent success anywhere, an assertion on the answer a person receives, and a number for how good that answer is.

Both instincts point the same way. Stop grading the parts. Grade the whole.

The plan of record

Five commitments, each with a check that runs, because a resolution to be careful is not an engineering plan.

1. Any claim that can be enforced must be enforced by something that runs. If a page says a thing works, a test asserts it works, and the test asserts it at the layer a person experiences. This is the rule that would have prevented the support rail, both application forms, and the consent scopes.

2. Invert the pyramid for seams. We keep every unit test, they are cheap and they document intent. But every user visible journey gets an end to end case, and the definition of a feature being done includes an assertion that a person can reach it. The target is not a coverage percentage, it is that no journey on the scorecard is unwatched.

3. The orphan count ratchets downward. 59 today. The gate fails when it rises. Wiring intent.ts up took a day and changed what the product is; there are others in that list with the same potential, and every one of them is work already paid for and not yet collected.

4. The scorecard is the definition of done. The MVP closes when the live production host scores 100 twice in a row. Adding a requirement means adding a check, not writing a paragraph. The number goes in the release notes whether it flatters us or not, which is why 64.6 is written down.

5. Every release documents itself, and the ship enforces it. The notes are checked before the production promote, tolerantly enough not to deadlock, strictly enough to catch a 59 commit drift.

Connections, then communication, then commerce

The order matters and it is not negotiable, so it is worth saying where the money actually enters.

Connections is what we spent this night on and it is not finished. A person types a sentence and reaches somebody who can help. The remaining work here is reachability, which is a data problem as much as a code one. In Connecticut the regulator publishes phone numbers for zero of 821 trades listings. Ranking cannot fix that. Only better sources, and claims from the professionals themselves, can.

Communication is next and barely begun. Today we hand a person a phone number and get out of the way, which is honest and thin. The real product is an agent that reaches out on the person's behalf, under their consent, and comes back with an answer. That is where the consent receipts stop being a compliance artifact and start being the product.

Commerce is third, and we have not built it. Being precise about this matters because it is the part most easily overclaimed. We have Stripe in production today, but as the CRM, not as a payments rail for these transactions: contacts are Stripe customers and the pipeline lives in customer metadata. What does not exist yet is the agent to merchant payment path. AP2, the agent payments protocol, is exactly the shape of the problem, because the hard part is not moving money, it is proving that an agent had authority to spend and producing a receipt that survives a dispute. That is the same problem as our consent receipts, in a domain where being wrong costs money rather than trust.

So the honest sequencing is: consent receipts first, because they are the substrate; then a single payment rail done properly end to end with a real refund and dispute path; then breadth across point of sale systems and gateways. Breadth before depth here would give us a dozen half integrations and no story about what happens when a payment goes wrong. We would rather have one that we can defend.

How long until it is good enough that Jobs would ship it

The honest answer has three parts, and I would rather give the shape than a single number.

The core journey is close. A person can type a sentence and reach somebody. That went from broken to working in one night, and the scorecard says 97.9 with the remaining points being reachability. Call it weeks to a state where the seeker journey is genuinely good, and most of those weeks are data acquisition, not engineering.

The product around it is further. Communication is not built. Commerce is not built. Consent is published more completely than it is enforced, and the deepest tier of our own contract is deliberately not yet enforced, which we say out loud. Insurance covers six states and forty four have no inventory at all. Call that quarters, not weeks, and the pacing item is inventory and integrations rather than architecture.

The bar itself is the honest constraint. Jobs would not ask whether the features are present. He would open the page, and if the first thing he met was our taxonomy instead of his problem, he would say no. We failed that test as recently as yesterday, when the box that takes the answer sat below the fold under a six line paragraph about ourselves. Passing it reliably is a habit, not a milestone, and the way you acquire the habit is to look at the thing constantly and be willing to be embarrassed.

What I would not do is give a date. Every defect in this essay was invisible until somebody looked, and there is no reason to believe we have found the last of them. What I will commit to is the rate: the scorecard runs against the live site, the number is published, and it goes up.

The one sentence

Speed of authorship is no longer the constraint. Correctness of connection is. Build the gates accordingly, and look at the page.