We Measured the Data Before We Built the Product, and the Data Said No

A product we did not ship

We were going to build something called The Record.

The idea was good. Rather than another ranked list of lawyers, which is mostly a popularity contest with a payment tier, we would publish the documented court record. Every claim cited to a real case. Sorted by a fact the reader picks. Blind spots printed on the page in normal type, not in grey at the bottom.

The appeal is that it is falsifiable. A ranking is an opinion, and in the United States an opinion enjoys a certain amount of legal shelter. A record is a claim about the world, and anyone can check it. We chose the version with less shelter deliberately, because a directory you can check is worth something and a directory you cannot is worth nothing.

That choice sets the accuracy bar at absolute. Which is why, before writing any of it, we went and measured the data.

What the measurement found

The corpus is the public domain bulk release from CourtListener. Cleanly licensed, no agreement to negotiate, genuinely excellent public infrastructure.

We sampled 166,806 rows. Here is the fill rate of the attorneys column by year.

• 2018: 83.6 percent
• 2019: 69.0 percent
• 2020: 0.5 percent
• 2021 through 2026: 0.0 percent

The cases run through June 2026. Only the counsel information stops.

We expected a gap. We did not expect a cliff, so we went looking for why, and the why is the part that settles it. Every row that carries counsel comes from one upstream source: the Harvard Caselaw Access Project. Rows from the live court scraper carry counsel 0.0 percent of the time. Rows from the federal docket system, 0.0 percent. Harvard's own metadata ends in 2020.

This is not a backlog anyone is going to clear. The ongoing pipeline does not extract a counsel block at all. A product built on this would be a picture of who practised law in 2018, presented to someone looking for a lawyer today.

The failure mode that ended the discussion

Even inside the good years, the field is declared free text. Here is what that means in practice. Two real rows:

`` Jamie A. Khan, Ross Alan Appel, McCullough Khan, Charleston, SC Bret Leon Lusskin, Jr., Bret Lusskin, P.A., Aventura, FL ``

The first is two attorneys and a firm. The second is one attorney and one firm. The comma does double duty as a separator between people, a separator between a person and their firm, and a component of a name. Split naively and you produce an attorney called Jr. and an attorney called McCullough Khan who does not exist.

There is also no identity key anywhere in the release. No bar number, no attorney identifier, no address. Nothing ties a name in a 2017 case to a living person practising now. And the older text carries optical character recognition damage: lor plaintiff, J oseph J. Murphy.

So the honest summary is: a directory built on this would invent plausible-looking lawyers, attribute real cases to them, and present the whole thing as current. Every one of those is a fabrication, and a fabrication is exactly the failure that a falsifiable product cannot survive.

We did not build it. The research took a day. Building it first and finding out afterwards would have cost a great deal more than a day, and the people wrongly listed would have paid part of that bill.

Pahrump is not Las Vegas

A smaller one from the same week, and a good illustration of how these mistakes actually happen.

We were building a Clark County page. Clark County is Las Vegas. Nevada postcodes in the 890 and 891 ranges are, to a very good approximation, Clark County. You can see the shape of the reasoning, and it is the shape of most data bugs: a pattern that is right often enough to feel right.

The 890 range also reaches into Nye, Lincoln and Esmeralda counties. Pahrump is 89048. Alamo is 89001. Neither is Las Vegas, and filing their insurance producers under a Las Vegas page would have been a quiet lie about where a person works.

The fix is unglamorous. The United States Census Bureau publishes a relationship file mapping every postcode area to its county. It is public domain. We use it. Clark County is 76 postcode areas, of which our roster covers 69, and the page says both numbers.

The rule we took from it: if a boundary can be looked up, never infer it. A pattern that is right ninety-five percent of the time is a bug you have not met yet.

While we were there we also caught ourselves quoting Nevada's statewide producer count, 16,572, as though it were the Las Vegas figure. It is not. Clark County is 13,091. There is now a test asserting the county number stays strictly below the state number, because the way you stop repeating a mistake is not by resolving to be careful.

Three comments that were not true

The most uncomfortable finding of the week is a pattern rather than a bug.

Three times in one month, in three unrelated parts of the codebase, we found a comment that confidently described behaviour that nothing enforced.

A comment said dynamic routes were already covered by the sitemap generator. Nineteen pages honouring named people were public, indexable, and in no sitemap.

A comment in the build configuration asserted two lists were identical. They had drifted, and a guard we wrote specifically to prevent a past outage had never once run in continuous integration.

A comment in our application route said the product lived one click away from the marketing page. The click did not exist. The primary button on our home page said "Get Agent One, free" and opened a name reservation form whose confirmation screen linked only back to itself. Three levels deep from the front door there was no way to reach the actual product, which had been live the entire time.

The pattern is not carelessness. Each comment was true when it was written. The pattern is that a sentence in a source file is a claim with no mechanism behind it, and claims without mechanisms rot silently while looking fine.

Our response is a rule rather than more diligence. Any statement that can be enforced must be enforced by something that runs. A comment asserting coverage becomes a test asserting coverage. A number that can be computed gets computed rather than typed. The home page now has a test that fails if the primary call to action stops pointing at the product, because a label and a destination that can drift apart eventually will.

What we found out about our own promise

The one that mattered most this week was not a data question.

We tell people they can remove themselves from our directory. Free, no account, no retention flow, honoured within seventy two hours. We say it publicly because it is the main thing that distinguishes us from a category of sites that treat a person's public record as an asset to be held hostage.

Our records are rebuilt from public registries that still list the person. So a removal has to survive the rebuild, and our suppression module says exactly that in a comment at the top.

One of the three code paths that emits a person honoured it. The other two never imported the check. Anyone who asked to be taken down was quietly restored by the next refresh, with no error anywhere.

That is worse than a feature we had not built. It is a promise that reverts itself, and it had been live. Both paths now filter at load, keyed to the identifier a rebuild reproduces, and a test walks every place we emit a person so a fourth path cannot be added without one. Removal requests now reach a human being through the same durable pipeline as everything else on the site, rather than a log line nobody watches.

We are writing this down for the same reason we wrote down the corpus measurement. If we only publish the things that worked, nobody has any reason to believe the things that worked.

Simplicity is a result, not a starting point

There is a version of engineering writing that presents the final design as though it arrived clean. It never does. Every simple thing in a system is the residue of an argument someone lost.

Clark County is 76 postcode areas because we looked it up rather than guessing. The home page button goes to the product because a test now says it must. The removal check runs in three places because it ran in one and we found out.

None of that is clever. It is closer to housekeeping. But the difference between a system you can trust and one you cannot is almost never cleverness, and the work of making a thing simple is mostly the work of finding out what is actually true and then refusing to look away from it.

If that is the kind of engineering you want to do, we are hiring, and the codebase is more honest than this blog post because it has tests.