Entity Resolution Under a Right to Removal

Abstract. Entity resolution is normally posed as an accuracy problem: given records from heterogeneous sources, decide which refer to the same real-world entity, and maximize precision and recall. When the entities are working professionals who never asked to be in your index, a second constraint appears that the literature largely ignores — the entity can demand removal, and must be able to. We report on building a directory of 248,568 organizations and 616,145 named professionals across 52 U.S. metropolitan areas under that constraint, and argue that removability is a design constraint on the resolution architecture rather than a compliance feature added afterward. We describe the source model, a proximity-first index, the collapse of 737 regulatory taxonomy codes into 27 terms an ordinary person would use, and the decision to verify credentials at claim time against the authoritative regulator rather than at ingest. We state what we did not measure.

1. Introduction

A person looking for a chiropractor, an insurance producer, or a financial adviser near them is performing a query that ought to be easy and is not. The information exists — regulators publish registries, carriers publish locators, firms publish staff pages — but it is scattered across formats, keyed differently, and rarely searchable by the dimension that actually matters, which is how close is this person to me.

Assembling it is a classic entity-resolution problem with an unusual ethical structure. The subjects are not customers who opted in. They are professionals whose public credentials make them findable, and who have a legitimate interest in how they are represented by a third party they never engaged.

There is an established, commercially successful pattern here that we consider a cautionary example: harvest a public registry, construct a profile the professional never consented to, rank it in search, and then sell that professional the ability to control the page built on their own name. It is legal and it is lucrative. It is also, in our judgment, a hostage arrangement, and we set out to build the same utility without it.

That decision turned out to have engineering consequences well beyond policy, which is the subject of this paper.

2. Sources and their disagreements

We draw exclusively on public, citable sources: national provider registries, regulator-published licensing data, public firm disclosures, and the “find an agent” and “find an adviser” locators that carriers and institutions publish themselves. That last category is underrated — it is the most current and most defensible source for who actually represents an institution today, because the institution maintains it for its own customers.

The sources disagree in predictable ways. Name forms vary (legal, professional, and display names for the same human). Addresses may be a practice location, a mailing address, or a registered agent's office, and only the first is useful for proximity. Organizations and individuals are conflated — a solo practitioner frequently appears as both an organization and a person, and whether those are one entity or two is genuinely ambiguous rather than merely unresolved. Classification is expressed in regulator vocabularies designed for billing and oversight, not for a human deciding whom to call.

3. Resolution, and the deliberate decision to under-merge

Standard practice maximizes a matching objective. We adopted an asymmetric loss instead, because the two error types are not equally bad when the entities are people.

Over-merging is worse than under-merging.Wrongly fusing two professionals into one record attributes one person's credentials, location, or specialty to another. That is a false statement about a named individual, and it is the error most likely to cause real harm. Under-merging produces a duplicate — untidy, and resolvable by the professional at claim time, when they are the best-qualified party to say which records are theirs.

So we bias toward under-merging and treat claim as the point at which ambiguity resolves. This inverts the usual pipeline: rather than trying to be maximally correct at ingest, we accept known duplication and let the authoritative party — the human — collapse it. The person who knows whether those two records are the same person is the person.

We are not aware of this being an unusual technique so much as an unusual objective. The literature is largely written for settings where no such authority is available to consult.

4. Taxonomy collapse: 737 to 27

Regulatory classification systems are precise and unusable. Our healthcare source carries 737 distinct taxonomy codes, distinguishing subspecialties in ways that matter enormously for billing and not at all to a person with back pain.

We collapse them to 27 terms that an ordinary person would actually type or recognize — chiropractor, dentist, massage therapist, eye care, mental health and counseling, and so on. Three observations from doing it.

The mapping is lossy on purpose and irreversible in practice. Once you have collapsed a hierarchy to a vocabulary, you cannot recover the distinction without keeping the original, so we keep the original code on every record and treat the collapse as a view.

The mapping is not a tree cut. Some user-facing terms cut across the regulatory hierarchy: massage therapy sits under healthcare taxonomies but belongs, from the searcher's point of view, with personal care. We handle this with explicit overrides rather than pretending the hierarchy is respected, because a clean abstraction that misroutes real queries is worse than an honest special case.

The vocabulary is a product decision wearing a data-engineering costume. Choosing 27 terms is choosing what people are allowed to look for, and no amount of source fidelity substitutes for getting that vocabulary right.

5. Proximity as the primary key

The query is “near me,” so geography is not a filter applied after retrieval — it is the organizing principle of the index. This has an architectural payoff we did not fully appreciate until an unrelated failure forced us to: because queries are geographically bounded, a geographic partition is a natural lazy-loading boundary. A proximity query touches a handful of postal regions, so the individual corpus can be sharded by region and loaded on demand rather than held resident.

We arrived at that design by taking production down, which is documented separately. The point worth keeping here is that the shape of the query should determine the physical layout, and when it does, an accuracy decision and a resource decision turn out to be the same decision.

  RESIDENT     national organization index      248,568 records
               (every query needs it)

  ON DEMAND    individuals sharded by region    616,145 records
               (a proximity query touches ~12 of ~1000 shards)

6. Verification at claim time

The design decision with the largest downstream consequences is when credentials get verified.

The obvious approach verifies at ingest: check every record against the authoritative registry before publishing. We do not do this, for two reasons. The authoritative registries are, in several cases, restricted — bulk-copying them is not permitted, and building a product on a restricted bulk copy would contradict the entire premise. And ingest-time verification degrades immediately, since a license verified at ingest may lapse the following week while the record continues to assert it.

Instead, verification happens at claim time, against the authoritative regulator, at the moment a human asserts a record is theirs — the state insurance department or the national producer registry for insurance; the securities regulator's adviser registry or the broker-check system for financial professionals.

This produces a directory with an explicit two-tier trust model, which we consider a feature and state plainly on every listing: unclaimed records are assembled from cited public sources and carry exactly that much authority; claimed records have been verified against the regulator by the human who owns them.

7. Removability as an architectural constraint

Every listing is removable on request, honored without a retention flow. Stating that is easy; the engineering consequences are not, and they are the part we most want to record.

Derived artifacts must be reconstructible. Anything computed from the corpus — indices, caches, shards, search structures — must be rebuildable without the removed record. A pipeline with an unreproducible derived artifact cannot honor removal, only approximate it.

Provenance must be per-record and durable. To honor a removal you must know every place a record propagated, which means provenance is not a nice-to-have for debugging but the mechanism removal depends on.

Removal must be idempotent under re-ingest. This is the subtle one. If the source still lists the professional, the next refresh will happily re-add them, so removal must be recorded as a durable suppression that survives every future ingest. Otherwise removal is a temporary state that silently reverts — a bug that presents as a betrayal.

Under-merging helps here too.A conservatively-merged record is easier to remove cleanly than an aggressively-fused one, whose removal risks taking someone else's data with it. The asymmetric loss from §3 pays off a second time.

None of these are hard individually. Together they mean removability cannot be added late — a system built without them can promise removal but cannot deliver it, and will not discover this until someone asks.

8. What we have and have not measured

We report scale honestly and evaluation honestly, including the parts that are missing.

Measured: 248,568 organizations and 616,145 individuals across all 52 major U.S. metropolitan areas; 737 source taxonomy codes mapped to 27 user-facing terms; proximity queries returning correct nearest-first results verified in production.

Not measured, and we should be direct about this: we have no labeled ground truth, so we cannot report precision and recall for the resolution step. We have not measured the duplicate rate our deliberate under-merging produces, which is the number most directly implied by the design choice in §3. We have not measured claim-rate or removal-rate, both of which are the real test of whether the ethical architecture is any good in practice. And we have not validated the 27-term vocabulary against actual search behavior — it is currently our judgment, not a finding.

A directory of this kind ought to be evaluated on those numbers, and we intend to publish them when we have them rather than describe the system as validated in the meantime.

9. Related work

Entity resolution has a long literature, from the Fellegi–Sunter probabilistic record-linkage framework through modern blocking, clustering, and learned-matching approaches. Our resolution machinery is unremarkable within that tradition. Data provenance and lineage research supplies the tracking that our removability requirement depends on. Work on the right to erasure has largely examined it as a legal and compliance question; the systems consequences — reconstructible derivations, durable suppression under re-ingest — are less developed than we expected when we went looking, and that gap is the part of this paper we think is most worth someone else picking up.

10. Conclusion

The technical claim is small: none of these techniques are new. The design claim is not. Treating removability as a first-class constraint changes the resolution objective, the merge policy, the provenance requirements, and the ingest architecture — and it does so in ways that are difficult to retrofit.

The alternative pattern is well established and works commercially. We think a directory whose subjects can leave is a better artifact, and we would rather demonstrate that it is buildable at scale than argue about it.