We Tried to Break Our Own Ledger

Most companies publish security writing that is marketing with citations. The system is always sound, the architecture always elegant, the diagrams always clean. Nothing ever breaks.

We would rather show you the opposite. We went looking for ways to break our own consent ledger, and we found three. All three were real. All three are fixed. None of them were hypothetical, and the reason we are writing this down is that the findings generalise well beyond us — if you are building anything that logs consent, at least one of these is probably sitting in your codebase right now.

What the ledger is supposed to do

In the 🤫 Permission Gateway, every grant, read, and revocation writes an entry into an append-only log that belongs to the person the data is about. Each entry carries a hash of the entry before it, so the log forms a chain. Change any historical entry and every hash after it stops matching. That is the standard construction, and it is genuinely good at what it does.

The promise we wanted to make was simple: you can prove what happened to your data. So we asked the uncomfortable question — what could an attacker, including us, do to this log without you being able to tell?

Finding one: a hash chain cannot detect truncation

Here is the flaw, and it is worth sitting with because it is not obvious.

A prev-hash chain protects against insertion, mutation, and reordering. If someone rewrites entry 40, entry 41's stored hash no longer matches, and verification fails loudly. Good.

But now imagine entries 1 through 50, and an attacker deletes entries 48, 49, and 50. What is left is entries 1 through 47 — and that is a perfectly valid chain. Every hash links correctly to the one before it. Verification passes. The math is completely happy.

The three most recent accesses to your data have simply ceased to exist, and nothing in the cryptography objects.

This matters most in exactly the case the ledger exists for. The entries an attacker most wants gone are the newest ones — the read that just happened, the access that should not have. Tail-truncation is not an edge case. It is the attack.

The fix is to stop treating the chain as self-describing. We now enforce that sequence numbers run 1..N with no gaps, and we anchor the head — the client remembers the last sequence number and hash it saw, and a chain that comes back shorter or with a different head is reported as head_regressed or head_diverged rather than quietly passing. Truncation stops being invisible.

The honest limit: this works under trust-on-first-use. A client that has never seen your chain before has no anchor to compare against, and a server colluding with that client from the very beginning is not caught by this. We think that is a materially better position than where we started. We do not think it is airtight, and we are not going to describe it as though it were.

Finding two: revoke had a race

The second flaw is a classic, which is precisely why it is worth naming.

A subscriber asks to read your data. The code checks whether the grant is active, sees that it is, and proceeds to fetch the fields. Between the check and the fetch, you revoke.

That gap — time of check to time of use — is small. It is also entirely real, and under concurrency it will eventually be hit. The read completes against a grant that no longer exists. The person did everything right, hit revoke, and the data went out anyway.

The fix was to stop treating the check and the read as two separate things. Reads now serialise against revocation inside a single transaction, taking an advisory lock and holding a share lock on the grant row for the duration. A revoke that commits first makes the read fail. A read that is already in flight completes, and the revoke waits behind it. There is no window in between, because there is no longer an "in between."

There is a cost — reads now contend with each other on the same grant. We took it. A revocation that is only mostly effective is not a revocation.

Finding three: the error messages were an oracle

The third one is the subtlest, and the kind of thing that survives code review because every individual piece of it looks like good practice.

We had thoughtful, specific error codes. If a grant did not exist, you got one code. If it had expired, another. If it had been revoked, another. If the signature was bad, another. Helpful, precise, easy to debug.

They were also a free information service for anyone probing the system. Feed it a guessed handle: "does not exist" and "was revoked" are different answers, and the difference tells you a real fact about a real person — that they had a relationship with this subscriber and ended it. Nobody needs to breach anything. They just need to read the error messages carefully.

The fix was to collapse every denial on the read path into a single, undifferentiated FABRIC_READ_DENIED. The specific reason is still recorded internally, where operators can see it. The caller learns exactly one thing: not authorised. Debuggability lost a little. The person the data is about gained a lot, and that trade is not close.

How we checked the fixes

Claiming a fix and having a fix are different things, so we ran the whole lifecycle against a real Postgres instance rather than mocks: set a profile, grant scoped access, read it back, verify only granted fields come through, revoke, confirm the next read fails.

Then we did the part that matters. We pulled the raw rows out of the database and recomputed the entire hash chain and every signature from scratch, independently, rather than asking our own verification function whether our own verification function was happy. A system that grades its own homework is not evidence of anything. Thirty-one checks, all passing, computed by code that shares nothing with the code under test.

What we still cannot promise

Three things, stated plainly, because a security write-up that only lists wins is an advertisement.

Our receipts give you tamper-evidence, not non-repudiation. They are signed with a symmetric key that our servers hold. That is strong against an outside attacker and against accidental corruption. It is *not* proof against us, because anyone holding the key could in principle produce a valid signature. Moving to asymmetric signing with hardware-held keys is the correct answer and it is on our list. Until it lands, we are not going to imply we have a property we do not have.

Revocation stops future access. It cannot retract the past. If a subscriber read your data yesterday and you revoke today, they still have yesterday's copy. Nothing we build changes that. What we can do is show you exactly what was taken and when, so you know precisely what is out there — and that is what the receipt is for.

We cannot reach into a model someone else trained. If your data was used to train a system we do not operate, our revocation does not touch it. We can stop the flow from here forward. No one can honestly promise more.

Why publish this

Because the truncation finding is more valuable to everyone precisely because it was our bug. Because a consent system nobody can inspect is asking for the same trust the surveillance internet asked for, just more politely. And because if we tell you there is a receipt you can verify, you should be able to check the work — including the parts where we got it wrong first.

The code is small and legible on purpose. Go read it.