Deleting the Commit Did Not Delete the Secret

The incident began with a secret scanner flagging a credential in a repository, and the team's first response was correct and fast: rotate the credential. The second response, deleting the commit that contained it, felt like cleanup and was actually meaningless, because the secret was never in the commit in the sense that deleting the commit removes it. It is in a blob, and the blob is addressed by its content, and it remains reachable from history, from reflogs, from tags, and from every clone that ever existed.

The mental model error is treating git like a filesystem where deleting a file deletes the bytes. Git is a content addressed archive, and "deleting" only removes a reference. The bytes persist as long as any reference, local or remote, points at them.

This is the postmortem of that model error, and the checklist for what removal actually requires.

Why the bytes survive

When the secret was committed, git stored its file content as a blob object, named by the hash of that content. The commit points to a tree, the tree points to the blob. Deleting the commit, or amending history to drop the file, creates a new commit and tree, but the old blob object is not deleted. It becomes unreachable from the new history, yet it still exists in the object store until garbage collection prunes it, and garbage collection will not prune it while any reference, including the reflog, mentions a commit that reaches it.

More importantly, the object store on your machine is one copy. Every developer clone, every CI checkout, every fork on the hosting provider contains the blob, because they fetched the history that included it. Rewriting your local or origin history touches none of those copies.

So "we deleted the commit" is true on one machine and false everywhere else, which is the same as false for security purposes.

The order that matters: rotate first, always

The only response that reliably ends the exposure is rotating the credential, because rotation makes every copy of the secret worthless, regardless of how many copies exist and where they are. Everything else, the history rewrite, the pruning, the fork audit, is hygiene that reduces future surface but does not undo the exposure window.

The teams that get this wrong do the cleanup first and the rotation later, or never, because the cleanup feels like the fix. It is not. A secret in history is a leaked secret, and the response to a leak is revocation, in that order, every time. This is the same priority as secrets rotation without downtime, which is the mechanics of doing the rotation part safely.

What real removal looks like, after rotation

Once the credential is dead, the hygiene work reduces the chance the old blob resurfaces or is scanned.

Rewrite history with a proper tool. Filter the blob out of all history using a rewriting tool, not by hand. The modern tools remove the object from every commit that referenced it and update refs accordingly.

Expire the reflog and prune. The reflog keeps the old commits reachable. Expire it and run an aggressive garbage collection so the blob becomes truly unreachable and is then removed from the object store.

git filter-repo --invert-paths --path config/secrets.yml
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Force push and re clone. The rewritten history must replace the remote, and every clone must be discarded or rewritten, because a stale clone pushed later can resurrect the old history. This is the part that makes history rewriting a coordinated event rather than a command.

Audit forks and CI caches. Forks on the hosting provider are separate repositories that your force push does not touch. They must be deleted or rewritten by their owners. CI systems that cached the checkout may retain the blob in their caches, which should be purged.

Only after all of that is the blob gone from the places you control, and the places you do not control, other people's clones you never knew about, are precisely why rotation, not removal, is the fix.

The prevention that beats the cleanup

The reason secrets reach history is that they pass through a working tree at all. The durable fixes remove the path.

Secrets live in the environment or a secret manager, injected at runtime, never in files the repository sees. A pre commit hook that scans for credential shapes blocks the accident before it becomes an object, and the scanner on the hosting provider is the second line, catching what the hook missed.

And the cultural fix: a .env file committed once, years ago, by someone long gone, is the canonical origin of these incidents, so the rule that environment files are never tracked, enforced by template files and hooks, removes the most common vector entirely.

The rule I keep

A secret that touched git history is a leaked secret, full stop. Rotate first so every copy dies, then treat history rewriting, reflog expiry, pruning and fork audit as hygiene that shrinks future surface, never as the revocation itself.

The cleanup feels like the fix because it removes what you can see. The exposure is what you cannot see, the clones and forks and caches, and revocation is the only response that reaches all of them at once.

One more copy location deserves naming because it is the one teams forget in the audit: package artefacts. If the secret ever made it into a built image, a wheel, a jar or a published package, those artefacts live in registries with their own immutability and their own download history, and rewriting the repository touches none of them. Rotate covers it, as it covers everything, but the artefact audit belongs on the hygiene checklist next to forks and CI caches.

The adjacent failure, where the secret is not in history but in the layers around a request, is the logging PII story in the pull request that logged an email address.