advanced45 minby DevFox

Rewriting History at Scale

A runbook for purging a secret or a huge binary from every commit with git filter-repo, extracting a subdirectory, and coordinating the force-push without breaking the team.

Git
Security

A teammate pushes an .env file with a live cloud key. Someone commits a 400 MB render that has sat in every clone for two years. Legal asks you to delete a vendor's file from a 2016 repository. No new commit fixes these — the bytes are still in history, one git show away from anyone who has cloned you.

By the end of this lesson you can purge a secret or a huge binary from every commit, split a subdirectory into its own project with its history intact, and run the whole thing as a coordinated operation that does not break your team's clones.

When a whole-history rewrite is justified

Rewriting every commit is a destructive migration, closer to a schema change with downtime than to a normal Git operation. Three cases justify it.

A leaked credential

Committed and pushed.

And the rewrite is cleanup, not remediation — the next section is about why.

A file that must not exist

Legal, or simply too large.

A licence violation, a takedown, a deletion request — or a few blobs that make every clone unusable.

Restructuring repositories

Splitting or merging projects.

Extracting a directory into its own repository, or combining two while keeping both histories.

That is the whole list. "The history is ugly" is not on it.

Cosmetic complaints do not justify a cost that everyone pays.

The blast radius, before you touch anything

A commit's hash covers its tree, message, author, and parents. Change one commit and its hash changes, which changes the next commit's parent, which changes that hash too. Every commit after the earliest change gets a new identity.

Think poured concrete, not stacked blocks: you cannot chip one piece out, you re-pour what sits above it.

Open pull requests
Their base and head commits no longer exist server-side, so hosts show them as wildly conflicted.
Tags, release tags included
They point at commits that are gone, unless you re-push them too.
Every clone, plus CI caches
Including the laptop of whoever is offline today, who will come back and push the old history straight into the repo.
Every recorded hash
Tracker links, changelog SHAs, pinned commits in deploy scripts, submodule pointers, blame permalinks.
Signed commits
Their signatures no longer verify, because the thing that was signed no longer exists.

Walk this list before you run anything. Every item is somebody else's afternoon.

Rotate the credential first

If the reason is a leaked secret, the rewrite is not the fix. It was pushed to a server, fetched by CI, cloned by people who have since left. You cannot un-distribute it.

The host does not forget on your schedule either. On GitHub a commit unreachable from any branch or tag is still served by its hash until garbage collection runs, on their timetable — so anyone holding the SHA still reads the blob after your rewrite. Purging it for certain takes a support request.

The tool is git filter-repo

git filter-repo is the tool for this job: a rewriter built on git fast-export, fast and correct where it counts — tags get rewritten, merges handled, empty commits pruned. It is not part of core Git, so install it:

bash

pip install git-filter-repo      # any platform, Python 3.6+
brew install git-filter-repo     # macOS

For the narrow "delete these files, scrub these strings" case, BFG Repo-Cleaner is a simpler alternative: a small Java tool (--delete-files, --replace-text, --strip-blobs-bigger-than 100M) that refuses to touch the commit HEAD points at. It cannot rename paths or extract directories, so structural work stays here.

Run filter-repo in your everyday working copy and it stops: this is not a fresh clone. That is deliberate, because a working copy holds state the rewrite cannot fix — stashes, reflogs, extra remotes, worktrees — any of which can resurrect the blob.

bash

git clone --no-local /path/to/repo repo-rewrite   # copy, not link
cd repo-rewrite

--no-local copies objects rather than hardlinking into the source, so a mistake in the rewrite cannot damage the original.

Removing a file from every commit

--invert-paths flips --path from "keep only this" to "drop it":

bash

git filter-repo --invert-paths --path .env

Paths match from the repository root. For patterns use --path-glob; any number of path options combine:

bash

git filter-repo --invert-paths --path-glob '*.pem' \
  --path assets/promo-render.mov

Quote globs so the shell does not expand them locally. Run git filter-repo --analyze first: it ranks blobs by size under .git/filter-repo/analysis.

Redacting a secret in place

Sometimes the file stays and only one string inside it must go — a key in a config sample, a password in a fixture. --replace-text takes a rules file, one rule per line:

bash

git filter-repo --replace-text rules.txt

text

literal:AKIAIOSFODNN7EXAMPLE
regex:AKIA[0-9A-Z]{16}
glob:*BEGIN RSA PRIVATE KEY*
hunter2==>correct-horse-battery-staple

A bare line is a literal. Without ==> the match becomes ***REMOVED***; with it, the text after the arrow is used.

Be honest about what this does: it changes file content in every historical commit. A build from an old tag now compiles different bytes and a test asserting on that literal fails, so if you pin old revisions, re-check them before pushing.

Extracting a subdirectory into its own repo

--subdirectory-filter keeps one directory and lifts it to the root:

bash

git clone --no-local git@github.com:acme/monorepo.git ui-repo
cd ui-repo
git filter-repo --subdirectory-filter packages/ui

What about commits that never touched packages/ui? After filtering, their tree matches their parent's, so filter-repo prunes them by default — as it does merge commits whose sides collapse to one tree. You keep the commits that changed the UI package, authors and dates intact.

To land the code somewhere other than the root, add a rename:

bash

git filter-repo --path packages/ui --path-rename packages/ui:src

Merging repositories reverses the manoeuvre: rewrite the incoming project into its target subdirectory, then merge with --allow-unrelated-histories.

Fixing author and committer identities

A mailmap file maps old identities to new ones, one per line, and applying it rewrites the author and committer of each commit:

text

Ada Lovelace <ada@acme.com> <ada@old-laptop.local>

bash

git filter-repo --mailmap ../mailmap

The coordinated rewrite runbook

Run it in this order.

Rotate any leaked credential. Nothing below substitutes.
Freeze the repository. Announce a window and lock the branches in your host's settings rather than trusting everyone to read the email. Ask people to push or drop open work.
Notify with specifics — the time, the reason, the commands each person runs afterwards. Surprised people force-push the old history back.

Rewrite on a fresh clone.

bash

git clone --no-local git@github.com:acme/app.git app-rewrite
cd app-rewrite
git filter-repo --invert-paths --path .env

Verify before pushing: shape, absence, size.

bash

git log --all --oneline | wc -l    # commit count sane?
git log --all --oneline -- .env    # must print nothing
git grep -I "AKIAIOSFODNN7EXAMPLE" $(git rev-list --all)
git count-objects -vH              # compare size-pack

Push every branch and every tag.

bash

git remote add origin git@github.com:acme/app.git
git push --force origin --all
git push --force origin --tags

Unfreeze, reopen PRs, purge caches. SHA-keyed CI caches are stale, and only your host can collect the unreachable objects.

Every teammate now holds a mismatched history. The safe answer is a fresh clone; for in-flight work, replant the branch:

bash

git switch feature/checkout   # branch-only, never a file path
git rebase --onto origin/main <old-main-sha> feature/checkout

That replays the commits after <old-main-sha> — the last commit shared with the old main — onto the new one. Nobody should run git pull after a rewrite: it merges the two histories and restores every commit you deleted.

Grafts and git replace: rewriting nothing

git replace records a substitution — "when you read commit A, use commit B" — leaving objects untouched and hashes valid, and travelling only if you push refs/replace/* deliberately. The classic use is stitching a truncated history onto an archived one: your repo opens with an import, the prehistory lives elsewhere.

bash

git remote add legacy /path/to/old-repo
git fetch legacy
git replace --graft <first-commit> <legacy-tip>
git log --oneline | tail -5     # prehistory now appears

git log, git blame and git bisect all follow the graft, since replacement is on by default (--replace-objects). To see the raw truth, run git --no-replace-objects log or set GIT_NO_REPLACE_OBJECTS=1. Being a view rather than a change, grafting is right when the goal is readability and wrong when the goal is that bytes stop existing.

Confirming the object is really gone

filter-repo expires reflogs and repacks for you; in a clone you rewrote with --force, do it yourself:

bash

git reflog expire --expire=now --all
git gc --prune=now --aggressive
git fsck --full --unreachable
git count-objects -vH

git gc --prune=now drops unreachable objects immediately rather than honouring the usual two-week grace period, and it permanently destroys anything not reachable from a ref, including work you stashed and forgot. Set the resulting size-pack against the number you recorded before: 400 MB down to 40 MB is your receipt — proof your copy is clean, not the internet's.

A cheat sheet to keep

bash

pip install git-filter-repo           # install the tool
git clone --no-local <url> work       # always a fresh clone
git filter-repo --analyze             # what costs space
git filter-repo --invert-paths --path .env         # drop a file
git filter-repo --invert-paths --path-glob '*.pem' # drop a glob
git filter-repo --replace-text rules.txt           # redact text
git filter-repo --subdirectory-filter packages/ui  # extract dir
git filter-repo --path-rename old:new # move a path
git filter-repo --mailmap ../mailmap  # rewrite identities
git log --all --oneline | wc -l       # commit count check
git count-objects -vH                 # repo size check
git remote add origin <url>           # filter-repo drops it
git push --force origin --all         # push every branch
git push --force origin --tags        # push every tag
git rebase --onto <new> <old> <ref>   # replant live work
git replace --graft <commit> <parent> # non-destructive stitch
git --no-replace-objects log          # see the raw history
git reflog expire --expire=now --all  # forget old positions
git gc --prune=now --aggressive       # delete unreachable data
git fsck --full --unreachable         # what is left over

Where to go next

You can now strip anything from a repository's history, split a project out of a monorepo, and coordinate the migration without stranding your team. The judgement is the harder half: most requests to rewrite history are best answered with "no, and here is the .mailmap or .gitignore that solves it."

Next comes merge strategies, drivers, and rerere — how Git decides what a merge means, how to teach it to resolve project-specific files on its own, and how to stop re-solving the same conflict on every rebase. Before that, practise on something disposable: clone a public repository, strip its largest blob, compare git count-objects -vH, then graft the old history back on. Doing it once where nobody depends on the result is what makes it calm the day it matters.