advanced40 minby DevFox

Scaling Git for Large Repositories

Diagnose why a repository is slow — too much history, too many files, or too much binary — then apply partial clones, sparse checkout, git maintenance, or Git LFS.

DevOps
Git

You clone the repository, make coffee, come back, and it is still cloning. Later, git status takes eleven seconds. Everyone calls this "the repo is too big" — as useful as "the car makes a noise."

By the end of this lesson you can measure a slow repository, name which of three real problems it has, and apply the fix that matches, instead of running git gc --aggressive and hoping.

Three problems wearing one name

A house is hard to move out of for unrelated reasons: boxes in the attic, too many rooms to sweep, a piano in the living room.

Repositories are the same, and every remedy in this lesson sits in exactly one of these columns.

Too much history

Clone and fetch crawl. The tree is fine.

A deep commit graph and a fat packfile.

Fixed by shallow and partial clones — you take less of the past.

Too many files

git status and checkout crawl.

A wide tree, hundreds of thousands of paths.

Fixed by sparse checkout and a filesystem monitor — you take less of the present.

Too much binary content

Everything is heavy, forever.

Blobs Git cannot delta-compress, stored whole on every revision.

Fixed by LFS, and better by not committing them.

Applying the wrong remedy costs you effort and changes nothing you can measure

Measure before you change anything

Start with Git's storage, then the shape of history and tree:

bash

du -sh .git                 # total on-disk cost
git count-objects -vH       # loose vs packed, size-pack
git rev-list --all --count  # how many commits
git ls-files | wc -l        # how many tracked files

Read those against each other. Half a million commits and forty thousand files is a history problem; nine thousand commits and six hundred thousand files is a working-tree problem. A large size-pack with modest counts in both means something heavy is hiding, and git verify-pack -v names it (size in column 3):

bash

git verify-pack -v .git/objects/pack/*.idx \
  | awk '/blob/ {print $3, $1}' \
  | sort -rn \
  | head -10

Turn an ID into a path by grepping git rev-list --objects --all for it; usually it is a vendored binary, a stray CSV, or an old dist/ folder.

Time what hurts, then see where it goes. If refreshing the index dominates, the problem is file count:

Too much history: shallow and partial clones

The oldest fix is the shallow clone: recent commits only, the rest left on the server:

That suits a build that compiles the tip. But a shallow repo is truncated: git log stops at the boundary, bisect has nothing to search, and merges across it fail for want of a merge base. git fetch --unshallow undoes it.

The partial clone is the modern answer: every commit, file contents deferred and fetched on demand via a recorded filter.

A blobless clone keeps the full commit graph and every directory listing. A treeless clone defers directories too, so path-related work like git log -- some/path hits the wire.

Too many files: sparse checkout

Partial clone shrinks the download. Sparse checkout shrinks what lands on disk: some directories populated, the rest tracked and indexed but unwritten. Use cone mode, whole directories:

Cone mode beats the older free-form pattern mode structurally: Git resolves a directory in one lookup instead of testing every path against every pattern. With a partial clone it is the standard monorepo setup:

--no-checkout matters: without it Git writes the whole tree first and you pay the cost you were avoiding. That git checkout is population, not a branch switch — hence not git switch.

Making the working tree stop crawling

git status answers "what changed?" by walking the tree and calling stat on everything. Two settings replace that walk:

The untracked cache remembers scans per directory, so unchanged directories are skipped. core.fsmonitor true starts Git's built-in file-system monitor, a daemon subscribed to OS change notifications: rather than sweep every room, Git asks the doorbell what came in. It works on macOS and Windows only.

feature.manyFiles switches on index version 4, the untracked cache, and a fetch negotiation suited to deep histories; version 4 prefix-compresses path names, shrinking a huge index. status.showUntrackedFiles no is fastest and most regrettable: git status stops naming new files at all.

Maintenance, and the commit-graph

Repositories degrade on their own: every fetch adds loose objects and packs, so Git opens more files per question. Schedule the cleanup rather than remembering it:

That registers jobs with your platform's scheduler: prefetch (downloads objects quietly, so your next fetch is near-instant), commit-graph, loose-objects, incremental-repack. It never runs a full gc.

The commit-graph is a binary file listing every commit with its parents, generation numbers, and dates — an index at the front of a reference book, not the book. With it, git log --graph, merge-base, and --contains are instant:

Repacking by hand, know what you ask for. git gc prunes unreachable objects and repacks conservatively. git repack -adf builds one pack (-a), drops redundant ones (-d), and recomputes deltas from scratch (-f) — the post-rewrite sledgehammer. gc.auto is the threshold at which Git gcs mid-command; 0 disables it.

You almost never need git gc --aggressive: it re-deltas the whole repository with a large window, running for hours and pinning memory, and on a repacked repo it buys a few percent.

Too much binary content: Git LFS

Git stores text beautifully because version 41 of a file is a small delta against version 40. A 200 MB PSD has no useful delta against the previous, so ten revisions cost two gigabytes in every clone, forever.

Git LFS (Large File Storage) swaps the payload for a receipt. Git tracks a pointer file naming a hash and a size; the bytes sit on an LFS server and arrive only for revisions you check out: a coat-check ticket, not the coat.

git lfs track writes a filter=lfs diff=lfs merge=lfs -text rule into .gitattributes. Commit that file, or the rule applies only to you. In CI you want pointers, not payloads, then git lfs pull --include="assets/ui" if bytes are needed:

The honest caveats: LFS needs server support and a quota that costs money, counting bandwidth as well as storage. Its objects are stored whole per version, not deduplicated across history the way blobs are, so it moves weight rather than removes it. Files already in history need a rewrite:

Hygiene that prevents all three

Everything above is treatment; the cure is not committing the weight. Keep build output out — dist/, target/, binaries — it is reproducible by definition and the commonest cause of a bloated pack. Keep dependencies in a registry, not a vendor/ directory, and generated assets behind a build step.

Then ask whether the monorepo needs to be one repository. Often it does — atomic cross-service changes are real — but that cost is worth paying deliberately, not by accident.

Scalar: the whole setup in one command

Scalar is Microsoft's packaging of this lesson, built for the Windows and Office repositories and now shipped with Git. scalar clone <url> does a blobless partial clone, enables cone-mode sparse checkout, turns on the monitor and untracked cache, and registers background maintenance. scalar register applies it to an existing repo.

A cheat sheet to keep

Where to go next

You can now walk into a slow repository, measure, and say which of the three problems it has — then reach for partial clones, sparse checkout, maintenance, or LFS.

Next comes securing your history with signed commits: proving the commits in all these clones are actually yours. Once a repo needs CI clones, shallow mirrors, and automated rewrites, "who wrote this commit?" stops being rhetorical.

Run the measurement block against the biggest repository you can reach and find its ten largest blobs. The answer usually surprises you — that surprise is the habit worth keeping.