advanced40 minby DevFox

Scaling Git for Large Repositories

Diagnose why a repository is slow — too much history, too many files, or too much binary — then apply partial clones, sparse checkout, git maintenance, or Git LFS.

Git
DevOps

You clone the repository, make coffee, come back, and it is still cloning. Later, git status takes eleven seconds. Everyone calls this "the repo is too big" — as useful as "the car makes a noise."

By the end of this lesson you can measure a slow repository, name which of three real problems it has, and apply the fix that matches, instead of running git gc --aggressive and hoping.

Three problems wearing one name

A house is hard to move out of for unrelated reasons: boxes in the attic, too many rooms to sweep, a piano in the living room.

Repositories are the same, and every remedy in this lesson sits in exactly one of these columns.

Too much history

Clone and fetch crawl. The tree is fine.

A deep commit graph and a fat packfile.

Fixed by shallow and partial clones — you take less of the past.

Too many files

git status and checkout crawl.

A wide tree, hundreds of thousands of paths.

Fixed by sparse checkout and a filesystem monitor — you take less of the present.

Too much binary content

Everything is heavy, forever.

Blobs Git cannot delta-compress, stored whole on every revision.

Fixed by LFS, and better by not committing them.

Applying the wrong remedy costs you effort and changes nothing you can measure

Measure before you change anything

Start with Git's storage, then the shape of history and tree:

bash

du -sh .git                 # total on-disk cost
git count-objects -vH       # loose vs packed, size-pack
git rev-list --all --count  # how many commits
git ls-files | wc -l        # how many tracked files

Read those against each other. Half a million commits and forty thousand files is a history problem; nine thousand commits and six hundred thousand files is a working-tree problem. A large size-pack with modest counts in both means something heavy is hiding, and git verify-pack -v names it (size in column 3):

bash

git verify-pack -v .git/objects/pack/*.idx \
  | awk '/blob/ {print $3, $1}' \
  | sort -rn \
  | head -10

Turn an ID into a path by grepping git rev-list --objects --all for it; usually it is a vendored binary, a stray CSV, or an old dist/ folder.

Time what hurts, then see where it goes. If refreshing the index dominates, the problem is file count:

bash

time git status --porcelain         # a baseline
GIT_TRACE_PERFORMANCE=1 git status  # timing per phase

Too much history: shallow and partial clones

The oldest fix is the shallow clone: recent commits only, the rest left on the server:

bash

git clone --depth=1 <url>                   # newest commit only
git clone --shallow-since=2025-01-01 <url>  # by date
git clone --single-branch --branch main <url>

That suits a build that compiles the tip. But a shallow repo is truncated: git log stops at the boundary, bisect has nothing to search, and merges across it fail for want of a merge base. git fetch --unshallow undoes it.

The partial clone is the modern answer: every commit, file contents deferred and fetched on demand via a recorded filter.

bash

git clone --filter=blob:none <url>  # blobless: commits + trees
git clone --filter=tree:0 <url>     # treeless: commits only

A blobless clone keeps the full commit graph and every directory listing. A treeless clone defers directories too, so path-related work like git log -- some/path hits the wire.

Too many files: sparse checkout

Partial clone shrinks the download. Sparse checkout shrinks what lands on disk: some directories populated, the rest tracked and indexed but unwritten. Use cone mode, whole directories:

bash

git sparse-checkout init --cone
git sparse-checkout set services/api libs/auth
git sparse-checkout add tools/codegen  # widen later
git sparse-checkout list               # what is populated
git sparse-checkout disable            # back to a full tree

Cone mode beats the older free-form pattern mode structurally: Git resolves a directory in one lookup instead of testing every path against every pattern. With a partial clone it is the standard monorepo setup:

bash

git clone --filter=blob:none --no-checkout <url> mono
cd mono
git sparse-checkout init --cone
git sparse-checkout set services/api libs/auth
git checkout main

--no-checkout matters: without it Git writes the whole tree first and you pay the cost you were avoiding. That git checkout is population, not a branch switch — hence not git switch.

Making the working tree stop crawling

git status answers "what changed?" by walking the tree and calling stat on everything. Two settings replace that walk:

bash

git config core.untrackedCache true  # remember untracked scans
git config core.fsmonitor true       # ask the OS what changed

The untracked cache remembers scans per directory, so unchanged directories are skipped. core.fsmonitor true starts Git's built-in file-system monitor, a daemon subscribed to OS change notifications: rather than sweep every room, Git asks the doorbell what came in. It works on macOS and Windows only.

bash

git config feature.manyFiles true  # umbrella for big trees
git config index.version 4         # compressed path names
git config status.showUntrackedFiles no

feature.manyFiles switches on index version 4, the untracked cache, and a fetch negotiation suited to deep histories; version 4 prefix-compresses path names, shrinking a huge index. status.showUntrackedFiles no is fastest and most regrettable: git status stops naming new files at all.

Maintenance, and the commit-graph

Repositories degrade on their own: every fetch adds loose objects and packs, so Git opens more files per question. Schedule the cleanup rather than remembering it:

bash

git maintenance start                    # register background maintenance
git maintenance stop
git maintenance run --task=commit-graph  # run one task now

That registers jobs with your platform's scheduler: prefetch (downloads objects quietly, so your next fetch is near-instant), commit-graph, loose-objects, incremental-repack. It never runs a full gc.

The commit-graph is a binary file listing every commit with its parents, generation numbers, and dates — an index at the front of a reference book, not the book. With it, git log --graph, merge-base, and --contains are instant:

bash

git commit-graph write --reachable

Repacking by hand, know what you ask for. git gc prunes unreachable objects and repacks conservatively. git repack -adf builds one pack (-a), drops redundant ones (-d), and recomputes deltas from scratch (-f) — the post-rewrite sledgehammer. gc.auto is the threshold at which Git gcs mid-command; 0 disables it.

You almost never need git gc --aggressive: it re-deltas the whole repository with a large window, running for hours and pinning memory, and on a repacked repo it buys a few percent.

Too much binary content: Git LFS

Git stores text beautifully because version 41 of a file is a small delta against version 40. A 200 MB PSD has no useful delta against the previous, so ten revisions cost two gigabytes in every clone, forever.

Git LFS (Large File Storage) swaps the payload for a receipt. Git tracks a pointer file naming a hash and a size; the bytes sit on an LFS server and arrive only for revisions you check out: a coat-check ticket, not the coat.

bash

git lfs install        # install the filters, once
git lfs track "*.psd"  # writes a rule to .gitattributes
git add .gitattributes
git lfs ls-files       # what is currently under LFS
git lfs pull           # fetch payloads for this checkout

git lfs track writes a filter=lfs diff=lfs merge=lfs -text rule into .gitattributes. Commit that file, or the rule applies only to you. In CI you want pointers, not payloads, then git lfs pull --include="assets/ui" if bytes are needed:

bash

GIT_LFS_SKIP_SMUDGE=1 git clone <url>

The honest caveats: LFS needs server support and a quota that costs money, counting bandwidth as well as storage. Its objects are stored whole per version, not deduplicated across history the way blobs are, so it moves weight rather than removes it. Files already in history need a rewrite:

bash

git lfs migrate import --include="*.psd"

Hygiene that prevents all three

Everything above is treatment; the cure is not committing the weight. Keep build output out — dist/, target/, binaries — it is reproducible by definition and the commonest cause of a bloated pack. Keep dependencies in a registry, not a vendor/ directory, and generated assets behind a build step.

Then ask whether the monorepo needs to be one repository. Often it does — atomic cross-service changes are real — but that cost is worth paying deliberately, not by accident.

Scalar: the whole setup in one command

Scalar is Microsoft's packaging of this lesson, built for the Windows and Office repositories and now shipped with Git. scalar clone <url> does a blobless partial clone, enables cone-mode sparse checkout, turns on the monitor and untracked cache, and registers background maintenance. scalar register applies it to an existing repo.

A cheat sheet to keep

bash

# --- measure first ---
du -sh .git                 # total Git storage
git count-objects -vH       # loose/packed breakdown
git rev-list --all --count  # commit count
git ls-files | wc -l        # tracked file count
git verify-pack -v .git/objects/pack/*.idx \
  | awk '/blob/ {print $3, $1}' | sort -rn | head -10
git rev-list --objects --all | grep <oid>  # oid -> path
time git status --porcelain                # baseline timing
GIT_TRACE_PERFORMANCE=1 git status

# --- too much history ---
git clone --depth=1 <url>           # shallow, tip only
git clone --shallow-since=2025-01-01 <url>
git clone --single-branch --branch main <url>
git fetch --unshallow               # recover full history
git clone --filter=blob:none <url>  # blobless: developers
git clone --filter=tree:0 <url>     # treeless: CI

# --- too many files ---
git sparse-checkout init --cone  # fast cone mode
git sparse-checkout set <dir>    # replace the set
git sparse-checkout add <dir>    # widen the set
git sparse-checkout list         # what is populated
git sparse-checkout disable      # restore the full tree

# --- working-tree speed ---
git config core.untrackedCache true      # cache untracked scans
git config core.fsmonitor true           # built-in FS monitor
git config feature.manyFiles true        # big-tree defaults
git config index.version 4               # compressed index
git config status.showUntrackedFiles no  # fast, regrettable

# --- maintenance ---
git maintenance start               # schedule background jobs
git maintenance stop                # unschedule them
git maintenance run --task=commit-graph
git commit-graph write --reachable  # build the graph index
git gc                              # prune + repack normally
git repack -adf                     # full repack after a rewrite
git config gc.auto 0                # stop opportunistic gc

# --- too much binary content ---
git lfs install        # install LFS filters
git lfs track "*.psd"  # write .gitattributes rule
git lfs ls-files       # list LFS-managed files
git lfs pull           # fetch payloads for HEAD

GIT_LFS_SKIP_SMUDGE=1 git clone <url>     # pointers only
git lfs migrate import --include="*.psd"  # rewrites history

# --- all at once ---
scalar clone <url>  # partial + sparse + maint.
scalar register     # apply to an existing repo

Where to go next

You can now walk into a slow repository, measure, and say which of the three problems it has — then reach for partial clones, sparse checkout, maintenance, or LFS.

Next comes securing your history with signed commits: proving the commits in all these clones are actually yours. Once a repo needs CI clones, shallow mirrors, and automated rewrites, "who wrote this commit?" stops being rhetorical.

Run the measurement block against the biggest repository you can reach and find its ten largest blobs. The answer usually surprises you — that surprise is the habit worth keeping.