Virtual Environments and Dependency Management
Why installing packages globally eventually breaks something, what an environment actually is, and how lockfiles turn 'works on my machine' into a reproducible install.
Why installing packages globally eventually breaks something, what an environment actually is, and how lockfiles turn 'works on my machine' into a reproducible install.
Two projects on your laptop. One was written last year against an older version of a library; the other needs the newest one. You install what the second needs, and the first stops working. You install what the first needs, and now the second is broken.
There is no arrangement of a single shared set of packages that makes both work, and no amount of care avoids the collision. By the end of this lesson your projects will be isolated from each other, and your installs will be reproducible on someone else's machine — which is a different problem from isolation, and the one people skip.
When you run pip install requests, it goes into one place
belonging to that Python. Every project using that Python sees
it, and there can only be one version of it.
That is fine until two projects disagree. The failure is not subtle:
ImportError: cannot import name 'Retry' from 'urllib3.util'Something upgraded a dependency of a dependency, and code that
never mentioned urllib3 stopped working.
You cannot solve this by being careful, because you do not control it — installing one package upgrades its dependencies, which are other packages' dependencies too. The only fix is to stop sharing.
A virtual environment is a folder containing its own copy of the Python interpreter's package directory. Packages installed while it is active go there, and nowhere else.
python3 -m venv .venv # create it, in this project
source .venv/bin/activate # macOS, Linux
.venv\Scripts\activate # WindowsYour prompt changes to show the environment name. Now:
python -m pip install requests # goes into .venv, not system-wide
python -m pip list # only what this project has
deactivate # back to normalThree things follow from this that are worth knowing explicitly.
python and pip now mean the environment's. Inside an
active environment you can drop the 3, because the environment
puts its own interpreter first on your PATH. This is why the
system Python stops being a hazard — you are no longer touching
it.
The folder is disposable. If an environment gets into a confusing state, delete it and make a new one. It contains nothing you wrote.
It never goes in version control. Add .venv/ to
.gitignore. It is large, it is platform-specific, and it is
rebuilt from a file instead — which is the next section.
Isolation solves your machine. It does nothing for the next person, or for the server, or for you in six months. For that you need the dependencies written down.
The oldest form is a requirements.txt:
python -m pip install requests
python -m pip freeze > requirements.txt# on another machine
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txtThat works, and it has a flaw people meet later rather than sooner.
Bad — recording only what you asked for, unpinned.
requests
pandasGood — recording exactly what was installed, pinned.
requests==2.32.3
urllib3==2.2.2
certifi==2024.7.4
charset-normalizer==3.3.2
idna==3.7The first file installs whatever versions exist on the day someone runs it. Six months later that is different software — and the failure lands on whoever set up the project most recently, usually a new starter on their first day, with an error deep inside a library nobody has heard of. The second pins every package including the ones you never asked for directly, so the install is the same today, in production, and next year.
That pinned list has a weakness of its own: it does not
distinguish the packages you chose from the ones that came
along. requests is a decision. charset-normalizer is a
consequence.
When you want to remove requests, pip freeze cannot tell you
which of the other four to remove with it. And when you want to
upgrade, you cannot see which lines are yours to think about.
The answer is two files:
requirements.in what you actually chose
requirements.txt everything, pinned, generated# requirements.in
requests>=2.32
pandas>=2.0pip-compile requirements.in # produces the pinned .txt
pip-sync requirements.txt # makes the venv match it exactlyWhat you chose, loosely.
requests>=2.32. Short, hand-edited, and readable as a list
of decisions.
Removing a line here is a decision you can make; removing one from a pinned list is guesswork.
Everything, exactly, generated.
Your dependencies and theirs, at exact versions, so two machines install the same bytes.
Never hand-edited. A tool produces it from the file on the left.
pip-sync is worth noticing: it does not just install, it
removes anything not in the file. An environment where you
once installed something to try it, and never removed it, is a
machine that works for a reason nobody has written down.
Newer tools do environment creation, dependency declaration and locking as one thing.
uv is the fastest and the closest to plain pip in
concepts:
uv venv # create .venv
uv pip install requests # install, very fast
uv pip compile requirements.in -o requirements.txt
uv sync # match the environment to the lockPoetry and PDM take over the project file too,
declaring dependencies in pyproject.toml and writing a lock
file beside it:
poetry add requests # declares AND installs AND locks
poetry install # recreate from the lock fileThey do more and ask you to adopt more. The trade is real either way, and any of these is fine.
What matters is not which you pick but that you pick one and the
project uses it consistently. A repository with a
requirements.txt, a poetry.lock and a Pipfile is a
repository where nobody knows which is true.
Pinning trades one problem for another: nothing updates until you do it. That is the correct trade — updates become a task with a diff and a test run, instead of something that happens to you — but the task has to actually happen.
python -m pip list --outdated # what has moved onThe habit that works: upgrade deliberately, on a branch, one meaningful change at a time, with the tests as the gate. Read the changelog for anything that jumps a major version, because that is the number that means "we broke something on purpose".
Security updates are the exception to leisurely scheduling.
Tools like pip-audit check your locked versions against known
vulnerabilities:
pip-audit # any known CVEs in what I have?Running that in CI turns "we should look at dependencies sometime" into a thing that tells you when it matters.
CREATING AND USING
python3 -m venv .venv create, in the project
source .venv/bin/activate macOS, Linux
.venv\Scripts\activate Windows
deactivate leave it
rm -rf .venv it is disposable; rebuild freely
.venv/ goes in .gitignore never commit it
INSIDE AN ACTIVE ENVIRONMENT
python means the venv's python
python -m pip install X goes in the venv only
python -m pip list what this project has
python -m pip list --outdated what has moved on
RECORDING
pip freeze > requirements.txt everything, pinned
pip install -r requirements.txt recreate
requirements.in what you CHOSE <- you edit
requirements.txt pinned resolution <- generated
pip-compile / pip-sync generate, and match exactly
commit BOTH
MODERN EQUIVALENTS
uv venv / uv pip install / uv sync
poetry add / poetry install
pick one; a repo with three is a repo with none
RULES
never install into the system python
pin everything for anything that deploys
distinguish declared from locked
upgrade on a branch, with tests, reading changelogs
pip-audit in CI for known vulnerabilitiesYour projects no longer interfere with each other, and an install can be reproduced rather than approximated. The idea underneath — declare what you chose, lock what that resolved to, commit both — is the same in every language you will meet, under different filenames.
Next is Packaging Your Project, which takes the other half
of the question. This lesson was about the code you depend on;
that one is about making your code something others can depend
on — pyproject.toml, the layout that avoids one specific and
nasty import bug, and what actually happens when someone runs
pip install on your work.
Before you move on, take a project you already have and give it
an environment: create .venv, install what it needs, freeze,
delete the environment entirely, and rebuild it from the file.
That delete-and-rebuild step is the one that proves the file is
complete, and it is the step everyone skips until the day a
colleague cannot run their code.