Python § modules

Packaging and the build

Python’s packaging — the distribution, dependency-management, and build story — is famously elaborate. The conventional contemporary toolchain is built on pip (the package installer), venv (virtual environments), and pyproject.toml (the standard configuration format). Modern tools — uv, poetry, pdm, hatch, rye — provide higher-level workflows that admit project-aware dependency resolution, lockfiles, and reproducible builds. The official package registry is PyPI (the Python Package Index); packages are distributed as wheels (.whl, the conventional binary format) or source distributions (.tar.gz).

This page covers the package model, the conventional contemporary tooling, the pyproject.toml format, virtual environments, and distribution. The treatment of modules within a project (import, packages, __init__.py) is in Modules.

The package model

A package is a unit of distribution. A package contains:

One or more modules — Python source files.
Metadata — name, version, author, dependencies, license.
Optionally: extension modules (compiled C code), data files, console scripts.

The conventional layout:

mypackage/
├── pyproject.toml           # the build configuration
├── README.md
├── LICENSE
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── core.py
│       └── util.py
└── tests/
    └── test_core.py

The src/ layout (with the package nested one level under) is the conventional contemporary form; it prevents accidental imports from the project root and forces explicit installation.

`pyproject.toml`

The conventional contemporary configuration format (PEP 518, PEP 621):

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
description = "A short description"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [
    { name = "Alice Author", email = "alice@example.com" }
]
dependencies = [
    "requests>=2.31",
    "click>=8.1",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "mypy>=1.0",
    "ruff>=0.1",
]

[project.scripts]
mycli = "mypackage.cli:main"

[project.urls]
Homepage = "https://example.com"
Repository = "https://github.com/user/mypackage"

The principal sections:

[build-system] — declares the build backend (setuptools, hatchling, poetry-core, pdm-backend).
[project] — the package metadata (PEP 621).
[project.optional-dependencies] — extras that may be installed via pip install mypackage[dev].
[project.scripts] — console scripts that the install creates.
[tool.<name>] — tool-specific configuration (ruff, mypy, pytest, etc.).

The pyproject.toml is the conventional contemporary configuration; older setup.py and setup.cfg are still supported but increasingly uncommon in new projects.

`pip`

pip is the conventional package installer:

pip install requests                          # install latest
pip install requests==2.31.0                   # specific version
pip install "requests>=2.31,<3"               # version constraint
pip install -e .                               # editable install of current project
pip install -r requirements.txt                 # from a requirements file
pip install ".[dev]"                            # current project with extras

pip uninstall requests
pip list                                        # installed packages
pip show requests                               # details about a package
pip freeze                                      # produce requirements.txt

The -e (“editable”) install creates a link to the source rather than copying; changes to the source are immediately visible without reinstalling.

The requirements.txt (legacy convention):

requests>=2.31
click>=8.1
flask>=3.0

For modern projects, dependencies are declared in pyproject.toml’s [project.dependencies] section; the requirements file remains for compatibility and certain workflows.

Virtual environments

A virtual environment is an isolated Python installation with its own packages:

python -m venv .venv          # create
source .venv/bin/activate     # activate (Linux/Mac)
.venv\Scripts\activate         # activate (Windows)

pip install -e .                # install the project in this environment
pip install pytest

deactivate                      # exit the environment

The convention is one virtual environment per project. The .venv/ directory holds the environment; conventionally added to .gitignore.

Virtual environments isolate packages — pip install modifies the active environment, not the system Python. The mechanism prevents cross-project conflicts (project A wants requests==2.20; project B wants requests==2.31).

Modern tooling

Several higher-level tools provide better workflows than pip + venv:

`uv`

The fastest contemporary Python package manager (written in Rust):

uv venv                          # create a virtual environment
uv pip install requests          # install
uv pip install -e .              # editable install

uv add requests                   # add to pyproject.toml and install
uv remove requests                # remove
uv sync                           # sync environment with pyproject.toml + lockfile
uv lock                           # update lockfile

uv run script.py                  # run with the project's environment
uv run pytest                     # run tests

uv is substantially faster than pip (10-100x in many cases), produces a lockfile (uv.lock), and admits rye-style project workflows. The conventional contemporary choice for new projects targeting modern tooling.

`poetry`

A widely-used project manager:

poetry new mypackage              # create a new project
poetry add requests                # add a dependency
poetry remove requests
poetry install                     # install from pyproject.toml + lockfile
poetry update                      # update lockfile

poetry run python script.py
poetry shell                       # enter the venv

Poetry has its own metadata format (in pyproject.toml under [tool.poetry]); recently it adopted the standard PEP 621 format too. The lockfile is poetry.lock.

`pdm`

Similar to Poetry but uses standard PEP 621 metadata throughout:

pdm init
pdm add requests
pdm install
pdm run python script.py

Lockfile: pdm.lock.

`hatch`

Project management with a focus on building and publishing:

hatch new mypackage
hatch shell                        # enter the environment
hatch run pytest
hatch build                        # build distributions
hatch publish                      # upload to PyPI

`rye`

A Rust-based all-in-one (similar to uv); the project that motivated uv’s creation. Largely superseded by uv.

The conventional contemporary choice depends on preference:

uv for speed and modern PEP 621-based workflows.
poetry for stability and a mature ecosystem.
pdm for standards-conformant tooling.
hatch for a balanced workflow that prioritises building and publishing.
pip + venv for the canonical, no-extra-dependency approach.

Lockfiles

A lockfile records the exact versions of every direct and transitive dependency, ensuring reproducible installs across machines:

uv.lock — uv’s lockfile.
poetry.lock — Poetry’s lockfile.
pdm.lock — PDM’s lockfile.
requirements.txt (with pinned versions, often produced by pip freeze) — the legacy form.

The conventional contemporary discipline is to commit the lockfile to version control; the lockfile pinpoints the dependency graph at install time.

For applications, locked dependencies are conventional. For libraries, the conventional advice is to specify version ranges in pyproject.toml (e.g., requests>=2.31,<3) and let downstream consumers’ lockfiles do the pinning.

Distribution

To publish a package:

Build the distributions:

python -m build                    # produces dist/*.whl and dist/*.tar.gz
# Or with hatch:
hatch build
# Or with poetry:
poetry build
# Or with uv:
uv build

Upload to PyPI:

python -m pip install twine
twine upload dist/*

# Or with hatch:
hatch publish

# Or with poetry:
poetry publish

Test uploads first to TestPyPI:

twine upload --repository testpypi dist/*

The pypi.org and test.pypi.org are separate registries.

For private distribution, internal package indexes (DevPI, simpleindex, GitHub Packages) admit hosting packages that are not on public PyPI.

The `init.py` and packages

A package is a directory with __init__.py:

mypackage/
├── __init__.py
├── core.py
└── sub/
    ├── __init__.py
    └── deeper.py

The __init__.py may be empty or may re-export selected names:

# mypackage/__init__.py

from .core import API, run
from .sub.deeper import helper

__version__ = "1.0.0"
__all__ = ["API", "run", "helper"]

Treated in Modules.

For packages without __init__.py (PEP 420 namespace packages), the directory contents are still importable but the package is not a single regular module. The conventional contemporary advice is to use __init__.py for regular packages; namespace packages are appropriate only for plugin systems.

Wheels and source distributions

Two distribution formats:

Wheel (.whl) — a zip archive of pre-built (architecture-specific or pure-Python) package contents. Fast to install; no build step required.
Source distribution (.tar.gz or sdist) — an archive of the source. Requires the build backend at install time.

For pure-Python packages, the wheel and the sdist contain similar content (the wheel admits installing without running the build). For packages with C extensions, the wheel is the conventional form for binary distribution; sdists require the user to have a C compiler.

The conventional contemporary advice is to publish both: wheels (one per supported platform) and an sdist for the source. hatch build, uv build, and python -m build produce both by default.

C extensions and compiled code

For packages with compiled code:

C extensions — .c files compiled by setuptools to .so (Linux/Mac) or .pyd (Windows).
Cython — Python-like syntax compiled to C.
PyO3 — Rust-based extensions; popular in modern projects.
maturin — the build backend for Rust extensions; integrates with pyproject.toml.
scikit-build — CMake-based extension building.

The conventional uses are performance-critical numeric work (NumPy, SciPy, pandas), bindings to existing C/Rust libraries (cryptography, pillow, lxml), and (increasingly) modern projects that use Rust extensions for safety and speed.

`pycache` and `.pyc`

Python caches compiled bytecode in __pycache__/:

mypackage/
├── __init__.py
├── __pycache__/
│   ├── __init__.cpython-312.pyc
│   └── core.cpython-312.pyc
└── core.py

The .pyc files are produced automatically and re-used as long as the source’s modification time is unchanged. The conventional .gitignore:

__pycache__/
*.pyc
*.pyo

The cache is a CPython optimisation; the program runs the same with or without it.

Common patterns

A modern Python project layout

mypackage/
├── pyproject.toml
├── README.md
├── LICENSE
├── .gitignore
├── .python-version            # for pyenv users
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── core.py
│       └── cli.py
└── tests/
    ├── __init__.py
    └── test_core.py

The src/ layout is the conventional contemporary choice; it prevents the project’s source from being importable via the project root (forcing the user to install the package or use pip install -e .).

Using `uv` for a new project

uv init mypackage              # creates pyproject.toml and src/
cd mypackage
uv add requests click           # adds dependencies
uv add --dev pytest mypy ruff   # adds dev dependencies
uv run pytest                   # runs in the project environment

Console scripts

Declared in pyproject.toml:

[project.scripts]
mycli = "mypackage.cli:main"

After install, a mycli command is on the path; calls mypackage.cli.main().

Versioning

# mypackage/__init__.py
__version__ = "1.0.0"

Or use importlib.metadata:

import importlib.metadata
__version__ = importlib.metadata.version("mypackage")

The latter avoids duplicating the version across pyproject.toml and the source.

Optional dependencies

[project.optional-dependencies]
plot = ["matplotlib>=3.0"]
fast = ["numpy>=1.20", "numba>=0.55"]
all = ["mypackage[plot,fast]"]

pip install mypackage             # core
pip install mypackage[plot]        # core + plot
pip install mypackage[all]         # everything

The pattern admits “lightweight default with optional features”.

Development dependencies

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov",
    "mypy>=1.0",
    "ruff>=0.1",
    "pre-commit",
]

pip install -e ".[dev]"

The pattern admits installing development tools alongside the project.

Tool configuration

[tool.ruff]
line-length = 100
select = ["E", "F", "W", "I", "N", "UP"]

[tool.mypy]
python_version = "3.12"
strict = true

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --cov=mypackage"

Most modern Python tools (ruff, mypy, pytest, black, isort) consume their configuration from pyproject.toml.

Build backends

The [build-system] section declares the build backend:

Backend	Notes
`setuptools`	The original; widely supported, somewhat verbose
`hatchling`	Modern, simpler, used by `hatch`
`flit-core`	Minimal; for pure-Python packages
`pdm-backend`	Used by `pdm`
`poetry-core`	Used by Poetry
`maturin`	For Rust extensions
`scikit-build-core`	For CMake-based builds

For pure-Python packages, the choice is largely cosmetic; hatchling is the conventional contemporary choice for new projects. For C/Rust extensions, the choice depends on the extension build system.

A note on the broader ecosystem

The Python ecosystem has substantial breadth:

Web frameworks: Flask, FastAPI, Django, Litestar.
Data science: NumPy, pandas, scipy, scikit-learn.
Plotting: matplotlib, seaborn, plotly.
HTTP: requests, httpx, aiohttp.
Async: asyncio, anyio, trio.
CLI: click, typer, argparse.
Validation: pydantic, attrs, marshmallow.
Testing: pytest, unittest, hypothesis.
Linting/formatting: ruff, black, mypy, pyright.
Logging: logging (stdlib), loguru, structlog.

Most modern projects use the conventional standard tooling — uv or poetry for environment, ruff for linting and formatting, mypy or pyright for type checking, pytest for testing — plus the appropriate domain-specific libraries.

A note on the discipline

The contemporary Python packaging advice:

Use pyproject.toml — the standard format for new projects.
Use a modern tool — uv, poetry, pdm, or hatch; avoid the bare pip + venv approach for non-trivial projects.
Use a lockfile and commit it to version control.
Use the src/ layout for non-trivial projects.
Use editable installs (pip install -e . or uv sync) for development.
Pin Python version with .python-version (pyenv) or in pyproject.toml (requires-python).
Test with multiple Python versions using tox or nox for libraries that support a range.

The combination — modern tooling, standard configuration, lockfile-based reproducibility — admits Python projects that are reproducible across machines and have well-managed dependencies. The cost is the substantial number of choices to make; the benefit is a development workflow that is increasingly comparable to that of more rigorously-tooled languages.