Python § io

I/O

Python’s I/O facilities are built on the open() builtin (which produces file objects), the pathlib module (the modern path API), and the io module (the lower-level abstractions). The conventional contemporary form for file I/O is pathlib.Path plus with open(path) as f:; the resulting file objects support iteration, line-by-line reading, and binary or text modes. For network I/O, the standard library’s urllib is workable but the third-party requests and httpx are conventionally preferable. For asynchronous I/O, asyncio (treated in Async and concurrency) admits high-throughput I/O on a single thread.

This page covers file I/O, pathlib, the io module’s abstractions, network I/O briefly, and the conventional patterns. The treatment of bytes/text encoding is in Strings.

`open()` and file objects

The open() builtin returns a file object:

f = open("input.txt", "r")        # text mode, read
contents = f.read()
f.close()

The conventional form uses with for automatic cleanup:

with open("input.txt", "r") as f:
    contents = f.read()
# f is closed automatically, even on exception

The open() parameters:

path — the file path (a str, bytes, or Path).
mode — the open mode:
- "r" — read text (default).
- "w" — write text; truncate or create.
- "a" — append text.
- "x" — write text; fail if the file exists.
- "r+" / "w+" — read and write.
- "b" suffix — binary mode ("rb", "wb", etc.).
- "t" suffix — text mode (default).
encoding — for text mode; "utf-8" is the conventional contemporary choice.
errors — error handling for encoding ("strict", "replace", "ignore").
newline — line-ending handling (None, "", "\n", "\r", "\r\n").
buffering — buffer size; -1 for default.

# Conventional contemporary form:
with open(path, "r", encoding="utf-8") as f:
    contents = f.read()

with open(path, "w", encoding="utf-8") as f:
    f.write("hello\n")

with open(path, "rb") as f:
    bytes_data = f.read()

The conventional discipline is to specify the encoding explicitly (avoiding surprises from the system default).

File-object methods

The principal methods:

Method	Effect
`f.read(n)`	Read up to `n` chars/bytes (or all if `n` omitted).
`f.readline()`	Read one line (including the newline).
`f.readlines()`	Read all lines into a list.
`f.write(s)`	Write a string or bytes.
`f.writelines(lines)`	Write each line; does not add newlines.
`f.seek(pos, whence)`	Seek to position.
`f.tell()`	Current position.
`f.flush()`	Flush buffers.
`f.close()`	Close the file.
`f.fileno()`	Underlying OS file descriptor.
`f.readable()`, `f.writable()`, `f.seekable()`	Capability queries.
`iter(f)`	Iterate line by line.

The conventional pattern for reading line-by-line:

with open(path) as f:
    for line in f:
        process(line.rstrip())            # strip the trailing newline

The for line in f: form iterates lazily — the entire file is not loaded at once. The rstrip() removes the trailing newline (and other whitespace).

`pathlib`

The conventional contemporary path API:

from pathlib import Path

p = Path("/var/log/app.log")
p.exists()                                 # True/False
p.is_file()                                 # bool
p.is_dir()                                  # bool
p.stat().st_size                            # bytes
p.stat().st_mtime                           # modification time

# Construction:
p = Path("/var") / "log" / "app.log"        # path joining
p = Path.home() / ".config" / "app"
p = Path.cwd()                              # current directory

# Reading and writing:
text = p.read_text(encoding="utf-8")
data = p.read_bytes()
p.write_text("contents", encoding="utf-8")
p.write_bytes(b"\x00\x01\x02")

# Components:
p.name                                       # filename with extension
p.stem                                       # filename without extension
p.suffix                                      # extension
p.suffixes                                    # all extensions ([.tar, .gz])
p.parent                                      # parent directory
p.parents                                      # all ancestors

# Manipulation:
p.with_suffix(".bak")                        # change extension
p.with_name("new_name.txt")                   # change filename
p.with_stem("new_stem")                       # change stem (3.9+)

# Iteration:
for f in Path(".").iterdir():
    print(f)

for f in Path(".").glob("*.py"):
    print(f)

for f in Path(".").rglob("*.py"):             # recursive
    print(f)

# Operations:
p.mkdir(parents=True, exist_ok=True)
p.rmdir()                                     # only if empty
p.unlink()                                    # delete file
p.rename(new_path)
p.replace(new_path)                            # like rename, replaces existing

# Resolution:
p.absolute()                                   # absolute path
p.resolve()                                    # absolute + resolve symlinks
p.relative_to(other)                            # relative path

# Tests:
p.match("*.py")                                # glob match
p.is_absolute()
p.is_relative_to(other)

The Path class is the conventional contemporary path-handling API; substantially clearer than the older os.path functions.

The `io` module

The io module provides the underlying abstractions:

Class	Purpose
`BytesIO`	Binary in-memory file
`StringIO`	Text in-memory file
`BufferedReader`, `BufferedWriter`	Binary buffered file
`TextIOWrapper`	Text wrapper around binary file

from io import BytesIO, StringIO

# In-memory binary:
buf = BytesIO()
buf.write(b"hello")
buf.seek(0)
buf.read()                                   # b"hello"

# In-memory text:
sbuf = StringIO()
sbuf.write("line 1\n")
sbuf.write("line 2\n")
sbuf.getvalue()                              # "line 1\nline 2\n"

The in-memory variants are the conventional Python form for testing file-handling code without touching the filesystem.

Common patterns

Read entire file

with open(path, encoding="utf-8") as f:
    contents = f.read()

# Or:
contents = Path(path).read_text(encoding="utf-8")

The Path.read_text form is the conventional contemporary choice.

Read lines

# Lazy (line-by-line):
with open(path) as f:
    for line in f:
        process(line.rstrip())

# Eager (all into a list):
with open(path) as f:
    lines = f.readlines()

# Or:
lines = Path(path).read_text().splitlines()

For very large files, the lazy form is preferable.

Write entire file

with open(path, "w", encoding="utf-8") as f:
    f.write(contents)

# Or:
Path(path).write_text(contents, encoding="utf-8")

Append to file

with open(path, "a", encoding="utf-8") as f:
    f.write("additional content\n")

The "a" mode appends to the existing content.

Atomic file replacement

import tempfile
import os
from pathlib import Path

def atomic_write(path: Path, content: str):
    tmp = path.parent / f".{path.name}.tmp"
    tmp.write_text(content)
    tmp.replace(path)            # atomic rename

The pattern admits writing the new contents to a temporary file and then renaming over the original; either the new file or the old, never a half-written file.

Read fixed-size records

RECORD_SIZE = 64

with open(path, "rb") as f:
    while record := f.read(RECORD_SIZE):
        if len(record) < RECORD_SIZE:
            break        # incomplete record at EOF
        process(record)

The walrus operator (:=) admits the read-and-test pattern in the loop condition.

Streaming a large file

def stream_lines(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip()

# The caller iterates without loading the whole file:
for line in stream_lines("huge.txt"):
    process(line)
    if should_stop():
        break

The generator-based form admits processing arbitrarily-large files in constant memory.

Glob and find

from pathlib import Path

# Match files with a pattern:
for f in Path(".").rglob("*.py"):
    print(f)

# Filter:
for f in Path(".").rglob("*.py"):
    if "test" in f.name:
        continue
    process(f)

The rglob is recursive; glob is single-level.

Temporary files

import tempfile

# Temporary file:
with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
    f.write("contents")
    path = f.name
# the file persists; use path

# Auto-cleaned temporary file:
with tempfile.NamedTemporaryFile(mode="w") as f:
    f.write("contents")
    path = f.name
    process(path)
# file deleted at the end of with

# Temporary directory:
with tempfile.TemporaryDirectory() as tmpdir:
    work_in(tmpdir)
# directory and contents deleted

The tempfile module admits temporary files and directories with automatic cleanup.

Reading binary structured data

import struct

with open("data.bin", "rb") as f:
    # Read a header: 32-bit magic + 16-bit version + 16-bit flags
    header = f.read(8)
    magic, version, flags = struct.unpack(">IHH", header)
    # Read records (24-byte each):
    while True:
        record = f.read(24)
        if not record:
            break
        # Parse the record...

The struct module admits packing and unpacking binary data with format strings (similar to C’s printf-style format).

Compressed I/O

import gzip
import json

# Read a gzipped JSON file:
with gzip.open("data.json.gz", "rt", encoding="utf-8") as f:
    data = json.load(f)

# Write:
with gzip.open("data.json.gz", "wt", encoding="utf-8") as f:
    json.dump(data, f)

The gzip, bz2, lzma, zipfile modules admit working with compressed files. They share the file-object interface.

Network I/O

For network I/O, the standard library provides:

socket — low-level TCP/UDP.
urllib.request — HTTP client (workable but limited).
http.client — lower-level HTTP.
http.server — basic HTTP server.
asyncio — async networking.

For non-trivial HTTP, the conventional third-party choice is:

import requests        # synchronous, the most popular

response = requests.get("https://api.example.com/data")
response.raise_for_status()
data = response.json()

# POST:
response = requests.post(
    "https://api.example.com/items",
    json={"name": "alice"}
)

# Sessions for connection pooling:
with requests.Session() as session:
    session.headers.update({"User-Agent": "MyApp"})
    for url in urls:
        r = session.get(url)
        process(r.json())

For modern async-capable HTTP:

import httpx

# Sync:
response = httpx.get("https://api.example.com/data")

# Async:
async with httpx.AsyncClient() as client:
    response = await client.get("https://api.example.com/data")

httpx is the conventional contemporary choice for new code; it provides both sync and async APIs with the same interface.

Async I/O

For high-concurrency I/O, asyncio admits cooperative concurrency:

import asyncio
import aiofiles

async def read_file(path):
    async with aiofiles.open(path) as f:
        return await f.read()

async def main():
    contents = await read_file("data.txt")
    process(contents)

The aiofiles (third-party) provides async file I/O. The treatment is in Async and concurrency.

Encoding considerations

Text I/O requires an encoding; the conventional contemporary choice is UTF-8:

with open(path, "r", encoding="utf-8") as f:
    contents = f.read()

Without an explicit encoding, Python uses the system default — which varies by platform and locale, producing surprising behaviour. The conventional discipline is to specify the encoding explicitly.

For files that may contain invalid UTF-8, the errors argument:

with open(path, encoding="utf-8", errors="replace") as f:
    contents = f.read()      # invalid bytes become U+FFFD

The options are:

"strict" — raise UnicodeDecodeError (default).
"ignore" — silently drop invalid bytes.
"replace" — substitute with U+FFFD.
"backslashreplace" — substitute with \xNN escapes.

For binary data, no encoding is needed:

with open(path, "rb") as f:
    bytes_data = f.read()

The treatment is in Strings.

Beyond pathlib:

os.path — the older path module; still available.
glob — file-name pattern matching (pathlib.Path.glob is preferable).
fnmatch — Unix-style filename matching.
tempfile — temporary files and directories.
shutil — high-level file operations (copy, remove tree).

import shutil
from pathlib import Path

# Copy:
shutil.copy("source.txt", "dest.txt")
shutil.copy2(src, dst)               # also copies metadata
shutil.copytree("src_dir", "dst_dir")

# Move:
shutil.move("source.txt", "dest.txt")

# Remove:
shutil.rmtree("dir_to_remove")        # recursive
Path("file.txt").unlink()             # single file

# Disk usage:
shutil.disk_usage("/")

The shutil provides high-level file-system operations not covered by pathlib directly.

`select` and async I/O

For low-level async I/O:

import select

# Wait for any of multiple file descriptors:
ready, _, _ = select.select([sock1, sock2], [], [], timeout)
for fd in ready:
    data = fd.recv(1024)
    process(data)

select is the conventional Unix select-based polling. For non-trivial async I/O, asyncio is preferable.

Logging output

import logging

logger = logging.getLogger(__name__)

# Log to a file:
file_handler = logging.FileHandler("app.log")
file_handler.setFormatter(logging.Formatter(
    "%(asctime)s %(levelname)s %(name)s: %(message)s"
))
logger.addHandler(file_handler)
logger.setLevel(logging.INFO)

logger.info("starting")

The logging is the conventional Python form for application logging; preferable to print-based output for non-trivial code.

A note on the conventional discipline

The contemporary Python I/O advice:

Use pathlib for paths.
Specify encoding explicitly (encoding="utf-8").
Use with for resource cleanup.
Use Path.read_text/write_text for simple cases.
Iterate line-by-line for large text files.
Use requests or httpx for HTTP.
Use asyncio plus aiofiles for high-concurrency I/O.
Use tempfile for temporary files (admits automatic cleanup).
Use atomic rename for safe file writes.

The combination — pathlib for paths, explicit encoding, with-based cleanup, line-by-line streaming for large files — is the conventional contemporary Python I/O toolkit.

open() and file objects