I/O

The principal I/O abstraction is the io.Reader and io.Writer interfaces. The unification admits substantial flexibility — files, network connections, byte buffers, compressed streams, encrypted streams, gzip pipelines — all share the same byte-oriented interface. The bufio package admits buffered I/O for line-based reading and substantial output. The os package provides file and process I/O; net/http provides the conventional HTTP surface; encoding/json and similar packages admit serialisation. The combination — Reader/Writer as the foundation, buffered wrappers for efficiency, the os and net/http packages for the conventional sources — covers the I/O surface.

The Reader and Writer interfaces

The two foundational interfaces:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

The semantics:

• Read(p) fills p with up to len(p) bytes; returns the number read and any error. io.EOF indicates end of stream; n > 0 may be returned together with an error.
• Write(p) writes len(p) bytes from p; returns the number written and any error. A short write is reported as an error.

Implementing types:

• *os.File — files (read and write).
• *bytes.Buffer — in-memory byte buffer.
• *strings.Reader — read from a string.
• *bytes.Reader — read from a []byte.
• *bufio.Reader, *bufio.Writer — buffered wrappers.
• *gzip.Reader, *gzip.Writer — compression.
• *tls.Conn, net.Conn — network connections.
• *http.Request.Body, *http.Response.Body — HTTP message bodies.

The conventional discipline is to take io.Reader or io.Writer parameters for substantial flexibility:

func Process(r io.Reader) error {
    /* works for files, network, strings, buffers, compressed streams, etc. */
}

Standard streams

import "os"

os.Stdin                                          // *os.File for stdin
os.Stdout
os.Stderr

// Print to stdout:
fmt.Println("hello")

// Print to stderr:
fmt.Fprintln(os.Stderr, "error message")

Reading

Whole-file read

import "os"

data, err := os.ReadFile("file.txt")             // []byte (Go 1.16+)
if err != nil {
    return err
}

contents := string(data)

The os.ReadFile (since Go 1.16) replaces the older ioutil.ReadFile. Conventional for small to medium files.

Streaming read

import "os"

f, err := os.Open("file.txt")
if err != nil {
    return err
}
defer f.Close()

buf := make([]byte, 1024)
for {
    n, err := f.Read(buf)
    if n > 0 {
        process(buf[:n])
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
}

The pattern admits processing files larger than memory.

Buffered line-based read

The conventional Go pattern for line-based input:

import (
    "bufio"
    "os"
)

f, err := os.Open("file.txt")
if err != nil {
    return err
}
defer f.Close()

scanner := bufio.NewScanner(f)
for scanner.Scan() {
    line := scanner.Text()
    process(line)
}
if err := scanner.Err(); err != nil {
    return err
}

The Scanner is the conventional choice for “read line by line”; it handles buffering internally.

A subtlety: Scanner has a default buffer limit (64 KiB by default). For long lines, bufio.Reader is the alternative:

reader := bufio.NewReader(f)
for {
    line, err := reader.ReadString('\n')
    if err != nil {
        if err == io.EOF {
            break
        }
        return err
    }
    process(strings.TrimRight(line, "\n"))
}

Read all

data, err := io.ReadAll(reader)                   // (Go 1.16+)

The io.ReadAll reads until EOF; conventional when the size is unknown but expected to be modest.

Read into a struct

For fixed-format binary data, encoding/binary:

import "encoding/binary"

var header struct {
    Magic   uint32
    Version uint16
    Flags   uint16
}

err := binary.Read(reader, binary.LittleEndian, &header)

Writing

Whole-file write

err := os.WriteFile("output.txt", []byte("hello\n"), 0644)

The os.WriteFile (Go 1.16+) replaces the older ioutil.WriteFile. Atomic in the sense that it creates a new file (with truncation if existing).

Streaming write

f, err := os.Create("output.txt")
if err != nil {
    return err
}
defer f.Close()

f.WriteString("hello\n")
fmt.Fprintln(f, "world")
fmt.Fprintf(f, "count: %d\n", n)

Buffered write

For substantial output, bufio.Writer:

import "bufio"

f, _ := os.Create("output.txt")
defer f.Close()

w := bufio.NewWriter(f)
defer w.Flush()                                   // ensures everything is written

for _, line := range lines {
    fmt.Fprintln(w, line)
}

The bufio.Writer accumulates writes; without Flush, data may remain in the buffer when the function returns. The conventional discipline is defer w.Flush() immediately after construction — though that defers Flush errors. For error checking, explicit Flush is conventional:

w := bufio.NewWriter(f)
for _, line := range lines {
    fmt.Fprintln(w, line)
}
if err := w.Flush(); err != nil {
    return err
}

Append to a file

f, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
    return err
}
defer f.Close()

fmt.Fprintln(f, "new entry")

File operations

import "os"

// Open / Create:
f, err := os.Open("file.txt")                    // read-only
f, err := os.Create("file.txt")                  // write, truncate
f, err := os.OpenFile("file.txt", os.O_RDWR|os.O_CREATE, 0644)

// Information:
info, err := os.Stat("file.txt")
fmt.Println(info.Size())
fmt.Println(info.ModTime())
fmt.Println(info.IsDir())

// Operations:
os.Remove("file.txt")
os.Rename("old.txt", "new.txt")
os.Mkdir("dir", 0755)
os.MkdirAll("a/b/c", 0755)
os.RemoveAll("dir")

// Copy:
src, _ := os.Open("source.txt")
defer src.Close()
dst, _ := os.Create("dest.txt")
defer dst.Close()
n, err := io.Copy(dst, src)

The io.Copy admits streaming copy without buffering the entire source.

Directory traversal

import (
    "os"
    "path/filepath"
)

entries, err := os.ReadDir(".")                   // (Go 1.16+)
for _, entry := range entries {
    fmt.Println(entry.Name(), entry.IsDir())
}

// Recursive:
err := filepath.WalkDir(".", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }
    fmt.Println(path)
    return nil
})

The WalkDir (Go 1.16+) is more efficient than the older Walk.

In-memory I/O

The bytes and strings packages admit treating in-memory data as Reader/Writer:

import (
    "bytes"
    "strings"
)

// Read from a string:
r := strings.NewReader("hello world")
buf := make([]byte, 5)
r.Read(buf)                                       // "hello"

// Read from a byte slice:
r := bytes.NewReader([]byte{1, 2, 3, 4, 5})

// Write to an in-memory buffer:
var b bytes.Buffer
b.WriteString("hello")
b.WriteByte(' ')
fmt.Fprintf(&b, "%d", 42)
result := b.String()                              // "hello 42"

// Or build a string with strings.Builder:
var sb strings.Builder
sb.WriteString("hello")
fmt.Fprintf(&sb, " %d", 42)
result := sb.String()

The conventional choice:

• strings.Builder for building strings.
• bytes.Buffer for building byte slices.
• bytes.NewReader/strings.NewReader for reading from in-memory data.

JSON I/O

The encoding/json package admits structured serialisation:

import "encoding/json"

type User struct {
    Name  string `json:"name"`
    Email string `json:"email,omitempty"`
    Age   int    `json:"age"`
}

// Marshal:
data, err := json.Marshal(user)
data, err := json.MarshalIndent(user, "", "  ")  // pretty-printed

// Unmarshal:
var u User
err := json.Unmarshal(data, &u)

// Streaming (preferred for files):
file, _ := os.Create("user.json")
defer file.Close()

encoder := json.NewEncoder(file)
encoder.SetIndent("", "  ")
encoder.Encode(user)

file, _ := os.Open("user.json")
defer file.Close()

decoder := json.NewDecoder(file)
err := decoder.Decode(&u)

For unknown structure, decode into map[string]interface{} or interface{}:

var data interface{}
json.Unmarshal(rawJSON, &data)

switch v := data.(type) {
case map[string]interface{}:
    fmt.Println("object")
case []interface{}:
    fmt.Println("array")
case string:
    fmt.Println("string")
}

Network I/O

import "net"

// TCP client:
conn, err := net.Dial("tcp", "example.com:80")
if err != nil {
    return err
}
defer conn.Close()

fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n")
data, _ := io.ReadAll(conn)

// TCP server:
listener, err := net.Listen("tcp", ":8080")
if err != nil {
    return err
}
for {
    conn, err := listener.Accept()
    if err != nil {
        continue
    }
    go handle(conn)
}

func handle(conn net.Conn) {
    defer conn.Close()
    buf := make([]byte, 1024)
    n, _ := conn.Read(buf)
    conn.Write(buf[:n])
}

net.Conn implements both io.Reader and io.Writer, admitting use with bufio and io.Copy.

HTTP I/O

import (
    "io"
    "net/http"
)

// Client:
resp, err := http.Get("https://example.com")
if err != nil {
    return err
}
defer resp.Body.Close()

body, err := io.ReadAll(resp.Body)
fmt.Println(string(body))

// Server:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello, world!")
})

http.ListenAndServe(":8080", nil)

The http.ResponseWriter is an io.Writer; r.Body is an io.ReadCloser.

Compression

import "compress/gzip"

// Compress:
f, _ := os.Create("output.gz")
defer f.Close()

w := gzip.NewWriter(f)
defer w.Close()

w.Write(data)

// Decompress:
f, _ := os.Open("input.gz")
defer f.Close()

r, _ := gzip.NewReader(f)
defer r.Close()

data, _ := io.ReadAll(r)

Common patterns

Read all lines from stdin

scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
    process(scanner.Text())
}
if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

Read a configuration file

data, err := os.ReadFile("config.json")
if err != nil {
    return err
}

var config Config
if err := json.Unmarshal(data, &config); err != nil {
    return fmt.Errorf("parse config: %w", err)
}

Atomic file write

func atomicWrite(path string, data []byte) error {
    tmp := path + ".tmp"
    if err := os.WriteFile(tmp, data, 0644); err != nil {
        return err
    }
    return os.Rename(tmp, path)
}

The pattern admits “either the new content is fully written or the old file remains”.

Copy with progress

type ProgressWriter struct {
    Total uint64
    Last  time.Time
}

func (p *ProgressWriter) Write(b []byte) (int, error) {
    p.Total += uint64(len(b))
    if time.Since(p.Last) > 100*time.Millisecond {
        fmt.Printf("\rcopied %d bytes", p.Total)
        p.Last = time.Now()
    }
    return len(b), nil
}

src, _ := os.Open("source")
defer src.Close()
dst, _ := os.Create("dest")
defer dst.Close()

io.Copy(io.MultiWriter(dst, &ProgressWriter{}), src)

The io.MultiWriter admits writing to multiple writers simultaneously.

Reading a CSV

import "encoding/csv"

f, _ := os.Open("data.csv")
defer f.Close()

reader := csv.NewReader(f)
records, _ := reader.ReadAll()

for _, record := range records {
    process(record)
}

Pipe between two streams

r, w := io.Pipe()

go func() {
    defer w.Close()
    fmt.Fprintln(w, "hello")
    fmt.Fprintln(w, "world")
}()

io.Copy(os.Stdout, r)

The io.Pipe admits a synchronous in-memory pipe between writer and reader.

Limited reader

limited := io.LimitReader(reader, 1024)            // read at most 1024 bytes
data, _ := io.ReadAll(limited)

Tee reader

var captured bytes.Buffer
tee := io.TeeReader(reader, &captured)

processor.Process(tee)                             // reads from reader

// captured now has a copy of everything that was read

HTTP file download

resp, err := http.Get(url)
if err != nil {
    return err
}
defer resp.Body.Close()

f, err := os.Create(filename)
if err != nil {
    return err
}
defer f.Close()

n, err := io.Copy(f, resp.Body)

The io.Copy streams the download to disk without buffering the whole response.

Reading process output

import "os/exec"

cmd := exec.Command("ls", "-la")
out, err := cmd.Output()
if err != nil {
    return err
}
fmt.Println(string(out))

// Streaming:
cmd := exec.Command("ls", "-la")
stdout, _ := cmd.StdoutPipe()
cmd.Start()

scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

cmd.Wait()

A note on the conventional discipline

The contemporary Go I/O advice:

• Take io.Reader and io.Writer parameters for flexibility.
• Use os.ReadFile and os.WriteFile for whole-file I/O.
• Use bufio.Scanner for line-based input.
• Use bufio.NewReader/Writer for streaming I/O.
• Use defer file.Close() immediately after opening.
• Always Flush bufio.Writer before closing.
• Use io.Copy for streaming copies.
• Use encoding/json with structs for JSON.
• Use strings.Builder for building strings.
• Use bytes.Buffer for in-memory byte construction.
• Check errors at every step — I/O is the conventional source of errors.

The combination — Reader/Writer as the foundation, buffered wrappers for efficiency, encoding/json and similar for structured data, the os/net/net/http packages for OS-level sources, in-memory I/O via bytes and strings — is the substance of Go’s I/O surface. The discipline produces flexible, composable, testable I/O code.