Rust Standard Library Tutorials

A structured series on the Rust standard library, targeting Rust 1.99 (compiler 1.99.0, edition 2024). The material is a Prologue plus 26 domains — thematic areas that group related types, traits, functions, and macros by use case rather than by where they live in the std module hierarchy.

Each tutorial is a ~15-minute lesson. The sidebar is the table of contents: open a domain, then a chapter. Every chapter ends with a Code Examples table of runnable binaries.

The examples live in this GitHub repository and verify their own behavior with assert! / assert_eq! — cargo run exiting successfully is the test.

# Requires Rust >= 1.99
git clone https://github.com/feamcor/rust-standard-library-tutorials.git
cd rust-standard-library-tutorials

cargo build --workspace --bins
cargo run -p domain-01-ownership-mechanics --bin 01_01_copy_semantics

Nightly-only samples (unstable APIs in domains 2, 5, 6, 7, and 25) are skipped on stable. Build them with:

cargo +nightly build -p domain-02-error-handling --features nightly --bins

HTML comments like  mark slides for the in-repo presenter; you can ignore them while reading.

Start with P.1 · What the Prelude Imports and Why, or pick a domain in the sidebar.

Rust 1.99 Standard Library Tutorial Curriculum

Toolchain: Rust 1.99 (MSRV; compiler 1.99.0), edition 2024. Format: Each tutorial is designed as a lecture of 15 minutes (hard cap: 30 minutes). If a subject cannot fit under the cap, split it into multiple tutorials and renumber — there is no limit on tutorial count, only on per-tutorial duration. Tutorials are grouped by domain — a thematic area that crosses module boundaries, connecting related types, traits, functions, and macros by use case rather than by where they live in the std hierarchy.

Prologue — The Prelude and Implicit Imports

P.1 · What does the Prelude Import and Why?

What's covered?

Edition-dependent preludes: v1 (2015/2018), rust_2021, rust_2024.
What changed in 2021 (added TryFrom, TryInto, FromIterator).
What changed in 2024 (added Future, IntoFuture).
The prelude traits that for loops, the ? operator, collect, and .await use, and the names that need no explicit import.
Why understanding the prelude prevents "where does this trait come from?" confusion.

Learn this before Domain 1 if you have ever wondered why certain traits seem to appear from nowhere.

Library Components:

std::prelude::v1
std::prelude::rust_2021
std::prelude::rust_2024

Domain 1 — Ownership Mechanics in the Standard Library

1.1 · Clone, Copy, and Drop: The Lifecycle Trio

What's covered?

How Clone, Copy, and Drop interact.
When the compiler inserts implicit copies vs. when you must clone explicitly.
The relationship between Copy and Drop (mutual exclusion).
Shallow vs. deep cloning with clone_from.
Performance implications of Clone on collection types.

Library Components:

std::clone::Clone
std::marker::Copy
std::ops::Drop
clone_from

1.2 · Borrow, ToOwned, and Cow: Flexible Ownership Boundaries

What's covered?

The Borrow and BorrowMut traits as abstraction over ownership.
ToOwned for going from borrowed to owned.
Cow<'a, B> (clone-on-write) as the bridge that defers allocation.
Real use case: building APIs that accept both &str and String without forcing the caller to allocate.

Library Components:

std::borrow::Borrow
std::borrow::BorrowMut
std::borrow::ToOwned
std::borrow::Cow

1.3 · AsRef, AsMut, Into, From: The Conversion Matrix

What's covered?

When to use AsRef<T> vs. Borrow<T>.
The blanket implementations that make From/Into work.
TryFrom/TryInto for fallible conversions — including numeric range checks, impl TryFrom<char> for usize (stabilized in 1.94), and impl TryFrom<{integer}> for bool (stabilized in 1.95).
as casting behavior (truncation, sign extension) and when try_into() is safer.
Infallible as the error type for conversions that cannot fail and its relationship to the never type !.
convert::identity as a typed no-op useful in higher-order functions.
Designing public APIs with conversion traits to maximize ergonomics.

Library Components:

std::convert::AsRef
std::convert::AsMut
std::convert::From
std::convert::Into
std::convert::TryFrom
std::convert::TryInto
std::convert::Infallible
std::convert::identity

Domain 2 — Error Handling Ecosystem

2.1 · Result and Option: Beyond the Basics

What's covered?

The full combinator API on Result and Option — and_then, or_else, map_or, unwrap_or_default, transpose, flatten, inspect, zip, unzip.
Chaining results with ?.
The FromResidual trait behind ?.
When Option is more appropriate than Result and vice versa.
bool::ok_or / ok_or_else (stabilized in 1.98) for turning a predicate into Result<(), E>.

Library Components:

std::option::Option
std::result::Result

2.2 · The Error Trait and Error Propagation Patterns

What's covered?

Implementing std::error::Error.
The source() chain for error causes.
Display vs. Debug for error formatting.
Downcasting errors with downcast_ref.
Building custom error hierarchies without third-party crates.
The deprecated description() method and why it was superseded.
Error::provide for supplying typed context including backtraces (see 2.4 for backtrace capture mechanics).

Library Components:

std::error::Error
std::fmt::Display
std::fmt::Debug
<dyn Error>::downcast_ref / downcast_mut / Box<dyn Error>::downcast (inherent methods; the same TypeId technique as std::any::Any)

2.3 · Panic Infrastructure: Hooks, Unwind, and Abort

What's covered?

panic!, catch_unwind, resume_unwind, set_hook, take_hook.
PanicHookInfo (added in 1.81; the old name PanicInfo is deprecated since 1.82).
The difference between std::panic::PanicHookInfo and core::panic::PanicInfo.
UnwindSafe and RefUnwindSafe marker traits; AssertUnwindSafe and its From<T> constructor (stabilized in 1.96).
When catch_unwind is appropriate (FFI boundaries, thread pools).

Library Components:

std::panic
std::panic::catch_unwind
std::panic::set_hook
std::panic::PanicHookInfo
std::panic::UnwindSafe
std::panic::RefUnwindSafe
std::panic::AssertUnwindSafe

2.4 · Backtraces: Capturing Stack Traces

What's covered?

Backtrace::capture and Backtrace::force_capture.
The RUST_BACKTRACE environment variable.
BacktraceStatus — Unsupported, Disabled, Captured.
Embedding backtraces in custom error types via Error::provide.
When backtraces help (production debugging) and when they hurt (performance).
How this integrates with the source() chain from 2.2.

Library Components:

std::backtrace::Backtrace
std::backtrace::BacktraceStatus

Domain 3 — Strings and Text Processing

3.1 · String, &str, and the UTF-8 Contract

What's covered?

Internal representation of String and &str.
Why indexing by usize doesn't work.
char_indices, chars, bytes.
String slicing and the panic on non-boundary indices.
str::substr_range (1.98) to recover a substring's byte range by pointer arithmetic.
String::from_utf8 vs. from_utf8_unchecked.
as_bytes, into_bytes, from_utf8_lossy.
String::from_utf8_lossy_owned and FromUtf8Error::into_utf8_lossy (1.99) for lossy conversion of an owned Vec<u8> without a copy of valid input.
String::from_utf16le / from_utf16be and their lossy siblings (1.98) for decoding endian-tagged UTF-16 byte streams.

Library Components:

std::string::String
primitive str
std::str::from_utf8
std::str::Utf8Error
FromUtf8Error

3.2 · Formatting Mastery: fmt, write!, format_args!

What's covered?

The fmt module architecture — Display, Debug, Binary, Octal, LowerHex, UpperHex, LowerExp, UpperExp, Pointer.
The Formatter struct and its options (padding, alignment, precision, sign, fill).
format_args! as a zero-allocation formatting primitive.
{integer}::format_into with NumBuffer (1.98; re-exported from std::fmt since 1.99) for allocation-free decimal integer formatting.
Writing custom Display and Debug implementations.
DebugStruct, DebugTuple, DebugList, DebugMap, DebugSet builder helpers.

fmt::Write (for formatting to strings/buffers) and io::Write (for byte streams) are distinct traits — write! and writeln! work with both via separate blanket implementations; see 8.1 for the io::Write side.

Library Components:

std::fmt
std::fmt::NumBuffer
all formatting traits
std::fmt::Formatter
std::fmt::Arguments
write!
writeln!
format!
format_args!

3.3 · Pattern Matching on Strings and FromStr Parsing

What's covered?

The Pattern trait and what can be used as a string pattern (char, &str, &[char], closures).
Methods that use patterns: contains, starts_with, find, rfind, split, splitn, trim_matches, strip_prefix, strip_suffix, strip_circumfix (1.98 — prefix and suffix in one call).
FromStr for parsing arbitrary types from strings.
str::parse::<T>() ergonomics.

Library Components:

std::str::pattern::Pattern
std::str::FromStr
str methods

3.4 · OsString, CStr, CString: Non-UTF-8 String Types

What's covered?

Why OsStr/OsString exist (platform-dependent encoding).
Converting between OsStr, Path, str.
CStr and CString for C interop — null termination guarantees, as_ptr, from_ptr.
Comparison traits between CStr, CString, and Cow<CStr> (stabilized recently).
When to use each string type.

Library Components:

std::ffi::OsStr
std::ffi::OsString
std::ffi::CStr
std::ffi::CString
std::ffi::NulError
std::ffi::FromBytesWithNulError

3.5 · ASCII Operations and Character Classification

What's covered?

The std::ascii module.
char methods for classification (is_alphabetic, is_numeric, is_ascii_*, is_uppercase, etc.).
Case conversion with to_lowercase, to_uppercase (iterator-based because of Unicode).
make_ascii_lowercase, make_ascii_uppercase on [u8] and str.
EscapeDefault, EscapeDebug, EscapeUnicode iterators.

Library Components:

std::ascii
std::char
primitive char methods
char::from_digit
char::to_digit

Domain 4 — Collections Deep Dive

4.1 · `Vec<T>`: The Workhorse Collection

What's covered?

Internal layout (pointer, length, capacity).
Growth strategy.
with_capacity, reserve, shrink_to_fit, shrink_to.
retain, dedup, dedup_by, dedup_by_key.
drain, splice, split_off.
push_mut and insert_mut for getting &mut T to the just-inserted element (stabilized in 1.95).
into_boxed_slice.
into_parts and from_parts (1.99) to decompose a Vec into (NonNull<T>, length, capacity) and rebuild it.
Vec as a Deref<Target = [T]> — the slice method inheritance.
extend_from_slice vs. extend.

Library Components:

std::vec::Vec
std::vec::IntoIter
std::vec::Drain
std::vec::Splice

4.2 · HashMap and HashSet: Hashing Internals and Custom Hashers

What's covered?

SwissTable-based implementation (quadratic probing + SIMD lookup).
The Entry API (or_insert, or_insert_with, and_modify, or_default).
RandomState and BuildHasher.
Providing custom hashers.
The Hash trait and implementing it correctly (the hash/eq contract).
HashSet as HashMap<K, ()> — when sets beat maps.
For in-depth coverage of implementing custom Hasher state machines and BuildHasher, see 13.2.

Library Components:

std::collections::HashMap
std::collections::HashSet
std::collections::hash_map::Entry
std::hash::Hash
std::hash::Hasher
std::hash::BuildHasher
std::hash::RandomState

4.3 · BTreeMap, BTreeSet, and Ordered Collections

What's covered?

B-Tree vs. hash-based collections — when ordering matters.
range, split_off, append.
The Ord contract for keys.
Performance characteristics (O(log n) vs. O(1) amortized).
BTreeMap::entry.
Iterating in sorted order.
Use cases: interval queries, leaderboards, ordered deduplication.

Library Components:

std::collections::BTreeMap
std::collections::BTreeSet
std::collections::btree_map::Entry

4.4 · VecDeque, LinkedList, and BinaryHeap

What's covered?

VecDeque as a growable ring buffer — push_front, push_back, make_contiguous, as_slices, plus the push_front_mut/push_back_mut/insert_mut family (stabilized in 1.95).
VecDeque::retain_back (1.99) — the front-side counterpart of truncate: keep only the last n elements.
Why LinkedList is almost never what you want (cache locality, allocation overhead).
BinaryHeap as a max-heap priority queue — peek, push, pop, into_sorted_vec.
Relaxed T: Ord bounds in 1.94 for some BinaryHeap methods.

Library Components:

std::collections::VecDeque
std::collections::LinkedList
std::collections::BinaryHeap

4.5 · Slices and Arrays: The Foundation Types

What's covered?

Slice methods — sort, sort_by, sort_unstable, binary_search, chunks, windows, array_windows (stabilized in 1.94), split, contains, rotate_left, rotate_right, fill, swap, reverse.
element_offset (new in 1.94).
array::from_fn, array::each_ref, array::each_mut, and the stable alternative to array::try_from_fn (still unstable as of 1.99).
Const generic arrays.
Range syntax (a..b, a..=b, ..) appears throughout slice indexing — see 14.2 for the full RangeBounds trait and Bound type behind it.
For advanced slice algorithms (partition-point, unstable select, group-by), see 25.2.

Library Components:

primitive [T] (slice)
primitive [T; N] (array)
std::slice
std::array

Domain 5 — Iterators and Lazy Computation

5.1 · Iterator Fundamentals: The Trait and Its Adapters

What's covered?

The Iterator trait and next().
Core adapters: map, filter, filter_map, flat_map, flatten, enumerate, zip, chain, take, skip, take_while, skip_while, step_by, cycle, fuse.
Laziness — no work happens until consumption.
Why iterators are zero-cost abstractions.

Library Components:

std::iter::Iterator
adapter types in std::iter

5.2 · Consumers, Collectors, and the FromIterator/Extend Traits

What's covered?

Terminal operations: collect, fold, reduce, for_each, sum, product, count, any, all, find, position, min, max, min_by, max_by, min_by_key, max_by_key, nth, last, unzip.
FromIterator for custom collection types.
Extend for appending.
collect::<Result<Vec<_>, _>>() pattern.

Library Components:

std::iter::FromIterator
std::iter::Extend
std::iter::Sum
std::iter::Product

5.3 · Advanced Iterators: Peekable, Chain, Scan, and Inspect

What's covered?

Peekable and its peek, peek_mut, next_if, next_if_eq, next_if_map (new in 1.94).
scan for stateful transformations.
inspect for debugging pipelines.
by_ref to borrow an iterator.
cloned vs. copied.
intersperse, intersperse_with, and Iterator::array_chunks (all still unstable as of 1.99 — nightly-gated example).

Library Components:

std::iter::Peekable
std::iter::Scan
std::iter::Inspect
std::iter::Intersperse

5.4 · DoubleEndedIterator, ExactSizeIterator, and FusedIterator

What's covered?

DoubleEndedIterator and next_back — iterating from both ends.
rev().
ExactSizeIterator and len() — when the compiler knows the exact remaining count.
FusedIterator — guaranteeing None forever after the first None; StepBy is fused when I is fused (1.99).
How these traits enable optimizations in collect and extend.
TrustedLen (unstable but worth knowing about).

Library Components:

std::iter::DoubleEndedIterator
std::iter::ExactSizeIterator
std::iter::FusedIterator

5.5 · Creating Iterators: once, empty, repeat, successors, from_fn

What's covered?

Factory functions for iterators: iter::once, iter::once_with, iter::empty, iter::repeat, iter::repeat_with, iter::repeat_n (whose RepeatN gained Default in 1.97), iter::successors, iter::from_fn, iter::from_coroutine (unstable).
IntoIterator and why for loops work.
Implementing Iterator for custom types.

Library Components:

std::iter::once
std::iter::empty
std::iter::repeat
std::iter::successors
std::iter::from_fn
std::iter::IntoIterator

5.6 · ControlFlow: Short-Circuiting Iterator Operations

What's covered?

ControlFlow<B, C> as the generalization of break and continue.
How ControlFlow powers the Try trait machinery under ?.
Using ControlFlow with Iterator::try_for_each and Iterator::try_fold for early-exit iterations without forcing Result or Option semantics.
Implementing custom iterators that respect ControlFlow.
How ControlFlow::Break and ControlFlow::Continue replace ad-hoc boolean flags.
Relationship to FromResidual (see 2.1).

Library Components:

std::ops::ControlFlow
std::iter::Iterator::try_for_each
std::iter::Iterator::try_fold

Domain 6 — Concurrency and Parallelism

6.1 · Threads: Spawning, Joining, and Thread-Local Storage

What's covered?

thread::spawn, JoinHandle, thread::Builder (naming, stack size).
thread::current, thread::park, thread::unpark.
thread_local! macro and LocalKey<T> (including LocalKey<Cell<T>>::update, stable since 1.99).
Scoped threads with thread::scope — borrowing stack data across threads without Arc.
When thread::spawn returns an error.

Library Components:

std::thread
std::thread::spawn
std::thread::Builder
std::thread::JoinHandle
std::thread::scope
thread_local!
std::thread::LocalKey

6.2 · Mutex, RwLock, and Poison Recovery

What's covered?

Mutex<T> — wrapping data, lock(), try_lock().
MutexGuard and RAII.
RwLock<T> — read(), write(), try_read(), try_write().
Poison: what it is, PoisonError::into_inner() for recovery.
Condvar for wait/notify patterns.
When RwLock beats Mutex and when it doesn't (writer starvation).

Library Components:

std::sync::Mutex
std::sync::MutexGuard
std::sync::RwLock
std::sync::RwLockReadGuard
std::sync::RwLockWriteGuard
std::sync::Condvar
std::sync::PoisonError
std::sync::TryLockError

6.3 · Atomic Types and Their Operations

What's covered?

The atomic types: AtomicBool, AtomicI8 through AtomicI64, AtomicU8 through AtomicU64, AtomicPtr, AtomicUsize, AtomicIsize.
Operations: load, store, swap, fetch_add, fetch_sub, fetch_or, fetch_and, fetch_xor, fetch_update (a legacy spelling, deprecated since 1.99 in favor of try_update), and the closure-based update/try_update (stabilized in 1.95).
from_mut / from_mut_slice / get_mut_slice (1.98) to view exclusive &mut integers as atomics and back.
Lock-free counters and flags using only Relaxed and the "when in doubt" default SeqCst.
When an atomic beats a Mutex and when it doesn't.
Memory orderings appear only as a usage rule of thumb here — the full ordering model is Tutorial 6.4.

Library Components:

std::sync::atomic::AtomicBool
std::sync::atomic::AtomicUsize
std::sync::atomic::AtomicPtr
std::sync::atomic::Ordering

6.4 · Memory Ordering: Relaxed, Acquire/Release, and SeqCst

What's covered?

What each Ordering guarantees and when to use it: Relaxed, Acquire, Release, AcqRel, SeqCst.
The happens-before relation illustrated with the message-passing pattern (data + ready-flag).
compare_exchange vs. compare_exchange_weak and the spurious-failure retry loop idiom.
fence and compiler_fence.
Common bugs: assuming Relaxed orders unrelated memory, mixing orderings inconsistently between load and store sides.

Library Components:

std::sync::atomic::Ordering
std::sync::atomic::fence
std::sync::atomic::compiler_fence

6.5 · Channels: mpsc and mpmc Message Passing

What's covered?

mpsc::channel (unbounded) and mpsc::sync_channel (bounded).
Sender, SyncSender, Receiver.
send, try_send, recv, try_recv, recv_timeout.
The mpmc module (multi-producer, multi-consumer — still unstable as of 1.99; demonstrated with a nightly-gated example, with the "clone the Receiver" pattern it will enable).
Choosing between channels and shared state.
Backpressure with bounded channels.

Library Components:

std::sync::mpsc
std::sync::mpmc
Sender
Receiver
SyncSender
RecvError
SendError
TryRecvError
TrySendError

6.6 · Once, OnceLock, LazyLock, LazyCell: One-Time Initialization

What's covered?

The full picture of lazy and once-initialization types.
OnceCell (single-threaded, no closure) vs. LazyCell (single-threaded, with closure).
OnceLock (multi-threaded, no closure) vs. LazyLock (multi-threaded, with closure).
Once and call_once for global one-shot init.
get, get_mut, force, force_mut (new in 1.94) — checking initialization state without forcing.
From<T> for pre-initialized LazyCell/LazyLock (stabilized in 1.96).
When to use static with LazyLock vs. a field with LazyCell.
Poisoning behavior differences between LazyLock and Mutex.
The four-cell decision matrix.

Library Components:

std::sync::Once
std::sync::OnceLock
std::sync::LazyLock
std::cell::OnceCell
std::cell::LazyCell

6.7 · Barrier and Rendezvous Synchronization

What's covered?

Barrier for synchronizing a fixed number of threads at a checkpoint — all threads block until the last one arrives.
BarrierWaitResult and is_leader() for designating one thread to do post-rendezvous work.
Constructing phased algorithms (map-reduce, parallel simulation steps) with repeated barrier usage.
Comparison with Condvar (6.2) and channels (6.5) for synchronization: when Barrier is the right tool.

Library Components:

std::sync::Barrier
std::sync::BarrierWaitResult

6.8 · Send, Sync, and the Marker Trait Contracts

What's covered?

Why Send and Sync are auto traits.
What breaks Send (e.g., Rc<T>).
What breaks Sync (e.g., Cell<T>, RefCell<T>).
PhantomData to manually opt out.
Unsafe implementations of Send/Sync and when they're justified.
How Arc<Mutex<T>> satisfies both.
The role of Unpin in this landscape.

Library Components:

std::marker::Send
std::marker::Sync
std::marker::Unpin
std::marker::PhantomData
std::marker::PhantomPinned

Domain 7 — Smart Pointers and Heap Allocation

7.1 · `Box<T>`: The Simplest Heap Allocation

What's covered?

Box::new, Box::into_raw, Box::from_raw, Box::leak.
Box::into_non_null / Box::from_non_null (1.99): the same ownership round trip with a NonNull<T> pointer.
Boxing trait objects (Box<dyn Trait>).
Boxing recursive types.
Box::into_inner (still unstable as of 1.99 — nightly-gated example; the stable equivalent is the deref-move *boxed).
The Deref and DerefMut implementations.
Stack vs. heap placement semantics.

Library Components:

std::boxed::Box

7.2 · `Rc<T>` and `Arc<T>`: Reference Counting Strategies

What's covered?

Rc<T> for single-threaded reference counting.
Arc<T> for thread-safe reference counting.
strong_count, weak_count, Rc::try_unwrap, Arc::try_unwrap.
make_mut (clone-on-write).
Weak<T> references to break cycles.
Rc::new_cyclic for self-referential data.
Why Rc is not Send.

Library Components:

std::rc::Rc
std::rc::Weak
std::sync::Arc
std::sync::Weak

7.3 · `Cell<T>` and `RefCell<T>`: Interior Mutability

What's covered?

Cell<T> — get, set, replace, take.
Why Cell requires Copy for get.
RefCell<T> — borrow, borrow_mut, try_borrow, try_borrow_mut.
Runtime borrow checking.
BorrowError and BorrowMutError.
OnceCell and LazyCell for single-assignment cells (covered in full at 6.6).
Array-of-cells conversions: Cell<[T; N]> as [Cell<T>; N] via AsRef (stabilized in 1.95) alongside as_slice_of_cells.
The UnsafeCell<T> primitive underneath.

Library Components:

std::cell::Cell
std::cell::RefCell
std::cell::Ref
std::cell::RefMut
std::cell::UnsafeCell

Domain 8 — I/O System

8.1 · Read, Write, BufRead: The I/O Trait Hierarchy

What's covered?

The three core traits: Read (bytes in), Write (bytes out), BufRead (buffered reading with read_line, lines, split).
BufReader and BufWriter wrappers.
read_to_string, read_to_end, read_exact.
write_all vs. write.
flush.
Chaining readers with chain.
take to limit reads.
io::copy for streaming data between a Read and a Write without intermediate allocation (it specializes internally for buffered sources and sinks; as of 1.99, std has no public io::copy_buf).
Note: io::Write (byte streams) is distinct from fmt::Write (string/buffer formatting) — both work with write! and writeln! macros via separate trait implementations; see 3.2 for the fmt::Write side.

Library Components:

std::io::Read
std::io::Write
std::io::BufRead
std::io::BufReader
std::io::BufWriter
std::io::LineWriter
std::io::copy

8.2 · Cursor, Sink, Empty, Repeat: In-Memory I/O and Binary Parsing

What's covered?

Cursor<T> for treating Vec<u8> or &[u8] as a reader/writer with seek support.
io::empty() as /dev/null for reads.
io::sink() as /dev/null for writes.
io::repeat(byte) for infinite byte streams.
Binary format parsing: using Cursor<&[u8]> to sequentially parse a binary format — combining Read with Seek to navigate offset tables, reading u32s and f64s in specific byte orders (see 18.1 for endian conversion helpers), writing binary data back with Cursor<Vec<u8>>.
Use case: testing I/O code without files and parsing binary file headers (e.g., BMP, PNG).

Library Components:

std::io::Cursor
std::io::Empty
std::io::Sink
std::io::Repeat
std::io::empty
std::io::sink
std::io::repeat

What's covered?

The Seek trait and SeekFrom enum (Start, End, Current).
stream_position as a convenience.
Using Seek with BufReader (the gotcha of stale buffers).
seek_relative on BufReader.
Use case: parsing binary file formats with offset tables.

Library Components:

std::io::Seek
std::io::SeekFrom

8.4 · std::io::Error: Anatomy of I/O Errors

What's covered?

io::Error construction: Error::new, Error::other, Error::from_raw_os_error.
ErrorKind enum — all variants and when each occurs.
error.kind() matching.
Custom error payloads.
Converting other error types into io::Error.
The io::Result<T> alias.

Library Components:

std::io::Error
std::io::ErrorKind
std::io::Result

8.5 · stdin, stdout, stderr: Standard Streams

What's covered?

io::stdin(), io::stdout(), io::stderr() — their locking model (lock() for performance).
print!, println!, eprint!, eprintln!.
StdinLock, StdoutLock.
Why println! panics on broken pipes and how to handle it.
Buffering behavior differences between stdout and stderr.

Library Components:

std::io::stdin
std::io::stdout
std::io::stderr
std::io::Stdin
std::io::Stdout
std::io::Stderr
print!
println!
eprint!
eprintln!

Domain 9 — Filesystem Operations

9.1 · Reading, Writing, and Creating Files

What's covered?

File::open, File::create, File::create_new.
OpenOptions builder — read, write, append, truncate, create, create_new.
Reading a file to string with fs::read_to_string.
Writing with fs::write.
File as Read + Write + Seek.
Setting permissions on open.

Library Components:

std::fs::File
std::fs::OpenOptions
std::fs::read_to_string
std::fs::read
std::fs::write

9.2 · Directory Operations and Metadata

What's covered?

fs::create_dir, fs::create_dir_all, fs::remove_dir, fs::remove_dir_all.
fs::read_dir and the DirEntry iterator.
fs::metadata, fs::symlink_metadata.
The Metadata struct — is_file, is_dir, is_symlink, len, modified, created, accessed, permissions.
fs::set_permissions.
fs::set_times and fs::set_times_nofollow (1.99) with the FileTimes builder, to set timestamps by path.

Library Components:

std::fs
std::fs::DirEntry
std::fs::Metadata
std::fs::Permissions
std::fs::FileType
std::fs::FileTimes

9.3 · Path and PathBuf: Cross-Platform Path Manipulation

What's covered?

Path vs. PathBuf (borrowed vs. owned).
join, push, pop, set_extension, set_file_name.
Components: parent, file_name, file_stem, extension, components, ancestors.
canonicalize for resolving symlinks.
Platform differences (separators, roots, prefixes).
Path::display for lossy printing.

Library Components:

std::path::Path
std::path::PathBuf
std::path::Component
std::path::Prefix
std::path::MAIN_SEPARATOR

9.4 · Symbolic Links, Hard Links, and File Copying

What's covered?

fs::copy, fs::rename, fs::hard_link.
Platform-specific symlink functions in std::os::unix::fs::symlink and std::os::windows::fs::symlink_file/symlink_dir.
fs::read_link.
Handling cross-device moves (rename fails, must copy + delete).
Atomicity guarantees (or lack thereof).

Library Components:

std::fs::copy
std::fs::rename
std::fs::hard_link
std::fs::read_link
std::os::unix::fs
std::os::windows::fs

Domain 10 — Networking

10.1 · TCP: TcpListener and TcpStream

What's covered?

TcpListener::bind, incoming(), accept().
TcpStream::connect, connect_timeout.
Reading and writing on streams.
set_nonblocking, set_read_timeout, set_write_timeout, set_nodelay.
shutdown with Shutdown::Read/Write/Both.
peek.
try_clone for sharing a socket.

Library Components:

std::net::TcpListener
std::net::TcpStream
std::net::Shutdown

10.2 · UDP: Connectionless Communication

What's covered?

UdpSocket::bind, send_to, recv_from.
connect + send/recv for connected UDP.
set_broadcast, set_multicast_loop_v4, join_multicast_v4, leave_multicast_v4.
set_nonblocking.
Use case: building a simple DNS resolver or game server.

Library Components:

std::net::UdpSocket

10.3 · IP Addresses and Socket Addresses

What's covered?

IpAddr, Ipv4Addr, Ipv6Addr — parsing, constructing, classification (is_loopback, is_multicast, is_private, is_global).
SocketAddr, SocketAddrV4, SocketAddrV6.
ToSocketAddrs trait for resolution (including DNS).
Why ToSocketAddrs blocks and how to handle that.

Library Components:

std::net::IpAddr
std::net::Ipv4Addr
std::net::Ipv6Addr
std::net::SocketAddr
std::net::SocketAddrV4
std::net::SocketAddrV6
std::net::ToSocketAddrs
std::net::AddrParseError

Domain 11 — Time and Duration

11.1 · Instant, SystemTime, and Duration

What's covered?

Instant::now() for monotonic timing (benchmarks, timeouts).
SystemTime::now() for wall-clock time.
UNIX_EPOCH.
Duration — from_secs, from_millis, from_nanos, as_secs_f64, as_secs_f32.
Arithmetic on Instant and SystemTime (addition, subtraction, elapsed).
checked_add, checked_sub, saturating_*.
SystemTimeError and its duration() method for negative differences.

Library Components:

std::time::Instant
std::time::SystemTime
std::time::Duration
std::time::UNIX_EPOCH
std::time::SystemTimeError

11.2 · Thread Sleep, Timeouts, and Timed Operations

What's covered?

thread::sleep (and thread::sleep_until, still unstable as of 1.99 — mention only).
Using Duration with Mutex::try_lock, Condvar::wait_timeout, mpsc::Receiver::recv_timeout, TcpStream::set_read_timeout.
Building a simple rate limiter with Instant.
Spin-wait vs. sleep tradeoffs.
hint::spin_loop for busy waiting.

Library Components:

std::thread::sleep
std::time::Duration
std::hint::spin_loop

Domain 12 — Process and Environment Interaction

12.1 · Environment Variables, Args, and Current Directory

What's covered?

env::var, env::var_os, env::vars, env::vars_os.
env::set_var (unsafe since 1.84), env::remove_var.
env::args, env::args_os.
env::current_dir, env::set_current_dir.
env::current_exe.
env::temp_dir, env::home_dir (un-deprecated in 1.87 after its Windows behavior was fixed).
The safety concerns around set_var in multithreaded contexts.

Library Components:

std::env
std::env::Args
std::env::ArgsOs
std::env::Vars
std::env::VarsOs
std::env::VarError

12.2 · Spawning and Managing Child Processes

What's covered?

Command::new, arg, args, env, current_dir, stdin, stdout, stderr.
Stdio::piped, Stdio::null, Stdio::inherit.
spawn vs. output vs. status.
Child — wait, wait_with_output, kill, try_wait.
Piping between processes.
ExitStatus and ExitCode.
Termination — the trait that allows fn main() -> Result<(), E> and custom process exit codes; how ExitCode::from and Termination::report work together.

Library Components:

std::process::Command
std::process::Child
std::process::Output
std::process::Stdio
std::process::ExitStatus
std::process::ExitCode
std::process::Termination
std::process::exit
std::process::abort

Domain 13 — Comparing, Ordering, and Hashing

13.1 · PartialEq, Eq, PartialOrd, Ord: The Comparison Hierarchy

What's covered?

Why four traits? The partial equivalence relation (NaN ≠ NaN).
Deriving vs. manual implementation.
Ord::cmp, PartialOrd::partial_cmp.
Reverse for descending sorts.
cmp::min/cmp::max as standalone functions and Ord::clamp (plus the float inherent clamp).
Implementing comparisons across different types (PartialEq<Rhs>).
Consistency contract: Eq requires reflexivity; Ord requires totality.

Library Components:

std::cmp::PartialEq
std::cmp::Eq
std::cmp::PartialOrd
std::cmp::Ord
std::cmp::Ordering
std::cmp::Reverse
std::cmp::min
std::cmp::max
std::cmp::min_by
std::cmp::max_by
std::cmp::min_by_key
std::cmp::max_by_key

13.2 · Custom Hashing: Implementing Hash and Building Hashers

What's covered?

The Hash trait and its hash method.
Hasher trait for state machines.
BuildHasher for constructing hashers.
DefaultHasher (SipHash 1-3).
The hash/eq contract (equal values must have equal hashes).
Writing a custom Hasher.
hash_map::RandomState for DOS resistance.
When and why you'd want a non-random hasher (deterministic tests, performance).
The Hash/Hasher/BuildHasher trio is introduced in context of HashMap at 4.2; this tutorial goes deeper on custom implementations.

Library Components:

std::hash::Hash
std::hash::Hasher
std::hash::BuildHasher
std::hash::BuildHasherDefault
std::hash::RandomState
std::hash::DefaultHasher

Domain 14 — Operator Overloading

14.1 · Arithmetic and Bitwise Operators

What's covered?

Add, Sub, Mul, Div, Rem and their *Assign variants.
Neg for unary minus.
Bitwise: BitAnd, BitOr, BitXor, Not, Shl, Shr and *Assign.
Implementing operators for custom numeric types.
Operator overloading with different RHS types.

Library Components:

std::ops::Add through std::ops::ShrAssign
std::ops::Neg
std::ops::Not

14.2 · Index, Deref, and the Range Operator Family

What's covered?

Index and IndexMut for [] syntax.
Deref and DerefMut — deref coercion rules and the "smart pointer" pattern.
Range, RangeInclusive, RangeFrom, RangeTo, RangeToInclusive, RangeFull — construction, iteration, and use in slice indexing.
The RangeBounds<T> trait — start_bound, end_bound, contains — and how to write functions that accept any range expression.
Bound::Included, Bound::Excluded, Bound::Unbounded.
Using ranges for slicing, iteration, and pattern matching.
Exclusive range patterns in match (stabilized 1.80).
The .. and ..= operators as syntactic sugar for these types.
The new core::range generation (stabilized in 1.96, RFC 3550): range::Range, range::RangeFrom, range::RangeInclusive, range::RangeToInclusive are Copy and implement IntoIterator instead of Iterator — why that matters, converting between legacy and new range types, and the future-edition migration outlook.
std::range::legacy (1.98) re-exports the iterator-style ops range types under a name that marks them as the generation .. sugar still builds.

Library Components:

std::ops::Index
std::ops::IndexMut
std::ops::Deref
std::ops::DerefMut
std::ops::Range
std::ops::RangeInclusive
std::ops::RangeFrom
std::ops::RangeTo
std::ops::RangeToInclusive
std::ops::RangeFull
std::ops::RangeBounds
std::ops::Bound
std::range
std::range::legacy

14.3 · Fn, FnMut, FnOnce: Closures as Trait Objects

What's covered?

The three closure traits and their hierarchy (FnOnce ⊇ FnMut ⊇ Fn).
Why closures implement different traits based on capture behavior.
Using closures in function signatures (impl Fn, Box<dyn FnMut>, fn pointer).
The move keyword.
Closure size and when they allocate.
Returning closures from functions.

Library Components:

std::ops::Fn
std::ops::FnMut
std::ops::FnOnce

Domain 15 — Asynchronous Programming Primitives

15.1 · Future, Poll, and the Async Machinery

What's covered?

Future trait and poll.
Poll::Ready vs. Poll::Pending.
Context and Waker.
Why std provides the traits but no runtime.
The relationship to async/await syntax.
async fn desugaring.
Building a minimal block_on executor with only the standard library (std::task::Wake + Arc + thread::park/unpark) — no external crates.

Library Components:

std::future::Future
std::future::poll_fn
std::future::pending
std::future::ready
std::task::Poll
std::task::Context
std::task::Waker
std::task::RawWaker
std::task::RawWakerVTable
std::task::Wake

15.2 · Pin: Why Futures Need Pinning

What's covered?

The self-referential future problem, shown concretely with an async block that borrows across an .await.
The Pin contract — what it prevents and what it promises.
Unpin as the escape hatch and why almost every type is Unpin.
pin! macro for stack pinning; Box::pin for heap pinning.
Pin::new (requires Unpin).
PhantomPinned to opt out of Unpin.

Library Components:

std::pin::Pin
std::pin::pin!
std::boxed::Box::pin
std::marker::Unpin
std::marker::PhantomPinned

15.3 · Advanced Pinning: Unsafe Construction and Structural Projection

What's covered?

Pin::new_unchecked and the invariants the caller must uphold.
Structural vs. non-structural pinning — deciding per field whether pinning propagates.
Hand-writing safe pin projections (the pattern behind the pin-project crate).
Pin::as_mut, Pin::get_mut, Pin::into_inner_unchecked.
Building a correct self-referential type with PhantomPinned + NonNull.

Library Components:

std::pin::Pin
std::ptr::NonNull
std::marker::PhantomPinned

Domain 16 — Memory Management and Unsafe Primitives

16.1 · std::mem: Swaps, Sizes, and Transmutes

What's covered?

mem::size_of, mem::size_of_val, mem::align_of, mem::align_of_val.
mem::size_of_val_raw, mem::align_of_val_raw, and Layout::for_value_raw (1.99): size and alignment from a raw pointer, when a reference is not permitted.
mem::swap, mem::replace, mem::take (replaces with default).
mem::drop (explicit drop).
mem::forget (leaking intentionally).
mem::transmute — power and peril.
mem::zeroed, mem::uninitialized (deprecated).
mem::ManuallyDrop for controlled destruction order (including the 1.98-documented guarantee that moving ManuallyDrop<Box<T>> after dropping the box is not UB).
mem::discriminant for comparing enum variants.

Library Components:

std::mem::*

16.2 · MaybeUninit: Safe Patterns for Uninitialized Memory

What's covered?

Why MaybeUninit<T> exists (replacing mem::uninitialized).
MaybeUninit::uninit, MaybeUninit::new, assume_init, assume_init_ref, assume_init_mut, assume_init_drop.
Initializing arrays element-by-element; the MaybeUninit<[T; N]> ⇄ [MaybeUninit<T>; N] conversions (stabilized in 1.95).
MaybeUninit::zeroed.
Use case: building a fixed-capacity buffer without default constructors.

Library Components:

std::mem::MaybeUninit

16.3 · Raw Pointers in Practice: NonNull, Copying, and Raw References

What's covered?

ptr::null, ptr::null_mut, and NonNull<T> for non-null guarantees (plus the niche optimization it enables).
Creating raw pointers with &raw const / &raw mut (stabilized 1.82, superseding addr_of!).
ptr::read and ptr::write for moving values through raw pointers.
ptr::copy vs. ptr::copy_nonoverlapping (memmove vs. memcpy semantics).
ptr::drop_in_place.
ptr::from_ref, ptr::from_mut.
The patterns most likely to appear in safe wrapper code.

Library Components:

std::ptr::NonNull
std::ptr::null
std::ptr::read
std::ptr::write
std::ptr::copy
std::ptr::copy_nonoverlapping
std::ptr::drop_in_place
&raw operators

16.4 · Pointer Arithmetic, Volatile, and Provenance

What's covered?

Pointer arithmetic — offset, add, sub, wrapping_add — and the UB conditions that separate them.
read_volatile/write_volatile and what volatile does (and does not) guarantee.
Strict provenance APIs (stabilized 1.84): addr, with_addr, map_addr; exposed provenance: expose_provenance, with_exposed_provenance.
Why pointer↔integer round-trips are subtler than they look.
ptr::dangling for well-aligned sentinel pointers.

Library Components:

std::ptr
pointer primitive methods
std::ptr::dangling

16.5 · Global Allocator and Allocation APIs

What's covered?

The GlobalAlloc trait: alloc, dealloc, realloc, alloc_zeroed.
Layout — size and alignment requirements; composing layouts with Layout::extend (stable since 1.44) and with Layout::repeat, Layout::repeat_packed, Layout::extend_packed, and Layout::dangling_ptr (stabilized in 1.95); Layout::for_value_raw (1.99, shown in 16.1).
#[global_allocator] attribute for replacing the default allocator.
alloc::alloc, alloc::dealloc, alloc::handle_alloc_error.
Why Allocator (the parameterized allocator trait) is still unstable but worth tracking.

Library Components:

std::alloc::GlobalAlloc
std::alloc::Layout
std::alloc::alloc
std::alloc::dealloc
std::alloc::realloc
std::alloc::handle_alloc_error
std::alloc::System

Domain 17 — Type System Utilities

17.1 · Any and Dynamic Typing

What's covered?

Any trait and TypeId.
Downcasting with downcast_ref, downcast_mut, downcast (on Box<dyn Any>).
TypeId::of::<T>() for runtime type identification.
type_name::<T>() and type_name_of_val for human-readable type names in diagnostics and generic error messages.
Limitations of type_name: output format is not guaranteed to be stable or unique across crates — do not use for serialization.
Use cases: heterogeneous collections, plugin architectures, error downcasting, logging.
The 'static requirement on Any.

Library Components:

std::any::Any
std::any::TypeId
std::any::type_name
std::any::type_name_of_val

17.2 · Marker Traits: Sized, Copy, Send, Sync, Unpin

What's covered?

Sized — the implicit bound and ?Sized for unsized types.
Copy semantics vs. Clone.
Send and Sync (covered in depth in 6.8 but touched on here from the type system perspective).
Unpin and its relationship to Pin.
PhantomData<T> for expressing unused type parameters, variance, and lifetime relationships.
PhantomPinned.

Library Components:

std::marker::Sized
std::marker::Copy
std::marker::Send
std::marker::Sync
std::marker::Unpin
std::marker::PhantomData
std::marker::PhantomPinned

17.3 · The Default Trait and Default Values

What's covered?

Deriving Default.
Implementing Default manually.
Option::unwrap_or_default().
HashMap::entry(..).or_default().
Using Default with mem::take.
Interaction with #[non_exhaustive].
Use case: builder pattern fallbacks, struct initialization with ..Default::default().

Library Components:

std::default::Default

Domain 18 — Numeric Types and Math

18.1 · Integer Methods: Checked, Wrapping, Saturating, Overflowing

What's covered?

The four arithmetic strategies on every integer type.
checked_add, wrapping_mul, saturating_sub, overflowing_div.
pow, isqrt.
leading_zeros, trailing_zeros, count_ones, rotate_left, rotate_right, reverse_bits, swap_bytes.
The bit-query family stabilized in 1.97: bit_width, highest_one, lowest_one, isolate_highest_one, isolate_lowest_one (also on NonZero integers).
Endian conversion: to_be, to_le, from_be_bytes, to_le_bytes, to_ne_bytes, from_le_bytes, from_ne_bytes — the primary tools for binary protocol encoding/decoding and working with [u8] byte buffers.
[u8]::escape_ascii for displaying raw byte sequences.
NonZero types, their MIN/MAX constants, const methods, and the niche optimization (Option<NonZeroU32> is the same size as u32).
Associated constants (i32::MAX) are the only current spelling: the legacy module constants (std::i32::MAX) and the min_value()/max_value() functions are deprecated since 1.99.

Library Components:

Primitive integer types
std::num::NonZero*
std::num::Wrapping
std::num::Saturating

18.2 · Floating-Point: IEEE 754, Special Values, and Math Constants

What's covered?

f32 and f64 methods: floor, ceil, round, trunc, fract, abs, signum, copysign, sqrt, cbrt, ln, log2, log10, exp, exp2, sin, cos, tan, asin, acos, atan, atan2, hypot, powi, powf, mul_add (const in 1.94).
Algebraic operators algebraic_{add,sub,mul,div,rem} (1.98) that permit reassociation and similar optimizations.
NAN, INFINITY, NEG_INFINITY, EPSILON, MIN, MAX, MIN_POSITIVE.
New constants: EULER_GAMMA, GOLDEN_RATIO (stabilized in 1.94).
is_nan, is_finite, is_infinite, is_subnormal, is_sign_positive, classify, total_cmp.

Library Components:

f32
f64
std::f32::consts
std::f64::consts

18.3 · Parsing Numbers: FromStr, Radix, and Formatting

What's covered?

str::parse::<i32>() and FromStr.
i32::from_str_radix for non-decimal bases.
NonZero*::from_str_radix (1.98) so a successful parse is already nonzero.
ParseIntError, ParseFloatError.
Formatting numbers: padding, precision, sign display.
num::IntErrorKind — distinguishing empty, invalid digit, overflow, underflow, zero.
Converting between integer sizes safely with TryFrom (introduced in Tutorial 1.3); since 1.99 the TryFromIntError message says if the value is too large or too small.

Library Components:

std::str::FromStr
integer and float from_str_radix
std::num::ParseIntError
std::num::ParseFloatError
std::num::IntErrorKind

Domain 19 — Macros from the Standard Library

19.1 · Essential Macros: assert, dbg, todo, unimplemented, unreachable

What's covered?

assert!, assert_eq!, assert_ne! — custom messages, when they compile away (debug_assert_*).
assert_matches! and debug_assert_matches! for pattern-based assertions (stabilized in 1.96; not in the prelude — must be imported).
dbg! — how it prints file/line/expression and returns the value.
todo! vs. unimplemented! vs. unreachable! — intent signaling.
panic! message formatting.

Library Components:

assert!
assert_eq!
assert_ne!
assert_matches!
debug_assert!
debug_assert_eq!
debug_assert_ne!
debug_assert_matches!
dbg!
todo!
unimplemented!
unreachable!
panic!

19.2 · Compile-Time Macros: cfg, env, include, concat, stringify

What's covered?

cfg! for runtime config checks.
cfg_select! (stabilized in 1.95) as a compile-time "match on cfg predicates" that replaces the cfg-if crate.
env! and option_env! for compile-time env vars.
include_str! and include_bytes! for embedding files.
concat! for compile-time string concatenation.
stringify! for turning tokens into a string.
file!, line!, column!, module_path!.
compile_error! for custom compile-time errors.
cfg! vs. #[cfg(...)] vs. cfg_select!.

Library Components:

cfg!
cfg_select!
env!
option_env!
include_str!
include_bytes!
include!
concat!
stringify!
file!
line!
column!
module_path!
compile_error!

19.3 · vec!, format!, write!, and Other Constructor Macros

What's covered?

vec![1, 2, 3] and vec![0; n].
format! for string building.
write! and writeln! for writing to any fmt::Write or io::Write.
matches! for boolean pattern matching.
todo! as a typed placeholder.
thread_local!.
Using these macros effectively in real code.

Library Components:

vec!
format!
write!
writeln!
matches!
thread_local!

Domain 20 — FFI (Foreign Function Interface)

20.1 · C Types and Calling Conventions

What's covered?

std::ffi C-compatible types: c_char, c_int, c_long, c_float, c_double, c_void, etc.
extern "C" blocks.
Calling conventions: extern "C", extern "system", and extern "stdcall" (a hard error on targets that don't support it — extern "system" is the portable spelling).
ABI compatibility and #[repr(C)].
Passing and returning structs across the FFI boundary.
C-variadic function definitions (unsafe extern "C" fn f(n: c_int, args: ...)) and VaList (stabilized in 1.99).

Library Components:

std::ffi::c_char
std::ffi::c_int
std::ffi::c_void
std::ffi::VaList
etc.

20.2 · String Marshalling: CString/CStr and OsString/OsStr at the Boundary

What's covered?

Creating CString from Rust strings.
Handling interior null bytes.
Passing CStr to C functions.
Receiving C strings as *const c_char and wrapping in CStr.
OsStr/OsString for OS-native strings.
Platform-specific extensions: OsStrExt on Unix (as raw bytes), OsStringExt on Windows (wide chars).

Library Components:

std::ffi::CString
std::ffi::CStr
std::ffi::NulError
std::ffi::OsStr
std::ffi::OsString
std::os::unix::ffi::OsStrExt
std::os::windows::ffi::OsStringExt

Domain 21 — OS-Specific Extensions

21.1 · Unix Extensions: File Descriptors, Permissions, Signals

What's covered?

std::os::unix — fs::PermissionsExt (mode bits), fs::MetadataExt (inode, dev, nlink, uid, gid).
io::AsRawFd, FromRawFd, IntoRawFd, OwnedFd, BorrowedFd.
Unix domain sockets via std::os::unix::net (the abstract-namespace SocketAddrExt is Linux-only, living in std::os::linux::net).
process::CommandExt — uid, gid, pre_exec.
Pipe and signal concepts.

Library Components:

std::os::unix::fs
std::os::unix::io
std::os::unix::net
std::os::unix::process

21.2 · Windows Extensions: Handles, Wide Strings, and Process Creation

What's covered?

std::os::windows — io::AsRawHandle, FromRawHandle, IntoRawHandle, OwnedHandle, BorrowedHandle.
OsStrExt::encode_wide and OsStringExt::from_wide.
fs::MetadataExt (file attributes, creation time).
process::CommandExt — creation_flags, raw_arg.
Differences from Unix when writing cross-platform code.

Library Components:

std::os::windows::fs
std::os::windows::io
std::os::windows::process
std::os::windows::ffi

Domain 22 — Compiler Hints and Low-Level Intrinsics

22.1 · std::hint: Guiding the Optimizer

What's covered?

The complete std::hint module.
hint::unreachable_unchecked — promises to the compiler (and the UB if you're wrong).
hint::spin_loop — yield for spin-wait loops (see also 11.2).
hint::black_box — preventing dead code elimination in benchmarks.
hint::assert_unchecked for precondition assertions.
hint::cold_path (stabilized in 1.95) for marking unlikely branches.
When and why to use each.

Library Components:

std::hint::unreachable_unchecked
std::hint::spin_loop
std::hint::black_box
std::hint::assert_unchecked
std::hint::cold_path

22.2 · SIMD with std::arch: Platform Intrinsics

What's covered?

Overview of std::arch for x86, x86_64, ARM, AArch64, WebAssembly.
Feature detection at runtime: is_x86_feature_detected!, is_aarch64_feature_detected!.
Using SSE/AVX intrinsics within unsafe blocks.
AVX-512 FP16 intrinsics (stabilized in 1.94) and AArch64 NEON FP16.
#[target_feature(enable = "...")].
Why this matters: hot loops in signal processing, compression, cryptography.

Library Components:

std::arch::x86_64
std::arch::aarch64
std::arch::wasm32
is_x86_feature_detected!
is_aarch64_feature_detected!

Domain 23 — Testing, Debugging, and Pattern Matching

23.1 · Debug Formatting and Diagnostic Output

What's covered?

#[derive(Debug)] and when to implement Debug manually.
Pretty-printing with {:#?}.
dbg! for quick debugging.
Writing custom Debug implementations with DebugStruct, DebugTuple, DebugList, DebugMap.
fmt::Pointer for printing addresses.
type_name::<T>() and type_name_of_val for runtime type inspection (see 17.1 for deeper coverage).

Library Components:

std::fmt::Debug
std::fmt::DebugStruct
std::fmt::DebugTuple
std::fmt::DebugList
std::fmt::DebugMap
std::any::type_name
dbg!

23.2 · Assertions, Panics, and Test Harness Integration

What's covered?

assert!, assert_eq!, assert_ne! with custom messages.
assert_matches! (1.96) for asserting on enum shapes in tests.
debug_assert_* variants that compile away in release.
#[should_panic] test attribute.
#[test] and #[cfg(test)].
Using Result<(), E> in test functions.
Capturing panics in tests with catch_unwind.
Building a custom test framework (overview).

Library Components:

assert!
assert_eq!
assert_ne!
debug_assert!
std::panic::catch_unwind
#[test]
#[should_panic]

23.3 · matches!, if let Chains, and Exhaustive Matching Strategies

What's covered?

The matches! macro for boolean pattern tests.
Using matches! with guards.
Nested pattern matching.
Exclusive range patterns (start..end, stabilized in 1.80).
Combining patterns with |.
@ bindings in patterns.
The if let and let else constructs; let chains (1.88) and if-let guards in match arms (stabilized in 1.95).
How Option, Result, and custom enums interact with pattern matching.

Library Components:

matches!
std::option::Option
std::result::Result
language-level pattern syntax

Domain 24 — Dynamic Dispatch and Trait Objects

24.1 · dyn Trait, Vtables, and Object Safety

What's covered?

What makes a trait "object safe."
The vtable layout — function pointers + size + alignment.
Box<dyn Trait>, &dyn Trait, Arc<dyn Trait>.
dyn Trait + Send + Sync + 'static bounds.
Downcasting with Any.
Performance implications of dynamic dispatch vs. monomorphization.
dyn Fn(), dyn FnMut(), dyn FnOnce() as callable trait objects.

Library Components:

std::any::Any
dyn keyword
std::boxed::Box
std::ops::Fn* traits

Domain 25 — The Contiguous Memory Triumvirate

25.1 · Vec, Boxed Slices, and Arrays: Choosing the Right Container

What's covered?

Vec<T> — growable.
Box<[T]> — heap-allocated, fixed after conversion.
[T; N] — stack-allocated, compile-time size.
Conversions between them: Vec::into_boxed_slice(), into_vec() on a Box<[T]>, <[T; N]>::as_slice().
When each is optimal (memory overhead, flexibility, stack limits).
Using array::from_fn for constructing arrays from closures.
Iterating a boxed array: IntoIterator for Box<[T; N]>, &Box<[T; N]>, and &mut Box<[T; N]> (1.99).

Library Components:

std::vec::Vec
std::boxed::Box
primitive [T; N]
std::array::from_fn
std::array::try_from_fn (unstable as of 1.99 — nightly-gated example)
std::boxed::BoxedArrayIntoIter

25.2 · Slice Algorithms: Advanced Sorting, Searching, and Splitting

What's covered?

Advanced algorithms not covered in 4.5.
sort_by_key, sort_unstable_by_key.
binary_search_by, binary_search_by_key, partition_point (find the split point for a sorted predicate).
split_at, split_at_mut, split_first, split_last, split_first_mut, split_last_mut.
[T]::subslice_range and [T]::strip_circumfix (1.98).
chunks_exact, chunks_exact_mut, and the as_chunks family (stabilized in 1.88; Iterator::array_chunks remains unstable as of 1.99).
chunk_by (stabilized in 1.77 — the rename of the formerly unstable group_by) for splitting on adjacent-element predicates.
select_nth_unstable, select_nth_unstable_by, select_nth_unstable_by_key for partial sorting.

Library Components:

Primitive [T] methods
std::slice

Domain 26 — Cross-Cutting Patterns

26.1 · The Newtype Pattern with Standard Library Traits

What's covered?

Wrapping a type to change its behavior (sorting, hashing, display).
Deriving vs. forwarding standard traits on newtypes.
Deref/DerefMut for transparent access.
From/Into for conversions.
Use case: Password(String) that redacts in Debug, Meters(f64) with type-safe arithmetic, SortByName(Person) with custom Ord.

Library Components:

std::ops::Deref
std::fmt::Debug
std::fmt::Display
std::cmp::Ord
std::convert::From

26.2 · Builder Patterns with Default and Option

What's covered?

Using Default::default() for struct initialization with ..Default::default().
Option<T> fields for optional configuration.
Combining Default with the builder pattern.
take and replace on Option for consuming builders.
unwrap_or_default chains.
Real use case: HTTP request builder, database connection options.

Library Components:

std::default::Default
std::option::Option
std::mem::take
std::mem::replace

26.3 · The Iterator + Collect Pattern as a Data Pipeline

What's covered?

Composing iterator chains as functional data pipelines.
Collecting into different types: Vec<T>, HashMap<K, V>, BTreeMap<K, V>, HashSet<T>, String, Result<Vec<T>, E>, Option<Vec<T>>.
Iterator::partition for splitting.
Iterator::unzip for destructuring pairs.
Performance: avoiding intermediate allocations with extend and chain.

Library Components:

std::iter::Iterator
std::iter::FromIterator
std::iter::Extend

Appendix A — Stabilizations Highlighted from Rust 1.94–1.99

Note: The curriculum targets Rust 1.99 (MSRV; compiler 1.99.0). This table maps stabilizations from releases 1.94 through 1.99 (all verified against the official release notes) to the tutorials that showcase them.

API	Stabilized	Domain	Tutorial
`<[T]>::array_windows`	1.94	Collections, Slices	4.5, 25.2
`<[T]>::element_offset`	1.94	Collections, Slices	4.5
`LazyCell`/`LazyLock` `get`, `get_mut`, `force_mut`	1.94	Lazy Evaluation	6.6
`Peekable::next_if_map`	1.94	Iterators	5.3
`impl TryFrom<char> for usize`	1.94	Conversions	1.3
`f32`/`f64` `consts::EULER_GAMMA`, `consts::GOLDEN_RATIO`	1.94	Numerics	18.2
`f32::mul_add` / `f64::mul_add` (const)	1.94	Numerics	18.2
AVX-512 FP16 + AArch64 NEON FP16 intrinsics	1.94	SIMD	22.2
Relaxed `T: Ord` on `BinaryHeap` methods	1.94	Collections	4.4
`cfg_select!`	1.95	Macros	19.2
if-let guards in `match` arms	1.95	Pattern Matching	23.3
`Atomic*::update` / `try_update`	1.95	Concurrency	6.3
`Vec::push_mut`, `Vec::insert_mut`	1.95	Collections	4.1
`VecDeque`/`LinkedList` `push_*_mut`, `insert_mut`	1.95	Collections	4.4
`impl TryFrom<{integer}> for bool`	1.95	Conversions	1.3
`MaybeUninit<[T; N]>` ⇄ `[MaybeUninit<T>; N]` conversions	1.95	Unsafe Memory	16.2
`Cell<[T; N]>: AsRef<[Cell<T>]>` family	1.95	Interior Mutability	7.3
`hint::cold_path`	1.95	Compiler Hints	22.1
`Layout::repeat`, `Layout::repeat_packed`, `Layout::extend_packed`, `Layout::dangling_ptr`	1.95	Allocation	16.5
`assert_matches!`, `debug_assert_matches!`	1.96	Macros, Testing	19.1, 23.2
`core::range::{Range, RangeFrom, RangeInclusive, RangeToInclusive}` (RFC 3550)	1.96	Operators	14.2
`From<T> for LazyCell`/`LazyLock`	1.96	Lazy Evaluation	6.6
`From<T> for AssertUnwindSafe<T>`	1.96	Panics	2.3
Integer `bit_width`, `highest_one`, `lowest_one`, `isolate_highest_one`, `isolate_lowest_one` (incl. `NonZero`)	1.97	Numerics	18.1
`Default for iter::RepeatN`	1.97	Iterators	5.5
`char::is_control` (const)	1.97	Text	3.5
`str::substr_range`, `[T]::subslice_range`	1.98	Text, Slices	3.1, 25.2
`str::strip_circumfix`, `[T]::strip_circumfix`	1.98	Text, Slices	3.3, 25.2
`core::fmt::NumBuffer`, `{integer}::format_into`	1.98	Formatting	3.2
`{f32,f64}::algebraic_{add,sub,mul,div,rem}`	1.98	Numerics	18.2
`NonZero<{integer}>::from_str_radix`	1.98	Numerics	18.3
`String::from_utf16le` / `from_utf16be` (and lossy)	1.98	Text	3.1
`Atomic*::from_mut`, `from_mut_slice`, `get_mut_slice`	1.98	Concurrency	6.3
`std::range::legacy`	1.98	Operators	14.2
`bool::ok_or`, `bool::ok_or_else`	1.98	Error Handling	2.1
`ManuallyDrop<Box<T>>` move-after-drop documented as not UB	1.98	Memory	16.1
`String::from_utf8_lossy_owned`, `FromUtf8Error::into_utf8_lossy`	1.99	Text	3.1
`NumBuffer` re-exported from `std::fmt` (and `alloc::fmt`)	1.99	Formatting	3.2
`Vec::into_parts`, `Vec::from_parts`	1.99	Collections	4.1
`VecDeque::retain_back`	1.99	Collections	4.4
`FusedIterator for StepBy<I>`	1.99	Iterators	5.4
`LocalKey<Cell<T>>::update`	1.99	Concurrency, Macros	6.1, 19.3
`Atomic*::fetch_update` deprecated (renamed to `try_update`)	1.99	Concurrency	6.3
`Box::into_non_null`, `Box::from_non_null`	1.99	Smart Pointers	7.1
`fs::set_times`, `fs::set_times_nofollow`	1.99	Filesystem	9.2
`mem::size_of_val_raw`, `mem::align_of_val_raw`, `Layout::for_value_raw`	1.99	Memory, Allocation	16.1, 16.5
`TryFromIntError` messages distinguish "too large" and "too small"	1.99	Conversions, Numerics	1.3, 18.3
Legacy integer and float module constants (`std::i32::MAX`, `std::f64::EPSILON`) deprecated	1.99	Numerics	18.1
C-variadic function definitions, `core::ffi::VaList`	1.99	FFI	20.1
`IntoIterator` for `Box<[T; N]>`, `&Box<[T; N]>`, `&mut Box<[T; N]>`	1.99	Contiguous Memory	25.1

Appendix B — Tutorial Index by Library Module

Module	Tutorials
`std::alloc`	16.5
`std::any`	17.1, 23.1
`std::arch`	22.2
`std::array`	4.5, 25.1
`std::ascii`	3.5
`std::backtrace`	2.4
`std::borrow`	1.2
`std::boxed`	7.1, 25.1
`std::cell`	6.6, 7.3
`std::char`	3.5
`std::clone`	1.1
`std::cmp`	13.1
`std::collections`	4.1–4.4
`std::convert`	1.3
`std::default`	17.3, 26.2
`std::env`	12.1
`std::error`	2.2
`std::ffi`	3.4, 20.1, 20.2
`std::fmt`	3.2, 23.1
`std::fs`	9.1–9.4
`std::future`	15.1
`std::hash`	4.2, 13.2
`std::hint`	11.2, 22.1
`std::io`	8.1–8.5
`std::iter`	5.1–5.6, 26.3
`std::marker`	6.8, 17.2
`std::mem`	16.1, 16.2, 26.2
`std::net`	10.1–10.3
`std::num`	18.1, 18.3
`std::ops`	5.6, 14.1–14.3
`std::option`	2.1, 23.3
`std::os`	21.1, 21.2
`std::panic`	2.3, 23.2
`std::path`	9.3
`std::pin`	15.2, 15.3
`std::prelude`	P.1
`std::process`	12.2
`std::ptr`	16.3, 16.4
`std::rc`	7.2
`std::result`	2.1, 23.3
`std::slice`	4.5, 25.2
`std::str`	3.1, 3.3
`std::string`	3.1
`std::range`	14.2
`std::sync`	6.1–6.8
`std::task`	15.1
`std::thread`	6.1, 11.2
`std::time`	11.1, 11.2
`std::vec`	4.1, 25.1

Total: 26 domains + prologue, 88 tutorials (~22 hours of video content at an average of 15 min each)

P.1 · What the Prelude Imports and Why

Prologue — The Prelude and Implicit Imports Duration: ~15 minutes Library components: std::prelude::v1, std::prelude::rust_2021, std::prelude::rust_2024

Introduction

Many names are available in each Rust file without a use statement. Examples are println!, Vec, Option, String, and Clone, and there are dozens of other names.

These names come from the prelude. The prelude is a small set of items that Rust automatically imports into each module of each crate. The effect is the same as if each file started with this hidden line:

// The compiler adds this import to each module. The module name depends on the edition.
use std::prelude::rust_2024::*;

Knowledge of the prelude prevents a type of confusion that occurs even for experienced Rust developers: "Where does this trait come from? I did not import it." When you know the contents of the prelude, you know the source of these traits.

This tutorial shows:

The contents of the v1 prelude (the baseline for all editions)
The additions in the 2021 edition prelude
The additions in the 2024 edition prelude
The prelude traits that for loops, ?, collect(), and .await use

The Prelude Is an Implicit Use

The Rust compiler automatically adds the prelude import before it processes your file. The standard library documentation shows the full list for your edition under std::prelude.

The #![no_implicit_prelude] attribute fully removes the prelude from a specific file. Then dozens of usually implicit names are not in scope, and code that uses them does not compile.

The prelude is edition-dependent. When your Cargo.toml sets edition = "2024", you get the 2024 prelude. The 2024 prelude is a superset of the 2021 prelude, and the 2021 prelude is a superset of the v1 prelude.

Figure: Rust Prelude by Edition — Each Layer Inherits the One Below Prelude edition layers

The v1 Prelude: The Foundation

The v1 prelude (the prelude of the 2015 and 2018 editions) is the baseline.
Each item in it is available in each edition.
The subsections that follow show the primary categories.

Macros

The macros that Rust programs use most are prelude items:

println!, print!, eprintln!, eprint!: formatted output to stdout or stderr
vec!: a short form of Vec::new() and a sequence of push calls
format!: builds a String from a format template
assert!, assert_eq!, assert_ne!: runtime tests of invariants
dbg!: prints the expression and its value to stderr, and returns the value
todo!, unimplemented!, unreachable!, panic!: control flow markers

Types and Structs

Option<T> with Some(T) and None
Result<T, E> with Ok(T) and Err(E)
String: an owned, heap-allocated UTF-8 string
Vec<T>: a growable, heap-allocated array
Box<T>: a heap pointer with one owner

Traits

This category is the most important. Normal Rust syntax uses these traits:

Clone: the explicit .clone() method
Copy: an implicit bitwise copy on assignment
Drop: the destructor that runs when a value leaves scope
Iterator: the .map(), .filter(), .collect() chain
IntoIterator: the for x in collection { } syntax
From<T>: String::from("hello") and other conversions
Into<T>: "hello".into() (a blanket implementation from From)
PartialEq, Eq: the == and != operators
PartialOrd, Ord: the <, >, <=, >= comparisons
Send, Sync: marker traits for thread safety
Sized: an implicit bound on most generics

A for loop compiles without an import, and the trait that it uses is IntoIterator. The ? operator propagates an error automatically, and the trait that it uses is From. The operator calls From::from() on the error value. This call converts the value to the error type that the function returns.

Use case: What the v1 prelude enables, visualized

fn main() {
    // println!, eprintln!, print!, eprint!: write to stdout or stderr
    println!("=== Macros from the prelude ===");
    println!("println! works with no import");
    eprintln!("eprintln! also works — goes to stderr");

    // vec!: constructs a Vec<T> inline
    let numbers = vec![10, 20, 30, 40, 50];
    println!("vec!: {numbers:?}");

    // format!: constructs a String
    let greeting = format!("Hello, {}!", "prelude");
    println!("format!: {greeting:?}");

    // assert!, assert_eq!, assert_ne!: panic if the condition is false
    let computed = 2_i32 + 2;
    assert_eq!(computed, 4);
    assert_ne!(2 + 2, 5);
    assert!(greeting.contains("Hello"));
    println!("Assertions passed");

    // dbg!: prints the source location, the expression, and the value to stderr.
    // Then it returns the value.
    let doubled = dbg!(6 * 7);
    assert_eq!(doubled, 42);

    println!("\n=== Types from the prelude ===");

    // Option<T>: no import is necessary
    let maybe: Option<i32> = Some(99);
    let nothing: Option<i32> = None;
    println!("Option: maybe={maybe:?}, nothing={nothing:?}");
    assert!(maybe.is_some());
    assert!(nothing.is_none());

    // Result<T, E>: no import is necessary
    let ok: Result<i32, &str> = Ok(42);
    let err: Result<i32, &str> = Err("something went wrong");
    println!("Result: ok={ok:?}, err={err:?}");
    assert!(ok.is_ok());
    assert!(err.is_err());

    // String: an owned heap string, no import is necessary
    let owned_str = String::from("owned string");
    println!("String: {owned_str:?}");

    // Box<T>, Vec<T>: owned heap types, no import is necessary
    let boxed: Box<i32> = Box::new(100);
    let bytes: Vec<u8> = vec![1, 2, 3];
    println!("Box: {boxed:?}, Vec: {bytes:?}");

    println!("\n=== Traits from the prelude ===");

    // Clone: explicit duplication
    let original = String::from("hello");
    let cloned = original.clone(); // Clone is in scope, so the method call resolves
    assert_eq!(original, cloned);
    println!("Clone: original={original:?}, cloned={cloned:?}");

    // Copy: implicit bitwise duplication
    let orig: i32 = 7;
    let copy_of_orig = orig; // i32 is Copy, so `orig` is still valid
    assert_eq!(orig, copy_of_orig);
    println!("Copy: orig={orig}, copy_of_orig={copy_of_orig}");

    // Drop: runs automatically when a value leaves scope
    {
        struct Loud(i32);
        impl Drop for Loud {
            fn drop(&mut self) {
                println!("  Drop called for Loud({})", self.0);
            }
        }
        let _loud = Loud(1);
        println!("About to leave inner scope...");
    } // <- Drop::drop runs here automatically
    println!("Left inner scope");

    // Iterator: its methods map and sum are in scope without an import
    let sum: i32 = numbers.iter().map(|&n| n * 2).sum();
    println!("\nfor loop via IntoIterator (no import needed):");
    // IntoIterator: the `for` loop calls into_iter on `&numbers`
    for n in &numbers {
        print!("  {n}");
    }
    println!();
    println!("sum of doubled: {sum}");
    assert_eq!(sum, 300); // 10+20+30+40+50 = 150, doubled = 300

    // From<T> / Into<T>: infallible conversions
    let from_i32: i64 = i64::from(42_i32);    // From is in the prelude
    let owned_hello: String = "hello".into(); // Into, through the blanket implementation
    println!("\nFrom/Into: from_i32={from_i32}, owned_hello={owned_hello:?}");
    assert_eq!(from_i32, 42);
    assert_eq!(owned_hello, "hello");

    println!("\nAll assertions passed.");
}

00_01_prelude_basics.rs prints the lines below. The eprintln! line and the dbg! line go to stderr. The dbg! line shows the path and the position in the example file. The path depends on the directory from which you build the example:

=== Macros from the prelude ===
println! works with no import
eprintln! also works — goes to stderr   # (stderr)
vec!: [10, 20, 30, 40, 50]
format!: "Hello, prelude!"
Assertions passed
[prologue-00-the-prelude/examples/src/bin/00_01_prelude_basics.rs:33:19] 6 * 7 = 42   # (stderr, path varies)

=== Types from the prelude ===
Option: maybe=Some(99), nothing=None
Result: ok=Ok(42), err=Err("something went wrong")
String: "owned string"
Box: 100, Vec: [1, 2, 3]

=== Traits from the prelude ===
Clone: original="hello", cloned="hello"
Copy: orig=7, copy_of_orig=7
About to leave inner scope...
  Drop called for Loud(1)
Left inner scope

for loop via IntoIterator (no import needed):
  10  20  30  40  50
sum of doubled: 300

From/Into: from_i32=42, owned_hello="hello"

All assertions passed.

Examine the for loop. You write this loop:

// numbers: the Vec<i32> from the example above
for n in &numbers { ... }

The compiler desugars the loop to code that is almost the same as this:

// `&Vec<i32>` implements IntoIterator, and into_iter returns a slice iterator.
let mut iter = IntoIterator::into_iter(&numbers);
// next returns Some(&i32) for each element, and then None.
while let Some(n) = iter.next() { ... }

The compiler finds IntoIterator itself, so the loop does not need the trait name in scope. The prelude puts the same trait in scope for your code. Thus you can also write the desugared form, or call .into_iter(), without an import.

The 2021 Prelude: TryFrom, TryInto, FromIterator

Rust 2021 added three traits to the prelude. Before this edition, these traits were so common that a manual use statement for them was boilerplate in almost all code.

TryFrom and TryInto

Before edition 2021, you had to write use std::convert::TryFrom; before a call such as u8::try_from(some_value). In edition 2021 and later, TryFrom and TryInto are in the prelude.

Rust 1.34 stabilized the TryFrom and TryInto traits, but the prelude could include them only in a new edition. The traits introduced new method names (try_from, try_into), and these names could have conflicted with existing code.

Use case: Numeric narrowing and validated types without imports

// Converts an i64 to a u8, or returns the conversion error.
fn narrow_checked(v: i64) -> Result<u8, std::num::TryFromIntError> {
    let n: u8 = v.try_into()?; // on Err, ? returns the error to the caller
    Ok(n)
}

fn main() {
    println!("=== TryFrom (2021 prelude addition) ===");

    // In edition 2018 you had to write: use std::convert::TryFrom;
    // In edition 2021 and later, the trait is in scope.

    let big: i32 = 300;
    let result: Result<u8, _> = u8::try_from(big); // 300 is larger than u8::MAX (255)
    println!("u8::try_from(300i32) = {result:?}");
    assert!(result.is_err());

    let small: i32 = 42;
    let result: Result<u8, _> = u8::try_from(small); // 42 fits in a u8
    println!("u8::try_from(42i32)  = {result:?}");
    assert_eq!(result, Ok(42));

    // Signed narrowing: -1i32 cannot fit in u8
    let neg: i32 = -1;
    let result: Result<u8, _> = u8::try_from(neg);
    println!("u8::try_from(-1i32)  = {result:?}");
    assert!(result.is_err());

    println!("\n=== TryInto (2021 prelude addition) ===");

    // TryInto is the blanket counterpart of TryFrom.
    // The type annotation on the left selects the target type.
    let wide: i64 = 1_000_000;
    let narrow: Result<i16, _> = wide.try_into(); // i16::MAX is 32_767
    println!("1_000_000i64.try_into::<i16>() = {narrow:?}");
    assert!(narrow.is_err());

    let fine: i64 = 100;
    let narrow: Result<i16, _> = fine.try_into();
    println!("100i64.try_into::<i16>()       = {narrow:?}");
    assert_eq!(narrow, Ok(100i16));

    // The ? operator works with TryInto in functions that return Result.
    println!("narrow_checked(200) = {:?}", narrow_checked(200));
    println!("narrow_checked(999) = {:?}", narrow_checked(999)); // 999 does not fit in a u8
    assert!(narrow_checked(200).is_ok());
    assert!(narrow_checked(999).is_err());

    println!("\n=== FromIterator (2021 prelude addition) ===");

    // collect() is a method of Iterator, and FromIterator is its trait bound.
    // The type annotation selects the FromIterator implementation.

    // Collect into Vec<T>
    let squares: Vec<u32> = (1..=5).map(|n| n * n).collect();
    println!("squares: {squares:?}");
    assert_eq!(squares, [1, 4, 9, 16, 25]);

    // Collect into String: String implements FromIterator<char>
    let vowels: String = "hello world".chars().filter(|c| "aeiou".contains(*c)).collect();
    println!("vowels: {vowels:?}");
    assert_eq!(vowels, "eoo");

    // Collect into Result<Vec<T>, E>: stops at the first Err
    let inputs = ["1", "2", "3", "4"];
    let parsed: Result<Vec<i32>, _> = inputs.iter().map(|s| s.parse::<i32>()).collect();
    println!("parsed ok: {parsed:?}");
    assert_eq!(parsed, Ok(vec![1, 2, 3, 4]));

    let mixed = ["1", "oops", "3"]; // "oops" is not a number
    let parsed_err: Result<Vec<i32>, _> = mixed.iter().map(|s| s.parse::<i32>()).collect();
    println!("parsed err: {parsed_err:?}");
    assert!(parsed_err.is_err());

    // Collect into HashMap<K, V>: each (key, value) tuple becomes one entry
    let map: std::collections::HashMap<&str, usize> =
        ["alpha", "beta", "gamma"].iter().enumerate().map(|(i, s)| (*s, i)).collect();
    println!("map: {map:?}"); // the order of the entries changes between runs
    assert_eq!(map["alpha"], 0);
    assert_eq!(map["gamma"], 2);

    println!("\nAll assertions passed.");
}

00_02_prelude_2021.rs prints:

=== TryFrom (2021 prelude addition) ===
u8::try_from(300i32) = Err(TryFromIntError(PosOverflow))
u8::try_from(42i32)  = Ok(42)
u8::try_from(-1i32)  = Err(TryFromIntError(NegOverflow))

=== TryInto (2021 prelude addition) ===
1_000_000i64.try_into::<i16>() = Err(TryFromIntError(PosOverflow))
100i64.try_into::<i16>()       = Ok(100)
narrow_checked(200) = Ok(200)
narrow_checked(999) = Err(TryFromIntError(PosOverflow))

=== FromIterator (2021 prelude addition) ===
squares: [1, 4, 9, 16, 25]
vowels: "eoo"
parsed ok: Ok([1, 2, 3, 4])
parsed err: Err(ParseIntError { kind: InvalidDigit })
map: {"gamma": 2, "beta": 1, "alpha": 0}   # (order varies)

All assertions passed.

Why FromIterator needed a new edition

FromIterator is the trait that collect() uses to build its result. collect() is a method of Iterator, so it compiles in all editions without an import of FromIterator. Before edition 2021, code that used the trait name directly needed use std::iter::FromIterator;. Two examples are the call Vec::from_iter(iter) and a C: FromIterator<T> bound.

The 2021 prelude removed the need for this import. The change needed a new edition for the same reason as TryFrom and TryInto: the method name from_iter could conflict with existing code.

The 2024 Prelude: Future, IntoFuture

Rust 2024 adds two traits for async code. Async Rust matured, and Future became common in function signatures and trait bounds. Thus an explicit import was an unnecessary step.

Future

Future is the core trait of the async system of Rust. A Future<Output = T> represents an asynchronous computation that will produce a T at a later time. Each async fn returns an anonymous type that implements Future.

In editions before 2024, a function that accepts or names a Future in its signature needs use std::future::Future. In edition 2024, the trait is in scope automatically.

IntoFuture

IntoFuture is the trait that .await uses internally. When you write some_value.await, Rust calls IntoFuture::into_future(some_value) and then polls the result. Thus builders and other types can be directly awaitable, although they are not a Future themselves. This pattern is common in async database libraries and async HTTP libraries.

Use case: Future and IntoFuture in function signatures

use std::pin::Pin;
use std::task::Context;
use std::task::Poll;

// A minimal manual Future that completes on the first poll.
// The name `Future` is in scope without a `use` statement in edition 2024.
struct Ready<T>(Option<T>);

impl<T: Unpin> Future for Ready<T> {
    type Output = T;
    fn poll(mut self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<T> {
        // take() moves the value out of the Option and leaves None.
        Poll::Ready(self.0.take().expect("polled after completion"))
    }
}

// `impl Future<Output = i32>` in a signature needs no import in edition 2024.
fn make_ready(value: i32) -> impl Future<Output = i32> {
    Ready(Some(value))
}

// A builder type that becomes awaitable through IntoFuture.
struct FetchBuilder {
    url: String,
    timeout_ms: u64,
}

// The Future that a FetchBuilder becomes.
struct FetchFuture {
    url: String,
    timeout_ms: u64,
}

impl Future for FetchFuture {
    type Output = String;
    fn poll(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<String> {
        // A simulated result: real code does async I/O here.
        Poll::Ready(format!("response from {} (timeout={}ms)", self.url, self.timeout_ms))
    }
}

impl IntoFuture for FetchBuilder {
    type Output = String;
    type IntoFuture = FetchFuture;

    // Moves the fields of the builder into the Future.
    fn into_future(self) -> FetchFuture {
        FetchFuture { url: self.url, timeout_ms: self.timeout_ms }
    }
}

fn main() {
    println!("=== Future (2024 prelude addition) ===");

    // This example has no async runtime, so it does not poll `fut`.
    let fut = make_ready(99);
    // The two lines print a size in bytes. A Ready<i32> has the size of an Option<i32>.
    println!("Created a Future: {}", std::mem::size_of_val(&fut));
    println!("Future size in bytes: {}", std::mem::size_of::<Ready<i32>>());
    assert_eq!(std::mem::size_of::<Ready<i32>>(), std::mem::size_of::<Option<i32>>());

    println!("\n=== IntoFuture (2024 prelude addition) ===");

    // FetchBuilder is not a Future, but it implements IntoFuture.
    // In async code you write: let response = builder.await;
    // This example calls into_future() directly:
    let builder = FetchBuilder { url: String::from("https://example.com"), timeout_ms: 5000 };
    let future = IntoFuture::into_future(builder); // future: FetchFuture
    println!("Created FetchFuture from FetchBuilder via IntoFuture");
    // On a 64-bit target: String (24 bytes) + u64 (8 bytes).
    println!("IntoFuture size: {} bytes", std::mem::size_of_val(&future));

    println!("\n=== What changes in edition 2024 ===");

    println!("Edition 2024 prelude summary:");
    println!("  Added to prelude: Future, IntoFuture");
    println!("  Inherited from 2021: TryFrom, TryInto, FromIterator");
    println!("  Inherited from v1: Option, Result, String, Vec, Clone, Copy,");
    println!("                     Drop, Iterator, IntoIterator, From, Into,");
    println!("                     println!, vec!, format!, assert!, ...");

    // The additions of all three prelude layers, with no `use` statement for them:
    let squares: Vec<u32> = (1_u32..=4).map(|n| n * n).collect(); // FromIterator (2021)
    let n: Result<u8, _> = u8::try_from(255_u32);                 // TryFrom (2021)
    let s: String = "hello".into();                               // Into (v1)

    assert_eq!(squares, [1, 4, 9, 16]);
    assert_eq!(n, Ok(255));
    assert_eq!(s, "hello");

    println!("\nAll assertions passed.");
}

00_03_prelude_2024.rs prints:

=== Future (2024 prelude addition) ===
Created a Future: 8
Future size in bytes: 8

=== IntoFuture (2024 prelude addition) ===
Created FetchFuture from FetchBuilder via IntoFuture
IntoFuture size: 32 bytes

=== What changes in edition 2024 ===
Edition 2024 prelude summary:
  Added to prelude: Future, IntoFuture
  Inherited from 2021: TryFrom, TryInto, FromIterator
  Inherited from v1: Option, Result, String, Vec, Clone, Copy,
                     Drop, Iterator, IntoIterator, From, Into,
                     println!, vec!, format!, assert!, ...

All assertions passed.

The "Mystery Trait" Problem

Figure: Diagnosing "Where Does This Trait Come From?"

00_1_prelude_1

Code such as this is one of the most common causes of confusion in Rust:

// This code has no `use` statement, but each line compiles.
// some_iterator: an iterator of i32 values. my_vec: a Vec.
let v: Vec<i32> = some_iterator.collect();
let s: String = "hello".into();
for item in my_vec { ... }
let x: u8 = 255_u32.try_into().unwrap(); // 255 fits in a u8, so unwrap does not panic

The prelude gives the explanation:

collect() works because Iterator is in the prelude (v1). It builds the Vec through FromIterator (in the prelude since 2021).
.into() works because Into is in the prelude (v1).
for uses IntoIterator, which is in the prelude (v1).
.try_into() works because TryInto is in the prelude (2021+).

When a method or a type has no visible source, examine std::prelude first.

Summary

Prelude contents	Since edition
Core types: `Option`, `Result`, `String`, `Vec`, `Box`	v1 (all editions)
Core traits: `Clone`, `Copy`, `Drop`, `Send`, `Sync`, `Sized`	v1
Conversion traits: `From`, `Into`	v1
Iterator traits: `Iterator`, `IntoIterator`	v1
Comparison traits: `PartialEq`, `Eq`, `PartialOrd`, `Ord`	v1
Macros: `println!`, `vec!`, `format!`, `assert!`, `dbg!`, `todo!`, …	v1
Fallible conversions: `TryFrom`, `TryInto`	2021
Collection building: `FromIterator`	2021
Async primitives: `Future`, `IntoFuture`	2024

In the remainder of this course, when a trait method or a type appears without a use statement, it comes from the prelude. This table shows the primary items, and std::prelude shows the full list.

Code Examples

File	Description
`00_01_prelude_basics.rs`	v1 prelude: macros, types, Clone, Copy, Drop, Iterator, From/Into
`00_02_prelude_2021.rs`	2021 additions: TryFrom, TryInto, FromIterator without imports
`00_03_prelude_2024.rs`	2024 additions: Future and IntoFuture, and all three layers together

1.1 · Clone, Copy, and Drop: The Lifecycle Trio

Domain 1 — Ownership Mechanics in the Standard Library Duration: ~15 minutes Library components: std::clone::Clone, std::marker::Copy, std::ops::Drop, clone_from

Introduction

Each value in Rust has exactly one owner. When that owner goes out of scope, Rust destroys the value. Three traits of the standard library control how Rust duplicates and destroys values: Copy, Clone, and Drop. The three traits interact, and the interactions can cause mistakes, also for experienced developers.

This tutorial shows:

When the compiler makes implicit copies, and when you must clone explicitly.
The rule that Copy and Drop are mutually exclusive.
How clone_from prevents unnecessary allocations.
The performance cost of Clone on collection types in real programs.

This tutorial assumes that you know ownership and borrowing. It examines the standard library traits that extend those concepts.

Figure: Copy, Clone, or Move?

01_1_clone_copy_drop_1

Copy: Implicit Bitwise Duplication

Copy is a marker trait: it has no methods. When a type implements Copy, the compiler can duplicate a value with a bitwise memcpy. The compiler does this when you assign the value, pass it to a function, or return it. The copy is implicit. You never call a .copy() method.

Which types are Copy?

These types are Copy:

All primitive scalar types (i32, f64, bool, char).
References (&T).
Raw pointers.
Tuples and arrays that contain only Copy types.

String, Vec<T>, and Box<T> are not Copy. No type that owns heap memory is Copy.

Use case: Primitive values survive assignment

When you assign an integer to a second variable, the two variables stay usable. The reason is that i32 is Copy. Without that trait, the assignment would be a move.

fn main() {
    let x: i32 = 42;
    let y = x; // implicit copy: x stays valid
    println!("x = {x}, y = {y}");
    assert_eq!(x, 42);
    assert_eq!(y, 42);

    // A tuple of Copy types is also Copy.
    let point = (3.0_f64, 4.0_f64);
    let other = point; // implicit copy: point stays valid
    println!("point = {point:?}, other = {other:?}");
    assert_eq!(point, other);

    // String is not Copy: the assignment moves ownership.
    let s1 = String::from("hello");
    let s2 = s1; // move: s1 is not valid after this line
    // println!("{s1}");  // compile error E0382: borrow of moved value: `s1`
    println!("s2 = {s2}");

    // A struct can derive Copy when all its fields are Copy.
    // Clone is a supertrait of Copy, so the derive list has the two traits.
    #[derive(Debug, Clone, Copy)]
    struct Pixel { r: u8, g: u8, b: u8 }

    let red = Pixel { r: 255, g: 0, b: 0 };
    let also_red = red; // implicit copy: red stays valid
    println!("red = {red:?}, also_red = {also_red:?}");
    assert_eq!(red.r, also_red.r);

    // A function call copies a Copy argument.
    fn double(n: i32) -> i32 { n * 2 }

    let val = 10;
    let doubled = double(val); // the function gets a copy of val
    println!("val = {val}, doubled = {doubled}"); // val is still usable
    assert_eq!(val, 10);
}

01_01_copy_semantics.rs prints:

x = 42, y = 42
point = (3.0, 4.0), other = (3.0, 4.0)
s2 = hello
red = Pixel { r: 255, g: 0, b: 0 }, also_red = Pixel { r: 255, g: 0, b: 0 }
val = 10, doubled = 20

The important point: Copy is an opt-in promise to the compiler that a bitwise duplicate of your type is semantically correct. The compiler accepts the promise. Thus it does not call custom code. It only copies the bytes.

Clone: Explicit, Potentially Expensive Duplication

Clone is a regular trait with a method: fn clone(&self) -> Self. A call to clone() is always explicit, which is different from Copy. The implementation can do any operation. For example, it can allocate heap memory, copy nested structures, or increment reference counts.

Each Copy type is also Clone, because Clone is a supertrait of Copy. But not all Clone types are Copy.

Use case: Duplicating heap-owning types

String, Vec<T>, and each struct that contains them need an explicit .clone() call to make a second independent copy.

fn main() {
    // String is Clone but NOT Copy.
    let original = String::from("deep sea");
    let cloned = original.clone(); // allocates a new heap buffer
    println!("original = {original:?}");
    println!("cloned   = {cloned:?}");
    assert_eq!(original, cloned);

    // The two strings are independent: a change to one does not change the other.
    let mut cloned = cloned; // moves the clone into a mutable binding
    cloned.push_str(" diving");
    println!("after mutation:");
    println!("  original = {original:?}");
    println!("  cloned   = {cloned:?}");
    assert_ne!(original, cloned);

    // A struct can derive Clone when all its fields are Clone.
    #[derive(Debug, Clone)]
    struct Document {
        title: String,
        pages: Vec<String>,
    }

    let doc = Document {
        title: String::from("Rust Handbook"),
        pages: vec![String::from("Chapter 1"), String::from("Chapter 2")],
    };

    let doc2 = doc.clone(); // deep clone: duplicates the title AND each page
    println!("\ndoc  = {doc:?}");
    println!("doc2 = {doc2:?}");
    assert_eq!(doc.title, doc2.title);
}

01_02_clone_vs_copy.rs prints these lines for the code above:

original = "deep sea"
cloned   = "deep sea"
after mutation:
  original = "deep sea"
  cloned   = "deep sea diving"

doc  = Document { title: "Rust Handbook", pages: ["Chapter 1", "Chapter 2"] }
doc2 = Document { title: "Rust Handbook", pages: ["Chapter 1", "Chapter 2"] }

The derived Clone does not show an important cost. A clone of a Document does 2 + N allocations. One is for the title, one is for the Vec buffer, and N are for the page strings. The cost increases linearly with the quantity of data. A later section of this tutorial measures this cost.

The Copy/Drop Mutual Exclusion

Drop is the trait that gives a type a destructor: fn drop(&mut self). The compiler calls the destructor automatically when a value goes out of scope.

The rule is: a type cannot implement Copy and Drop together. The compiler enforces this rule.

The reason is that Copy means "a copy of the bytes is a valid duplicate". If a type has a destructor, a bitwise copy would make two owners of the same resource. When the two owners go out of scope, the destructor runs two times. The result is a double free, a double close, or a second run of some other resource cleanup.

Figure: Copy and Drop Are Mutually Exclusive

01_1_clone_copy_drop_2

Use case: Why file handles are not Copy

fn main() {
    // A type that prints a line when Rust drops it.
    #[derive(Debug)]
    struct Noisy { id: u32 }

    impl Drop for Noisy {
        fn drop(&mut self) {
            println!("  Dropping Noisy(id={})", self.id);
        }
    }

    // Noisy implements Drop, so it CANNOT implement Copy.
    // impl Copy for Noisy {}  // compile error E0184: the type has a destructor

    println!("Creating two Noisy values:");
    {
        let a = Noisy { id: 1 };
        let b = Noisy { id: 2 };
        println!("  a = {a:?}, b = {b:?}");
        // The scope ends here: b drops first (reverse declaration order), then a.
    }
    println!();

    // Contrast: a Copy type has no destructor.
    #[derive(Debug, Clone, Copy)]
    struct Point { x: f32, y: f32 }

    let p1 = Point { x: 1.0, y: 2.0 };
    let p2 = p1; // bitwise copy: no destructor runs
    let p3 = p1; // a second copy: p1 is still valid
    println!("p1 = {p1:?}");
    println!("p2 = {p2:?}");
    println!("p3 = {p3:?}");
}

01_03_copy_drop_exclusion.rs prints these lines for the code above:

Creating two Noisy values:
  a = Noisy { id: 1 }, b = Noisy { id: 2 }
  Dropping Noisy(id=2)
  Dropping Noisy(id=1)

p1 = Point { x: 1.0, y: 2.0 }
p2 = Point { x: 1.0, y: 2.0 }
p3 = Point { x: 1.0, y: 2.0 }

This mutual exclusion is not an arbitrary rule. It is a direct result of memory safety. If std::fs::File were Copy, an assignment would silently make two file handles for the same OS descriptor. If the program then closed the two handles, the behavior would be undefined.

Drop Order

It is also important to know when destructors run and in what order.

There are three rules:

Local variables drop in reverse declaration order: the last variable drops first.
Struct fields drop in declaration order: the first field drops first.
Vec elements drop in front-to-back order: index 0 drops first.

To destroy a value before the end of its scope, call std::mem::drop(value).

Use case: Releasing a lock before doing more work

fn main() {
    // A type that prints its name when Rust drops it.
    #[derive(Debug)]
    struct Named(&'static str);

    impl Drop for Named {
        fn drop(&mut self) {
            println!("  Dropped: {}", self.0);
        }
    }

    // Local variables: reverse declaration order.
    println!("=== Local variable drop order ===");
    {
        let _first = Named("first");
        let _second = Named("second");
        let _third = Named("third");
        println!("  All three are alive");
        // The scope ends here: the drop order is third, second, first.
    }

    // Manual drop before the end of the scope.
    println!("\n=== Early drop with std::mem::drop ===");
    {
        let guard = Named("mutex_guard"); // represents a lock guard
        let _resource = Named("resource");
        println!("  Both alive — doing work...");
        drop(guard); // moves guard into drop(), which destroys it immediately
        println!("  Guard released — resource still alive");
        // The scope ends here: only _resource drops.
    }

    // Vec elements: front to back.
    println!("\n=== Vec element drop order ===");
    {
        let _v = vec![Named("vec[0]"), Named("vec[1]"), Named("vec[2]")];
        println!("  Vec is alive");
        // The scope ends here: the drop order is vec[0], vec[1], vec[2].
    }
}

01_06_drop_order.rs prints these lines for the code above:

=== Local variable drop order ===
  All three are alive
  Dropped: third
  Dropped: second
  Dropped: first

=== Early drop with std::mem::drop ===
  Both alive — doing work...
  Dropped: mutex_guard
  Guard released — resource still alive
  Dropped: resource

=== Vec element drop order ===
  Vec is alive
  Dropped: vec[0]
  Dropped: vec[1]
  Dropped: vec[2]

Figure: Drop Order — Locals vs. Struct Fields vs. Vec Elements

01_1_clone_copy_drop_3

clone_from: Reusing Existing Allocations

Most developers know .clone(). Fewer developers know .clone_from(). Its signature is:

// A provided method of the Clone trait. `self` is the destination.
fn clone_from(&mut self, source: &Self) { ... }

The default implementation is *self = source.clone(). It discards the old value and makes a new allocation. But String and Vec override the method to use the existing heap buffer again when the buffer is sufficiently large. This prevents one deallocation and one new allocation.

Use case: A processing loop that reuses a buffer each iteration

fn main() {
    // The destination: a string with a heap buffer of at least 100 bytes.
    let mut buffer = String::with_capacity(100);
    buffer.push_str("old content that allocated a nice big buffer");

    let ptr_before = buffer.as_ptr(); // the address of the heap buffer
    let cap_before = buffer.capacity();
    println!("Before clone_from:");
    println!("  buffer = {buffer:?}");
    println!("  capacity = {cap_before}");

    let source = String::from("new");
    buffer.clone_from(&source); // copies "new" into the existing buffer

    let ptr_after = buffer.as_ptr();
    let cap_after = buffer.capacity();
    println!("\nAfter clone_from:");
    println!("  buffer = {buffer:?}");
    println!("  capacity = {cap_after}");

    assert_eq!(buffer, "new");
    // The pointer is the same: clone_from used the same allocation.
    assert_eq!(ptr_before, ptr_after);
    // The capacity is the same: clone_from did not shrink the buffer.
    assert_eq!(cap_before, cap_after);

    // A processing loop that uses one buffer for all the messages.
    println!("\n--- Simulated processing loop ---");
    let incoming_messages = vec![
        String::from("msg-alpha"),
        String::from("msg-beta-longer"),
        String::from("msg-gamma"),
    ];

    let mut work_buffer = String::with_capacity(64);
    for msg in &incoming_messages {
        // Each message fits in the 64 bytes, so this call does not allocate.
        work_buffer.clone_from(msg);
        println!("  Processing: {work_buffer:?} (cap={})", work_buffer.capacity());
    }
}

01_04_clone_from.rs prints these lines for the code above:

Before clone_from:
  buffer = "old content that allocated a nice big buffer"
  capacity = 100

After clone_from:
  buffer = "new"
  capacity = 100

--- Simulated processing loop ---
  Processing: "msg-alpha" (cap=64)
  Processing: "msg-beta-longer" (cap=64)
  Processing: "msg-gamma" (cap=64)

The capacity stays at 64 in all three iterations. Without clone_from, each iteration would allocate a new buffer and deallocate the old buffer.

The general rule: when you overwrite a variable that already has an allocation, prefer dest.clone_from(&src) to dest = src.clone().

Performance Implications of Clone on Collections

A clone has a cost, and the cost depends very much on the contents of the collection.

Vec<i32>.clone() does one memcpy of the backing buffer. It is fast.
Vec<String>.clone() clones each String in the vector, and each clone allocates. The cost is O(n) in the element count and in the total string length.
HashMap<String, Vec<u8>>.clone() clones each key and each value. It is expensive.

Use case: Measuring and avoiding clone costs

use std::time::Instant;

fn main() {
    // 100,000 strings: each string has its own heap buffer.
    let big_vec: Vec<String> = (0..100_000).map(|i| format!("element-{i:06}")).collect();

    let start = Instant::now();
    let cloned_vec = big_vec.clone(); // clones each String: one allocation for each element
    let clone_time = start.elapsed(); // the Duration of the clone
    println!(
        "Cloning Vec<String> with {} elements: {:?}",
        big_vec.len(),
        clone_time
    );
    assert_eq!(big_vec.len(), cloned_vec.len());

    // 100,000 integers: all the data is in one buffer.
    let int_vec: Vec<i32> = (0..100_000).collect();

    let start = Instant::now();
    let cloned_ints = int_vec.clone(); // one memcpy of the backing buffer
    let int_clone_time = start.elapsed();
    println!(
        "Cloning Vec<i32> with {} elements:    {:?}",
        int_vec.len(),
        int_clone_time
    );
    assert_eq!(int_vec.len(), cloned_ints.len());

    // `.max(1)` prevents a division by zero if the measured time is 0 ns.
    let ratio = clone_time.as_nanos() as f64 / int_clone_time.as_nanos().max(1) as f64;
    println!("\nVec<String> clone was roughly {ratio:.0}x slower than Vec<i32> clone");
}

01_05_clone_performance.rs prints these lines for the code above. The times and the ratio are different in each run:

Cloning Vec<String> with 100000 elements: 1.46925ms   # (varies)
Cloning Vec<i32> with 100000 elements:    30.75µs   # (varies)

Vec<String> clone was roughly 48x slower than Vec<i32> clone   # (varies)

(The exact numbers vary by machine, but the order-of-magnitude difference is consistent.)

Strategies to avoid unnecessary clones

Pass a slice (&[T]), and do not clone the collection. If you only read the data, borrow it.
Clone one element, not the full collection. If you need one item, clone that item.
Consume the collection with into_iter(). If you do not need the original collection after this point, move the elements and do not clone them.
Use clone_from when you overwrite a value that has an allocation (see the clone_from section).

Summary

Trait	Method	Implicit?	Cost	Requires
`Copy`	None (the compiler inserts a memcpy)	Yes	Low (bitwise copy)	All fields must be `Copy`, and the type has no `Drop` impl
`Clone`	`.clone()`	No (explicit call)	Arbitrary (may allocate)	All fields must be `Clone`
`Drop`	`.drop()` (the compiler calls it)	Yes	Arbitrary (cleanup code)	The type cannot be `Copy`

The three traits make one system:

Copy means that duplication is trivially safe: the compiler only copies the bytes.
Clone means that duplication needs work: you call the clone method.
Drop means that destruction needs work: the compiler calls the destructor.
Copy together with Drop is not permitted, because trivial duplication and non-trivial destruction are incompatible.

clone_from is the performance tool for Clone types. It lets you use an existing allocation again when you overwrite a value.

Code Examples

File	Description
`01_01_copy_semantics.rs`	Implicit copies of primitives, tuples, and custom structs
`01_02_clone_vs_copy.rs`	Explicit clones of `String`, `Vec`, and nested structs
`01_03_copy_drop_exclusion.rs`	Why a type cannot implement `Copy` and `Drop` together
`01_04_clone_from.rs`	How `clone_from` uses an existing allocation again
`01_05_clone_performance.rs`	The clone cost of collections, and how to prevent unnecessary clones
`01_06_drop_order.rs`	Drop order of local variables, struct fields, and `Vec` elements, and the early drop

1.2 · Borrow, ToOwned, and Cow: Flexible Ownership Boundaries

Domain 1 — Ownership Mechanics in the Standard Library Duration: ~15 minutes Library components: std::borrow::Borrow, std::borrow::BorrowMut, std::borrow::ToOwned, std::borrow::Cow

Introduction

The ownership model of Rust gives you safety guarantees. It also causes a practical problem: a function frequently must accept data in borrowed form and in owned form. For example, you must select between a &str parameter and a String parameter, or between &[u8] and Vec<u8>.

The std::borrow module has three tools that solve this problem:

Borrow<T> gives you a &T. The trait promises that the &T hashes and compares the same as the original value.
ToOwned makes the related owned value from a borrowed value.
Cow<'a, B> contains borrowed data or owned data. The program selects the variant at runtime, and Cow allocates only when it is necessary.

This tutorial explains how each tool operates and when to use it. It also explains how Cow removes a common class of unnecessary allocations.

Borrow: The Hash/Eq Contract

The definition of the Borrow trait is short:

// `?Sized` permits unsized targets such as `str` and `[T]`.
pub trait Borrow<Borrowed: ?Sized> {
    fn borrow(&self) -> &Borrowed;
}

String implements Borrow<str>. Vec<T> implements Borrow<[T]>. AsRef also converts references. The difference between the two traits is a semantic contract. The compiler does not enforce this contract, but each implementor must obey it:

If T: Borrow<Q>, then t.borrow() must give a value that hashes identically and compares equally to t.

HashMap::get depends on this contract. When you call .get("some_key") on a HashMap<String, V>, the map hashes your &str. Then it searches for an equal String key. If the Borrow<str> implementation of String did not keep the hash and the equality consistent, lookups would silently fail.

Use case: Looking up HashMap entries with &str keys

use std::borrow::Borrow;
use std::collections::HashMap;

fn main() {
    let mut scores: HashMap<String, u32> = HashMap::new();
    scores.insert(String::from("Alice"), 100);
    scores.insert(String::from("Bob"), 85);

    // HashMap::get accepts a &Q for each Q where String: Borrow<Q>.
    // String: Borrow<str>, so a &str is a valid key. You do not need a String.
    let alice_score = scores.get("Alice"); // Option<&u32>
    println!("Alice's score: {alice_score:?}");
    assert_eq!(alice_score, Some(&100));

    // A generic function with the same bounds as HashMap::get.
    fn find_score<Q>(map: &HashMap<String, u32>, key: &Q) -> Option<u32>
    where
        String: Borrow<Q>,
        Q: std::hash::Hash + Eq + ?Sized, // ?Sized permits Q = str
    {
        map.get(key).copied() // Option<&u32> becomes Option<u32>
    }

    let s1 = find_score(&scores, "Alice");      // key is &str: Q = str
    let owned_key = String::from("Bob");
    let s2 = find_score(&scores, &owned_key);   // key is &String: Q = String
    println!("find_score(&str):    {s1:?}");
    println!("find_score(&String): {s2:?}");
    assert_eq!(s1, Some(100));
    assert_eq!(s2, Some(85));

    // Check the hash/eq contract.
    use std::hash::{DefaultHasher, Hash, Hasher};

    // Returns the hash of one value. ?Sized permits T = str.
    fn compute_hash<T: Hash + ?Sized>(val: &T) -> u64 {
        let mut h = DefaultHasher::new();
        val.hash(&mut h);
        h.finish()
    }

    let owned = String::from("rust");
    let borrowed: &str = owned.borrow();
    let h1 = compute_hash(&owned);   // the hash of the String
    let h2 = compute_hash(borrowed); // the hash of the str
    println!("Hash of String 'rust': {h1}");
    println!("Hash of &str 'rust':   {h2}");
    assert_eq!(h1, h2, "Borrow contract: hashes must match");
}

01_07_borrow_trait.rs prints these lines for the code above. The two hash values are always equal, but the value can change between Rust releases:

Alice's score: Some(100)
find_score(&str):    Some(100)
find_score(&String): Some(85)
Hash of String 'rust': 18297823557882888176   # (can change between Rust releases)
Hash of &str 'rust':   18297823557882888176   # (can change between Rust releases)

The important difference from AsRef: implement Borrow<Q> for your type only if you can guarantee that the borrowed form hashes and compares identically. Tutorial 1.3 explains when AsRef is the correct selection.

ToOwned: From Borrowed to Owned

ToOwned is the inverse of Borrow. Borrow goes from owned to borrowed. ToOwned goes from borrowed to owned:

pub trait ToOwned {
    // The owned type. It must be able to give a borrow of `Self` back.
    type Owned: Borrow<Self>;
    fn to_owned(&self) -> Self::Owned;
    // (The trait also has the provided method `clone_into`.)
}

The standard library has these implementations:

Borrowed type	`to_owned()` returns
`str`	`String`
`[T]`	`Vec<T>`
`Path`	`PathBuf`
`OsStr`	`OsString`
`CStr`	`CString`

Why Clone is not sufficient

Clone on a reference type clones the reference, not the data behind the reference. A clone of a &str gives you a second &str. ToOwned gives you a String.

Use case: Going from borrowed to owned across type boundaries

use std::borrow::Borrow;
use std::borrow::ToOwned;
use std::path::Path;

fn main() {
    // &str → String
    let greeting: &str = "hello";
    let owned: String = greeting.to_owned(); // allocates and copies the bytes
    println!("borrowed: {greeting:?}");
    println!("owned:    {owned:?}");
    assert_eq!(greeting, owned);

    // Contrast: Clone on a &str gives a second &str, not a String.
    // `greeting.clone()` is the same call, but the compiler warns that it does nothing.
    let cloned: &str = Clone::clone(&greeting);
    println!("cloned (still &str): {cloned:?}");

    // &[i32] → Vec<i32>
    println!("\n--- Slice to Vec ---");
    let slice: &[i32] = &[1, 2, 3, 4, 5];
    let vec: Vec<i32> = slice.to_owned();
    println!("slice: {slice:?}");
    println!("vec:   {vec:?}");
    assert_eq!(slice, vec.as_slice());

    // &Path → PathBuf
    println!("\n--- Path to PathBuf ---");
    let path: &Path = Path::new("/usr/local/bin");
    let path_buf: std::path::PathBuf = path.to_owned();
    println!("path:     {}", path.display());
    println!("path_buf: {}", path_buf.display());
    assert_eq!(path, path_buf);

    // Round trip: to_owned makes the String, borrow gives a &str back.
    println!("\n--- ToOwned ↔ Borrow round-trip ---");
    let original: &str = "round trip";
    let owned: String = original.to_owned();
    let back: &str = owned.borrow();
    println!("original:  {original:?}");
    println!("owned:     {owned:?}");
    println!("back:      {back:?}");
    assert_eq!(original, back);

    println!("\nAll assertions passed.");
}

01_08_to_owned.rs prints:

borrowed: "hello"
owned:    "hello"
cloned (still &str): "hello"

--- Slice to Vec ---
slice: [1, 2, 3, 4, 5]
vec:   [1, 2, 3, 4, 5]

--- Path to PathBuf ---
path:     /usr/local/bin
path_buf: /usr/local/bin

--- ToOwned ↔ Borrow round-trip ---
original:  "round trip"
owned:     "round trip"
back:      "round trip"

All assertions passed.

Figure: ToOwned and Borrow — The Ownership Round-Trip

01_2_borrow_toowned_cow_1

Cow: Clone-on-Write

Cow<'a, B> is an enum with two variants:

pub enum Cow<'a, B: ?Sized + ToOwned> {
    Borrowed(&'a B),              // a reference to the data: no allocation
    Owned(<B as ToOwned>::Owned), // the owned type: String when B is str
}

Thus a Cow<'a, str> is a &'a str or a String. A Cow<'a, [T]> is a &'a [T] or a Vec<T>.

The advantage of Cow is deferred allocation. You start with borrowed data. If you never mutate the data, you never allocate. If you must mutate the data, call .to_mut(). It clones the data into the owned variant and gives you a mutable reference.

Use case: A text function that usually returns the input unchanged

This is the standard use of Cow. A function sometimes must change its input, but usually it does not. Without Cow, you have two alternatives. You can always allocate a new String, which is wasteful. Or you can pass lifetimes through your API in complex ways.

use std::borrow::Cow;

fn main() {
    // Returns the input unchanged, or a changed copy of the input.
    fn remove_profanity(input: &str) -> Cow<'_, str> {
        if input.contains("darn") {
            // A change is necessary: `replace` allocates a new String.
            Cow::Owned(input.replace("darn", "****"))
        } else {
            // No change is necessary: borrow the input, no allocation.
            Cow::Borrowed(input)
        }
    }

    let clean = "have a nice day";
    let dirty = "oh darn, that's broken";

    let result1 = remove_profanity(clean); // Cow::Borrowed
    let result2 = remove_profanity(dirty); // Cow::Owned

    // matches! returns true when the value has the given variant.
    println!("clean input → {:?} (borrowed: {})", result1, matches!(result1, Cow::Borrowed(_)));
    println!("dirty input → {:?} (borrowed: {})", result2, matches!(result2, Cow::Borrowed(_)));

    assert!(matches!(result1, Cow::Borrowed(_)));
    assert!(matches!(result2, Cow::Owned(_)));
    assert_eq!(&*result2, "oh ****, that's broken"); // &* derefs the Cow to a &str
}

01_09_cow_basics.rs prints these lines for the code above:

clean input → "have a nice day" (borrowed: true)
dirty input → "oh ****, that's broken" (borrowed: false)

Some programs process millions of log lines, and only a small fraction of the lines need a change. For such a program, this pattern prevents millions of unnecessary allocations.

Cow with slices

Cow accepts each type that implements ToOwned, and slices are such a type:

use std::borrow::Cow;

fn main() {
    // Returns the input if it is sorted, or a sorted copy of the input.
    fn ensure_sorted(data: &[i32]) -> Cow<'_, [i32]> {
        // windows(2) gives each pair of adjacent elements.
        if data.windows(2).all(|w| w[0] <= w[1]) {
            Cow::Borrowed(data) // already sorted: no allocation
        } else {
            let mut owned = data.to_vec(); // copies the slice into a new Vec
            owned.sort_unstable();
            Cow::Owned(owned)
        }
    }

    let sorted = [1, 2, 3, 4, 5];
    let unsorted = [5, 3, 1, 4, 2];

    let r1 = ensure_sorted(&sorted);   // Cow::Borrowed
    let r2 = ensure_sorted(&unsorted); // Cow::Owned

    println!("already sorted → borrowed: {}", matches!(r1, Cow::Borrowed(_)));
    println!("unsorted       → borrowed: {}", matches!(r2, Cow::Borrowed(_)));
    assert_eq!(&*r2, &[1, 2, 3, 4, 5]); // r2 contains a sorted copy
}

01_09_cow_basics.rs prints these lines for the code above:

already sorted → borrowed: true
unsorted       → borrowed: false

Cow in API Design

Cow is most useful together with Into<Cow<'a, str>> in function signatures. &str and String each implement Into<Cow<str>>. Thus your API accepts the two types, and the caller does not have to select one.

Use case: A configuration struct that avoids allocating for literal keys

use std::borrow::Cow;
use std::collections::HashMap;

#[derive(Debug)]
struct Config<'a> {
    // Each key and each value is borrowed data or owned data.
    entries: HashMap<Cow<'a, str>, Cow<'a, str>>,
}

impl<'a> Config<'a> {
    fn new() -> Self {
        Config { entries: HashMap::new() }
    }

    // A &'a str becomes Cow::Borrowed. A String becomes Cow::Owned.
    fn set(&mut self, key: impl Into<Cow<'a, str>>, value: impl Into<Cow<'a, str>>) {
        self.entries.insert(key.into(), value.into());
    }

    // Cow<str> implements Borrow<str>, so a &str is a valid lookup key.
    fn get(&self, key: &str) -> Option<&str> {
        self.entries.get(key).map(|v| v.as_ref()) // &Cow<str> becomes &str
    }
}

fn main() {
    let mut config = Config::new();

    // String literals become Cow::Borrowed: no allocation.
    config.set("host", "localhost");
    config.set("port", "8080");

    // A String that the program makes at runtime becomes Cow::Owned.
    let user = std::env::var("USER").unwrap_or_else(|_| "anonymous".to_string());
    config.set("user", user);

    // A literal key with a runtime value.
    let db_url = format!("postgres://{}@localhost/mydb", config.get("user").unwrap());
    config.set("database_url", db_url);

    println!("Config entries:");
    for (k, v) in &config.entries {
        let kind = if matches!(v, Cow::Borrowed(_)) { "borrowed" } else { "owned" };
        println!("  {k} = {v} ({kind})");
    }

    assert_eq!(config.get("host"), Some("localhost"));
    assert_eq!(config.get("port"), Some("8080"));
}

01_10_cow_api_design.rs prints these lines for the code above. The order of the four entries changes between runs, because the iteration order of a HashMap is not specified. The user name is the value of the USER environment variable (here, admin):

Config entries:
  host = localhost (borrowed)   # (order varies)
  user = admin (owned)   # (varies with USER)
  port = 8080 (borrowed)
  database_url = postgres://admin@localhost/mydb (owned)   # (varies with USER)

Cow Mutation: to_mut and into_owned

Two methods control the change from borrowed to owned:

to_mut(&mut self) -> &mut B::Owned: If the Cow is borrowed, the method clones the data into the owned variant. Then it returns a mutable reference. If the Cow is already owned, the method only returns the mutable reference. This clone is the "clone-on-write" step.
into_owned(self) -> B::Owned: The method consumes the Cow. If the Cow is borrowed, the method clones the data. If the Cow is already owned, the method returns the owned value and does not clone.

Use case: Conditionally mutating borrowed data

use std::borrow::Cow;

fn main() {
    let mut cow: Cow<str> = Cow::Borrowed("immutable");
    println!("Before to_mut: {:?} (borrowed: {})", cow, matches!(cow, Cow::Borrowed(_)));

    // The first to_mut call clones "immutable" into a String.
    let mutable_ref = cow.to_mut(); // &mut String
    mutable_ref.push_str(" → now mutable!");
    println!("After to_mut:  {:?} (borrowed: {})", cow, matches!(cow, Cow::Borrowed(_)));
    assert!(matches!(cow, Cow::Owned(_)));

    // The second to_mut call does NOT clone: the Cow is already owned.
    println!("\n--- Second to_mut: no extra clone ---");
    let mutable_ref = cow.to_mut();
    mutable_ref.push_str(" (still the same allocation)");
    println!("After second to_mut: {cow:?}");

    // into_owned consumes the Cow and returns the owned value.
    println!("\n--- into_owned ---");
    let cow1: Cow<str> = Cow::Borrowed("borrow me");
    let cow2: Cow<str> = Cow::Owned(String::from("own me"));

    let s1: String = cow1.into_owned(); // borrowed: clones the data
    let s2: String = cow2.into_owned(); // owned: moves the String, no clone
    println!("s1 = {s1:?}");
    println!("s2 = {s2:?}");
}

01_11_cow_mutation.rs prints these lines for the code above:

Before to_mut: "immutable" (borrowed: true)
After to_mut:  "immutable → now mutable!" (borrowed: false)

--- Second to_mut: no extra clone ---
After second to_mut: "immutable → now mutable! (still the same allocation)"

--- into_owned ---
s1 = "borrow me"
s2 = "own me"

Figure: Cow State Transitions

01_2_borrow_toowned_cow_2

When to Use Each

Situation	Use
Collection lookups (`HashMap::get`)	`Borrow<T>` (the hash/eq contract is necessary)
Conversion of `&str` to `String`, or of `&[T]` to `Vec<T>`	`ToOwned`
A function that usually returns its input unchanged	`Cow<T>` (deferred allocation)
An API that accepts `&str` and `String`	`impl Into<Cow<str>>`, or the simpler `impl AsRef<str>` if you only read the data
A processing pipeline with conditional mutation	`Cow<T>` with `to_mut()`

A common mistake is to use Cow when a &str parameter is sufficient. Cow adds complexity. Use it when you have measured evidence that deferred allocation is important for your workload. Also use it when the API must be able to return borrowed data or owned data.

Summary

The std::borrow module connects owned data and borrowed data:

Borrow<T> is a contract: the borrowed form and the owned form are interchangeable for hashing and comparison. This contract is the reason that HashMap lookups are easy to write.
ToOwned is the conversion from borrowed to owned: it gives you the owned version of borrowed data.
Cow<'a, B> is the optimization tool: it holds borrowed data by default and allocates only when a mutation makes it necessary.

Together, they let you write APIs that are flexible and efficient. The APIs accept each ownership form, and they do not allocate until it is necessary.

Code Examples

File	Description
`01_07_borrow_trait.rs`	The `Borrow` trait with `HashMap` lookups, and the hash/eq contract
`01_08_to_owned.rs`	`ToOwned` for `&str`, `&[T]`, and `&Path`, and the round trip with `Borrow`
`01_09_cow_basics.rs`	`Cow` basics: the borrowed and owned variants, and deferred allocation
`01_10_cow_api_design.rs`	APIs with `Cow`: a `Config` struct and function return types
`01_11_cow_mutation.rs`	`to_mut()`, `into_owned()`, and conditional normalization

1.3 · AsRef, AsMut, Into, From: The Conversion Matrix

Domain 1 — Ownership Mechanics in the Standard Library Duration: ~15 minutes Library components: std::convert::AsRef, std::convert::AsMut, std::convert::From, std::convert::Into, std::convert::TryFrom, std::convert::TryInto, std::convert::Infallible, std::convert::identity

Introduction

The type system of Rust is strict, and this is intentional. But a strict type system without conversions makes APIs difficult to use. The std::convert module has a family of traits that convert values between types safely. Each trait has clear semantics:

Trait	Direction	Cost	Fallible?
`AsRef<T>`	`&Self → &T`	Cheap (reference cast)	No
`AsMut<T>`	`&mut Self → &mut T`	Cheap (reference cast)	No
`From<T>`	`T → Self`	May allocate	No
`Into<T>`	`Self → T`	May allocate	No
`TryFrom<T>`	`T → Result<Self, Err>`	May allocate	Yes
`TryInto<T>`	`Self → Result<T, Err>`	May allocate	Yes

This tutorial explains:

When to use each trait.
The blanket implementations that connect the traits.
How to design APIs that combine the traits and are easy to call.

AsRef: Cheap Reference Conversion

AsRef<T> means that a type can give you a &T at low cost. There is no allocation and no copy of the data. It is only a conversion between references.

// `?Sized` permits unsized targets such as `str`, `[T]`, and `Path`.
pub trait AsRef<T: ?Sized> {
    fn as_ref(&self) -> &T;
}

The standard library implements AsRef<Path> for &str, String, PathBuf, OsStr, and OsString. Thus a function that accepts impl AsRef<Path> accepts all of them.

When to use AsRef vs. Borrow

This selection confuses most Rust developers. The rule is:

AsRef<T>: Use it when you need a cheap reference conversion with no contract about hashing or equality. This is the common case for function parameters.
Borrow<T>: Use it when the borrowed form must hash and compare identically to the original. This is the case for HashMap and HashSet.

An example of a violation is a string wrapper that compares without case sensitivity. The wrapper could implement AsRef<str> to give access to the underlying bytes. But it must not implement Borrow<str>. Its Eq implementation (case-insensitive) is different from the Eq implementation of str (case-sensitive). If you used the wrapper as a HashMap key through Borrow<str>, lookups would not operate correctly.

Use case: A path utility function that accepts anything path-like

use std::path::Path;

fn main() {
    // Accepts each type that can give a &Path.
    fn file_exists(path: impl AsRef<Path>) -> bool {
        let p: &Path = path.as_ref(); // a reference conversion: no allocation
        println!("  Checking path: {}", p.display());
        p.exists()
    }

    // All of these calls compile:
    println!("--- AsRef<Path> in action ---");
    let _ = file_exists("/tmp");                            // &str
    let _ = file_exists(String::from("/tmp"));              // String
    let _ = file_exists(std::path::PathBuf::from("/tmp"));  // PathBuf

    // AsRef<str>: accepts each string-like type.
    fn shout(text: impl AsRef<str>) {
        println!("  {}", text.as_ref().to_uppercase());
    }

    println!("\n--- AsRef<str> ---");
    shout("hello");                            // &str
    shout(String::from("world"));              // String
    shout(std::borrow::Cow::Borrowed("cow"));  // Cow<str>

    // A custom AsRef implementation for a wrapper type.
    struct EmailAddress(String);

    impl AsRef<str> for EmailAddress {
        fn as_ref(&self) -> &str { &self.0 } // borrows the inner String as &str
    }

    println!("\n--- Custom AsRef implementation ---");
    let email = EmailAddress("user@example.com".to_string());
    shout(email.as_ref()); // passes the &str
}

01_12_asref_vs_borrow.rs prints these lines for the code above:

--- AsRef<Path> in action ---
  Checking path: /tmp
  Checking path: /tmp
  Checking path: /tmp

--- AsRef<str> ---
  HELLO
  WORLD
  COW

--- Custom AsRef implementation ---
  USER@EXAMPLE.COM

AsMut

AsMut<T> is the mutable equivalent. It is less common, because conversions between mutable references are rarer. But it has the same pattern:

// Accepts each type that can give a &mut [u8], for example Vec<u8> or [u8; 4].
fn clear_buffer(buf: &mut impl AsMut<[u8]>) {
    buf.as_mut().fill(0); // sets each byte to 0
}

From and Into: Infallible Value Conversions

From<T> defines how to make a Self from a value of type T. It is the conversion trait that you implement.

pub trait From<T>: Sized {
    fn from(value: T) -> Self; // takes ownership of `value`
}

Into<T> is the opposite direction: it converts self into a T. You almost never implement Into directly, because the standard library has a blanket implementation:

// For each pair of types where U: From<T>, the type T gets Into<U>.
impl<T, U> Into<U> for T where U: From<T> {
    fn into(self) -> U {
        U::from(self)
    }
}

Implement From. You get Into at no cost.

Use case: Custom type conversions and ergonomic APIs

fn main() {
    // From implementations of the standard library.
    println!("=== From conversions ===");
    let s: String = String::from("hello");       // From<&str> for String
    println!("String::from(\"hello\") = {s:?}");
    let f: f64 = f64::from(42_i32);              // From<i32> for f64
    println!("f64::from(42i32) = {f}");
    let wide: u32 = u32::from(255_u8);           // From<u8> for u32 (lossless widening)
    println!("u32::from(255u8) = {wide}");

    // Into: the blanket implementation.
    println!("\n=== Into (blanket impl) ===");
    let s: String = "hello".into();              // the type annotation selects Into<String>
    println!("\"hello\".into() = {s:?}");
    let f: f64 = 42_i32.into();                  // the type annotation selects Into<f64>
    println!("42i32.into() = {f}");

    // A custom From implementation in each direction.
    #[derive(Debug)]
    struct Celsius(f64);
    #[derive(Debug)]
    struct Fahrenheit(f64);

    impl From<Celsius> for Fahrenheit {
        fn from(c: Celsius) -> Self {
            Fahrenheit(c.0 * 9.0 / 5.0 + 32.0)
        }
    }

    impl From<Fahrenheit> for Celsius {
        fn from(f: Fahrenheit) -> Self {
            Celsius((f.0 - 32.0) * 5.0 / 9.0)
        }
    }

    println!("\n=== Custom From implementation ===");
    let boiling = Celsius(100.0);
    let boiling_f: Fahrenheit = boiling.into(); // blanket Into: moves `boiling`
    println!("Celsius(100.0) = {boiling_f:?}");
    assert_eq!(boiling_f.0, 212.0);

    let freezing_c = Celsius::from(Fahrenheit(32.0));
    println!("Fahrenheit(32.0) = {freezing_c:?}");
    assert_eq!(freezing_c.0, 0.0);

    // Into in a function parameter: the caller passes a Celsius or a Fahrenheit.
    fn set_temperature(temp: impl Into<Celsius>) {
        let celsius: Celsius = temp.into();
        println!("  Temperature set to {celsius:?}");
    }

    println!("\n=== Into in function parameters ===");
    set_temperature(Celsius(25.0));        // already a Celsius: no conversion
    set_temperature(Fahrenheit(98.6));     // converts through From<Fahrenheit>
}

01_13_from_into.rs prints these lines for the code above:

=== From conversions ===
String::from("hello") = "hello"
f64::from(42i32) = 42
u32::from(255u8) = 255

=== Into (blanket impl) ===
"hello".into() = "hello"
42i32.into() = 42

=== Custom From implementation ===
Celsius(100.0) = Fahrenheit(212.0)
Fahrenheit(32.0) = Celsius(0.0)

=== Into in function parameters ===
  Temperature set to Celsius(25.0)
  Temperature set to Celsius(37.0)

From for error conversion

One of the most important uses of From in real Rust code is the conversion of error types. The ? operator calls From::from() on the error. This call converts the error to the error type that the function returns.

fn main() {
    // One application error type that contains two library error types.
    #[derive(Debug)]
    #[allow(dead_code)] // this example never makes an `Io` value
    enum AppError {
        Io(std::io::Error),
        Parse(std::num::ParseIntError),
    }

    impl From<std::io::Error> for AppError {
        fn from(e: std::io::Error) -> Self { AppError::Io(e) }
    }

    impl From<std::num::ParseIntError> for AppError {
        fn from(e: std::num::ParseIntError) -> Self { AppError::Parse(e) }
    }

    fn parse_port(s: &str) -> Result<u16, AppError> {
        // `parse` returns Result<u16, ParseIntError>.
        // On Err(e), `?` returns Err(AppError::from(e)) from the function.
        let port: u16 = s.parse()?;
        Ok(port)
    }

    println!("=== From for error conversion ===");
    match parse_port("8080") {
        Ok(p) => println!("  Parsed port: {p}"),
        Err(e) => println!("  Error: {e:?}"),
    }
    match parse_port("not_a_number") {
        Ok(p) => println!("  Parsed port: {p}"),
        Err(e) => println!("  Error: {e:?}"),
    }
}

01_13_from_into.rs prints these lines for the code above:

=== From for error conversion ===
  Parsed port: 8080
  Error: Parse(ParseIntError { kind: InvalidDigit })

This pattern is very common. Most Rust applications define a central error enum, with a From implementation for each error type that they get.

TryFrom and TryInto: Fallible Conversions

From and Into are for conversions that always succeed. When a conversion can fail (numeric narrowing, parsing, validation), use TryFrom and TryInto:

pub trait TryFrom<T>: Sized {
    type Error; // the type of the value in the Err result
    fn try_from(value: T) -> Result<Self, Self::Error>;
}

As for From and Into, there is a blanket implementation. Implement TryFrom, and you get TryInto at no cost.

Use case: Safe numeric narrowing

// The prelude of edition 2021 and later has these two traits. The imports are optional.
use std::convert::TryFrom;
use std::convert::TryInto;

fn main() {
    println!("=== Numeric narrowing with TryFrom ===");

    let big: i32 = 300;
    let result: Result<u8, _> = u8::try_from(big); // 300 is larger than u8::MAX (255)
    println!("u8::try_from(300i32) = {result:?}");
    assert!(result.is_err());

    let small: i32 = 42;
    let result: Result<u8, _> = u8::try_from(small); // 42 fits in a u8
    println!("u8::try_from(42i32)  = {result:?}");
    assert_eq!(result, Ok(42));

    // TryInto comes from the blanket implementation, as Into does.
    println!("\n=== TryInto ===");
    let val: i64 = 1_000_000;
    let narrowed: Result<u16, _> = val.try_into(); // the type annotation selects u16
    println!("1_000_000i64.try_into::<u16>() = {narrowed:?}");
    assert!(narrowed.is_err());
}

01_14_tryfrom_tryinto.rs prints these lines for the code above:

=== Numeric narrowing with TryFrom ===
u8::try_from(300i32) = Err(TryFromIntError(PosOverflow))
u8::try_from(42i32)  = Ok(42)

=== TryInto ===
1_000_000i64.try_into::<u16>() = Err(TryFromIntError(PosOverflow))

Use case: Validated domain types

TryFrom is the idiomatic method to make types that enforce their invariants at construction time.

use std::convert::TryFrom;

fn main() {
    // Invariant: the number in a Port is always 1024 or larger.
    #[derive(Debug, PartialEq)]
    struct Port(u16);

    #[derive(Debug, PartialEq)]
    enum PortError { Zero, SystemPort(u16) }

    impl std::fmt::Display for PortError {
        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
            match self {
                PortError::Zero => write!(f, "port cannot be zero"),
                PortError::SystemPort(p) => write!(f, "port {p} is a system port (< 1024)"),
            }
        }
    }

    impl TryFrom<u16> for Port {
        type Error = PortError;

        // The only code that makes a Port. It rejects each invalid number.
        fn try_from(value: u16) -> Result<Self, Self::Error> {
            match value {
                0 => Err(PortError::Zero),
                1..=1023 => Err(PortError::SystemPort(value)),
                _ => Ok(Port(value)), // 1024 to 65535
            }
        }
    }

    for &val in &[0_u16, 80, 443, 8080, 65535] {
        let port = Port::try_from(val); // Result<Port, PortError>
        println!("  Port::try_from({val:5}) = {port:?}"); // {val:5} pads val to 5 columns
    }

    assert_eq!(Port::try_from(0), Err(PortError::Zero));
    assert_eq!(Port::try_from(80), Err(PortError::SystemPort(80)));
    assert_eq!(Port::try_from(8080), Ok(Port(8080)));
}

01_14_tryfrom_tryinto.rs prints these lines for the code above:

  Port::try_from(    0) = Err(Zero)
  Port::try_from(   80) = Err(SystemPort(80))
  Port::try_from(  443) = Err(SystemPort(443))
  Port::try_from( 8080) = Ok(Port(8080))
  Port::try_from(65535) = Ok(Port(65535))

With this pattern, when you have a Port, you know that it is valid. The type system enforces the invariant. You do not need to examine the number again at runtime in many locations of your code.

Infallible: The "Cannot Fail" Error Type

std::convert::Infallible is a type with no possible values: an empty enum. It is the error type for conversions that can never fail:

impl From<&str> for String {
    // This conversion can never fail. Thus, as a TryFrom conversion,
    // its error type is Infallible.
}

Each From implementation also gives a TryFrom implementation with Error = Infallible. The standard library has a blanket implementation for this:

// Each From impl gives an Into impl, so this impl applies to each From conversion.
impl<T, U> TryFrom<U> for T where U: Into<T> {
    type Error = Infallible;
    fn try_from(value: U) -> Result<Self, Self::Error> {
        Ok(U::into(value)) // never returns Err
    }
}

Thus you can always call .try_into(), also for an infallible conversion, and the result is always Ok. Infallible lets generic code with a TryFrom bound operate the same for fallible and infallible conversions.

The never type (!) is not fully stable in Rust 1.99. The standard library documentation states a plan to make Infallible an alias for ! when ! becomes stable.

`as` Casting: The Low-Level Conversion

Before TryFrom and Into existed, the as keyword was the only method to convert between primitive types. as is still useful, but it has risks.

as always succeeds at compile time. It never returns a Result. When a value does not fit in the target type, as silently does one of these conversions:

Conversion	Behaviour
Integer widening (`u8` → `u32`)	Zero-extends (unsigned) or sign-extends (signed). The result is always exact.
Integer narrowing (`u32` → `u8`)	Truncates: `300u32 as u8 == 44` (300 % 256)
Signed to unsigned	Reinterprets the bits: `-1i8 as u8 == 255`
Float to integer	Truncates toward zero: `3.9f64 as i32 == 3`. NaN gives 0, and ∞ gives `T::MAX`.
Integer to float	Gives the nearest representable value (may lose precision for large integers)

Use case: When `as` gives a wrong value and `try_into()` prevents it

fn main() {
    println!("=== Integer truncation with `as` ===");

    let big: u32 = 300;
    let truncated = big as u8; // keeps the low 8 bits: 300 % 256 = 44
    println!("300u32 as u8  = {truncated}");
    assert_eq!(truncated, 44); // NOT 300 and NOT an error

    // Safe alternative: returns Err and does not truncate silently.
    let safe = u8::try_from(300u32);
    println!("u8::try_from(300u32) = {safe:?}");
    assert!(safe.is_err());

    println!("\n=== Sign semantics with `as` ===");

    let neg: i8 = -1; // bit pattern 0xFF
    let as_u8 = neg as u8; // reads the same bits as an unsigned value: 255
    println!("-1i8 as u8    = {as_u8}");
    assert_eq!(as_u8, 255);

    println!("\n=== Float to integer: truncation, not rounding ===");

    let f: f64 = 3.9;
    let i = f as i32; // truncates toward zero: gives 3, not 4
    println!("3.9f64 as i32  = {i}");
    assert_eq!(i, 3);

    // NaN and infinity saturate in Rust 1.45+ (before 1.45, the cast was UB).
    let nan = f64::NAN as i32;      // NaN becomes 0
    let inf = f64::INFINITY as i32; // +infinity becomes i32::MAX
    println!("NAN as i32      = {nan}");
    println!("INFINITY as i32 = {inf}");
    assert_eq!(nan, 0);
    assert_eq!(inf, i32::MAX);

    println!("\n=== When to use `as` vs `try_into()` ===");

    // GOOD: `as` for a widening conversion (a u8 always fits in a u32).
    let byte: u8 = 200;
    let wide: u32 = byte as u32;
    println!("u8 as u32 (safe widen): {wide}");
    assert_eq!(wide, 200);

    // GOOD: `as` when you want the truncated bits on purpose.
    let hash: u64 = 0xDEAD_BEEF_CAFE_1234;
    let low_byte = hash as u8; // keeps only the low byte: 0x34
    println!("hash as u8 (intentional mask): 0x{low_byte:02X}");

    // PREFER try_into(): when the value may be out of range at runtime.
    fn safe_index(v: &[i32], pos: i64) -> Option<i32> {
        // A negative pos gives Err. Then `.ok()?` returns None from the function.
        let idx: usize = pos.try_into().ok()?;
        v.get(idx).copied() // None if idx is out of bounds
    }

    let data = vec![10, 20, 30];
    println!("safe_index(data, 1)  = {:?}", safe_index(&data, 1));
    println!("safe_index(data, -1) = {:?}", safe_index(&data, -1));
}

01_16_as_casting.rs prints these lines for the code above:

=== Integer truncation with `as` ===
300u32 as u8  = 44
u8::try_from(300u32) = Err(TryFromIntError(PosOverflow))

=== Sign semantics with `as` ===
-1i8 as u8    = 255

=== Float to integer: truncation, not rounding ===
3.9f64 as i32  = 3
NAN as i32      = 0
INFINITY as i32 = 2147483647

=== When to use `as` vs `try_into()` ===
u8 as u32 (safe widen): 200
hash as u8 (intentional mask): 0x34
safe_index(data, 1)  = Some(20)
safe_index(data, -1) = None

The rule: use as when you can prove at the call site that the value fits (widening conversions, intentional masking). Use try_into() when there is any doubt.

Recent TryFrom Stabilizations: char and bool

The standard library got two TryFrom implementations recently. They are useful to know.

`TryFrom<char>` for `usize` (stabilized in 1.94)

Rust 1.94 stabilized impl TryFrom<char> for usize. It converts a char to its Unicode scalar value (code point) as a usize. This is the fallible equivalent of c as u32. On a 16-bit platform, usize has the size of u16, and the conversion fails if the code point is larger than 65535. On 32-bit and 64-bit platforms, the conversion always succeeds.

fn main() {
    let c = '7';
    let code_point = usize::try_from(c); // Result<usize, _>
    println!("usize::try_from('7') = {code_point:?}"); // Ok(55): U+0037, not 7
    assert_eq!(code_point, Ok(55));

    let emoji = '🦀'; // U+1F980 = 129_408
    let crab_cp = usize::try_from(emoji);
    println!("usize::try_from('🦀') = {crab_cp:?}");
    assert_eq!(crab_cp, Ok(129_408));

    // Practical use: a char as a hash bucket index.
    fn char_bucket(c: char, buckets: usize) -> usize {
        // unwrap_or(0) gives 0 if the code point does not fit in a usize.
        usize::try_from(c).unwrap_or(0) % buckets
    }
    println!("bucket('A', 8) = {}", char_bucket('A', 8)); // 65 % 8 = 1
    println!("bucket('a', 8) = {}", char_bucket('a', 8)); // 97 % 8 = 1
    println!("bucket('Z', 8) = {}", char_bucket('Z', 8)); // 90 % 8 = 2
    assert_eq!(char_bucket('🦀', 8), 129_408 % 8);        // 0
}

01_14_tryfrom_tryinto.rs prints these lines for the code above:

usize::try_from('7') = Ok(55)
usize::try_from('🦀') = Ok(129408)
bucket('A', 8) = 1
bucket('a', 8) = 1
bucket('Z', 8) = 2

Note: '7' gives the code point 55 (its Unicode value), not 7. To convert a digit character to its numeric value, use c.to_digit(10), or subtract '0' as usize from the code point.

TryFrom<{integer}> for bool (stabilized in 1.95)

Rust 1.95 stabilized TryFrom implementations from each fixed-width integer type (i8–i128, u8–u128) to bool:

0 converts to false.
1 converts to true.
All other values fail with a TryFromIntError.

These implementations do not include usize and isize. This conversion is the fallible equivalent of the lossy x != 0 idiom. It is useful when you parse binary formats or FFI flags. There, a value other than 0 or 1 indicates corrupted input, not a "true" value.

fn main() {
    assert_eq!(bool::try_from(0_u8), Ok(false));
    assert_eq!(bool::try_from(1_i32), Ok(true));
    println!("bool::try_from(0u8)  = {:?}", bool::try_from(0_u8));
    println!("bool::try_from(1i32) = {:?}", bool::try_from(1_i32));

    let invalid = bool::try_from(2_u64); // the error type is std::num::TryFromIntError
    println!("bool::try_from(2u64) = {invalid:?}");
    assert!(invalid.is_err());

    // Check the flag bytes of a binary header: only 0 and 1 are valid.
    let header = [0_u8, 1, 255];
    for byte in header {
        match bool::try_from(byte) {
            Ok(flag) => println!("byte {byte:3} → flag {flag}"),
            Err(e) => println!("byte {byte:3} → error: {e}"), // {e} prints the Display text
        }
    }
}

01_14_tryfrom_tryinto.rs prints these lines for the code above:

bool::try_from(0u8)  = Ok(false)
bool::try_from(1i32) = Ok(true)
bool::try_from(2u64) = Err(TryFromIntError(PosOverflow))
byte   0 → flag false
byte   1 → flag true
byte 255 → error: number too large to fit in target type

Since Rust 1.99, the Display text of TryFromIntError tells you if the value is too large or too small. The text is "number too large to fit in target type" or "number too small to fit in target type". Earlier releases printed one generic message for the two cases.

convert::identity: The Typed No-Op

std::convert::identity<T>(x: T) -> T is a function that returns its argument unchanged. Its body is only { x }.

This trivial function is in the standard library because its type signature is useful. Some contexts need a fn(T) -> T:

Use case: Filtering out None values from an iterator

use std::convert::identity;

// identity is a const fn, so you can call it in a constant.
const ANSWER: i32 = identity(42);
const ZERO: usize = identity(0);
const MAX_RETRIES: usize = identity(3);

fn main() {
    println!("=== Filtering None values with filter_map + identity ===");
    let mixed: Vec<Option<i32>> = vec![Some(1), None, Some(3), None, Some(5)];

    // filter_map calls the function on each item. It keeps x for Some(x) and drops None.
    // The items are already Options, so identity is the function.
    // (`.flatten()` gives the same result and is the form that Clippy recommends.)
    let present: Vec<i32> = mixed.into_iter().filter_map(identity).collect();
    println!("filter_map(identity): {present:?}");
    assert_eq!(present, [1, 3, 5]);

    // A higher-order function that needs a fn(i32) -> i32.
    fn apply_transform(values: Vec<i32>, transform: fn(i32) -> i32) -> Vec<i32> {
        values.into_iter().map(transform).collect()
    }

    println!("\n=== identity as a typed function pointer ===");
    let numbers = vec![1, 2, 3, 4, 5];
    let doubled = apply_transform(numbers.clone(), |n| n * 2);  // a real transform
    println!("doubled: {doubled:?}");
    let unchanged = apply_transform(numbers.clone(), identity); // no transform
    println!("unchanged: {unchanged:?}");
    assert_eq!(doubled, [2, 4, 6, 8, 10]);
    assert_eq!(unchanged, numbers);

    println!("\n=== identity is a const fn ===");
    println!("const ANSWER = {ANSWER}");
    println!("const ZERO = {ZERO}, const MAX_RETRIES = {MAX_RETRIES}");
}

01_17_convert_identity.rs prints these lines for the code above:

=== Filtering None values with filter_map + identity ===
filter_map(identity): [1, 3, 5]

=== identity as a typed function pointer ===
doubled: [2, 4, 6, 8, 10]
unchanged: [1, 2, 3, 4, 5]

=== identity is a const fn ===
const ANSWER = 42
const ZERO = 0, const MAX_RETRIES = 3

You will not use identity frequently. But when an API takes a fn(T) -> T and you need a function that returns its argument unchanged, identity is the function to use. Its name also documents the intent.

Designing APIs with Conversion Traits

These are the patterns, in the order of how frequently you should use them:

Pattern 1: `impl AsRef<T>` for read-only access

Use this pattern when you only read the data and do not need ownership. This is the cheapest option: it never allocates.

// Accepts each type that implements AsRef<Path>: &str, String, PathBuf, &Path.
fn print_extension(path: impl AsRef<std::path::Path>) {
    let p = path.as_ref(); // &Path
    match p.extension() {  // Option<&OsStr>
        Some(ext) => println!("  {}: extension = {}", p.display(), ext.to_string_lossy()),
        None => println!("  {}: no extension", p.display()),
    }
}
// print_extension("photo.jpg") prints "  photo.jpg: extension = jpg".
// print_extension("Makefile") prints "  Makefile: no extension".

Pattern 2: `impl Into<T>` for taking ownership

Use this pattern when you must store the data. The caller can pass borrowed data, which the function converts. Or the caller can pass owned data, which moves into the function at low cost.

struct LogEntry {
    level: String,
    message: String,
}

impl LogEntry {
    // Accepts each type that converts to a String, for example &str and String.
    fn new(level: impl Into<String>, message: impl Into<String>) -> Self {
        LogEntry {
            level: level.into(),     // a &str allocates here, a String moves
            message: message.into(),
        }
    }
}

// &str arguments: each `into()` call allocates a String.
let e1 = LogEntry::new("INFO", "server started");
// String arguments: each String moves into the struct.
let e2 = LogEntry::new(String::from("ERROR"), format!("port {} in use", 8080));

Pattern 3: Do not over-generalize

A common mistake is to use impl Into<T> on many parameters. This increases monomorphization and compile times. The recommendation:

1–2 parameters: impl Into<T> is acceptable.
3+ parameters: Consider concrete types or a builder pattern.
Read-only access: Use AsRef<T>, not Into<T>.

fn main() {
    // Good: AsRef to read, Into to store.
    fn process(
        input: impl AsRef<str>,   // read-only: no allocation
        label: impl Into<String>, // needs ownership: the type of the argument sets the cost
    ) {
        let _text = input.as_ref(); // &str
        let _label = label.into();  // String
    }

    process("hello", "label");           // &str for the two parameters: label allocates
    process("hello", String::from("x")); // label moves: `process` does not allocate
}

Figure: The Conversion Trait Matrix

01_3_asref_from_into_1

Overview: How the Blanket Implementations Connect

Three blanket implementations connect the conversion traits:

From<T> → Into<T>: Implement From, and you get Into at no cost.
TryFrom<T> → TryInto<T>: Implement TryFrom, and you get TryInto at no cost.
From<T> → TryFrom<T>: Each infallible conversion is also a fallible conversion that always returns Ok.

Thus you only need to implement From (for an infallible conversion) or TryFrom (for a fallible conversion). The blanket implementations supply the other traits automatically.

Figure: Blanket Implementation Chain — Implement From, Get Everything

01_3_asref_from_into_2

There is one more blanket implementation: each type implements From<T> for T (the identity conversion). Thus String::from(already_a_string) compiles, and let s: &str = "hello".into(); compiles too. They are no-ops, and the compiler removes them during optimization.

Summary

When you need...	Use...	You implement...
A cheap `&T` from `&Self`	`AsRef<T>`	`AsRef<T>`
A cheap `&mut T` from `&mut Self`	`AsMut<T>`	`AsMut<T>`
An infallible owned conversion	`Into<T>`	`From<T>` (the blanket implementation gives you `Into`)
A fallible owned conversion	`TryInto<T>`	`TryFrom<T>` (the blanket implementation gives you `TryInto`)
Key lookups in a `HashMap` or `HashSet`	`Borrow<T>`	`Borrow<T>` (with the hash/eq contract)

The conversion traits are the Rust alternative to function overloading. You do not write many function signatures. You write one function that accepts impl Into<T> or impl AsRef<T>. The type of the caller's argument selects the conversion, if a conversion is necessary. The cost is always visible:

AsRef has no cost.
From and Into may allocate.
TryFrom and TryInto may fail.

Code Examples

File	Description
`01_12_asref_vs_borrow.rs`	`AsRef` and `Borrow`: when to use each one, and a custom `AsRef`
`01_13_from_into.rs`	`From` and `Into`: the blanket implementation, custom types, and error conversion
`01_14_tryfrom_tryinto.rs`	`TryFrom` and `TryInto`: numeric narrowing, validated types, `TryFrom<char>` (1.94), and `TryFrom<{integer}>` for `bool` (1.95)
`01_15_conversion_api_design.rs`	How to combine conversion traits in the design of a public API
`01_16_as_casting.rs`	`as` casting: truncation, sign semantics, float to integer, and when to use `try_into()`
`01_17_convert_identity.rs`	`convert::identity`: `filter_map`, a typed `fn(T) -> T` no-op, and const contexts

2.1 · Result and Option: Beyond the Basics

Domain 2 — Error Handling Ecosystem Duration: ~15 minutes Library components: std::option::Option, std::result::Result, all their methods

Introduction

Option<T> and Result<T, E> are the two primary types of Rust error handling. Most introductions stop after unwrap and a basic match. With the full combinator API, you can write clear pipelines without nested if let blocks or many early returns.

This tutorial shows:

The full set of transformation methods on Option and Result.
When to chain with and_then and when to transform with map.
How to use transpose and flatten on nested types.
How to use inspect and inspect_err for zero-cost debugging.
The ? operator and the FromResidual trait that it uses.
How to select between Option and Result, and how to convert between them.

Option Combinators

Option<T> represents a value that is present (Some(T)) or absent (None). Its API has many methods that operate on the inner value while the value stays in the Option.

Figure: Option Combinator Pipeline

02_1_result_option_1

map: Transform the inner value

map applies a function to the value in Some(T) and returns Some(U). On None, it returns None and does not call the function.

// A lookup that can fail: id 1 is "alice", id 2 is "bob", each other id gives None.
fn find_user(id: u32) -> Option<String> { /* ... */ }

let length: Option<usize> = find_user(1).map(|u| u.len());
// Some("alice") → Some(5)

let missing: Option<usize> = find_user(99).map(|u| u.len());
// None → None (the closure does not run)

and_then: Chain operations that may themselves fail

Use and_then when the transformation function itself returns an Option. and_then is the flatMap operation of Option: it prevents a nested Option<Option<T>>.

// find_display_name(&str) -> Option<String>: only "alice" has a display name.

// find_user(1) → Some("alice") → find_display_name → Some("Alice Wonderland")
let display = find_user(1).and_then(|u| find_display_name(&u));

// find_user(2) → Some("bob") → find_display_name → None (no display name for bob)
let no_display = find_user(2).and_then(|u| find_display_name(&u));

// find_user(99) → None (and_then short-circuits and does not call the closure)
let no_user = find_user(99).and_then(|u| find_display_name(&u));

or_else: Provide a fallback on None

or_else supplies an alternative Option when the value is absent. The closure is lazy: it runs only when the value is None.

// find_user(99) is None, so the closure runs.
let fallback = find_user(99).or_else(|| Some(String::from("anonymous")));
// None → Some("anonymous")

filter, unwrap_or, unwrap_or_default, map_or, zip

// filter: keep Some only if the predicate returns true
let long = find_user(1).filter(|u| u.len() > 4); // Some("alice"): the length is 5
let short = find_user(1).filter(|u| u.len() > 10); // None

// unwrap_or: default value on None (eager: the argument is always evaluated)
let name = find_user(99).unwrap_or(String::from("guest")); // "guest"

// unwrap_or_default: use Default::default() on None
let empty: String = find_user(99).unwrap_or_default(); // ""

// unwrap_or_else: compute the default lazily (no allocation when the value is Some)
let lazy = find_user(99).unwrap_or_else(|| format!("user-{}", 99)); // "user-99"

// map_or: a default value for None and a map for Some, in one call
let len = find_user(1).map_or(0, |u| u.len()); // 5
let zero = find_user(99).map_or(0, |u| u.len()); // 0

// zip: combine two Options into Option<(A, B)>. One None gives None.
let pair = find_user(1).zip(Some(42_u32)); // Some(("alice", 42))

02_01_option_combinators.rs prints:

map: username length = Some(5)
map on None: None

and_then (found): Some("Alice Wonderland")
and_then (inner None): None
and_then (outer None): None

or_else: Some("anonymous")
or_else (Some already): Some("alice")

filter (passes): Some("alice")
filter (fails): None

unwrap_or: "guest"
unwrap_or_default: ""
unwrap_or_else: "user-99"

map_or (Some): 5
map_or (None): 0

zip (both Some): Some(("alice", 42))
zip (one None): None

All assertions passed.

Result Combinators

Result<T, E> represents success (Ok(T)) or failure (Err(E)). Its combinator API has the same structure as the Option API, with more methods that operate on the error.

map and map_err

// lookup(key: &str) -> Result<String, ConfigError>
// "port" gives Ok("8080") and "host" gives Ok("localhost").
// Each other key gives Err(ConfigError::Missing(key)).

// map: transform the Ok value
let upper = lookup("host").map(|s| s.to_uppercase());
// Ok("localhost") → Ok("LOCALHOST")

// map_err: transform the Err value (for example, to convert the error type)
// {e} uses the Display impl of ConfigError, which prints "missing key: missing".
let stringified = lookup("missing").map_err(|e| format!("config error: {e}"));
// Err(Missing("missing")) → Err("config error: missing key: missing")

and_then: Chain fallible operations

// parse_port(&str) -> Result<u16, ConfigError>
// It gives Err(ConfigError::Invalid(..)) if the text is not a u16.

// Look up the "port" string, then parse it as u16.
let port: Result<u16, ConfigError> = lookup("port").and_then(|s| parse_port(&s));
// Ok("8080") → Ok(8080)

let no_key = lookup("missing").and_then(|s| parse_port(&s));
// Err(Missing(...)) → and_then short-circuits and does not call parse_port

let bad_port: Result<u16, ConfigError> = Ok(String::from("not-a-number"))
    .and_then(|s| parse_port(&s));
// Ok("not-a-number") → Err(Invalid(...)): the second step fails

or_else, ok, err, map_or

// or_else: recover from an error
// The turbofish gives the error type of the new Result, which the compiler cannot infer.
let recovered =
    lookup("missing").or_else(|_| Ok::<String, ConfigError>(String::from("default")));
// Err(Missing(...)) → Ok("default")

// ok: Result<T, E> → Option<T>, discards the error
let opt: Option<String> = lookup("host").ok();     // Some("localhost")
let none: Option<String> = lookup("missing").ok(); // None

// err: Result<T, E> → Option<E>, discards the value
let err_opt: Option<ConfigError> = lookup("missing").err(); // Some(Missing(...))

// map_or: a default value for Err and a map for Ok, in one call
let len = lookup("host").map_or(0, |s| s.len()); // 9, the length of "localhost"

02_02_result_combinators.rs prints:

map Ok: Ok("LOCALHOST")
map Err (passthrough): Err(Missing("missing"))

map_err: Err("config error: missing key: missing")

and_then (success chain): Ok(8080)
and_then (first fails): Err(Missing("missing"))
and_then (second fails): Err(Invalid("\"not-a-number\" is not a valid port"))

or_else (recover): Ok("default")
or_else (Ok passthrough): Ok("localhost")

ok() on Ok: Some("localhost")
ok() on Err: None

err() on Err: Some(Missing("missing"))
err() on Ok: None

map_or (Ok): 9
map_or (Err): 0

unwrap_or: "127.0.0.1"
unwrap_or_else: "fallback (missing key: missing)"
unwrap_or_default: ""

All assertions passed.

transpose, flatten, and inspect

transpose: bridge Option and Result

transpose is a very useful conversion in the standard library that few programmers know. It exchanges the order of the two nested types:

Option<Result<T, E>> ↔ Result<Option<T>, E>

You need this conversion very frequently for optional configuration values. When such a value is present, it must also parse correctly:

// parse_timeout(&str) -> Result<u64, String>: parses a timeout in milliseconds.
// In each case, `map` gives an Option<Result<u64, String>>,
// and `transpose` changes it into a Result<Option<u64>, String>.

// Present and valid
let raw: Option<&str> = Some("500");
let parsed = raw.map(parse_timeout).transpose();
// Some(Ok(500)) → Ok(Some(500))

// Present but invalid
let bad: Option<&str> = Some("not-a-number");
let parsed_bad = bad.map(parse_timeout).transpose();
// Some(Err(..)) → Err("\"not-a-number\" is not a valid timeout in ms")

// Absent
let absent: Option<&str> = None;
let parsed_absent = absent.map(parse_timeout).transpose();
// None → Ok(None)

The pattern optional_value.map(parse).transpose()? is frequent in code that loads a configuration. It returns early on a parse error. An absent key is not an error: the result is Ok(None).

flatten: unwrap one level of nesting

// Option<Option<T>> → Option<T>
let nested: Option<Option<u32>> = Some(Some(42));
assert_eq!(nested.flatten(), Some(42));
assert_eq!(Some(None::<u32>).flatten(), None); // an inner None gives None

// Result<Result<T, E>, E> → Result<T, E> (the two error types are the same type E)
let nested_ok: Result<Result<u32, &str>, &str> = Ok(Ok(7));
assert_eq!(nested_ok.flatten(), Ok(7));
// Ok(Err("inner")) gives Err("inner"), and Err("outer") gives Err("outer")

inspect: peek without consuming

inspect calls a closure that has a side effect (for example, logging) on a reference to the inner value. The chain continues with the same value:

// `log` and `err_log` are empty Vec<String> values, declared with `let mut`.
let result = parse_timeout("250")
    .inspect(|ms| log.push(format!("parsed timeout: {ms}ms"))); // ms: &u64
// result is still Ok(250), and log has one new entry

let failed = parse_timeout("bad")
    .inspect_err(|e| err_log.push(format!("parse failed: {e}"))); // e: &String
// failed is still Err(...), and err_log has one new entry

inspect and inspect_err are especially useful when you debug a combinator chain. You can insert them at any position, and the structure of the code stays the same.

02_03_transpose_flatten_inspect.rs prints:

transpose Some(Ok):  Ok(Some(500))
transpose Some(Err): Err("\"not-a-number\" is not a valid timeout in ms")
transpose None:      Ok(None)

Option::flatten Some(Some): Some(42)
Option::flatten None:       None
Option::flatten Some(None): None

Result::flatten Ok(Ok):  Ok(7)
Result::flatten Ok(Err): Err("inner")
Result::flatten Err:     Err("outer")

inspect Ok: Ok(250)
inspect_err: Err("\"bad\" is not a valid timeout in ms")

Option inspect: Some(99)

All assertions passed.

The `?` Operator and FromResidual

The ? operator desugars approximately to a match expression with an early return. The figure shows the control flow, and the snippet after the figure shows the match.

Figure: How the ? Operator Works

02_1_result_option_2

// `expr?` on a Result is approximately this expression:
match expr {
    Ok(v)  => v,                         // success: `v` is the value of `expr?`
    Err(e) => return Err(From::from(e)), // failure: convert the error and return early
}

The From::from(e) call lets ? convert between error types automatically. The only condition is a From impl that connects the two error types:

// AppError is an enum with three variants: Parse(ParseIntError), NotFound(u32), Forbidden.
impl From<ParseIntError> for AppError {
    fn from(e: ParseIntError) -> Self { Self::Parse(e) }
}

// lookup_record(u32) -> Result<String, AppError>
// check_access(&str) -> Result<(), AppError>
fn handle_request(raw_id: &str) -> Result<String, AppError> {
    let id = raw_id.trim().parse::<u32>()?; // ParseIntError → AppError::Parse through From
    let record = lookup_record(id)?;         // Err(AppError::NotFound(..)) returns early
    check_access(&record)?;                  // Err(AppError::Forbidden) returns early
    Ok(format!("approved: {record}"))
}

? also operates on Option in a function that returns Option:

fn find_first_even(nums: &[i32]) -> Option<i32> {
    // `find` returns Option<i32>. On None, `?` returns None from the function.
    let even = nums.iter().copied().find(|n| n % 2 == 0)?;
    Some(even * 10)
}

The FromResidual trait is the mechanism that ? uses. Result<T, E> implements FromResidual<Result<Infallible, F>> for each F where E: From<F>. The trait is unstable (feature try_trait_v2), so an implementation for a custom type requires nightly as of Rust 1.99.

main() can return Result<(), E>, which lets you use ? at the top level:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // On Err, `?` converts the ParseIntError into a Box<dyn Error> and returns from main.
    // The process then prints the Debug form of the error and exits with code 1.
    let _n: u32 = "42".parse()?;
    Ok(())
}

02_04_question_mark.rs prints:

handle_request("2"): Ok("approved: order:keyboard")
handle_request("abc"): Err(Parse(ParseIntError { kind: InvalidDigit }))
handle_request("99"): Err(NotFound(99))
handle_request("1"): Err(Forbidden)

find_first_even([1,3,4,6]): Some(40)
find_first_even([1,3,5]):   None

All assertions passed.

Option vs Result: Choosing the Right Type

Scenario	Use
A value may be present or absent, and the absence needs no explanation	`Option<T>`
An operation may fail, and the failure has a reason that the caller might use	`Result<T, E>`
Map lookup, `Iterator::find`, optional struct field	`Option<T>`
I/O, parsing, network calls, validation	`Result<T, E>`

Convert between the two types at API boundaries:

// find_capital(&str) -> Option<&str>: a lookup in a map of capitals ("Germany" → "Berlin").
// parse_percentage(&str) -> Result<u8, ParseError>: accepts the integers 0 to 100.

// Option → Result: add an error value
let r: Result<&str, &str> = find_capital("Germany").ok_or("country not found");
// Some("Berlin") → Ok("Berlin")
let r2: Result<&str, String> = find_capital("Atlantis")
    .ok_or_else(|| "Atlantis is not in the database".to_owned());
// None → Err(..): ok_or_else makes the String only on None

// Result → Option: drop the error
let opt: Option<u8> = parse_percentage("50").ok();   // Some(50)
let none: Option<u8> = parse_percentage("abc").ok(); // None

// bool → Result<(), E> (1.98): a check, not a value
// `ready` is a bool and `name` is a &str. On Err, `?` returns early from the function.
ready.ok_or("not ready")?;               // true → Ok(()), false → Err
ready.ok_or_else(|| format!("{name} is not ready"))?;  // lazy error

General rule: Use Option if the only meaningful information at the call site is success (with the value) or no success. Use Result if the failure reason changes the response of the caller (retry, report, a different log entry).

Summary

Method	On `Option`	On `Result`
`map(f)`	`Some(x)` → `Some(f(x))`	`Ok(x)` → `Ok(f(x))`
`map_err(f)`	N/A	`Err(e)` → `Err(f(e))`
`and_then(f)`	flatMap over `Option`	flatMap over `Result`
`or_else(f)`	alternative `Option`	recovery from `Err`
`filter(p)`	`Some(x)` if `p(x)`, else `None`	N/A
`flatten()`	`Option<Option<T>>` → `Option<T>`	`Result<Result<T,E>,E>` → `Result<T,E>`
`transpose()`	`Option<Result<T,E>>` → `Result<Option<T>,E>`	`Result<Option<T>,E>` → `Option<Result<T,E>>`
`inspect(f)`	side effect on `Some`, value unchanged	side effect on `Ok`, value unchanged
`inspect_err(f)`	N/A	side effect on `Err`, value unchanged
`zip(other)`	`(Some(a), Some(b))` → `Some((a,b))`	N/A
`ok()`	N/A	`Result<T,E>` → `Option<T>`
`ok_or(e)`	`Option<T>` → `Result<T,E>`	N/A (`bool::ok_or` is 1.98: `true` → `Ok(())`)

Code Examples

File	Description
`02_01_option_combinators.rs`	`map`, `and_then`, `or_else`, `filter`, `unwrap_or*`, `map_or`, `zip`
`02_02_result_combinators.rs`	`map`, `map_err`, `and_then`, `or_else`, `ok`, `err`, `map_or`, `unwrap_or*`
`02_03_transpose_flatten_inspect.rs`	`transpose`, `flatten` on Option and Result, `inspect`, `inspect_err`
`02_04_question_mark.rs`	`?` operator, `From` conversions, `?` on `Option`, `main() -> Result`
`02_05_option_vs_result.rs`	How to select between `Option` and `Result`, and the conversions `ok_or`, `ok_or_else`, `ok`
`02_18_bool_ok_or.rs`	`bool::ok_or` / `ok_or_else` (1.98): predicates as `Result<(), E>`

2.2 · The Error Trait and Error Propagation Patterns

Domain 2 — Error Handling Ecosystem Duration: ~15 minutes Library components: std::error::Error, std::fmt::Display, std::fmt::Debug, <dyn Error>::downcast_ref (for downcasting), std::error::Request

Introduction

The std::error::Error trait lets Rust error types operate together. When you understand the trait, you can:

build structured error hierarchies,
examine error causes in code,
avoid Box<dyn Error> as the solution for all errors.

This tutorial shows:

What std::error::Error requires, and why.
The source() chain for error causes.
The difference between Display and Debug formatting for errors.
Downcasting with downcast_ref, which uses the same technique as std::any::Any.
How to build multi-level error hierarchies with From conversions.
The Error::provide method, which supplies typed context (nightly).

Implementing std::error::Error

Error has two required bounds, Display and Debug, and one optional method, source():

use std::error::Error;
use std::fmt;
use std::num::ParseIntError;

// Debug is one of the two required bounds. The derive implements it.
#[derive(Debug)]
enum CsvError {
    ColumnCount { expected: usize, got: usize },       // wrong number of columns
    ParseInt { column: usize, source: ParseIntError }, // a field is not an integer
    EmptyInput,                                        // the input has no data
}

// Display: the human-readable message (shown to end users)
impl fmt::Display for CsvError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::ColumnCount { expected, got } =>
                write!(f, "expected {expected} columns, got {got}"),
            // `..` ignores `source`: the message does not include the cause.
            Self::ParseInt { column, .. } =>
                write!(f, "column {column} is not a valid integer"),
            Self::EmptyInput =>
                write!(f, "input is empty"),
        }
    }
}

// Error: source() is optional. Implement it to give the cause of the error.
impl Error for CsvError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        match self {
            // `source` is a &ParseIntError. It coerces to &(dyn Error + 'static).
            Self::ParseInt { source, .. } => Some(source),
            _ => None, // the other variants have no cause
        }
    }
}

The Debug impl (from the derive) is for developers. {:?} prints the full structure with field names and values, which is useful in log files and test output. Display is for operators. {} prints the error message, which is applicable to output that users see.

02_06_impl_error.rs prints:

parse OK: Ok([10, 20, 30])

EmptyInput Display:  input is empty
EmptyInput Debug:    EmptyInput

ColumnCount Display: expected 3 columns, got 2
ColumnCount Debug:   ColumnCount { expected: 3, got: 2 }

ParseInt Display:    column 1 is not a valid integer
ParseInt Debug:      ParseInt { column: 1, source: ParseIntError { kind: InvalidDigit } }
source() Display:    invalid digit found in string

Boxed dyn Error Display: column 0 is not a valid integer
Boxed source(): Some("invalid digit found in string")

All assertions passed.

The source() Chain

source() returns the underlying cause of an error. The causes make a chain:

AppError → QueryError → io::Error

Each link implements Error and returns the subsequent link from source(). A caller follows the chain to get the full sequence of causes:

Figure: Error Source Chain

02_2_error_trait_1

fn print_error_chain(err: &dyn Error) {
    println!("  error: {err}");     // the Display message of the top-level error
    let mut current = err.source(); // Option<&(dyn Error + 'static)>
    let mut depth = 1;
    // The loop stops at the first error that has no source.
    while let Some(cause) = current {
        println!("  caused by [{depth}]: {cause}");
        current = cause.source(); // go to the subsequent cause
        depth += 1;
    }
}

Each layer wraps the error of the layer below it:

// Level 2: a query failure. It wraps the I/O error that caused it.
#[derive(Debug)]
struct QueryError {
    query: String,
    source: io::Error,
}

impl Error for QueryError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        Some(&self.source) // the cause is the io::Error
    }
}

// Level 1: an application failure. It wraps the QueryError.
#[derive(Debug)]
struct AppError {
    context: String,
    source: QueryError,
}

impl Error for AppError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        Some(&self.source) // the cause is the QueryError
    }
}

// Each type also has a Display impl (not shown here), because Error requires it.
// QueryError prints "query failed: ..." and AppError prints "operation failed: ...".

02_07_error_source_chain.rs prints:

=== Top-level error ===
  operation failed: fetch user list

=== source() at each level ===
level 1: operation failed: fetch user list
level 2: query failed: "SELECT * FROM users"
level 3: connection refused

=== Full error chain (walk) ===
  error: operation failed: fetch user list
  caused by [1]: query failed: "SELECT * FROM users"
  caused by [2]: connection refused

=== Chain through Box<dyn Error> ===
  error: operation failed: fetch user list
  caused by [1]: query failed: "SELECT 1"
  caused by [2]: connection refused

Chain depth: 3

All assertions passed.

The deprecated description() method

Display replaced the old fn description(&self) -> &str method. Rust 1.42 deprecated the method. Do not implement it in new code.

Downcasting with downcast_ref

When you receive a Box<dyn Error> or a &dyn Error, you can get the concrete type back with downcast_ref::<T>(). dyn Error + 'static has its own downcast_ref method. The method compares TypeId values, the same technique that std::any::Any uses. Any is not a supertrait of Error.

Figure: Downcasting dyn Error to Concrete Types

02_2_error_trait_2

// NetworkError (field `code: u32`) and ParseError (field `field: String`)
// are two structs that implement Error.
fn handle_plugin_error(err: &(dyn Error + Send + Sync + 'static)) {
    // downcast_ref gives Some(&NetworkError) only if the concrete type is NetworkError.
    if let Some(net) = err.downcast_ref::<NetworkError>() {
        println!("  → NetworkError (code {}): retrying...", net.code);
    } else if let Some(parse) = err.downcast_ref::<ParseError>() {
        println!("  → ParseError on field {:?}: not retrying.", parse.field);
    } else {
        // The concrete type is not known: only the Display message is available.
        println!("  → Unknown error type: {err}");
    }
}

downcast_ref::<T>() returns Option<&T>. The result is None if the concrete type is not T. To consume the box and get an owned Box<T>, use Box::downcast::<T>():

let net_err: Box<dyn Error + Send + Sync> = /* ... */;
match net_err.downcast::<NetworkError>() {
    Ok(concrete) => println!("code={}", concrete.code), // concrete: Box<NetworkError>
    Err(_box) => { /* the downcast failed, and _box is the original box */ }
}

Important: Downcasting requires a 'static bound. Box<dyn Error + 'static> (the default) permits it. Box<dyn Error + 'a> with a non-static lifetime does not.

02_08_downcast_errors.rs prints:

fetch ok: Ok("data payload")

Received: network error 503: service unavailable
  → NetworkError (code 503): retrying...

Received: parse error on field "timestamp": "not-a-date"
  → ParseError on field "timestamp": not retrying.

downcast consumed: code=503

io::Error downcast: kind=NotFound

All assertions passed.

Building Custom Error Hierarchies

A library with a good structure has one top-level error enum that contains all the sub-error types. From implementations let ? convert each sub-error with no explicit conversion at the call site:

// NetworkError and ProtocolError are the error enums of two layers.
// Each one implements Error and has its own source().
#[derive(Debug)]
enum ClientError {
    Network(NetworkError),   // wraps a sub-error
    Protocol(ProtocolError), // wraps a sub-error
    InvalidUrl(String),      // has no inner error
}

// The From impls let `?` wrap a sub-error automatically at each call site
impl From<NetworkError> for ClientError {
    fn from(e: NetworkError) -> Self { Self::Network(e) }
}
impl From<ProtocolError> for ClientError {
    fn from(e: ProtocolError) -> Self { Self::Protocol(e) }
}

// connect(&str) -> Result<(), NetworkError>
// parse_response(&str) -> Result<u64, ProtocolError>
fn fetch(url: &str) -> Result<u64, ClientError> {
    if !url.starts_with("http") {
        return Err(ClientError::InvalidUrl(url.to_owned()));
    }
    // connect returns Err(NetworkError::Timeout) for the host "down.example".
    connect("down.example")?;               // NetworkError → ClientError via From
    Ok(parse_response("Content-Length: 42")?) // ProtocolError → ClientError via From
}

The hierarchy has these parts:

Sub-error types are narrow and reusable, and each one has its own source() chain.
The top-level enum wraps them. Its Error impl has a source() method that returns the inner error.
Callers match on the top-level enum to decide what to do.
Logging code uses Box<dyn Error> to handle all errors in the same way.

02_09_error_hierarchies.rs prints the Display message of each error:

network path: network: network timeout
  source: network timeout

url error: invalid URL: "ftp://example.com"

protocol error: protocol: invalid status line: "Garbage: xyz"
  source: invalid status line: "Garbage: xyz"

boxed: invalid URL: "ftp://x"

All assertions passed.

Error::provide: Typed Context Beyond source() (nightly)

Note: Error::provide uses the error_generic_member_access feature (tracking issue #99301). This feature requires nightly as of Rust 1.99.

source() gives you the chain of causes. But sometimes you want to attach typed context: a request ID, a file path, or a backtrace. The caller must be able to read that context without a downcast to the concrete type. Error::provide does this:

// Nightly only: the crate root needs #![feature(error_generic_member_access)].
use std::error::{Error, Request};

// RequestId is a newtype: struct RequestId(pub String).
// ServiceError has a `request_id: RequestId` field and a Display impl.
impl Error for ServiceError {
    fn provide<'a>(&'a self, request: &mut Request<'a>) {
        // Offer a &RequestId to each caller that requests this type.
        request.provide_ref::<RequestId>(&self.request_id);
    }
}

Callers get the context with std::error::request_ref::<T>(&err):

fn extract_request_id(err: &dyn Error) -> Option<&RequestId> {
    // The result is None if the error does not provide a RequestId.
    std::error::request_ref::<RequestId>(err)
}

// This also works through a Box<dyn Error>. `my_err` is a ServiceError.
let boxed: Box<dyn Error> = Box::new(my_err);
let id = std::error::request_ref::<RequestId>(&*boxed); // &*boxed is a &dyn Error

This mechanism is the base for Backtrace integration. See tutorial 2.4.

02_10_error_provide.rs prints (on nightly):

error:      service error: upstream timeout
request_id: Some("req-abc-123")

error:      something went wrong
request_id: None

boxed error: service error: disk full
request_id:  Some("req-xyz-999")

All assertions passed.

Summary

Concept	What it gives you
`impl Display`	Human-readable error message through `{}`
`impl Debug` (usually derived)	Programmer-readable detail through `{:?}`
`fn source()`	Chain of causes. Follow it with a `while let` loop
`downcast_ref::<T>()`	The concrete type from a `&dyn Error`
`Box::downcast::<T>()`	The concrete type from a `Box<dyn Error>`. The call consumes the box
`From<SubError> for TopError`	Automatic wrapping through `?`
`Error::provide`	Typed context of any type, in addition to `source()` (nightly)

Code Examples

File	Description
`02_06_impl_error.rs`	How to implement `Error` with `Display`, `Debug`, and `source()`
`02_07_error_source_chain.rs`	Multi-level `source()` chains, and how to follow a chain in code
`02_08_downcast_errors.rs`	`downcast_ref` and `Box::downcast`, which give the concrete type back
`02_09_error_hierarchies.rs`	Top-level error enum with `From` conversions, and `Box<dyn Error>` at API boundaries
`02_10_error_provide.rs`	`Error::provide` for typed context (requires nightly)

2.3 · Panic Infrastructure: Hooks, Unwind, and Abort

Domain 2 — Error Handling Ecosystem Duration: ~15 minutes Library components: std::panic, std::panic::catch_unwind, std::panic::resume_unwind, std::panic::set_hook, std::panic::take_hook, std::panic::PanicHookInfo, std::panic::UnwindSafe, std::panic::RefUnwindSafe, std::panic::AssertUnwindSafe

Introduction

A panic is the Rust mechanism for an unrecoverable error. After such an error, the program cannot reasonably continue. Examples are an index that is out of bounds, an integer overflow in debug mode, and an explicit panic!. For specific use cases, the standard library gives you several tools that control the panic mechanism.

This tutorial shows:

How catch_unwind intercepts a panic at a boundary.
How set_hook and take_hook give you custom panic output.
The PanicHookInfo type and its contents.
UnwindSafe, RefUnwindSafe, and AssertUnwindSafe.
How resume_unwind starts a panic again with the original payload.
When each of these tools is applicable.

panic! and catch_unwind

panic! first calls the panic hook, and the default hook prints the panic message to stderr. Then panic! unwinds the call stack and runs the Drop implementations of the values on that stack. The unwind stops at one of two points:

A catch_unwind call intercepts the panic and returns Err(Box<dyn Any + Send>).
No catch_unwind call is on the stack, and the panic ends the thread.

Figure: Panic Unwinding Flow

02_3_panic_1

catch_unwind is not for general error handling. Use Result for that. catch_unwind has these uses:

Thread pools: a worker that panics must not stop the whole process.
FFI boundaries: C code cannot handle Rust panics. With catch_unwind, you convert a panic to an error code.
Test infrastructure: the test runner continues with the other tests when some tests panic.

use std::panic;

// Panics when `b` is 0. The panic message has a format argument.
fn divide(a: i32, b: i32) -> i32 {
    if b == 0 {
        panic!("division by zero: a={a}");
    }
    a / b
}

// catch_unwind returns Ok(value) if the closure returns, and Err(payload) if it panics.
let ok = panic::catch_unwind(|| divide(10, 2));
assert_eq!(ok.unwrap(), 5);

let caught = panic::catch_unwind(|| divide(10, 0));
assert!(caught.is_err()); // the panic stops here, and the program continues

// The payload is a Box<dyn Any + Send>. Downcast it to get the message.
let payload = panic::catch_unwind(|| divide(5, 0)).unwrap_err();
if let Some(msg) = payload.downcast_ref::<String>() {
    // A message with format arguments is a String: this branch runs.
    println!("panic payload (String): {msg:?}");
} else if let Some(msg) = payload.downcast_ref::<&str>() {
    // A literal message, such as panic!("text"), is a &'static str.
    println!("panic payload (&str):   {msg:?}");
}

The supervisor pattern processes a batch in which individual items may panic:

// process_item(item) panics for 0 and for a negative item.
// For each other item, it returns item * 2.
let items = [3, 0, 5, -1, 4];
for &item in &items {
    match panic::catch_unwind(|| process_item(item)) {
        Ok(v) => println!("item {item} → {v}"),
        Err(payload) => {
            // Get the message from a String payload or from a &str payload.
            let msg = payload
                .downcast_ref::<String>()
                .map(String::as_str)
                .or_else(|| payload.downcast_ref::<&str>().copied())
                .unwrap_or("(unknown panic)");
            println!("item {item} panicked: {msg}");
        }
    }
}
// Items 3, 5, and 4 succeed. Items 0 and -1 panic, but the loop continues.

todo!(), unreachable!(), and out-of-bounds indexing all cause panics that catch_unwind can catch.

02_11_panic_basics.rs prints:

catch_unwind (no panic): Ok(5)
catch_unwind (panic):    Err(Any { .. })
panic payload (String): "division by zero: a=5"

item 3 → 6
item 0 panicked: got zero, cannot continue
item 5 → 10
item -1 panicked: negative input: -1
item 4 → 8

todo! panic caught: true
unreachable! panic caught: true

All assertions passed.

This block shows stdout only. The default panic hook also prints the message of each caught panic to stderr.

Panic Hooks: set_hook and take_hook

By default, a panic prints a message to stderr, and optionally a backtrace. set_hook replaces this behavior fully.

use std::panic::{self, PanicHookInfo};

// take_hook removes the current hook (here, the default hook) and returns it.
// Keep the returned hook, so that you can restore it later.
let default_hook = panic::take_hook();

// Install your own hook. The runtime calls it one time for each panic.
panic::set_hook(Box::new(|info: &PanicHookInfo<'_>| {
    // info.payload()  is the panic payload (usually &str or String)
    // info.location() is the file, line, and column of the panic
    // This chain gets the message from a String payload or from a &str payload.
    // Since 1.91, info.payload_as_str() does the same in one call.
    let msg = info.payload()
        .downcast_ref::<String>().map(String::as_str)
        .or_else(|| info.payload().downcast_ref::<&str>().copied())
        .unwrap_or("(non-string payload)");

    match info.location() {
        Some(loc) => eprintln!("PANIC at {}:{}: {}", loc.file(), loc.line(), msg),
        None      => eprintln!("PANIC (no location): {msg}"),
    }
}));

// This panic calls your hook, not the default hook. catch_unwind stops the unwind.
let _ = panic::catch_unwind(|| panic!("something went wrong"));

// Restore the default hook.
panic::set_hook(default_hook);

take_hook() is the only function that gives you the current hook. It removes the hook from the runtime and returns it. To add to the default behavior (and not replace it), do these steps:

Take the default hook.
Call the default hook in your custom hook.
Install your custom hook.

PanicHookInfo (replaces PanicInfo since 1.81)

std::panic::PanicHookInfo has these methods:

payload() returns &(dyn Any + Send): the value that panic! received.
location() returns Option<&'static Location<'static>>: the file name, the line, and the column.
can_unwind() returns bool: true if the panic unwinds, false if it aborts. This method is still unstable in 1.99 (feature panic_can_unwind).

Rust 1.81 added the name PanicHookInfo, and Rust 1.82 deprecated the old name std::panic::PanicInfo. The new name prevents confusion with core::panic::PanicInfo. A #[panic_handler] function in a no_std environment receives that core type.

The hook in 02_12_panic_hooks.rs does not print. It adds one line for each panic to a shared Vec<String>. The example prints:

Captured 3 panic events:
  [0] panic at domain-02-error-handling/examples/src/bin/02_12_panic_hooks.rs:48: first failure
  [1] panic at domain-02-error-handling/examples/src/bin/02_12_panic_hooks.rs:49: second failure
  [2] panic at domain-02-error-handling/examples/src/bin/02_12_panic_hooks.rs:52: index out of bounds: the len is 0 but the index is 10

Hook restored. Log count unchanged: 3

All assertions passed.

loc.file() returns the path that the compiler received. Cargo compiles the example from the workspace root, so the path is relative to that directory.

UnwindSafe, RefUnwindSafe, and AssertUnwindSafe

catch_unwind requires an UnwindSafe closure. This trait marks the types that cannot leave shared state invalid if a panic occurs in the middle of the closure.

Which types are UnwindSafe?

i32, bool, String, and each type that contains only UnwindSafe types: yes.
&mut T: no. A panic during a change through a &mut T can leave the referenced value partially changed.
&RefCell<T>: no. A closure can also change the value through this shared reference, and RefCell does not record that a panic stopped a change.
&Mutex<T>: yes. A panic while the closure holds the guard poisons the Mutex, and subsequent lock calls return a PoisonError.

Figure: Is My Type UnwindSafe?

02_3_panic_2

AssertUnwindSafe: the explicit override

When you know that a captured value is safe, although its type does not implement UnwindSafe, wrap the closure:

use std::panic::{self, AssertUnwindSafe};

let mut counter = 0_u32;

// The closure captures `&mut counter`, so the closure is not UnwindSafe.
// The wrapper around the whole closure tells the compiler that you checked the safety.
let ok = panic::catch_unwind(AssertUnwindSafe(|| {
    counter += 10; // the only change: a panic cannot leave it partially done
    counter
}));
assert_eq!(ok.unwrap(), 10); // the closure returned the new value of `counter`

AssertUnwindSafe is your statement to the compiler that you verified the closure. The statement says that a panic in this closure leaves no broken invariant in the captured state. The compiler does not check the statement. If the statement is wrong, the result can be a logic error, but not undefined behavior in safe code.

Since Rust 1.96, AssertUnwindSafe<T> also implements From<T> for each T that is UnwindSafe. Thus let wrapped: AssertUnwindSafe<u32> = value.into(); makes the wrapper through the standard conversion traits. This is convenient in generic code that already accepts impl Into<AssertUnwindSafe<T>>.

A note on RefCell: a temporary borrow() or borrow_mut() guard drops at the semicolon of its statement. If no guard is alive when the panic occurs, no change of the value is in progress. Then the RefCell is safe to use again after catch_unwind. Use AssertUnwindSafe to tell the compiler about that reasoning.

02_13_unwind_safe.rs prints:

catch_unwind result: true
cell after catch:    1

catch_unwind with &mut (ok): Ok(10)
catch_unwind with &mut (panic): true

AssertUnwindSafe return: Ok("hello")
AssertUnwindSafe via From: 15

All assertions passed.

resume_unwind: Re-panicking with the Original Payload

catch_unwind catches a panic and gives you the payload. resume_unwind starts the panic again with that same payload, as if the original panic did not stop:

use std::{panic, thread};

fn spawn_worker(value: i32) -> thread::JoinHandle<i32> {
    thread::spawn(move || {
        // result: Result<i32, Box<dyn Any + Send>>
        let result = panic::catch_unwind(move || {
            // The worker logic. It panics when `value` is 0.
            if value == 0 {
                panic!("worker received zero");
            }
            value * 3
        });
        // This is the place to log or to clean up before the panic continues.
        match result {
            Ok(v)        => v,
            // Start the panic again in this worker thread, with the original payload.
            // join() in the parent thread then returns Err(payload).
            Err(payload) => panic::resume_unwind(payload),
        }
    })
}

resume_unwind(payload) and panic!(...) are different:

resume_unwind keeps the type and the message of the original payload.
panic! starts a new panic with a new message.

When to use resume_unwind vs. convert to error

Situation	Recommendation
A thread pool must propagate the panic to the caller	Call `resume_unwind` after the logging and the cleanup
FFI boundary (`extern "C"`)	Never resume the panic. Convert it to an error code
Test runner	Call `resume_unwind` to keep the original failure message
Custom retry logic	Examine the payload. Then resume the panic or discard the payload

02_14_resume_unwind.rs prints:

=== Logging runner ===
[add] completed normally
[divide] panicked: division by zero

=== Cross-thread resume_unwind ===
worker(7) → 21
worker(0) panicked: worker received zero

=== FFI boundary pattern ===
ffi_result code: -1

All assertions passed.

Summary

Tool	Purpose
`panic!`	Signals an unrecoverable error
`catch_unwind`	Intercepts a panic at a boundary (thread pool, FFI)
`resume_unwind`	Starts the panic again with the original payload, after you examine it
`set_hook`	Replaces the default panic output
`take_hook`	Removes the current hook and returns it, so that you can restore it
`PanicHookInfo`	The payload and the location that the hook function receives
`UnwindSafe`	Marker: the type cannot leave invalid state after a panic
`RefUnwindSafe`	Marker: the same property for `&T`
`AssertUnwindSafe`	Wrapper: you state that you verified the safety

The rule: use Result for expected failures. Use catch_unwind only at controlled boundaries that must stop a panic from untrusted or isolated code.

Code Examples

File	Description
`02_11_panic_basics.rs`	`panic!`, `catch_unwind`, payload extraction, supervisor pattern
`02_12_panic_hooks.rs`	`set_hook`, `take_hook`, `PanicHookInfo` location and payload
`02_13_unwind_safe.rs`	`UnwindSafe`, `RefUnwindSafe`, `AssertUnwindSafe`, `RefCell` interaction, `From<T>` constructor (1.96)
`02_14_resume_unwind.rs`	`resume_unwind`, thread-pool pattern, FFI boundary pattern

2.4 · Backtraces: Capturing Stack Traces

Domain 2 — Error Handling Ecosystem Duration: ~15 minutes Library components: std::backtrace::Backtrace, std::backtrace::BacktraceStatus

Introduction

A backtrace shows the call stack at the moment that the program created an error. It shows which function called which function, back to main. Without a backtrace, the diagnosis of a production failure in complex async code or library code can take hours. With a backtrace, the call site of the failure is immediately visible.

std::backtrace::Backtrace (stable since Rust 1.65) captures this information when you request it. With the Error::provide mechanism (nightly: error_generic_member_access), you can embed a backtrace in a custom error type. Then each consumer can get the backtrace and does not need to know the concrete type.

This tutorial shows:

Backtrace::capture and the RUST_BACKTRACE environment variable.
The BacktraceStatus variants: Captured, Disabled, Unsupported.
Backtrace::force_capture, and when to use it in place of capture.
The performance cost of a capture that is always on, compared with a conditional capture.
How to embed a Backtrace in a custom Error type through Error::provide.

Backtrace::capture and BacktraceStatus

Backtrace::capture() always returns a Backtrace object. Its contents depend on the RUST_BACKTRACE environment variable:

`RUST_BACKTRACE` value	`BacktraceStatus`	Frames available?
not set or `0`	`Disabled`	No
`1`	`Captured`	Yes
`full`	`Captured`	Yes (the same frames as `1`)
The platform has no backtrace support	`Unsupported`	No

Figure: Backtrace Capture Decision Flow

02_4_backtrace_1

use std::backtrace::{Backtrace, BacktraceStatus};

// capture() obeys RUST_BACKTRACE. It always returns a Backtrace object.
let bt = Backtrace::capture();

match bt.status() {
    BacktraceStatus::Captured  => println!("{bt}"),   // Display prints the frames
    BacktraceStatus::Disabled  => println!("run with RUST_BACKTRACE=1"),
    BacktraceStatus::Unsupported => println!("not supported here"),
    _ => {}   // BacktraceStatus is #[non_exhaustive], so the match needs this arm
}

The Display implementation of Backtrace prints the stack trace as text that you can read. {bt} prints the short form. The alternate form {bt:#} prints the full form, which adds the instruction address of each frame.

The value of RUST_BACKTRACE (1 or full) does not change this output. It changes only the backtrace that the default panic hook prints. With 1, that hook omits some frames. With full, it prints all frames (standard library internals included).

When to check status before logging

// `log::error!` is the macro of the external `log` crate. It is here as an example logger.
let bt = Backtrace::capture();
if bt.status() == BacktraceStatus::Captured {
    log::error!("backtrace:\n{bt}");   // only this branch formats the frames
} else {
    log::error!("(enable RUST_BACKTRACE=1 for backtrace)");
}

It is cheap to create the Backtrace each time and to check its status, because a status check only reads a flag. The format operation with {} does the work.

With RUST_BACKTRACE not set, 02_15_backtrace_capture.rs prints:

Backtrace status: Disabled (set RUST_BACKTRACE=1 to enable)
Backtrace::capture() returned a backtrace object.

To see frames, re-run with RUST_BACKTRACE=1

Backtrace not available; logging error message only.

All assertions passed.

(With RUST_BACKTRACE=1, the status is Captured, and the example prints the start of the trace.)

Backtrace::force_capture

force_capture() always captures a full backtrace and ignores RUST_BACKTRACE. The status of the result is always BacktraceStatus::Captured (when the platform supports backtraces).

let normal = Backtrace::capture();       // obeys RUST_BACKTRACE
let forced = Backtrace::force_capture(); // always captures

// True for each value of RUST_BACKTRACE, on a platform that supports backtraces.
assert_eq!(forced.status(), BacktraceStatus::Captured);

capture() vs force_capture(): when to use each

Scenario	Use
Production error types	`capture()`: cheap when disabled, available when necessary
Test assertion helpers	`force_capture()`: you always want the call site
Diagnostic tools (always on)	`force_capture()`
Hot paths where performance is important	`capture()` with `RUST_BACKTRACE=0`

Performance cost

force_capture() walks the native stack on each call. When backtraces are disabled, capture() does not walk the stack, so its cost is almost zero. The difference is typically 100x–1000x. With RUST_BACKTRACE not set, 02_16_force_capture.rs prints:

capture()       status: Disabled
force_capture() status: Captured

capture()       x1000: 16.791µs   # (varies)
force_capture() x1000: 9.044083ms   # (varies)

With RUST_BACKTRACE unset, capture() is ~538x faster than force_capture().   # (varies)

force_capture Display (first 200 chars):
   0: std::backtrace_rs::backtrace::libunwind::trace   # (varies)
             at /rustc/b940084d7eb6a299eb4bfeb8e34901bc051e7ac4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9   # (varies)
   1: std::backtra   # (varies)

All assertions passed.

The times and the ratio vary with the machine and the stack depth. The frame lines vary with the platform and the toolchain.

For production error types that a program may construct millions of times per second, use capture(). Use force_capture() only for code paths that always need the trace, whatever the environment of the operator is.

Embedding a Backtrace in a Custom Error

The standard pattern has two steps. First, capture the backtrace when you construct the error. Then, make the backtrace available to consumers through Error::provide.

Note: Error::provide requires the error_generic_member_access nightly feature (tracking issue #99301).

#![feature(error_generic_member_access)] // nightly only

use std::backtrace::Backtrace;
use std::error::{Error, Request};
use std::io;

#[derive(Debug)]
struct StorageError {
    operation: String,
    source: io::Error,    // the cause: source() returns it
    backtrace: Backtrace, // the context: provide() gives it to consumers
}

impl StorageError {
    fn new(operation: impl Into<String>, source: io::Error) -> Self {
        Self {
            operation: operation.into(),
            source,
            backtrace: Backtrace::capture(), // capture at the construction site
        }
    }
}

// The Error trait also requires a Display implementation. It is not shown here.
impl Error for StorageError {
    fn source(&self) -> Option<&(dyn Error + 'static)> {
        Some(&self.source)
    }

    // A consumer requests a value by its type. This method answers a request for Backtrace.
    fn provide<'a>(&'a self, request: &mut Request<'a>) {
        request.provide_ref::<Backtrace>(&self.backtrace);
    }
}

A consumer gets the backtrace and does not need to know the concrete type:

// Works for each error type. Returns None if the error provides no Backtrace.
fn get_backtrace(err: &dyn Error) -> Option<&Backtrace> {
    std::error::request_ref::<Backtrace>(err)
}

// read_record(0) returns Err(StorageError), because record 0 does not exist.
let err = read_record(0).unwrap_err();
if let Some(bt) = get_backtrace(&err) {
    println!("backtrace status: {:?}", bt.status()); // Disabled or Captured
    println!("{bt}"); // prints "disabled backtrace" when the status is Disabled
}

// The same function works through Box<dyn Error>.
let boxed: Box<dyn Error> = Box::new(read_record(0).unwrap_err());
let bt = get_backtrace(&*boxed); // Some(&Backtrace)

Why capture at construction, not at the call site?

A program can create an error at a deep level of a call chain:

main → fetch → read_record → StorageError::new

A capture in StorageError::new gives you the frame that caused the failure. A capture at a higher level (for example, in fetch) shows only the call to fetch, and not the internal path.

The integration with the error chain

source() and provide() are complementary:

source() gives the cause: a different error.
provide() gives the context: typed values that this specific error holds.

A consumer that logs all of this information can use this function:

// `log::error!` is the macro of the external `log` crate. It is here as an example logger.
fn log_error(err: &dyn Error) {
    log::error!("error: {err}");

    // Log the backtrace if the error provides one.
    if let Some(bt) = std::error::request_ref::<Backtrace>(err) {
        log::error!("backtrace:\n{bt}");
    }

    // Go through the chain of causes. source() returns None at the end of the chain.
    let mut cause = err.source();
    while let Some(c) = cause {
        log::error!("caused by: {c}");
        cause = c.source();
    }
}

With RUST_BACKTRACE not set, 02_17_backtrace_in_errors.rs prints:

read_record(42): Ok("record-42")

error: storage operation "read" failed
source: Some("record 0 does not exist")
backtrace status: Disabled
(backtrace disabled — run with RUST_BACKTRACE=1)

backtrace through Box<dyn Error>: Some(Disabled)
io::Error (no provide): None

All assertions passed.

(With RUST_BACKTRACE=1, the backtrace status is Captured, and the example prints the first frames.)

When to Capture a Backtrace and When Not To

Backtraces help when:

You debug unexpected errors in production (set RUST_BACKTRACE=1 in the environment).
You write tools for post-mortem analysis.
You write test assertion helpers that must show where an assertion failed.

Backtraces hurt when:

The program constructs errors on hot paths (thousands per second). Use capture(), not force_capture().
Memory is limited. Each Backtrace object holds a vector of frames.
The program expects the errors and handles them silently. A disabled Backtrace::capture() has almost zero cost, so this is a problem only for force_capture().

The standard practice for library authors is to embed Backtrace::capture() in error types. This has no cost when the operator did not enable RUST_BACKTRACE. It gives the full context when the operator enabled it.

Summary

API	What it does
`Backtrace::capture()`	Captures if `RUST_BACKTRACE` enables backtraces. If not, returns a disabled backtrace
`Backtrace::force_capture()`	Always captures, and ignores `RUST_BACKTRACE`
`bt.status()`	Returns `Captured`, `Disabled`, or `Unsupported`
`format!("{bt}")` / `println!("{bt}")`	Formats the stack trace as text
`Error::provide` + `request.provide_ref::<Backtrace>`	Embeds a backtrace in an error type (nightly)
`std::error::request_ref::<Backtrace>(err)`	Gets the backtrace from any `dyn Error` (nightly)

Code Examples

File	Description
`02_15_backtrace_capture.rs`	`Backtrace::capture`, `BacktraceStatus`, conditional logging
`02_16_force_capture.rs`	`force_capture` compared with `capture`, performance comparison
`02_17_backtrace_in_errors.rs`	A `Backtrace` in a custom `Error` through `Error::provide` (nightly)

3.1 · String, &str, and the UTF-8 Contract

Domain 3 — Strings and Text Processing Duration: ~15 minutes Library components: std::string::String, primitive str, std::str::from_utf8, std::str::Utf8Error, FromUtf8Error

Introduction

String and &str are not aliases of each other. They are two different things. String is a heap-allocated, growable buffer of UTF-8 bytes. &str is a borrowed slice of UTF-8 bytes. Those bytes can be anywhere: on the heap, on the stack, or in static memory.

Each design decision in their API comes from two guarantees. The contents are always valid UTF-8. The length that len() returns is always a byte count.

This tutorial shows:

The internal layout of String and &str.
Why an index of type usize is a compile error.
How to iterate over chars, bytes, and (byte offset, char) pairs.
How to slice safely, and which slice operations panic.
substr_range: how to get the byte offsets of a substring without a search.
How to convert between byte slices and String/&str, with UTF-16LE and UTF-16BE included.

Internal Representation

String is a thin wrapper around Vec<u8> that keeps the UTF-8 invariant. It has three fields: a heap pointer, a byte length, and a byte capacity.

&str is a fat pointer: a heap (or static) address together with a byte length. For a string literal such as "hello", the compiler stores the UTF-8 bytes in the binary. The literal is a &'static str that points into that read-only segment.

Figure: String vs &str Memory Layout

String vs &str memory layout

let owned: String = String::from("héllo");
println!("{}", owned.len());           // 6 bytes: 'h'=1, 'é'=2, 'l'=1, 'l'=1, 'o'=1
println!("{}", owned.chars().count()); // 5 chars

let literal: &str = "world";           // a &'static str: the bytes are in the binary
println!("{}", literal.len());         // 5 bytes and 5 chars (all ASCII)

len() always counts bytes, not characters. This is the most common cause of confusion for new Rust programmers.

Why indexing by usize is not permitted

The expression owned[0] is a compile error. If the compiler accepted it, the expression would return one u8. But one u8 from the middle of a multi-byte sequence is not a valid Unicode scalar value. There is no safe way to give it to the caller. Thus str does not implement Index<usize>, and the type system rejects the expression.

03_01_string_str_representation.rs prints:

owned.len()  = 6 (bytes)
owned.chars().count() = 5 (chars)

with_capacity(16): len=0 cap=16
after push_str:    len=5 cap=16

slice &owned[0..1] = "h"
literal &str: "world", len=5

Indexing by usize is intentionally a compile error.
Use .chars(), .bytes(), .char_indices(), or explicit byte slices.

Raw pointer to first byte: 0x100e48d20   # (varies)

All assertions passed.

chars(), bytes(), and char_indices()

Three iterators supply all the access patterns:

Iterator	Yields	Use when
`chars()`	`char` (Unicode scalar value)	You count characters or process code points.
`bytes()`	`u8`	You process raw bytes or ASCII-only data.
`char_indices()`	`(usize, char)`: byte offset and char	You slice the original string afterward.

let text = "café";   // 'é' is U+00E9: one char, two bytes in UTF-8

// chars: 4 Unicode scalar values
for ch in text.chars() {
    println!("  {ch:?}");   // 'c', 'a', 'f', 'é'
}

// bytes: 5 raw bytes (195 and 169 are the two bytes of 'é')
let byte_vec: Vec<u8> = text.bytes().collect(); // [99, 97, 102, 195, 169]

// char_indices: the byte offset where each char starts, and the char
for (byte_pos, ch) in text.char_indices() {
    println!("  byte {byte_pos}: {ch:?}");
}
// (0, 'c'), (1, 'a'), (2, 'f'), (3, 'é')

char_indices is the iterator to use when you want to slice the string afterward. The iterator guarantees that each byte offset is a valid char boundary.

Figure: UTF-8 Multi-Byte Encoding — "café"

03_1_string_str_utf8_1

03_02_chars_bytes_char_indices.rs prints:

chars():
  'c'
  'a'
  'f'
  'é'
  count = 4

bytes():
  [99, 97, 102, 195, 169]

char_indices():
  byte 0: 'c'
  byte 1: 'a'
  byte 2: 'f'
  byte 3: 'é'

suffix from 3rd char: "fé"

"rust" reversed: "tsur"

All assertions passed.

String Slicing and Byte Boundaries

A slice of a str with &s[start..end] is fast: it is only a pointer offset and a length adjustment. But start and end must each be on a char boundary, or the operation panics at run time.

let s = "héllo wörld";   // 'h' is byte 0, 'é' is bytes 1 and 2, the first 'l' is byte 3

let safe = &s[3..5]; // "ll": the two bounds are char boundaries
// let bad = &s[1..2]; // PANIC: byte 2 is inside 'é' (a 2-byte char)

is_char_boundary

// `s` is "héllo wörld" from the previous snippet.
// is_char_boundary(i) returns true when a char starts at byte offset i.
s.is_char_boundary(0); // true:  'h' starts here
s.is_char_boundary(1); // true:  'é' starts here
s.is_char_boundary(2); // false: the second byte of 'é'
s.is_char_boundary(3); // true:  'l' starts here

Non-panicking slicing: get()

str::get(range) returns Option<&str>. When the range is not valid, it returns None and does not panic:

// `s` is "héllo wörld" from the previous snippets.
let maybe = s.get(3..5); // Some("ll")
let bad   = s.get(1..2); // None: byte 2 is not a char boundary

substr_range (1.98) is the inverse of slicing. You give it a &str that already points into source. It returns the byte offsets as a std::range::Range. The method uses pointer arithmetic, not a search. A separate "hello" with the same text gives None. Code that splits text uses it to get the spans for diagnostics:

let source = "fn main() { hello }";
let hello = &source[12..17];   // "hello": a slice that points into `source`
// substr_range compares addresses. It does not search `source` for the text.
assert_eq!(source.substr_range(hello), Some(std::range::Range { start: 12, end: 17 }));

Safe N-char prefix using char_indices

// Returns the first `n` chars of `s`. The slice never ends inside a char.
fn first_n_chars(s: &str, n: usize) -> &str {
    // nth(n) gives the byte offset where the char with index n starts.
    match s.char_indices().nth(n) {
        Some((byte_pos, _)) => &s[..byte_pos], // the slice stops before that char
        None => s,                             // `s` has n chars or fewer: return all of it
    }
}
// first_n_chars("héllo wörld", 3) returns "hél" (4 bytes)

03_03_string_slicing.rs prints:

safe slice [3..5]: "ll"

is_char_boundary checks:
  byte 0: is_char_boundary = true
  byte 1: is_char_boundary = true
  byte 2: is_char_boundary = false
  byte 3: is_char_boundary = true
  byte 4: is_char_boundary = true

first 3 chars: "hél"
get(3..5): Some("ll")
get(1..2): None

All assertions passed.

UTF-8 Conversion API

The standard library has a full set of functions that convert between byte sequences (&[u8], Vec<u8>) and string types:

Function	Input	Output	Allocates?
`str::as_bytes()`	`&str`	`&[u8]`	No
`String::into_bytes()`	`String`	`Vec<u8>`	No (moves)
`str::from_utf8(&[u8])`	`&[u8]`	`Result<&str, Utf8Error>`	No
`String::from_utf8(Vec<u8>)`	`Vec<u8>`	`Result<String, FromUtf8Error>`	No (moves)
`String::from_utf8_lossy(&[u8])`	`&[u8]`	`Cow<str>`	Only for invalid input
`String::from_utf8_lossy_owned(Vec<u8>)` (1.99)	`Vec<u8>`	`String`	Only for invalid input
`FromUtf8Error::into_utf8_lossy()` (1.99)	`FromUtf8Error`	`String`	Yes
`String::from_utf16le` / `from_utf16be` (1.98)	`&[u8]`	`Result<String, FromUtf16Error>`	Yes
`from_utf16le_lossy` / `from_utf16be_lossy` (1.98)	`&[u8]`	`String`	Always

// as_bytes: borrows the UTF-8 bytes of a &str (no copy, no allocation)
let bytes: &[u8] = "hello".as_bytes();

// from_utf8: validates the bytes and borrows them as a &str (no allocation)
match std::str::from_utf8(bytes) {
    Ok(s)  => { /* s: &str, the bytes are valid UTF-8 */ }
    Err(e) => { /* e: Utf8Error, e.valid_up_to() is the length of the valid prefix */ }
}

// from_utf8_lossy: always succeeds. It replaces each invalid sequence with U+FFFD.
let lossy = String::from_utf8_lossy(b"hel\xFF\xFElo");   // 0xFF and 0xFE are not valid UTF-8
// lossy == "hel\u{FFFD}\u{FFFD}lo"

The two strict conversions have one important difference. str::from_utf8 returns a &str that borrows from the input slice (no allocation). String::from_utf8 consumes a Vec<u8> and wraps it (no copy). When String::from_utf8 fails, it returns a FromUtf8Error that contains the original bytes. Call .into_bytes() on the error to recover them.

03_04_utf8_conversion.rs prints the lines below. The terminal prints each U+FFFD as the character �:

as_bytes: [104, 101, 108, 108, 111]
into_bytes: [114, 117, 115, 116]

from_utf8 valid: "héllo"
from_utf8 invalid: error at byte 0

String::from_utf8 valid: Ok("hello")
String::from_utf8 invalid: true
recovered bytes from FromUtf8Error: [255]

from_utf8_lossy: "hel��lo"
from_utf8_lossy (all valid): "clean"

All assertions passed.

Rust 1.99 adds the owned lossy conversions. String::from_utf8_lossy borrows a &[u8] and returns Cow<str>. When you own the bytes as a Vec<u8>, String::from_utf8_lossy_owned returns a String directly. If the bytes are valid UTF-8, the String reuses the buffer of the Vec, and the function copies no bytes. FromUtf8Error::into_utf8_lossy gives the same repair after a strict conversion fails:

// Valid input: the buffer of the Vec becomes the buffer of the String (no copy).
let text = String::from_utf8_lossy_owned(b"status: ok".to_vec());
assert_eq!(text, "status: ok");

// Invalid input: each bad sequence becomes U+FFFD, as with from_utf8_lossy.
let repaired = String::from_utf8_lossy_owned(b"status: \xFFok".to_vec());
assert_eq!(repaired, "status: \u{FFFD}ok");

// Strict conversion first, lossy conversion as the alternative. The error keeps the bytes.
let wire: Vec<u8> = b"id=\xF0\x90\x80;end".to_vec();   // a 4-byte char without its last byte
let recovered = match String::from_utf8(wire) {
    Ok(text) => text,
    Err(error) => error.into_utf8_lossy(),   // consumes the error, returns the repaired text
};
assert_eq!(recovered, "id=\u{FFFD};end");

UTF-16 data in a network protocol is a byte stream with an endianness. Since 1.98, String::from_utf16le(&[u8]) and from_utf16be decode it directly. An odd length or a lone surrogate gives Err. The _lossy variants replace invalid sequences with U+FFFD and always allocate a String. Unlike from_utf8_lossy, they have no borrowed fast path.

Summary

Concept	Key point
`String`	Heap-allocated `Vec<u8>` with the UTF-8 invariant
`&str`	Fat pointer (address and byte length) into valid UTF-8
`len()`	Always the byte count, never the char count
Indexing by `usize`	Compile error. Use iterators or slices.
`chars()`	Unicode scalar values
`bytes()`	Raw UTF-8 bytes
`char_indices()`	`(byte_offset, char)` pairs, safe for subsequent slicing
`get(range)`	Slice that does not panic. It returns `Option<&str>`.
`substr_range` (1.98)	Byte `Range` of a substring, from pointer arithmetic, not a search
`from_utf8`	Validates bytes. It returns `Result<&str, Utf8Error>` (no allocation).
`from_utf8_lossy`	Always succeeds. It returns `Cow<str>`.
`from_utf8_lossy_owned` (1.99)	Converts an owned `Vec<u8>` to a `String`. Valid input reuses the buffer.
`from_utf16le` / `from_utf16be` (1.98)	Decodes UTF-16 bytes (`&[u8]`) of a known endianness into a `String`

Code Examples

File	Description
`03_01_string_str_representation.rs`	`String` layout, `len()` compared with `chars().count()`, `with_capacity`, raw pointer
`03_02_chars_bytes_char_indices.rs`	`chars()`, `bytes()`, `char_indices()`, suffix slicing, `rev()`
`03_03_string_slicing.rs`	Byte-boundary slicing, `is_char_boundary`, `get()`, safe N-char prefix
`03_04_utf8_conversion.rs`	`as_bytes`, `into_bytes`, `from_utf8`, `String::from_utf8`, `from_utf8_lossy`
`03_19_substr_range.rs`	`str::substr_range` (1.98): get the offsets of split tokens without a search
`03_21_from_utf16_endian.rs`	`from_utf16le`/`from_utf16be` and their lossy variants (1.98)
`03_23_from_utf8_lossy_owned.rs`	`String::from_utf8_lossy_owned` and `FromUtf8Error::into_utf8_lossy` (1.99): lossy conversion of an owned `Vec<u8>`, buffer reuse for valid input

3.2 · Formatting Mastery: fmt, write!, format_args!

Domain 3 — Strings and Text Processing Duration: ~15 minutes Library components: std::fmt, std::fmt::NumBuffer, all formatting traits, std::fmt::Formatter, std::fmt::Arguments, write!, writeln!, format!, format_args!

Introduction

The Rust formatting system is more than println!. The std::fmt module defines a family of traits, and a format specifier selects each one:

Display and Debug
Binary, Octal, LowerHex, and UpperHex
LowerExp and UpperExp
Pointer

The Formatter struct gives access to all the options that the caller supplied (width, alignment, precision, fill, sign). When you know these traits and options, you can write custom types that behave exactly like built-in types.

This tutorial explains:

How to implement Display and Debug for custom types.
How to use the Formatter options: width, alignment, fill, precision, sign, zero-padding.
The numeric format traits and the pointer format trait.
format_args! for formatting without allocation.
{integer}::format_into and std::fmt::NumBuffer for decimal integers without allocation.
write! and writeln! for fmt::Write targets and for io::Write targets.
The debug builder helpers (DebugStruct, DebugTuple, DebugList, DebugMap, DebugSet).

Display and Debug

Display is for output to the end user. Debug is for developers. A Debug implementation should always give a representation that is similar to syntactically valid Rust.

use std::fmt;

struct Color { r: u8, g: u8, b: u8 }

// Display: the text for the end user, a CSS-style hex color such as #FF0000.
impl fmt::Display for Color {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        // {:02X} writes one u8 as 2 uppercase hex digits, with a leading zero if necessary.
        write!(f, "#{:02X}{:02X}{:02X}", self.r, self.g, self.b)
    }
}

// Debug: the text for developers, with the type name and the name of each field.
impl fmt::Debug for Color {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_struct("Color")     // starts the output with the type name
            .field("r", &self.r)    // adds `r: <Debug text of self.r>`
            .field("g", &self.g)
            .field("b", &self.b)
            .finish()               // completes the output and returns fmt::Result
    }
}

{} calls Display. {:?} calls Debug. {:#?} enables pretty-printing, and the debug builder helpers do the pretty-printing automatically.

An implementation of Display also gives the type a to_string() method at no cost. The blanket impl<T: fmt::Display> ToString for T supplies the method.

03_05_display_debug_traits.rs also formats a Point type and a Meters type. It prints:

Display: #FF0000
Display: #008080

Debug: Color { r: 255, g: 0, b: 0 }
Debug pretty: Color {
    r: 0,
    g: 128,
    b: 128,
}

Point Display: (3, -1.5)
Point Debug:   Point { x: 3.0, y: -1.5 }

Meters Display: 12.35m
Meters Debug:   Meters(12.345678)

formatted: The color is #FF0000 and the point is (3, -1.5)

All assertions passed.

Formatter Options

The format string {:fill<align><sign><#><0><width>.<precision>type} maps to fields on the Formatter struct.

Figure: Format Specifier Syntax

03_2_formatting_mastery_1

Specifier	Meaning	Example
`<` `>` `^`	Left, right, or center alignment	`{:<10}`
`fill`	Fill char, written before the alignment character	`{:*>10}`
`+`	Always print the sign	`{:+}`
`0`	Zero-padding	`{:08}`
`width`	Minimum field width	`{:10}`
`.prec`	Decimal precision, or string truncation	`{:.2}` / `{:.4}`
`width$`	Width from an argument	`{:>width$}`

// Alignment in a field that is 10 characters wide
format!("{:<10}", "left")   // "left      "
format!("{:>10}", "right")  // "     right"
format!("{:^10}", "center") // "  center  "

// Fill character: it goes before the alignment character
format!("{:*<10}", "fill")  // "fill******"
format!("{:0>6}", 42)       // "000042"

// Precision
format!("{:.2}", 3.14159)   // "3.14": 2 digits after the decimal point
format!("{:.4}", "hello")   // "hell": for a string, the precision is the maximum length

// Sign
format!("{:+}", 42_i32)     // "+42": a positive number also gets its sign

// Combination: fill `*`, right alignment, sign, width 10
format!("{:*>+10}", 42_i32) // "*******+42"

// Width at run time: `width$` reads the width from the argument with the name `width`
let w = 12_usize;
format!("{:>width$}", "dynamic", width = w) // "     dynamic"

To obey the options of the caller in a custom Display, call f.pad(&s) with your formatted string. pad applies the alignment and the fill automatically.

03_06_formatter_options.rs prints each formatted value on a line of its own. The last three lines come from a Padded(i64) type that calls f.pad:

left         # (6 spaces follow the text)
     right
  center     # (2 spaces follow the text)
fill******
000042
----hi-----

pi = 3.14
pi = 3.14159
pi = 3
truncated: truncate
truncated: trun

{:+} of 42 = +42
{:+} of -42 = -42
00000042
00002.50

*******+42
     dynamic

Padded left:   [42      ]
Padded right:  [      42]
Padded center: [   42   ]

All assertions passed.

Numeric and Pointer Format Traits

The fmt module defines more traits than Display and Debug. A type character in the format string selects each one:

Trait	Specifier	Example
`Binary`	`{:b}`	`"10101100"`
`Octal`	`{:o}`	`"755"`
`LowerHex`	`{:x}`	`"deadbeef"`
`UpperHex`	`{:X}`	`"DEADBEEF"`
`LowerExp`	`{:e}`	`"1.235e6"`
`UpperExp`	`{:E}`	`"1.235E6"`
`Pointer`	`{:p}`	`"0x7ff..."`

The # flag adds the usual prefix (0b, 0o, 0x).

let n: u32 = 0b1010_1100;             // 172 in decimal
format!("{:b}", n)                    // "10101100"
format!("{:#b}", n)                   // "0b10101100": `#` adds the 0b prefix
format!("{:#X}", 0xDEAD_BEEFu32)      // "0xDEADBEEF": the prefix stays lowercase
format!("{:.3e}", 1_234_567.89_f64)   // "1.235e6": 3 digits after the decimal point

let x = 42u32;
format!("{:p}", &x)                   // the address of x, for example "0x16ee01754" (varies)

You can implement each of these traits on your own types. A common pattern is to implement LowerHex and UpperHex on a newtype for a byte buffer. Then the type formats as a hex string.

03_07_numeric_format_traits.rs prints the lines below. The Rgb lines come from a newtype with custom LowerHex and UpperHex implementations:

Binary:    10101100
Binary 0b: 0b10101100
Binary 8w: 10101100

Octal:    755
Octal 0o: 0o755

LowerHex:    deadbeef
UpperHex:    DEADBEEF
LowerHex 0x: 0xdeadbeef
UpperHex 0X: 0xDEADBEEF
LowerHex 10: 00deadbeef

Rgb lower: ff7f00
Rgb upper: FF7F00

LowerExp: 1.23456789e6
UpperExp: 1.23456789E6
Prec 3:   1.235e6

Pointer: 0x16ee01754   # (varies)

All assertions passed.

format_args! and write!

format! always allocates a String. format_args! captures the format string and its arguments as a fmt::Arguments<'_> value and does not allocate. You can pass the value to any type that implements fmt::Write or io::Write.

// The two traits have the same name. The aliases let you import them together.
use std::fmt::Write as FmtWrite;
use std::io::Write as IoWrite;

// Write to a String (String implements fmt::Write)
let mut buf = String::new();
write!(buf, "Hello, world!").unwrap();         // buf is "Hello, world!"
writeln!(buf, " Next line.").unwrap();         // appends the text and a '\n'

// Write to a Vec<u8> (Vec<u8> implements io::Write)
let mut bytes: Vec<u8> = Vec::new();
write!(bytes, "bytes: {}", 255_u8).unwrap();   // bytes is b"bytes: 255"

// A logger with no intermediate String: `args` goes directly to the sink
fn log_to(sink: &mut dyn FmtWrite, args: std::fmt::Arguments<'_>) {
    sink.write_fmt(args).unwrap();
}
let mut output = String::new();
log_to(&mut output, format_args!("event={} level={}", "login", "INFO"));
// output is "event=login level=INFO"

fmt::Write and io::Write are different traits. write! works with the two traits. The macro calls write_fmt from the trait that is in scope. To prevent ambiguity, import only the trait that you need, or import the two traits with aliases.

03_08_format_args.rs prints:

value = 42
format_args passthrough: x = 1, y = 2
write! to String: "Hello, world! Next line.\n"
report:
  item 0: value=0
  item 1: value=10
  item 2: value=20

write! to Vec<u8>: [98, 121, 116, 101, 115, 58, 32, 50, 53, 53]
log_to output: "event=login level=INFO"

Named args: Alice scored 99 points
Positional: yes and no and yes

All assertions passed.

format_into (1.98) is the fast path for integers only. It writes the decimal text into a NumBuffer on the stack and returns a &str that borrows from the buffer. It uses no heap and no dyn Write. You can use the same buffer again: each call overwrites the previous text.

Rust 1.98 supplied the buffer type only as core::fmt::NumBuffer. Since Rust 1.99, std::fmt and alloc::fmt re-export it, so the usual std::fmt import path is valid.

use std::fmt::NumBuffer; // 1.99 re-export. On 1.98 the path is core::fmt::NumBuffer.

// The buffer is a stack value. Its size comes from the integer type (u32 here).
let mut buf = NumBuffer::new();   // the compiler infers NumBuffer<u32> from the next line
assert_eq!(1972u32.format_into(&mut buf), "1972");   // the &str borrows from `buf`
// A negative number includes its sign. This call uses a temporary NumBuffer<i32>.
assert_eq!((-1972i32).format_into(&mut NumBuffer::new()), "-1972");

Debug Builder Helpers

Manual formatting of {:?} output is tedious, and the code must also handle the {:#?} pretty-print flag. The builder helpers do this work automatically:

Helper	Created by	Use for
`DebugStruct`	`f.debug_struct("Name")`	Struct with named fields
`DebugTuple`	`f.debug_tuple("Name")`	Tuple struct
`DebugList`	`f.debug_list()`	Sequence of values
`DebugSet`	`f.debug_set()`	Unordered set of values
`DebugMap`	`f.debug_map()`	Key–value pairs

// QueryNode has the fields table: &'static str, limit: usize, columns: Vec<&'static str>.
impl fmt::Debug for QueryNode {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_struct("QueryNode")            // the type name in the output
            .field("table", &self.table)       // field name, then a value that implements Debug
            .field("limit", &self.limit)
            .field("columns", &self.columns)   // the Vec uses its own Debug implementation
            .finish()                          // completes the output and returns fmt::Result
    }
}

The same Debug implementation gives compact output with {:?} and indented output with {:#?}.

03_09_debug_builders.rs has one type for each helper. It prints:

DebugStruct compact:
QueryNode { table: "users", limit: 10, columns: ["id", "name", "email"] }

DebugStruct pretty:
QueryNode {
    table: "users",
    limit: 10,
    columns: [
        "id",
        "name",
        "email",
    ],
}

DebugTuple: Pair(3, 7)
DebugList: ["parse", "validate", "execute"]
DebugSet: {"async", "rust", "web"}
DebugMap: {"host": "localhost", "port": "8080"}

All assertions passed.

Summary

Tool	Purpose
`Display` (`{}`)	String representation for users
`Debug` (`{:?}`)	Representation for developers, which you can derive
`Formatter` options	Width, alignment, fill, precision, sign, zero-padding
`Binary`/`Octal`/`Hex`/`Exp`	Numeric format traits for `{:b}`, `{:o}`, `{:x}`, `{:e}`
`Pointer` (`{:p}`)	Raw memory address
`format_args!`	Argument capture without allocation
`format_into` + `NumBuffer` (1.98, `std::fmt` path since 1.99)	Decimal integers into a stack buffer. The `&str` borrows from the buffer.
`write!` / `writeln!`	Format into any `fmt::Write` or `io::Write` target
Debug builders	Automatic support for the `{:#?}` pretty-print flag

Code Examples

File	Description
`03_05_display_debug_traits.rs`	Custom `Display` and `Debug`, `to_string()`, `format!`
`03_06_formatter_options.rs`	Width, alignment, fill, precision, sign, zero-padding, run-time width
`03_07_numeric_format_traits.rs`	`Binary`, `Octal`, `LowerHex`, `UpperHex`, `LowerExp`, `UpperExp`, `Pointer`
`03_08_format_args.rs`	`format_args!`, `write!`, `writeln!`, a logger with no intermediate `String`
`03_09_debug_builders.rs`	`DebugStruct`, `DebugTuple`, `DebugList`, `DebugSet`, `DebugMap`
`03_22_format_into.rs`	`{integer}::format_into` and `std::fmt::NumBuffer` (1.98, `std::fmt` path since 1.99)

3.3 · Pattern Matching on Strings and FromStr Parsing

Domain 3 — Strings and Text Processing Duration: ~15 minutes Library components: std::str::pattern::Pattern, std::str::FromStr, str methods

Introduction

The string search methods of Rust accept a pattern, not one fixed char type or string type. A pattern can be:

a char
a &str
a &[char] slice
a |c: char| -> bool closure

You can pass each of these to contains, find, split, and related methods. The call site does not change.

For parsing, FromStr is the standard trait to parse a value of any type from a string. When you implement it, you get str::parse::<T>() at no cost. It also integrates with the ? operator.

This tutorial explains:

How to use char, &str, &[char], and closures as patterns.
contains, starts_with, ends_with, find, rfind.
split, splitn, split_once, rsplit, rsplitn, rsplit_once, split_whitespace.
trim, trim_start, trim_end, trim_matches, strip_prefix, strip_suffix, strip_circumfix.
How to implement FromStr and use str::parse::<T>().

The Pattern Abstraction

The Pattern trait (in std::str::pattern) unifies the types that you can search for in a string. The trait is currently unstable as a public trait, but all the APIs that use it are stable.

Figure: Pattern Types — What Can You Search For?

03_3_patterns_fromstr_1

let url = "https://example.com/path?key=value";

// char as pattern
url.contains('?')           // true
url.find('/')               // Some(6): the byte offset of the first '/'

// &str as pattern
url.starts_with("https://") // true
url.ends_with("value")      // true

// &[char] as pattern: a match is any one of the chars in the slice
let seps = &['/', '?', '='];   // seps: &[char; 3]
url.contains(seps.as_ref())    // true: as_ref() gives the &[char] slice

// Closure as pattern: the method calls the predicate for each char
url.contains(|c: char| c.is_ascii_digit())                    // false: the URL has no digit
"error 404: not found".contains(|c: char| c.is_ascii_digit()) // true

03_10_string_patterns.rs prints the lines below. The find and rfind lines search the sentence "the cat sat on the mat":

contains '?': true
starts_with "https://": true
ends_with "value": true
contains any separator: true
contains a digit: false
"error 404: not found" contains digit: true

find "cat": Some(4)
find ' ': Some(3)
find vowel: Some(2)

rfind ' ': Some(18)
rfind "at": Some(20)

query string: "key=value"

All assertions passed.

split and Its Variants

split returns a lazy iterator. It does not allocate until you call collect.

// Basic split
let names: Vec<&str> = "alice,bob,carol,dave".split(',').collect();
// ["alice", "bob", "carol", "dave"]

// A delimiter at an end, or two adjacent delimiters, give an empty item
",a,,b,".split(',').collect::<Vec<_>>()
// ["", "a", "", "b", ""]

// split_terminator: a delimiter at the end gives no empty item
"a,b,c,".split_terminator(',').collect::<Vec<_>>()
// ["a", "b", "c"]

// splitn: a maximum of n parts. The last part can contain the delimiter.
"key=value=extra".splitn(2, '=').collect::<Vec<_>>()
// ["key", "value=extra"]

// split_once: the idiomatic parse of key=value text. It splits at the first match.
"Content-Type: application/json".split_once(": ")
// Some(("Content-Type", "application/json"))

// rsplit / rsplitn: split from the right
"std.fmt.Display".rsplit('.').collect::<Vec<_>>()
// ["Display", "fmt", "std"]

// rsplit_once: splits at the last match
"archive.tar.gz".rsplit_once('.')
// Some(("archive.tar", "gz"))

// split_whitespace: splits on any whitespace and skips empty tokens
"  one  two\tthree\n".split_whitespace().collect::<Vec<_>>()
// ["one", "two", "three"]

03_11_split_and_search.rs prints:

split: ["alice", "bob", "carol", "dave"]
split on "::": ["home", "user", "documents", "file.txt"]
split with empties: ["", "a", "", "b", ""]
split_terminator: ["a", "b", "c"]

splitN(2): ["key", "value=extra"]

split_once key: "Content-Type"
split_once val: "application/json"

rsplit: ["Display", "fmt", "std"]
rsplitn(2): ["Display", "std.fmt"]

rsplit_once stem: "archive.tar"
rsplit_once ext:  "gz"

split_whitespace: ["one", "two", "three"]

All assertions passed.

trim and strip

trim removes Unicode whitespace from the two ends. trim_matches does the same for any pattern. The strip_* methods return an Option, for safe removal of a prefix or a suffix.

// trim: removes whitespace from the two ends
"  hello world  ".trim()           // "hello world"
"  left".trim_start()              // "left"
"right  ".trim_end()               // "right"

// trim_matches: removes every repeat of a char from the two ends
"\"value\"".trim_matches('"')      // "value"
"###hello###".trim_matches('#')    // "hello"

// trim_start_matches / trim_end_matches: one end only, with a pattern
"///path/to/file".trim_start_matches('/')  // "path/to/file": the inner '/' chars stay
"3.14000".trim_end_matches('0')            // "3.14"

// Closure: removes the chars that are not alphanumeric from the two ends
"...hello...".trim_matches(|c: char| !c.is_alphanumeric()) // "hello"

// strip_prefix: removes the prefix and returns an Option
"https://example.com".strip_prefix("https://") // Some("example.com")
"https://example.com".strip_prefix("ftp://")   // None: the prefix is absent

// strip_suffix: removes the suffix and returns an Option
"report.pdf".strip_suffix(".pdf") // Some("report")

// Chain: remove the angle brackets at the two ends
"<value>".strip_prefix('<').and_then(|s| s.strip_suffix('>'))
// Some("value")

// strip_circumfix (1.98): the two ends in one call. None unless the prefix *and* the suffix match.
"<value>".strip_circumfix('<', '>')             // Some("value")
"bar:hello:foo".strip_circumfix("bar:", ":foo") // Some("hello")
"hello".strip_circumfix('<', '>')               // None

03_12_trim_strip.rs prints:

trim: "hello world"
trim_start: "left"
trim_end:   "right"

trim_matches '"': "value"
trim_matches '#': "hello"
trim_start_matches '/': "path/to/file"
trim_end_matches '0': "3.14"
trim_matches closure: "hello"

strip_prefix "https://": Some("example.com")
strip_prefix "ftp://":  None

strip_suffix ".pdf": Some("report")
strip prefix + suffix: Some("value")

All assertions passed.

FromStr and str::parse

FromStr is the standard interface to parse a type from its string representation. An implementation of FromStr is the idiomatic alternative to standalone parse_X(s: &str) functions.

use std::str::FromStr;

#[derive(Debug, PartialEq)]
struct Color { r: u8, g: u8, b: u8 }

// The error type of the parse. It holds a message.
#[derive(Debug)]
struct ParseColorError(String);

// Parses "r,g,b" text such as "255,127,0" into a Color.
impl FromStr for Color {
    type Err = ParseColorError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        let parts: Vec<&str> = s.splitn(3, ',').collect();
        if parts.len() != 3 {
            return Err(ParseColorError(format!("expected 3 components, got {s:?}")));
        }
        // Parses one component. map_err converts the ParseIntError into a ParseColorError.
        let parse_u8 = |p: &str| {
            p.trim()             // permits spaces around the number
                .parse::<u8>()   // fails if the text is not an integer from 0 to 255
                .map_err(|_| ParseColorError(format!("{p:?} is not 0-255")))
        };
        Ok(Color {
            r: parse_u8(parts[0])?,   // `?` returns the error of the first bad component
            g: parse_u8(parts[1])?,
            b: parse_u8(parts[2])?,
        })
    }
}

The parse() method on str calls FromStr::from_str. It is a short form that accepts turbofish syntax:

// Built-in types: the type annotation or the turbofish selects the FromStr implementation
let n: i32  = "42".parse().unwrap();            // 42
let f: f64  = "1.75".parse().unwrap();          // 1.75
let port    = "8080".parse::<u16>().unwrap();   // 8080: the turbofish gives the type u16

// Custom type: parse() calls the from_str of Color
let color: Color = "255,127,0".parse().unwrap(); // Color { r: 255, g: 127, b: 0 }

The associated type Err is the error that from_str returns when the parse fails. To use it with ? in a function that returns Box<dyn Error>, the error type must implement std::error::Error. That trait requires Debug and Display.

03_13_fromstr_parse.rs prints:

"42".parse::<i32>() = 42
"1.75".parse::<f64>() = 1.75
port = 8080
bad parse: true

parsed Color: Color { r: 255, g: 127, b: 0 }
too few: Err(ParseColorError("expected 3 components, got \"255,0\""))
out of range: Err(ParseColorError("\"256\" is not 0-255"))

config key="background" color=Color { r: 30, g: 30, b: 30 }

All assertions passed.

Summary

Tool	Description
`char` / `&str` / `&[char]` / closure	Each one is a valid `Pattern` argument
`contains(pat)`	Returns `true` if the string contains the pattern
`find(pat)` / `rfind(pat)`	Byte offset of the first or the last match of the pattern
`split(pat)`	Iterator of substrings. It consumes all the delimiters.
`splitn(n, pat)`	A maximum of `n` parts. The last part may contain delimiters.
`split_once(pat)`	Splits at the first match. It returns `Option<(&str, &str)>`.
`split_whitespace()`	Splits on any whitespace and skips empty tokens
`trim` / `trim_matches`	Removes whitespace or a custom pattern from the ends
`strip_prefix` / `strip_suffix`	Safe removal of a prefix or a suffix. It returns `Option<&str>`.
`strip_circumfix` (1.98)	Removes a prefix and a suffix in one call. It returns `Option<&str>`.
`FromStr::from_str`	Parses a `&str` into a typed value
`str::parse::<T>()`	Convenient call site for `FromStr`

Code Examples

File	Description
`03_10_string_patterns.rs`	`char`, `&str`, `&[char]`, and closures as patterns, with `contains`, `find`, `rfind`
`03_11_split_and_search.rs`	`split`, `splitn`, `split_once`, `rsplit`, `rsplitn`, `split_whitespace`
`03_12_trim_strip.rs`	`trim`, `trim_matches`, `trim_start/end_matches`, `strip_prefix`, `strip_suffix`
`03_20_strip_circumfix.rs`	`str::strip_circumfix` (1.98): remove a prefix and a suffix, or get `None`
`03_13_fromstr_parse.rs`	`FromStr` implementation, `parse::<T>()`, error handling

3.4 · OsString, CStr, CString: Non-UTF-8 String Types

Domain 3 — Strings and Text Processing Duration: ~15 minutes Library components: std::ffi::OsStr, std::ffi::OsString, std::ffi::CStr, std::ffi::CString, std::ffi::NulError, std::ffi::FromBytesWithNulError

Introduction

String and &str guarantee valid UTF-8. Two important environments do not give this guarantee:

The operating system: On Unix, a filename is an arbitrary byte sequence (not necessarily UTF-8). On Windows, a filename is WTF-16. OsStr and OsString represent these platform-native strings.
C libraries: A C string is a null-terminated char * buffer with no encoding guarantee. CStr and CString model this buffer for safe FFI.

The two pairs have the same ownership pattern as &str and String. The borrowed type (OsStr, CStr) is a slice that you use through a reference. The owned type (OsString, CString) holds the allocation.

This tutorial describes:

OsStr and OsString for platform-native strings.
How to convert between OsStr, Path, and str.
CString::new and its check for NUL bytes.
CStr::from_bytes_with_nul and from_bytes_until_nul.
as_ptr, which gives the pointer that you pass to a foreign function.

OsStr and OsString

OsStr is a borrowed platform-native string. OsString is the owned version. Their internal representation is opaque and platform-specific. Do not write code that depends on it.

use std::ffi::{OsStr, OsString};

// OsStr::new borrows the &str. It does not copy the text.
let os_str: &OsStr = OsStr::new("hello.txt");

// Convert to &str when the content is valid UTF-8 (None if it is not).
let s: Option<&str> = os_str.to_str();  // Some("hello.txt")

// The lossy conversion always succeeds.
let lossy = os_str.to_string_lossy();   // Cow<str>: "hello.txt"

to_string_lossy returns a Cow<str>. If the bytes are valid UTF-8, it returns Cow::Borrowed and does not allocate. If the bytes are not valid UTF-8, it returns Cow::Owned. In that owned string, U+FFFD replaces each invalid sequence.

An OsString can grow:

// OsString::from copies the text into a new allocation.
let mut owned: OsString = OsString::from("my_file");
owned.push(".rs");   // appends to the end: owned is now "my_file.rs"

// into_string consumes the OsString and returns Result<String, OsString>.
owned.into_string()  // Ok("my_file.rs"): the content is valid UTF-8

OsStr and Path

Path and PathBuf are thin wrappers around OsStr and OsString. You can convert between them without restrictions:

use std::path::{Path, PathBuf};

// The parts of a path are &OsStr values, not &str values.
Path::new("/usr/local/bin").file_name()  // Some(OsStr::new("bin"))
Path::new("archive.tar.gz").extension()  // Some(OsStr::new("gz")): only the last extension

// PathBuf is the owned type, as OsString is the owned type for OsStr.
let mut pb = PathBuf::from("/home/user");
pb.push("documents");    // adds one path component
pb.push("report.pdf");   // pb is now "/home/user/documents/report.pdf"
pb.as_os_str()           // &OsStr: the OS string of the full path

In an API that receives filenames, accept &OsStr (or AsRef<OsStr>) and not &str. Then the API does not lose data on platforms that have non-UTF-8 filenames.

03_14_osstring_osstr.rs prints the lines below. The last two lines are from its describe_file function, which reads a filename with to_str.

OsStr: "hello.txt"
to_str: Some("hello.txt")
to_string_lossy: hello.txt

OsString after push: "my_file.rs"
OsString len: 10
into_string: Ok("my_file.rs")

Path::file_name: Some("bin")
extension: Some("gz")

PathBuf: /home/user/documents/report.pdf
as_os_str: "/home/user/documents/report.pdf"

main.rs is a Rust source file
data.csv is a regular file

All assertions passed.

CString and CStr

C functions typically accept const char *, which is a pointer to a null-terminated sequence of bytes. CString allocates such a buffer, and CStr borrows one.

CString::new

use std::ffi::CString;

// CString::new copies the text and appends the NUL terminator.
let cstr = CString::new("hello from Rust").unwrap();
// Internally: b"hello from Rust\0"

// ptr: *const c_char (c_char is i8 or u8, as the platform defines).
// You can pass ptr to a C function. It is valid only while `cstr` is alive.
let ptr = cstr.as_ptr();

CString::new returns the error NulError if the input contains an interior NUL byte. Such a byte would terminate the C string too early:

// The NUL byte is at index 3, after the 3 bytes of "bad".
let bad = CString::new("bad\0string"); // Err(NulError): its nul_position() returns 3

Recovering bytes

// Each method consumes the CString and returns its bytes as a Vec<u8>.
CString::new("rust").unwrap().into_bytes()           // [114, 117, 115, 116]: no NUL
CString::new("rust").unwrap().into_bytes_with_nul()  // [114, 117, 115, 116, 0]

Borrowing C Strings

CStr::from_bytes_with_nul

CStr::from_bytes_with_nul borrows a &[u8] as a &CStr. The slice must end with exactly one NUL. If the slice has an interior NUL or has no NUL at its end, the function returns FromBytesWithNulError:

// CStr is in std::ffi, as CString is.
let bytes: &[u8] = b"example\0";   // 8 bytes: the last byte is the NUL
// The &CStr borrows `bytes`. The function does not copy them.
let cstr: &CStr = CStr::from_bytes_with_nul(bytes).unwrap();
cstr.to_str().unwrap() // "example" (to_str returns Err if the bytes are not UTF-8)

CStr::from_bytes_until_nul (stabilized 1.69)

CStr::from_bytes_until_nul finds the first NUL byte and uses it as the terminator. Use this function for a C buffer of constant size that may contain unwanted bytes after the NUL:

// The buffer has two NUL bytes. Only the first NUL is the terminator.
let buf = b"hello\0ignored\0";
let found = CStr::from_bytes_until_nul(buf).unwrap();   // a &CStr that contains "hello"
found.to_str().unwrap() // "hello": the bytes after the first NUL are not included

CStr comparison

CStr implements PartialOrd and Ord, so you can compare C strings by byte value:

let a = CString::new("alpha").unwrap();
let b = CString::new("beta").unwrap();
// as_c_str borrows a CString as a &CStr.
a.as_c_str() < b.as_c_str() // true: the first bytes differ, and b'a' < b'b'

03_15_cstr_cstring.rs prints:

CString: "hello from Rust"
bytes_with_nul last: 0x00
as_ptr: 0x10542cd20   # (varies)

CString with interior NUL: true
NUL at position: 3

into_bytes: [114, 117, 115, 116]

CStr::from_bytes_with_nul: "example"
to_str: "example"

from_bytes_until_nul: "hello"

"alpha" < "beta": true

All assertions passed.

When to Use Each Type

Type	Use when
`&str` / `String`	The text is always valid UTF-8 (most Rust code)
`&OsStr` / `OsString`	The text is a filesystem path, an environment variable, or other platform-native text
`&CStr` / `CString`	You pass strings to C functions, or receive strings from C functions, through FFI

Figure: Choosing the Right String Type

03_4_osstring_cstring_1

Do not use String for filenames if portability is important to you. Always use Path or PathBuf (their storage is OsStr and OsString). Convert to str only at the point where you need a str. For that conversion, use to_str() (which returns an Option) or to_string_lossy() (which always succeeds).

Summary

Type	Borrowed/Owned	Encoding	Interior NUL allowed
`&str`	Borrowed	UTF-8 guaranteed	Yes
`String`	Owned	UTF-8 guaranteed	Yes
`&OsStr`	Borrowed	Platform-native	Yes
`OsString`	Owned	Platform-native	Yes
`&CStr`	Borrowed	Null-terminated, unspecified	No (ends at first NUL)
`CString`	Owned	Null-terminated, unspecified	No (`CString::new` rejects it)

Code Examples

File	Description
`03_14_osstring_osstr.rs`	`OsStr`/`OsString`, `to_str`, `to_string_lossy`, and their relation to `Path`/`PathBuf`
`03_15_cstr_cstring.rs`	`CString::new`, `NulError`, `CStr::from_bytes_with_nul`, `from_bytes_until_nul`, `as_ptr`

3.5 · ASCII Operations and Character Classification

Domain 3 — Strings and Text Processing Duration: ~15 minutes Library components: std::ascii, std::char, primitive char methods, char::from_digit, char::to_digit

Introduction

The char type of Rust represents a Unicode scalar value. Its classification methods apply the full Unicode standard. For example, is_alphabetic() returns true for 'é' and '中', not only for ASCII letters. Use these methods when you need behaviour that is correct for Unicode.

Some domains are strictly ASCII: network protocols, identifiers, numeric parsing. For these domains, the is_ascii_* family does the same checks, but only in the ASCII range. The to_ascii_* and make_ascii_* methods convert case fast and do not change non-ASCII bytes. The make_ascii_* methods convert in place.

This tutorial shows:

Unicode-aware char classification: is_alphabetic, is_numeric, is_alphanumeric, is_whitespace, is_uppercase, is_lowercase.
ASCII-specific classification: is_ascii, is_ascii_alphabetic, is_ascii_digit, is_ascii_punctuation, is_ascii_graphic, is_ascii_control.
char::from_digit and to_digit, which convert digits in a given base.
Unicode case conversion: to_lowercase, to_uppercase (these methods return iterators).
ASCII-only case conversion: make_ascii_lowercase, make_ascii_uppercase, to_ascii_lowercase, to_ascii_uppercase.
Escape iterators: EscapeDefault, EscapeDebug, EscapeUnicode.

Unicode Character Classification

The classification methods in this section apply the Unicode standard.

// Each method returns a bool.
'é'.is_alphabetic()    // true: alphabetic in Unicode, but not ASCII
'中'.is_alphabetic()   // true: a CJK ideograph
'1'.is_numeric()       // true
' '.is_whitespace()    // true
'\n'.is_whitespace()   // true
'A'.is_uppercase()     // true
'a'.is_lowercase()     // true

Method	Returns true for
`is_alphabetic()`	Each Unicode alphabetic character
`is_numeric()`	Each Unicode numeric character (fractions and superscripts included)
`is_alphanumeric()`	Each character that is alphabetic or numeric
`is_whitespace()`	Unicode whitespace (space, tab, newline, and others)
`is_uppercase()`	Uppercase Unicode characters
`is_lowercase()`	Lowercase Unicode characters
`is_control()`	Control characters (C0, C1)

Since Rust 1.97, char::is_control is a const fn, so the classification of control characters can run at compile time. For example, you can check a protocol delimiter constant while the program builds:

const DELIM: char = '\x1F'; // ASCII unit separator
// The assert! runs in const evaluation. If DELIM is not a control character,
// the build fails. The program does not start.
const _: () = assert!(DELIM.is_control(), "delimiter must be a control character");

The first part of the output of 03_16_char_classification.rs:

char   alphabetic   numeric    alphanumeric   whitespace   uppercase    lowercase
--------------------------------------------------------------------------------
'A'    true         false      true           false        true         false
'a'    true         false      true           false        false        true
'1'    false        true       true           false        false        false
' '    false        false      false          true         false        false
'\n'   false        false      false          true         false        false
'!'    false        false      false          false        false        false
'é'    true         false      true           false        false        true
'中'    true         false      true           false        false        false
'\t'   false        false      false          true         false        false

const is_control ('\u{1f}'): true (checked at compile time)

ASCII Classification

The is_ascii_* methods examine only the 128 ASCII code points. They return false for each non-ASCII character, even if the character satisfies the Unicode criterion:

Figure: Unicode vs ASCII Classification Scope

03_5_ascii_char_1

'é'.is_ascii_alphabetic()  // false: 'é' is alphabetic, but it is not ASCII
'A'.is_ascii_alphabetic()  // true
'5'.is_ascii_digit()       // true
'!'.is_ascii_punctuation() // true
'A'.is_ascii_graphic()     // true: a graphic character is printable and is not the space
' '.is_ascii_whitespace()  // true
'\x01'.is_ascii_control()  // true: the control characters are 0x00–0x1F and 0x7F

is_ascii() tells you if the code point of the character is in 0..=127. Each other is_ascii_* method tests a subset of that range.

char::from_digit and to_digit

These two functions convert between the numeric value of a digit and its character in a given base:

// char::from_digit(value, radix) returns Option<char>.
char::from_digit(10, 16)  // Some('a'): the hex digit for 10
char::from_digit(5, 10)   // Some('5'): the decimal digit for 5
char::from_digit(16, 16)  // None: 16 is not a digit in base 16

// to_digit(radix) returns Option<u32>.
'f'.to_digit(16)          // Some(15)
'9'.to_digit(10)          // Some(9)
'g'.to_digit(16)          // None: 'g' is not a hex digit

The two functions accept a radix from 2 to 36 (inclusive). The digits 10–35 correspond to the letters 'a'–'z'.

The second part of the output of 03_16_char_classification.rs:

is_ascii:
'A'.is_ascii(): true, 'é'.is_ascii(): false

ASCII classification:
  'A': alpha=true digit=false punct=false graphic=true control=false
  'z': alpha=true digit=false punct=false graphic=true control=false
  '5': alpha=false digit=true punct=false graphic=true control=false
  ' ': alpha=false digit=false punct=false graphic=false control=false
  '!': alpha=false digit=false punct=true graphic=true control=false
  '\u{1}': alpha=false digit=false punct=false graphic=false control=true
  '_': alpha=false digit=false punct=true graphic=true control=false
  '+': alpha=false digit=false punct=true graphic=true control=false

from_digit / to_digit:
from_digit(10, 16) = Some('a')
'f'.to_digit(16) = Some(15)

All assertions passed.

Unicode Case Conversion

char::to_lowercase and char::to_uppercase return iterators, not one char. The reason is that case conversion changes some Unicode characters into more than one character. The usual example is the German 'ß' (U+00DF): its uppercase form is "SS".

// Collect the iterator into a String.
let lower: String = 'A'.to_lowercase().collect();   // "a"
let upper: String = 'a'.to_uppercase().collect();   // "A"
let ss: String    = 'ß'.to_uppercase().collect();   // "SS": one char becomes two

// str::to_lowercase and str::to_uppercase allocate a new String.
"Héllo, Wörld!".to_lowercase()  // "héllo, wörld!"
"Héllo, Wörld!".to_uppercase()  // "HÉLLO, WÖRLD!"

To make a word title case, convert the first character with to_uppercase. Then append the remainder of the word:

let word = "hello";
let mut chars = word.chars();
// chars.next() removes the first char from the iterator.
// chars.as_str() is the remainder of the word as a &str ("ello").
let titled: String = match chars.next() {
    Some(c) => c.to_uppercase().collect::<String>() + chars.as_str(),
    None    => String::new(),   // the word is empty
};
// titled is "Hello"
// The example binary applies this code to each word of "hello world".

ASCII-Only Case Conversion

Sometimes you know that your data is ASCII, or you want non-ASCII bytes to stay as they are. In these cases, use the make_ascii_* and to_ascii_* variants. They operate on u8 values, do not use Unicode tables, and can convert in place:

// The make_ascii_* methods change the value in place.
// They are methods of [u8] and str, so a Vec<u8> and a String have them too.
let mut bytes = b"Hello, World!".to_vec();   // Vec<u8>
bytes.make_ascii_lowercase();  // b"hello, world!"
bytes.make_ascii_uppercase();  // b"HELLO, WORLD!"

let mut s = String::from("Hello, World!");
s.make_ascii_lowercase();      // "hello, world!"

// The to_ascii_* methods allocate and return a new String.
"RUST".to_ascii_lowercase()    // "rust"
"rust".to_ascii_uppercase()    // "RUST"

// The ASCII methods do not change non-ASCII characters.
"héllo".to_ascii_lowercase()   // "héllo" ('é' does not change)

03_17_case_conversion.rs prints:

'A'.to_lowercase(): "a"
'a'.to_uppercase(): "A"
'ß'.to_uppercase(): "SS"

"Héllo, Wörld!"
  to_lowercase: "héllo, wörld!"
  to_uppercase: "HÉLLO, WÖRLD!"

title case: ["Hello", "World"]

make_ascii_lowercase: "hello, world!"
make_ascii_uppercase: "HELLO, WORLD!"
str make_ascii_lowercase: "hello, world!"
to_ascii_lowercase: "rust"
to_ascii_uppercase: "RUST"
"héllo".to_ascii_lowercase(): "héllo" (é unchanged)

All assertions passed.

Escape Iterators

Three methods of char return iterators that produce escape sequences as text. The iterators yield char values and do not allocate:

Method	Behaviour
`escape_default()`	Rust literal style: `\n`, `\t`, `\\`, `\"`, and `\u{XXXX}` for characters that are not printable or not ASCII
`escape_debug()`	As `escape_default()`, but it does not change printable non-ASCII characters
`escape_unicode()`	Always `\u{XXXX}` for each character, ASCII included

// The results are written as Rust string literals: "\\n" is the 2 characters \ and n.
'\n'.escape_default().collect::<String>()  // "\\n"
'"'.escape_default().collect::<String>()   // "\\\""     (the 2 characters \ and ")
'é'.escape_default().collect::<String>()   // "\\u{e9}"  (not ASCII: a Unicode escape)
'é'.escape_debug().collect::<String>()     // "é"        (printable: no change)
'a'.escape_unicode().collect::<String>()   // "\\u{61}"  (ASCII is escaped too)
'中'.escape_unicode().collect::<String>()  // "\\u{4e2d}"

str also has escape_default(), escape_debug(), and escape_unicode(). Each one escapes the full string and returns an iterator that yields char values. Collect the iterator into a String, or write it directly to a fmt::Write sink.

The iterators are lazy. Thus you can count the length of the escaped text with no allocation:

// The escaped text is \u{1f600}: 9 characters.
'\u{1F600}'.escape_unicode().count()  // 9

03_18_escape_iterators.rs prints the lines below. The lines for one character show the escaped string in {:?} format, so each backslash of the result appears two times. The two str results use {} format.

escape_default per char:
  '\n' → "\\n"
  '\t' → "\\t"
  '\\' → "\\\\"
  '"' → "\\\""
  '\'' → "\\'"
  'a' → "a"
  'é' → "\\u{e9}"
  '中' → "\\u{4e2d}"

escape_debug per char:
  '\n' → "\\n"
  '\t' → "\\t"
  '\\' → "\\\\"
  '"' → "\\\""
  '\'' → "\\'"
  'a' → "a"
  'é' → "é"
  '中' → "中"

escape_unicode per char:
  'a' → "\\u{61}"
  'A' → "\\u{41}"
  'é' → "\\u{e9}"
  '中' → "\\u{4e2d}"
  '\n' → "\\u{a}"

str escape_default:
  tab:\there\nnewline and \"quotes\"
str escape_debug:
  tab:\there\nnewline and \"quotes\"

escape_unicode char count for U+1F600: 9

All assertions passed.

Summary

Topic	Key APIs
Unicode classification	`is_alphabetic`, `is_numeric`, `is_whitespace`, `is_uppercase`, `is_lowercase`
ASCII classification	`is_ascii`, `is_ascii_alphabetic`, `is_ascii_digit`, `is_ascii_graphic`, `is_ascii_control`
Const classification	`char::is_control` is a `const fn` since 1.97: you can classify at compile time
Digit conversion	`char::from_digit(n, radix)`, `char::to_digit(radix)`
Unicode case	`to_lowercase()` / `to_uppercase()` return iterators (the result may be more than one character)
ASCII case	`to_ascii_lowercase/uppercase`, `make_ascii_lowercase/uppercase` (in place)
Escape iterators	`escape_default`, `escape_debug`, `escape_unicode` return lazy `char` iterators

Code Examples

File	Description
`03_16_char_classification.rs`	Unicode and ASCII classification, `from_digit`, `to_digit`, const `is_control` (1.97)
`03_17_case_conversion.rs`	`to_lowercase`/`to_uppercase` (Unicode), `make_ascii_`/`to_ascii_`
`03_18_escape_iterators.rs`	`EscapeDefault`, `EscapeDebug`, `EscapeUnicode` on `char` and `str`

4.1 · `Vec<T>`: The Workhorse Collection

Domain 4 — Collections Deep Dive Duration: ~15 minutes Library components: std::vec::Vec, std::vec::IntoIter, std::vec::Drain, std::vec::Splice

Introduction

Vec<T> is the most common collection in Rust. It is a contiguous, growable array type, and its elements are on the heap. Almost every non-trivial Rust program uses it. Through Deref, a Vec also gets hundreds of methods from [T].

This tutorial shows:

The internal layout of Vec<T> (pointer, length, capacity), and why it is always 24 bytes on 64-bit platforms for each T.
How to divide a Vec into these three parts with Vec::into_parts, and how to build it again with Vec::from_parts (stabilized in 1.99).
The growth strategy, and how to control allocation with with_capacity, reserve, reserve_exact, shrink_to_fit, and shrink_to.
How to get a mutable reference to the new element with push_mut and insert_mut (stabilized in 1.95).
How to filter in place with retain, and how to remove consecutive duplicates with dedup, dedup_by, and dedup_by_key.
How to remove and replace many elements in one call with drain, splice, and split_off.
How Vec<T> implements Deref<Target = [T]> and thus gives you every slice method at no cost.
The difference between extend_from_slice and extend.
How to convert a Vec to a Box<[T]> with into_boxed_slice.

Internal Layout and Capacity Management

A Vec<T> is three machine words on the stack:

a pointer to the heap allocation
a usize length: the number of initialized elements
a usize capacity: the number of elements that the allocation can contain before a reallocation

Each of these values has the size of a pointer. Thus size_of::<Vec<T>>() is always 24 bytes on a 64-bit platform. The size is the same when T is u8 and when T is i64.

Figure: Vec<T> Memory Layout

Vec memory layout

use std::mem;

// The size of the Vec value does not include the heap buffer.
assert_eq!(mem::size_of::<Vec<u8>>(), mem::size_of::<Vec<i64>>());  // both are 24 on 64-bit

Since Rust 1.99, Vec::into_parts gives you these three words as values, and Vec::from_parts builds the Vec again. The pointer has the type NonNull<T>, because a Vec pointer is never null. Code that stores a buffer as a raw pointer (an FFI boundary, a custom data structure) uses this pair:

use std::ptr::NonNull;

let mut samples: Vec<u32> = Vec::with_capacity(8);
samples.extend([10, 20, 30]);   // len 3, capacity 8

// into_parts consumes the Vec. No destructor runs, so the buffer stays allocated.
let (ptr, len, cap): (NonNull<u32>, usize, usize) = samples.into_parts();
assert_eq!((len, cap), (3, 8));

// SAFETY: index 3 is inside the allocation of 8 elements.
unsafe { ptr.add(3).write(40) };

// SAFETY: the pointer and the capacity come from into_parts, and 4 elements are initialized.
let rebuilt: Vec<u32> = unsafe { Vec::from_parts(ptr, len + 1, cap) };
assert_eq!(rebuilt, [10, 20, 30, 40]);   // `rebuilt` owns and frees the buffer

04_20_vec_into_parts.rs prints:

parts: len = 3, cap = 8
rebuilt: [10, 20, 30, 40], cap = 8
empty Vec round trip: len = 0, cap = 0

All assertions passed.

into_raw_parts and from_raw_parts are the same pair with a *mut T pointer. Tutorial 16.3 explains NonNull.

with_capacity

When you know the necessary number of elements before you start, use Vec::with_capacity(n). It allocates space for n elements and does not initialize them. The length stays at zero. Subsequent pushes that stay in that capacity never reallocate.

let mut buf: Vec<u8> = Vec::with_capacity(1024);
assert_eq!(buf.len(), 0);            // no element is initialized
assert!(buf.capacity() >= 1024);     // the capacity can be larger than the request

let ptr_before = buf.as_ptr();       // the address of the heap buffer
buf.extend_from_slice(&[0xDE, 0xAD, 0xBE, 0xEF]);   // 4 bytes: they fit in the capacity
let ptr_after = buf.as_ptr();
assert!(std::ptr::eq(ptr_before, ptr_after)); // same address: no reallocation

Growth strategy

When a Vec has no more free capacity, it reallocates. The specification does not guarantee the exact strategy, but the current implementation approximately doubles the capacity. For 20 pushes into an empty Vec<i32>, the capacity changes as follows:

len= 1, capacity grew: 0 -> 4
len= 5, capacity grew: 4 -> 8
len= 9, capacity grew: 8 -> 16
len=17, capacity grew: 16 -> 32

Because the capacity doubles, push is amortized O(1). But if you prevent those intermediate allocations with with_capacity, you save time in performance-critical loops.

reserve, reserve_exact, shrink_to_fit, shrink_to

reserve(n) makes sure that there is space for at least n more elements. The allocator may give more.
reserve_exact(n) requests a capacity of exactly len + n. This value is still only a lower bound, because the allocator may round it up.
shrink_to_fit() releases all unused capacity. The capacity becomes as near to the length as the allocator permits.
shrink_to(min) releases capacity but keeps space for at least min elements. Use it when you know that the Vec will grow again to a moderate size.

let mut data = vec![1, 2, 3];
data.reserve(100);                      // space for 100 more elements
assert!(data.capacity() >= 103);        // len (3) + requested (100)

let mut bloated = Vec::with_capacity(1000);
bloated.extend_from_slice(&[1, 2, 3, 4, 5]);   // len 5, capacity 1000
bloated.shrink_to_fit();                // releases the unused capacity
assert!(bloated.capacity() <= 10);      // the capacity is now near the length

let mut partial = Vec::with_capacity(200);
partial.extend(0..10);                  // len 10, capacity 200
partial.shrink_to(50);                  // keeps a capacity of at least 50
assert!(partial.capacity() >= 50 && partial.capacity() < 200);

push_mut and insert_mut: a handle to the new element

Since Rust 1.95, push_mut and insert_mut do the same work as push and insert, but they return &mut T. This value is a mutable reference to the new element. You do not need v.push(x) and then v.last_mut().unwrap() to change the new element immediately:

let mut scores: Vec<i32> = Vec::new();
let newest = scores.push_mut(10);      // newest: &mut i32, points to the new 10
*newest += 5;                          // the element is now 15
let first = scores.insert_mut(0, 1);   // inserts 1 at index 0, first: &mut i32
*first *= 100;                         // the element at index 0 is now 100
assert_eq!(scores, [100, 15]);

The returned reference borrows the Vec mutably. Thus the borrow must end before you use the Vec again. The borrow checker applies the same rule as for last_mut.

04_01_vec_layout_capacity.rs prints:

size_of::<Vec<u8>>()  = 24 bytes
size_of::<Vec<i64>>() = 24 bytes

with_capacity(1024): len=0, cap=1024
after 4 pushes: len=4, cap=1024, ptr_moved=false

Growth observation:
  len= 1, capacity grew: 0 → 4
  len= 5, capacity grew: 4 → 8
  len= 9, capacity grew: 8 → 16
  len=17, capacity grew: 16 → 32

after reserve(100): len=3, cap=103
after reserve_exact(5): len=10, cap=15

before shrink: len=5, cap=1000
after shrink_to_fit: len=5, cap=5

shrink_to(50): len=10, cap=50

push_mut/insert_mut: [100, 15]

All assertions passed.

retain and dedup

retain: filtering in place

retain examines each element of the Vec and keeps only the elements for which the predicate returns true. It operates in place, with no new allocation. It keeps the relative order of the elements that stay.

// -999.0 is a sentinel value that represents a sensor error.
let mut readings = vec![22.1, -999.0, 23.4, -999.0, 24.0, 22.8];
readings.retain(|&x| x > -100.0);   // keeps the values above -100.0
assert_eq!(readings, [22.1, 23.4, 24.0, 22.8]);

The closure receives an immutable reference to each element. But the closure can capture mutable state from the surrounding scope. For example, it can count the elements that retain removes:

let mut nums = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let mut removed = 0;
nums.retain(|&x| {
    // For a multiple of 3: count it, and return false to remove it.
    if x % 3 == 0 { removed += 1; false } else { true }
});
// nums is now [1, 2, 4, 5, 7, 8, 10]
assert_eq!(removed, 3);   // 3, 6, and 9

dedup: collapsing consecutive duplicates

dedup() removes consecutive equal elements and keeps the first element of each run. It compares only adjacent pairs. Thus dedup removes all duplicates only if the Vec is sorted (or grouped in some other way).

let mut sorted = vec![1, 1, 2, 3, 3, 3, 4, 4, 5];
sorted.dedup();   // equal values are adjacent, so dedup removes all duplicates
assert_eq!(sorted, [1, 2, 3, 4, 5]);

// In unsorted data, the duplicates that are not adjacent stay.
let mut unsorted = vec![1, 2, 1, 3, 2, 3];
unsorted.dedup();
assert_eq!(unsorted, [1, 2, 1, 3, 2, 3]); // dedup removed nothing

dedup_by: custom equality

With dedup_by, you define when two consecutive elements are "the same". The closure receives mutable references to the pair (a, b). a is the current element, and b is the previous element that dedup_by kept.

let mut temps: Vec<f64> = vec![20.0, 20.3, 20.1, 21.5, 21.7, 23.0];
// If the closure returns true, dedup_by removes `a` and keeps `b`.
temps.dedup_by(|a, b| (*a - *b).abs() < 0.5);
// 20.3 and 20.1 are less than 0.5 from 20.0. 21.7 is less than 0.5 from 21.5.
assert_eq!(temps, [20.0, 21.5, 23.0]);

dedup_by_key: extract a comparison key

dedup_by_key gets a key from each element and removes the consecutive elements that have equal keys:

let mut words = vec!["Hello".to_string(), "hello".to_string(),
                     "HELLO".to_string(), "world".to_string(),
                     "World".to_string()];
words.dedup_by_key(|w| w.to_lowercase());   // the key ignores the letter case
assert_eq!(words.len(), 2); // "Hello" and "world" stay

04_02_vec_retain_dedup.rs prints:

before retain: [22.1, -999.0, 23.4, -999.0, 24.0, 22.8]
after retain:  [22.1, 23.4, 24.0, 22.8]

kept (not divisible by 3): [1, 2, 4, 5, 7, 8, 10], removed 3

before dedup: [1, 1, 2, 3, 3, 3, 4, 4, 5]
after dedup:  [1, 2, 3, 4, 5]

unsorted after dedup: [1, 2, 1, 3, 2, 3]

before dedup_by: [20.0, 20.3, 20.1, 21.5, 21.7, 23.0]
after dedup_by (Δ<0.5): [20.0, 21.5, 23.0]

before dedup_by_key: ["Hello", "hello", "HELLO", "world", "World"]
after dedup_by_key:  ["Hello", "world"]

All assertions passed.

drain, splice, and split_off

These three methods remove and replace many elements in one call. The same operations with individual remove calls are difficult to write and prone to errors.

Figure: drain, splice, and split_off Operations

04_1_vec_workhorse_1

drain: extract a range

drain(range) removes the elements in the specified range and returns a Drain iterator that yields them. The remaining elements move to close the gap, but the Vec keeps its allocation.

let mut queue = vec!["a", "b", "c", "d", "e", "f"];
// Remove indices 1, 2, and 3. The end of the range is exclusive.
let drained: Vec<&str> = queue.drain(1..4).collect();
assert_eq!(drained, ["b", "c", "d"]);
assert_eq!(queue, ["a", "e", "f"]);   // "e" and "f" moved to close the gap

A common pattern is drain(..), which removes all the elements and keeps the allocation for the next use. It is equivalent to a clear() that also gives you the removed elements:

// `queue` is the Vec<&str> from the previous snippet: ["a", "e", "f"].
let cap_before = queue.capacity();
let all: Vec<&str> = queue.drain(..).collect();   // all is ["a", "e", "f"]
assert!(queue.is_empty());
assert_eq!(queue.capacity(), cap_before);         // the allocation stays

splice: replace a range with new elements

splice(range, iter) does three things:

It removes the elements in range.
It inserts all the items of iter at that position.
It returns a Splice iterator that yields the removed elements.

The length of the replacement can be different from the length of the removed range.

let mut colors = vec!["red", "green", "blue", "yellow"];
// Replace the 2 elements at indices 1..3 with 3 new elements.
let replaced: Vec<&str> = colors.splice(1..3, ["cyan", "magenta", "black"]).collect();
assert_eq!(replaced, ["green", "blue"]);   // the removed elements
assert_eq!(colors, ["red", "cyan", "magenta", "black", "yellow"]);

Two special cases extend the use of splice:

Insertion with no removal: use an empty range at the insertion point.

let mut nums = vec![1, 2, 5, 6];
// The range 2..2 is empty: splice removes nothing and inserts at index 2.
nums.splice(2..2, [3, 4]);
assert_eq!(nums, [1, 2, 3, 4, 5, 6]);

Removal with no insertion: use an empty iterator as the replacement.

let mut letters = vec!['a', 'b', 'c', 'd', 'e'];
// The replacement `[]` is empty: splice only removes indices 1..4.
let cut: Vec<char> = letters.splice(1..4, []).collect();
assert_eq!(cut, ['b', 'c', 'd']);
assert_eq!(letters, ['a', 'e']);

split_off: divide a vec at an index

split_off(mid) divides the Vec into two. The original Vec keeps the elements 0..mid, and the returned Vec gets the elements mid..len. Each Vec owns its allocation.

let mut front = vec![10, 20, 30, 40, 50];
let back = front.split_off(3);     // back: Vec<i32>, the elements from index 3
assert_eq!(front, [10, 20, 30]);   // front keeps indices 0..3
assert_eq!(back, [40, 50]);

04_03_vec_drain_splice.rs prints:

before drain: ["a", "b", "c", "d", "e", "f"]
drained:      ["b", "c", "d"]
after drain:  ["a", "e", "f"]

drain(..): removed ["a", "e", "f"], len=0, cap=6

before splice: ["red", "green", "blue", "yellow"]
replaced:      ["green", "blue"]
after splice:  ["red", "cyan", "magenta", "black", "yellow"]

insert via splice: [1, 2, 3, 4, 5, 6], removed: []
delete via splice: ['a', 'e'], cut: ['b', 'c', 'd']

before split_off(3): [10, 20, 30, 40, 50]
front: [10, 20, 30]
back:  [40, 50]

All assertions passed.

Deref to Slice, extend, and into_boxed_slice

`Vec<T>` as `Deref<Target = [T]>`

Vec<T> implements Deref<Target = [T]>. Thus, where the code expects a &[T], the compiler automatically coerces a &Vec<T>. In practice, this is how Vec "inherits" every slice method and does not implement them again. Examples are contains, first, last, iter, windows, chunks, and binary_search, and there are hundreds more.

// The parameter is a slice, not a Vec.
fn sum_of(data: &[i32]) -> i32 {
    data.iter().sum()
}

let v = vec![10, 20, 30, 40];
let total = sum_of(&v);          // &Vec<i32> coerces to &[i32]
assert_eq!(total, 100);

assert!(v.contains(&20));        // a slice method, called on a Vec
assert_eq!(v.first(), Some(&10));
assert_eq!(v.last(), Some(&40));

The same function accepts arrays, because arrays also coerce to slices:

// `sum_of` is the function from the previous snippet.
let arr = [1, 2, 3];
assert_eq!(sum_of(&arr), 6);     // &[i32; 3] coerces to &[i32]

The design rule: write functions that take &[T], not &Vec<T>. Then a caller with a Vec, an array, or a different contiguous buffer can pass its data.

extend_from_slice vs. extend

The two methods append elements to a Vec, but from different sources:

extend_from_slice(&[T]) copies from a contiguous slice. For Copy types, the compiler can emit one memcpy. Thus it is the fastest method to append many elements from a slice.
extend(iter) accepts any IntoIterator<Item = T>. It is more flexible: it accepts ranges, other Vec values, hash maps, and all other iterable values. But the compiler may not always optimize it to a memcpy.

let mut a = vec![1, 2, 3];
a.extend_from_slice(&[4, 5, 6]);   // copies the three values from the slice
assert_eq!(a, [1, 2, 3, 4, 5, 6]);

let mut b = vec![1, 2];
b.extend([3, 4, 5]);   // an array implements IntoIterator
b.extend(6..=8);        // a range implements IntoIterator too
assert_eq!(b, [1, 2, 3, 4, 5, 6, 7, 8]);

When you have a slice, prefer extend_from_slice because it is clear and fast. When you must append from sources of different types, use extend.

into_boxed_slice

into_boxed_slice() converts a Vec<T> into a Box<[T]> and releases all excess capacity. The result is a slice on the heap with no capacity field: only a pointer and a length. Use it when the collection is complete and you want to keep it at its exact size.

let mut big = Vec::with_capacity(1000);
big.extend(0..5);                                 // len 5, capacity 1000
let boxed: Box<[i32]> = big.into_boxed_slice();   // releases the unused capacity
assert_eq!(&*boxed, &[0, 1, 2, 3, 4]);

// Convert the box to a Vec again if the collection must grow.
let restored: Vec<i32> = boxed.into_vec();
assert_eq!(restored, [0, 1, 2, 3, 4]);

Slice indexing on Vec

Because Vec derefs to [T], all the slice indexing syntax is available on a Vec:

let v = vec!['a', 'b', 'c', 'd', 'e'];
let middle = &v[1..4];               // middle: &[char], borrows indices 1, 2, and 3
assert_eq!(middle, ['b', 'c', 'd']);

// get() returns an Option, so an index that is out of range does not panic.
assert_eq!(v.get(2), Some(&'c'));
assert_eq!(v.get(99), None);

04_04_vec_deref_slice.rs prints:

sum_of(&v) = 100
v.contains(&20) = true, first = Some(10), last = Some(40)
sum_of(&[1,2,3]) = 6

after extend_from_slice: [1, 2, 3, 4, 5, 6]
after extend: [1, 2, 3, 4, 5, 6, 7, 8]

before into_boxed_slice: len=5, cap=1000
boxed slice len = 5
restored vec: [0, 1, 2, 3, 4]

v[1..4] = ['b', 'c', 'd']
v.get(2) = Some('c'), v.get(99) = None

All assertions passed.

Summary

Concept	Key point
Layout	Three machine words: pointer, length, capacity (24 bytes on 64-bit)
`into_parts` / `from_parts`	Divide a `Vec` into a `NonNull` pointer, a length, and a capacity, and build it again (1.99)
`with_capacity`	Allocates the space first, to prevent reallocations when you know the final size
Growth strategy	Approximately doubles the capacity at each reallocation: `push` is amortized O(1)
`reserve` / `reserve_exact`	Make sure that there is space for N more elements
`shrink_to_fit` / `shrink_to`	Release unused capacity
`push_mut` / `insert_mut`	Insert an element and return `&mut` to it in one call (1.95)
`retain`	Filters in place with a predicate
`dedup`	Removes consecutive duplicates (sort first to remove all duplicates)
`dedup_by` / `dedup_by_key`	Remove consecutive duplicates by a custom equality or by a key
`drain`	Removes a range, returns a `Drain` iterator, and keeps the allocation
`splice`	Replaces a range with the items of an arbitrary iterator
`split_off`	Divides a `Vec` into two owned `Vec` values at an index
`Deref<Target = [T]>`	`Vec` gets all slice methods automatically
`extend_from_slice`	Appends many elements from a slice: the fastest method for `Copy` types
`extend`	Appends many elements from any `IntoIterator`: the most flexible method
`into_boxed_slice`	Converts to `Box<[T]>` and releases the excess capacity

Code Examples

File	Description
`04_01_vec_layout_capacity.rs`	Internal layout, `with_capacity`, `reserve`, `shrink_to_fit`, `shrink_to`, `push_mut`/`insert_mut` (1.95)
`04_02_vec_retain_dedup.rs`	`retain`, `dedup`, `dedup_by`, `dedup_by_key`
`04_03_vec_drain_splice.rs`	`drain`, `splice`, `split_off`
`04_04_vec_deref_slice.rs`	`Deref` to slice, `extend_from_slice` vs `extend`, `into_boxed_slice`
`04_20_vec_into_parts.rs`	`Vec::into_parts` / `Vec::from_parts` (1.99): divide a `Vec` into a `NonNull` pointer, a length, and a capacity, then build it again

4.2 · HashMap and HashSet: Hashing Internals and Custom Hashers

Domain 4 — Collections Deep Dive Duration: ~15 minutes Library components: std::collections::HashMap, std::collections::HashSet, std::collections::hash_map::Entry, std::hash::Hash, std::hash::Hasher, std::hash::BuildHasher, std::hash::RandomState

Introduction

HashMap is the most common collection in Rust after Vec. Its lookups, insertions, and removals are amortized O(1). It has all that you need for key-value storage when the order is not important.

Internally, the standard library uses an implementation based on SwissTable (adopted from absl::flat_hash_map of Google). This implementation does quadratic probing, and SIMD instructions accelerate its metadata lookups. You do not need to know these internals to use the type. But they explain why the API has its shape, and why the iteration order is not predictable.

HashSet is literally HashMap<K, ()>. Each set operation is a thin wrapper around the related map operation, with no value type. When you need only membership, and not associated data, HashSet shows that intent clearly. It also has dedicated methods for set algebra.

This tutorial shows:

How to create a HashMap, and how to insert, find, remove, and iterate its entries.
The Entry API: or_insert, or_insert_with, and_modify, or_default.
RandomState and BuildHasher, the trait for hasher factories.
The Hash trait and the hash/eq contract.
How to supply custom hashers.
HashSet operations and set algebra.

HashMap Basics

Creation

HashMap::new() starts with zero capacity. HashMap::with_capacity(n) allocates space for at least n entries. Use it when the size is predictable, to prevent reallocations.

use std::collections::HashMap;

// An empty map does not allocate.
let mut hits: HashMap<&str, u64> = HashMap::new();
// len=0, cap=0

// One allocation with space for at least 100 entries.
let prealloc: HashMap<String, i32> = HashMap::with_capacity(100);
// len=0, cap >= 100

You can also make a map from an array of tuples with HashMap::from([(k, v), ...]). Or you can collect an iterator of (K, V) pairs into a map.

Insertion and Replacement

insert(key, value) returns Option<V>: Some(old) if the key was already present, and None if it was not. Thus you can detect a replacement, and you do not need a separate lookup.

// `hits` is the empty HashMap<&str, u64> from the previous snippet.
hits.insert("/index", 120);      // returns None: the key is new
hits.insert("/about", 45);
hits.insert("/api/data", 300);

let old = hits.insert("/about", 46);   // the key exists: 46 replaces 45
// old == Some(45), the previous value

Lookup

There are three ways to read a value:

Method	Returns	On missing key
`get(&key)`	`Option<&V>`	`None`
`map[&key]` (Index)	`&V`	panics
`contains_key(&key)`	`bool`	`false`

Use get() in production code. The index operator is convenient in tests and assertions, where you know that the key exists.

Removal

remove(&key) returns Option<V>. remove_entry(&key) returns Option<(K, V)>. Use remove_entry when you need the owned key again (for example, a String that you want to use again).

Iteration

The iteration order is non-deterministic. Each call to keys(), values(), or iter(), and each for loop, may yield the entries in a different order. The order may also change between two runs of the same binary. If you need sorted output, collect the keys into a Vec and sort the Vec.

With values_mut() and iter_mut(), you change the values in place. You do not remove the entries and insert them again.

04_05_hashmap_basics.rs prints the output below. A line with (order varies) shows a map in Debug format, and the order of its entries changes between runs:

empty map: len=0, cap=0
with_capacity(100): len=0, cap≥100=true

after inserts: {"/index": 120, "/api/data": 300, "/about": 45}   # (order varies)
replaced /about: old=Some(45), new=Some(46)

hits["/api/data"] = 300

removed /about: Some(46)
remove_entry: Some(("/health", 999))

scores:
  alice: 90
  bob: 85
  carol: 95
total score = 270
after +5 curve: total = 285

collected from tuples: {"y": 2, "x": 1}   # (order varies)

All assertions passed.

The Entry API

A very common pattern is: find a key, and if the key is absent, insert a default value. HashMap has a full sub-API for this pattern. map.entry(key) returns an Entry enum, which is Occupied (the key exists) or Vacant (the key does not exist). With the methods of Entry, you handle the two cases in one expression and with only one lookup.

Figure: The Entry API Flow

04_2_hashmap_hashset_1

or_insert

or_insert inserts its argument as the default value if the key is absent. Then it returns &mut V:

// word_count: HashMap<&str, u32>, initially empty.
let count = word_count.entry("the").or_insert(0);   // count: &mut u32
*count += 1;                                        // the first call changes 0 to 1

This is the usual pattern to count word frequencies. Rust always evaluates the argument of or_insert, also when the key exists. If the default value is expensive to create, use or_insert_with.

or_insert_with

or_insert_with accepts a closure and calls it only when the key is absent:

// cache: HashMap<String, Vec<u8>>. `key` is a String.
// `expensive_computation` represents a slow function that returns a Vec<u8>.
let data = cache.entry(key).or_insert_with(|| {
    expensive_computation()   // runs only on a cache miss
});
// data: &mut Vec<u8>, the value that is now in the cache

Thus, on a cache hit, the program does not allocate or compute a default value that it discards immediately.

or_default

or_default is the short form of or_insert_with(Default::default). Use it when the value type implements Default. Vec, String, u32, and most standard types implement Default:

// groups: HashMap<char, Vec<String>>. `name` is a &str and `initial` is its first char.
// For a new initial, or_default inserts an empty Vec. It returns &mut Vec<String>.
groups.entry(initial).or_default().push(name.to_string());

and_modify

and_modify runs a closure on the value if the key exists. Put it before or_insert or or_default, which handle the absent key:

// stock: HashMap<&str, i32>. `ticker` is a &str and `qty` is an i32.
stock.entry(ticker)
    .and_modify(|holding| *holding += qty)   // the key exists: add qty to the value
    .or_insert(qty);                         // the key is absent: insert qty

This chain has two cases. If the key exists, it updates the value. If the key does not exist, it sets the initial value.

key()

You can read the key of an Entry, and the call does not consume the Entry. Use key() for logging, or for a condition before you decide to insert:

// map: HashMap<String, i32>, initially empty.
let entry = map.entry("hello".to_string());   // a Vacant entry that owns the key
println!("entry key = {:?}", entry.key());    // key() borrows the key: "hello"
entry.or_insert(42);                          // consumes the entry and inserts 42

04_06_entry_api.rs prints:

word_count: {"the": 3, "on": 1, "mat": 1, "sat": 1, "cat": 2}   # (order varies)

  computing default for session_42...
cache hit for session_42: len=64

groups by initial: {'C': ["Charlie", "Cleo"], 'A': ["Alice", "Ada"], 'B': ["Bob", "Brenda"]}   # (order varies)

portfolio: {"AAPL": 7, "GOOG": 13, "MSFT": 20}   # (order varies)

page visits: {"/about": 1, "/home": 3}   # (order varies)

entry key = "hello"

All assertions passed.

SwissTable Internals (Conceptual)

In Rust 1.36, a SwissTable design replaced the Robin Hood hashing implementation of HashMap. The primary ideas are:

Flat, open-addressed layout. All entries are in one contiguous array of buckets. There are no linked lists and no chains on the heap. This layout gives very good cache locality.
Metadata bytes. Each bucket has a control field of one byte. The field contains a 7-bit part of the hash (the "H2" hash) and a status bit. One SIMD instruction can load and compare a group of control bytes. A group has 16 bytes with SSE2 on x86-64, and 8 bytes with NEON on AArch64. Thus the table examines all the candidate buckets of a group in parallel.
Quadratic probing. When a collision occurs, the table probes the subsequent groups at offsets that increase quadratically. In each group, the SIMD lookup examines all the candidates at the same time.
Growth policy. The table grows when the load factor is more than approximately 7/8. To grow, the table hashes each entry again into a new, larger table.

Figure: SwissTable Conceptual Layout

SwissTable layout

You never use these details directly. The important effect that you can see is that the iteration order is non-deterministic. The table layout depends on the insertion order, the capacity, and the random hash seed.

RandomState and BuildHasher

Each HashMap keeps a hasher factory together with its buckets. The default factory is RandomState. At construction time, RandomState seeds the hasher with random bytes from the OS. This seed is a defense against HashDoS attacks. In such an attack, an attacker sends many keys with the same hash value, to make your map lookups O(n). The attack fails because the hash function is different for each map instance.

BuildHasher is the trait that describes a hasher factory:

pub trait BuildHasher {
    type Hasher: Hasher;                      // the hasher type that the factory makes
    fn build_hasher(&self) -> Self::Hasher;   // makes one new hasher
    // The trait also has a provided method, `hash_one`.
}

RandomState implements BuildHasher. Each call to build_hasher() returns a Hasher with the seed of that RandomState. Two different RandomState instances typically give different hashes for the same input:

use std::hash::{BuildHasher, RandomState};

let rs1 = RandomState::new();   // each instance has its own seed
let rs2 = RandomState::new();
// hash(42) through rs1 != hash(42) through rs2 (usually)

Deterministic Hashing with BuildHasherDefault

For tests, benchmarks, or applications where HashDoS is not a risk, you can use BuildHasherDefault<DefaultHasher>. It gives you a deterministic hasher:

use std::hash::BuildHasherDefault;
use std::collections::HashMap;
use std::hash::DefaultHasher;

// The third type parameter of HashMap is the hasher factory.
// BuildHasherDefault makes each hasher with DefaultHasher::default(): no random seed.
let map: HashMap<&str, i32, BuildHasherDefault<DefaultHasher>> =
    HashMap::with_hasher(BuildHasherDefault::default());

Tutorial 13.2 explains in detail how to implement a custom Hasher state machine and a BuildHasher.

04_07_custom_hash.rs prints the output below. Its first lines use the GridPoint key type, which the subsequent section shows. The RandomState hashes change in each run. The DefaultHasher results are the same in each run, but a different Rust release can change them:

lookup (0,0) with different label: Some("home base")

hash(p1)          = 13646096770106105413   # (same value in each run)
hash(p1_relabeled) = 13646096770106105413   # (same value as hash(p1))
hashes match (label excluded): true

RandomState seed 1 → hash(42) = 14353867740142031612   # (varies)
RandomState seed 2 → hash(42) = 7071660002262858912   # (varies)

map with DefaultHasher: {"beta": 2, "alpha": 1}   # (same order in each run)

palette lookup red: Some("red")

All assertions passed.

The Hash Trait and the Hash/Eq Contract

Each type that you use as a HashMap key (or a HashSet element) must implement Hash and Eq. A strict contract connects these two implementations:

If a == b, then hash(a) == hash(b).

The contract does not require the converse. Different values may have the same hash, and collisions are normal. But if two equal values have different hashes, the map silently loses entries. It searches for the key in one bucket, but the entry is in a different bucket.

Deriving Hash

When equality uses all the fields, derive the two traits:

// The two derives use the same fields: r, g, and b.
#[derive(Hash, PartialEq, Eq)]
struct Color { r: u8, g: u8, b: u8 }

The derived Hash gives each field to the hasher in declaration order. The derived PartialEq compares each field. Thus the contract holds automatically.

Manual Hash Implementation

Sometimes only some fields define identity. For example, a struct can have a label field that is only text for the user, and not a part of logical equality. Then you must implement Hash manually and hash exactly the fields that PartialEq uses:

// `Hash` and `Hasher` are the traits from std::hash.
struct GridPoint {
    x: i32,
    y: i32,
    label: String, // not part of identity
}

// Equality compares x and y only.
impl PartialEq for GridPoint {
    fn eq(&self, other: &Self) -> bool {
        self.x == other.x && self.y == other.y
    }
}
impl Eq for GridPoint {}

// Hash the same fields that `eq` compares: x and y.
impl Hash for GridPoint {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.x.hash(state);
        self.y.hash(state);
        // `label` is absent on purpose
    }
}

Now two GridPoint values with the same (x, y) but different labels have the same hash and compare as equal. A lookup gives the correct result, and the label of the lookup key has no effect.

Common Mistakes

Hash uses more fields than Eq compares. This breaks the contract. Two values can be == and have different hashes. This is the dangerous case.
Hash uses fewer fields than Eq compares. This is not a correctness bug. Two values can be != and have the same hash. The contract permits this: it is only a collision.
Floating-point fields. f32 and f64 do not implement Hash. The reason is NaN != NaN, which makes the contract impossible to keep. If you need floats as keys, use a wrapper that orders the bits, or the ordered-float crate.

HashSet: When a Set Is Better Than a Map

HashSet<T> is a HashMap<T, ()> with a simpler API. Use it when you need membership and uniqueness but have no associated value.

Creation and Membership

use std::collections::HashSet;

let mut visited: HashSet<&str> = HashSet::new();
visited.insert("Paris");    // returns true: the value is new
visited.insert("Tokyo");
visited.insert("Berlin");
visited.insert("Paris");    // returns false: the value is already present

visited.contains("Tokyo");  // true
visited.remove("Berlin");   // true: the value was present

To remove duplicates in one line, collect an iterator into a HashSet:

let unique: HashSet<i32> = vec![1, 2, 2, 3, 3, 3, 4].into_iter().collect();
// unique contains 1, 2, 3, and 4, in an arbitrary order

Set Algebra

HashSet has the four standard operations of set algebra. Each method returns a lazy iterator:

Operation	Method	Operator	Meaning
Union	`a.union(&b)`	`&a \| &b`	Elements in either set
Intersection	`a.intersection(&b)`	`&a & &b`	Elements in both sets
Difference	`a.difference(&b)`	`&a - &b`	Elements in `a` but not in `b`
Symmetric difference	`a.symmetric_difference(&b)`	`&a ^ &b`	Elements in exactly one set

The method forms return iterators of &T. The operator forms (&, |, ^, -) return new owned HashSet<T> values. They are convenient, but they allocate.

Subset, Superset, and Disjoint

let small: HashSet<i32> = HashSet::from([3, 4]);
let big: HashSet<i32>   = HashSet::from([1, 2, 3, 4, 5]);

small.is_subset(&big);    // true: 3 and 4 are in `big`
big.is_superset(&small);  // true: the same test from the other side
small.is_disjoint(&HashSet::from([10, 20])); // true: the sets have no common element

is_subset checks that each element of self is in the other set. is_disjoint checks that the intersection is empty. The two methods stop at the first counterexample.

04_08_hashset.rs prints the output below. The Debug format of a HashSet shows the elements in an order that changes between runs:

visited: {"Berlin", "Paris", "Tokyo"}   # (order varies)
primes: {2, 11, 5, 13, 3, 7}   # (order varies)
unique: {2, 3, 4, 1}   # (order varies)

contains Tokyo: true
removed Berlin: true

A = {4, 2, 5, 3, 1}   # (order varies)
B = {7, 3, 4, 5, 6}   # (order varies)
A ∪ B = {6, 3, 2, 5, 1, 7, 4}   # (order varies)
A ∩ B = {5, 4, 3}   # (order varies)
A \ B = {2, 1}   # (order varies)
A △ B = {7, 1, 6, 2}   # (order varies)

{3,4} ⊂ {1..5}: true
disjoint with {10,20}: true

bitwise operators work too: &x & &y = {2, 3}   # (order varies)

All assertions passed.

Summary

Concept	Key point
`HashMap`	Amortized O(1) key-value storage. SwissTable with SIMD probing
`with_capacity`	Allocates the space first, to prevent rehashing when you know the size
`Entry` API	Insert or update with one lookup: `or_insert`, `or_insert_with`, `and_modify`, `or_default`
`RandomState`	Default hasher factory. A random seed for each map instance prevents HashDoS
`BuildHasher`	Trait for hasher factories. Supply a custom hasher with `with_hasher`
Hash/Eq contract	If `a == b` then `hash(a) == hash(b)`. A violation corrupts the map
`#[derive(Hash)]`	Safe when `PartialEq` uses all the fields
Manual `Hash`	Hash exactly the fields that `PartialEq` uses: no more, no fewer
`HashSet<T>`	`HashMap<T, ()>`. Use it when you need membership, not associated data
Set algebra	`union`, `intersection`, `difference`, `symmetric_difference`, and the operator forms

Code Examples

File	Description
`04_05_hashmap_basics.rs`	Creation, insert, get, remove, iteration, collect from tuples
`04_06_entry_api.rs`	`or_insert`, `or_insert_with`, `or_default`, `and_modify`, `key()`
`04_07_custom_hash.rs`	Manual `Hash` implementation, hash/eq contract, `RandomState`, `BuildHasherDefault`
`04_08_hashset.rs`	`HashSet` creation, membership, set algebra, bitwise operators

4.3 · BTreeMap, BTreeSet, and Ordered Collections

Domain 4 — Collections Deep Dive Duration: ~15 minutes Library components: std::collections::BTreeMap, std::collections::BTreeSet, std::collections::btree_map::Entry

Introduction

HashMap and HashSet are the default selections for key-value collections and membership collections. Their lookups are amortized O(1). But they cannot answer one simple query: "give me each entry between key X and key Y". When the order is important (range queries, sorted iteration, interval lookups), you need BTreeMap and BTreeSet.

These collections keep their entries in a B-Tree, a balanced tree structure that always keeps the keys sorted. Each operation (lookup, insertion, removal) is O(log n) in the worst case. There is no "amortized" qualifier, and no pathological hash collision can occur. The trade-off is that O(log n) is slower than O(1) for lookups only. But the sorted invariant makes possible a full class of operations that hash-based collections cannot do.

This tutorial shows:

How to create a BTreeMap and a BTreeSet. The insertion order has no effect: the iteration order is always sorted.
Range queries with range(), the primary advantage over hash-based collections.
first_key_value, last_key_value, pop_first, and pop_last, which give access to the smallest and largest entries.
The Entry API of BTreeMap, which has the same shape as the Entry API of HashMap.
split_off, which divides a map at a threshold, and append, which merges maps.
BTreeSet for ordered deduplication and sorted set operations.
The Ord contract, and why floating-point types cannot be B-Tree keys.
Performance characteristics: when to select a B-Tree and not a hash table.

Figure: B-Tree Node Structure (Conceptual)

04_3_btreemap_btreeset_1

BTreeMap Basics: Creation and Ordered Iteration

Insertion into a BTreeMap is the same as for HashMap. Call insert(key, value), or make the map from an array with BTreeMap::from(...). You see the difference only when you iterate: the entries always come in sorted key order.

use std::collections::BTreeMap;

let mut scores: BTreeMap<&str, u32> = BTreeMap::new();
scores.insert("charlie", 88);
scores.insert("alice", 95);   // inserted second, but iterates first
scores.insert("bob", 72);
scores.insert("diana", 91);

// `&scores` yields (&key, &value) pairs in ascending key order.
for (name, score) in &scores {
    println!("  {name}: {score}");
}
// Output: alice, bob, charlie, diana (lexicographic order)

With HashMap, the iteration order is arbitrary and changes between runs. With BTreeMap, the iteration order is deterministic and sorted. Thus the Debug output of a BTreeMap always shows the keys in order. This property is useful for snapshot tests and reproducible logs.

From arrays

// The compiler infers the type BTreeMap<&str, &str>.
let config = BTreeMap::from([
    ("host", "localhost"),
    ("port", "8080"),
    ("debug", "true"),
]);
// Debug output: {"debug": "true", "host": "localhost", "port": "8080"}

It is not necessary to sort the array elements. The B-Tree puts them in order internally.

Range Queries: The Primary Advantage

The range method is the primary difference between BTreeMap and HashMap. It accepts any range expression (.., start..end, start..=end, start.., ..end, ..=end). It returns an iterator that yields the key-value pairs in that range, in sorted order.

// The key is the hour (0 to 23). The value is the temperature in °C.
let mut temps: BTreeMap<u32, f64> = BTreeMap::new();
// (The example inserts one temperature for each hour here.)

// Query only 06:00 through 12:00 (inclusive).
// `hour` is &u32 and `temp` is &f64.
for (hour, temp) in temps.range(6..=12) {
    println!("  {hour:02}:00 → {temp:.1}°C");
}

This operation is O(log n + k), where k is the number of results. The B-Tree goes down to the start key in O(log n). Then it visits the subsequent entries in sequence. With a HashMap, you must iterate all the entries and filter them, which is O(n) each time.

Practical use cases for range

Time-series data: select all the events between two timestamps.
Leaderboards: find all the players with scores between 90 and 100.
Configuration: get all the keys that start with a prefix. String order is lexicographic, so range("db.".."db/") gives all the db.* keys (/ is the character after .).
Interval scheduling: find all the intervals that overlap a specified window.

Accessing Extremes

BTreeMap gives direct access to the smallest entry and the largest entry:

Method	Returns	Mutates?
`first_key_value()`	`Option<(&K, &V)>`	No
`last_key_value()`	`Option<(&K, &V)>`	No
`pop_first()`	`Option<(K, V)>`	Yes: removes the entry
`pop_last()`	`Option<(K, V)>`	Yes: removes the entry

These methods are O(log n). pop_first and pop_last are useful for patterns that are almost a priority queue, where you consume the smallest or the largest entry:

// The key is the priority. A smaller number is a higher priority.
let mut priorities: BTreeMap<u32, &str> =
    BTreeMap::from([(1, "critical"), (2, "high"), (3, "medium"), (5, "low")]);

let highest = priorities.pop_first(); // Some((1, "critical")): the smallest key
let lowest  = priorities.pop_last();  // Some((5, "low")): the largest key
// priorities is now {2: "high", 3: "medium"}

The Ord Contract

BTreeMap keys must implement Ord, which is a total order. This requirement is stricter than the Hash + Eq requirement of HashMap. The Ord trait guarantees that you can compare any two values, and that the result is consistent, transitive, and antisymmetric.

Most standard types implement Ord: integers, strings, char, bool, Vec<T> (where T: Ord), and tuples. But floating-point types (f32, f64) do not implement Ord, because NaN != NaN breaks the requirement of a total order. If you need floating-point keys, put them in a newtype that handles NaN. For example, use ordered_float::OrderedFloat from the ecosystem, or write the comparison manually with std::cmp::Ordering.

For custom types, derive Ord together with PartialOrd, Eq, and PartialEq:

// The derived order compares `points` first. If the points are equal, it compares `name`.
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)]
struct Score {
    points: u32,
    name: String,
}

The derived Ord compares the fields from top to bottom (lexicographic order on the struct fields). If you need a different order, implement Ord manually.

The Entry API

BTreeMap::entry operates the same as HashMap::entry. It returns an Entry enum. With it, you examine or change the value of a key with only one lookup:

use std::collections::BTreeMap;

let mut inventory: BTreeMap<&str, u32> = BTreeMap::new();
let items = ["apple", "banana", "apple", "cherry", "banana", "apple"];

for item in items {
    // If the key exists, add 1 to the count. If the key is absent, insert the count 1.
    inventory.entry(item).and_modify(|n| *n += 1).or_insert(1);
}
// inventory: {"apple": 3, "banana": 2, "cherry": 1}

The primary methods of Entry are:

Method	Behavior
`or_insert(val)`	Inserts `val` if the entry is vacant
`or_insert_with(f)`	Inserts `f()` if the entry is vacant, and calls `f` only then
`or_default()`	Inserts `Default::default()` if the entry is vacant
`and_modify(f)`	Applies `f` to the existing value if the entry is occupied

Grouping with or_default

or_default is a clear method to make grouped collections:

let mut lists: BTreeMap<char, Vec<String>> = BTreeMap::new();
for name in ["Alice", "Ada", "Bob", "Clara", "Cleo"] {
    let initial = name.chars().next().unwrap();   // the first char is the group key
    // For a new initial, or_default inserts an empty Vec. Then push adds the name.
    lists.entry(initial).or_default().push(name.to_string());
}
// lists: {'A': ["Alice", "Ada"], 'B': ["Bob"], 'C': ["Clara", "Cleo"]}

The map is a BTreeMap, so the groups are in key order: 'A', then 'B', then 'C'.

split_off: Dividing a Map at a Threshold

split_off(&key) divides a BTreeMap into two maps. The entries with keys strictly less than key stay in the original map. The entries with keys greater than or equal to key move into the returned map.

let mut all_scores: BTreeMap<u32, &str> =
    BTreeMap::from([(10, "F"), (30, "D"), (50, "C"), (70, "B"), (90, "A")]);

let high = all_scores.split_off(&50);   // high: BTreeMap<u32, &str>
// all_scores: {10: "F", 30: "D"}          (keys < 50)
// high:       {50: "C", 70: "B", 90: "A"} (keys >= 50)

The division of the tree is O(log n). But the current implementation also counts the entries of one of the two maps, and that step is O(n) in the worst case. Use split_off when you must divide data at a boundary:

grades that pass and grades that fail
events before and after a cutoff time
items below and above a price threshold

append: Merging Two Maps

append(&mut other) moves all the entries from other into self, and other becomes empty. If the two maps contain the same key, append keeps the value from other:

let mut base: BTreeMap<&str, i32> = BTreeMap::from([("a", 1), ("c", 3)]);
let mut extra: BTreeMap<&str, i32> = BTreeMap::from([("b", 2), ("d", 4)]);

base.append(&mut extra);   // moves "b" and "d" into `base`
// base:  {"a": 1, "b": 2, "c": 3, "d": 4}
// extra: {} (append emptied it)

The merged result is still sorted. append reads the two maps in key order and merges them in one pass. For two maps of similar size, this is faster than a separate insert call for each entry.

Conflict resolution

When the two maps have the same key, append keeps the value of the donor map:

let mut m1 = BTreeMap::from([("x", 1), ("y", 2)]);
let mut m2 = BTreeMap::from([("y", 99), ("z", 3)]);   // "y" is in the two maps
m1.append(&mut m2);
// m1["y"] == 99: the value from m2 replaces the value from m1
// m1 is now {"x": 1, "y": 99, "z": 3}

If you need a different merge rule (for example, the sum of the two values), iterate the donor map and use the Entry API.

BTreeSet: Ordered Deduplication and Set Operations

BTreeSet<T> has the same relation to BTreeMap<T, ()> as HashSet<T> has to HashMap<T, ()>. It is a collection of unique values with no associated data. The difference is that BTreeSet keeps its elements sorted.

Creation and ordering

use std::collections::BTreeSet;

// The array is not sorted, and it contains 1 and 5 two times.
let digits: BTreeSet<u8> = BTreeSet::from([5, 3, 1, 4, 1, 5, 9, 2, 6]);
// digits is {1, 2, 3, 4, 5, 6, 9}: sorted, with no duplicates

Iteration always yields the elements in ascending order.

Range queries on sets

As BTreeMap does, BTreeSet has a range method:

// `digits` is the BTreeSet<u8> from the previous snippet: {1, 2, 3, 4, 5, 6, 9}.
// range yields &u8. The pattern `&d` copies the value.
for &d in digits.range(3..=6) {
    print!("{d} ");
}
// 3 4 5 6

Set operations

BTreeSet has the four standard set operations. Each one returns a sorted iterator:

Method	Meaning	Mathematical notation
`union(&other)`	Elements in either set	A ∪ B
`intersection(&other)`	Elements in both sets	A ∩ B
`difference(&other)`	Elements in `self` but not in `other`	A \ B
`symmetric_difference(&other)`	Elements in one set but not in both	A △ B

let evens: BTreeSet<i32> = (0..10).filter(|n| n % 2 == 0).collect();    // {0, 2, 4, 6, 8}
let threes: BTreeSet<i32> = (0..10).filter(|n| n % 3 == 0).collect();   // {0, 3, 6, 9}

// Each method yields &i32. `copied()` makes i32 values for the new set.
let union: BTreeSet<i32> = evens.union(&threes).copied().collect();
let intersection: BTreeSet<i32> = evens.intersection(&threes).copied().collect();
let difference: BTreeSet<i32> = evens.difference(&threes).copied().collect();
let sym_diff: BTreeSet<i32> = evens.symmetric_difference(&threes).copied().collect();
// union:        {0, 2, 3, 4, 6, 8, 9}
// intersection: {0, 6}
// difference:   {2, 4, 8}
// sym_diff:     {2, 3, 4, 8, 9}

The sets use a B-Tree, so the results are already sorted. With HashSet, the same operations give results in arbitrary order, and a sorted result needs a separate sort step.

Subset and superset checks

// `evens` is the set {0, 2, 4, 6, 8} from the previous snippet.
let small: BTreeSet<i32> = BTreeSet::from([2, 4]);
small.is_subset(&evens);    // true: 2 and 4 are in `evens`
evens.is_superset(&small);  // true: the same test from the other side

first, last, pop_first, pop_last

As BTreeMap does, BTreeSet gives direct O(log n) access to the smallest element and the largest element. pop_first and pop_last remove and return them.

Ordered Deduplication Pattern

A common pattern has two steps. Collect a Vec into a BTreeSet to remove the duplicates. Then collect the set into a Vec again to get sorted, unique elements:

let data = vec![5, 3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5];
let deduped_sorted: Vec<i32> = data
    .into_iter()
    .collect::<BTreeSet<_>>()   // removes the duplicates and sorts the values
    .into_iter()                // yields the values in ascending order
    .collect();
// deduped_sorted is [1, 2, 3, 4, 5, 6, 9]

This pattern is simpler than a manual sort and dedup, and the intent is clearer. The cost is O(n log n), the same asymptotic complexity as sort and dedup. But the constant factors are higher, because of the allocation overhead of the tree. For small to medium collections, prefer the clarity. For very large collections, sort and dedup on a Vec may be faster in practice.

Performance: B-Tree vs. Hash

Operation	`BTreeMap`	`HashMap`
Lookup	O(log n)	Amortized O(1)
Insert	O(log n)	Amortized O(1)
Remove	O(log n)	Amortized O(1)
Range query	O(log n + k)	Not supported
Sorted iteration	O(n): already sorted	O(n log n): you must sort
Min / max	O(log n)	O(n)
Memory layout	Tree nodes, less cache-friendly	Flat array, cache-friendly

Use BTreeMap when:

You need range queries.
You need sorted iteration.
You need access to the minimum key or the maximum key.
You want a deterministic iteration order for reproducible output.
Your keys implement Ord but not Hash (rare, but possible with custom types).

Use HashMap when:

You only do point lookups by exact key.
You need the fastest possible lookups on large collections.
The order is not important.

Figure: Choosing Between HashMap and BTreeMap

04_3_btreemap_btreeset_2

In practice, HashMap is the default. Use BTreeMap when order is a requirement, and not only a convenience.

Summary

Concept	Key point
`BTreeMap<K, V>`	Sorted key-value map, O(log n) operations
`BTreeSet<T>`	Sorted set of unique values, O(log n) operations
`range(bounds)`	Iterates a key range: the primary advantage over hash collections
`split_off(&key)`	Divides a map at a key: `< key` stays, `>= key` moves to the new map
`append(&mut other)`	Merges a different map into `self`. On a key conflict, it keeps the donor value
Entry API	Same as `HashMap`: `or_insert`, `or_default`, `and_modify`
`Ord` contract	Keys must implement a total order. `f32` and `f64` do not
`pop_first` / `pop_last`	Remove and return the smallest or the largest entry in O(log n)
Ordered dedup	Collect into a `BTreeSet`, then collect into a `Vec` again
Performance	O(log n) vs. amortized O(1). Select a B-Tree when the order is important

Code Examples

File	Description
`04_09_btreemap_basics.rs`	Creation, ordered iteration, range queries, first/last, pop
`04_10_btreemap_entry_split.rs`	Entry API, `split_off`, `append`, merge conflict behavior
`04_11_btreeset.rs`	`BTreeSet`, ordered dedup, range, set operations, subset checks

Expected Output

`04_09_btreemap_basics`

leaderboard (sorted by name):
  alice: 95
  bob: 72
  charlie: 88
  diana: 91

config (sorted keys): {"debug": "true", "host": "localhost", "port": "8080"}

temperatures 06:00–12:00:
  06:00 → 30.0°C
  07:00 → 29.7°C
  08:00 → 29.0°C
  09:00 → 27.8°C
  10:00 → 26.3°C
  11:00 → 24.4°C
  12:00 → 22.5°C

afternoon max temp: 22.5°C

first key: alice, last key: diana

removed bob: Some(72)

pop_first (highest priority): Some((1, "critical"))
pop_last (lowest priority): Some((5, "low"))

All assertions passed.

`04_10_btreemap_entry_split`

inventory: {"apple": 3, "banana": 2, "cherry": 1}

grouped: {'A': ["Alice", "Ada"], 'B': ["Bob"], 'C': ["Clara", "Cleo"]}

before split_off(50): {10: "F", 30: "D", 50: "C", 70: "B", 90: "A"}
low  (<50):  {10: "F", 30: "D"}
high (>=50): {50: "C", 70: "B", 90: "A"}

before append:
  base:  {"a": 1, "c": 3}
  extra: {"b": 2, "d": 4}
after append:
  base:  {"a": 1, "b": 2, "c": 3, "d": 4}
  extra: {}

after conflicting append: m1 = {"x": 1, "y": 99, "z": 3}

All assertions passed.

`04_11_btreeset`

tags (sorted): {"collections", "rust", "tutorial"}
digits (sorted, deduped): {1, 2, 3, 4, 5, 6, 9}

digits in 3..=6:
3 4 5 6

after removing 'tutorial': {"collections", "rust"}
first digit: Some(1), last: Some(9)
after popping first and last: {2, 3, 4, 5, 6}

evens:  {0, 2, 4, 6, 8}
threes: {0, 3, 6, 9}
union:        {0, 2, 3, 4, 6, 8, 9}
intersection: {0, 6}
difference:   {2, 4, 8}
sym_diff:     {2, 3, 4, 8, 9}

{2,4} ⊂ evens: true

ordered dedup: [1, 2, 3, 4, 5, 6, 9]

All assertions passed.

4.4 · VecDeque, LinkedList, and BinaryHeap

Domain 4 — Collections Deep Dive Duration: ~15 minutes Library components: std::collections::VecDeque, std::collections::LinkedList, std::collections::BinaryHeap

Introduction

Vec is the most common sequence collection in Rust. The standard library has three more sequence collections, and each one has a specific use:

VecDeque is a growable ring buffer in one contiguous allocation. Push and pop are amortized O(1) at both ends. In a Vec, an operation at the front is O(n).
LinkedList is a doubly-linked list. Push and pop are O(1) at both ends, and the join of two lists is O(1). It is almost never the correct selection, because each node is a separate heap allocation. Separate allocations remove CPU cache locality.
BinaryHeap is a max-heap that you can use as a priority queue. push and pop are O(log n), and peek is O(1). The heap invariant keeps the largest element at the top.

This tutorial shows:

How to create and change a VecDeque: push_front, push_back, pop_front, pop_back, and the push_front_mut/push_back_mut/insert_mut variants (stabilized in 1.95).
How to keep only the newest elements with retain_back (stabilized in 1.99).
How to examine the internal layout of the ring buffer with as_slices, and how to make it contiguous with make_contiguous.
How to use a VecDeque for a sliding window.
Why LinkedList is rarely the correct selection, and the few cases where it is.
BinaryHeap as a max-heap priority queue: peek, push, pop, into_sorted_vec.
How to make a min-heap with the Reverse wrapper.
The relaxed T: Ord bounds of some BinaryHeap methods in Rust 1.94.

VecDeque: A Growable Ring Buffer

VecDeque<T> keeps its elements in one heap allocation. It uses a head pointer and a tail pointer, so the two ends of the buffer are accessible in amortized O(1) time. Internally, the buffer is circular. When the tail goes past the end of the allocation, it continues at the start.

Figure: VecDeque Ring Buffer Layout

VecDeque ring buffer

Creation

use std::collections::VecDeque;

// An empty deque of string slices.
let mut deque: VecDeque<&str> = VecDeque::new();
// An empty deque with space for at least 64 elements.
let prealloc: VecDeque<i32> = VecDeque::with_capacity(64);

with_capacity allocates space for at least the specified number of elements. Use it when you know the expected size, to prevent reallocations.

Push and Pop at Both Ends

The four basic operations are push_front, push_back, pop_front, and pop_back. Each one is amortized O(1):

// `deque` is the empty VecDeque<&str> from the previous snippet.
deque.push_back("middle");
deque.push_front("first");   // goes before "middle"
deque.push_back("last");     // goes after "middle"
// deque is now ["first", "middle", "last"]

deque.pop_front(); // returns Some("first")
deque.pop_back();  // returns Some("last")
// deque is now ["middle"]

Thus VecDeque is the usual selection for a FIFO queue (push at the back, pop from the front) and for access at both ends. A Vec does these operations efficiently only at the back. insert(0, _) on a Vec moves each element.

To remove many elements in one call, use truncate or retain_back. truncate(n) keeps the first n elements and drops the remainder from the back. retain_back(n) (stabilized in 1.99) keeps the last n elements and drops the remainder from the front. The capacity does not change. If n is not smaller than the length, the call does nothing:

use std::collections::VecDeque;

let mut oldest: VecDeque<u32> = (1..=6).collect();   // [1, 2, 3, 4, 5, 6]
oldest.truncate(2);                                  // keep the FIRST 2
assert_eq!(oldest, [1, 2]);

let mut newest: VecDeque<u32> = (1..=6).collect();   // [1, 2, 3, 4, 5, 6]
newest.retain_back(2);                               // keep the LAST 2
assert_eq!(newest, [5, 6]);

A log buffer that must keep only the most recent entries is the usual application. 04_19_vecdeque_retain_back.rs prints:

truncate(2):    [1, 2]
retain_back(2): [5, 6]
retain_back(10) on 2 elements: [5, 6]
recent log entries: ["event 6", "event 7", "event 8"]
drain(..len - 2) gives the same result: [5, 6]

All assertions passed.

The `*_mut` Variants: a Reference to the New Element

Since Rust 1.95, push_front_mut, push_back_mut, and insert_mut insert an element and return &mut T to that element. Vec::push_mut does the same for Vec. You do not need front_mut().unwrap() to change the new element immediately:

let mut scores: VecDeque<i32> = VecDeque::from([5]);
let front = scores.push_front_mut(1);   // front: &mut i32, points to the new 1
*front += 100;                          // the front element is now 101
let back = scores.push_back_mut(9);     // back: &mut i32, points to the new 9
*back *= 2;                             // the back element is now 18
let mid = scores.insert_mut(1, 42);     // inserts 42 at index 1
*mid += 1;                              // the element at index 1 is now 43
assert_eq!(scores, [101, 43, 5, 18]);

Indexing and Safe Access

VecDeque implements Index<usize>, so buf[i] is valid. get(i) is the checked form and returns Option<&T>. front() and back() return a reference to the element at each end:

let buf: VecDeque<i32> = VecDeque::from([10, 20, 30, 40, 50]);
assert_eq!(buf[0], 10);             // an index that is out of bounds panics
assert_eq!(buf.get(99), None);      // get returns None, it does not panic
assert_eq!(buf.front(), Some(&10)); // the first element
assert_eq!(buf.back(), Some(&50));  // the last element

Sliding Window Pattern

A common use of VecDeque is a sliding window with a constant size on a data stream. Push each new element at the back. When the window is full, pop the oldest element from the front:

let data_stream = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let window_size = 4;
let mut window: VecDeque<i32> = VecDeque::with_capacity(window_size);

for &val in &data_stream {
    // The window is full: remove the oldest value before the push.
    if window.len() == window_size {
        window.pop_front();
    }
    window.push_back(val);
    // Print only complete windows (the first 3 iterations print nothing).
    if window.len() == window_size {
        let sum: i32 = window.iter().sum();
        println!("  window={window:?}, sum={sum}");
    }
}

Each iteration does a maximum of one pop and one push, and each is O(1). With a Vec, the removal of the front element is O(n).

as_slices: Viewing the Ring Buffer

VecDeque is a circular buffer, so its elements are not always contiguous in memory. After some pushes and pops, the head pointer can be at a later position. Then the logical sequence continues from the end of the allocation to its start. as_slices() returns two &[T] slices, front and back. Together they contain all the elements in order:

let mut ring: VecDeque<u8> = VecDeque::with_capacity(4);
ring.push_back(1);
ring.push_back(2);
ring.push_back(3);
ring.pop_front();       // removes 1: the head moves forward
ring.push_back(4);
ring.push_back(5);      // the buffer can wrap to the start of the allocation

let (front, back) = ring.as_slices();
// front and back together contain [2, 3, 4, 5] in order

Use as_slices when an API needs slices, or when you do not want to copy the elements into a Vec.

make_contiguous: Linearizing the Buffer

If you need one contiguous &mut [T] slice (for example, to sort the elements in place), call make_contiguous(). It moves the elements in the internal buffer until they are in one sequence:

let mut buf: VecDeque<i32> = VecDeque::from([10, 20, 30, 40, 50]);
buf.push_front(0);    // [0, 10, 20, 30, 40, 50]
buf.push_front(-1);   // [-1, 0, 10, 20, 30, 40, 50]
let contiguous = buf.make_contiguous();
// contiguous is &mut [-1, 0, 10, 20, 30, 40, 50]

After make_contiguous, the second slice from as_slices() is empty.

Converting Between Vec and VecDeque

VecDeque implements From<Vec<T>>, and Vec<T> implements From<VecDeque<T>>. Thus each conversion is one call:

let v = vec![1, 2, 3, 4, 5];
let d: VecDeque<i32> = VecDeque::from(v);      // moves the Vec, no copy
let back_to_vec: Vec<i32> = Vec::from(d);      // moves the VecDeque

The conversion from Vec to VecDeque is O(1), because it uses the same allocation. The opposite conversion can need to move the elements of the ring buffer until they are contiguous. Then it gives the allocation to the Vec.

04_12_vecdeque.rs prints:

empty: len=0, with_capacity(64): cap≥64=true

after pushes: ["first", "middle", "last"]
after pops: ["middle"]

push_*_mut/insert_mut: [101, 43, 5, 18]

front=Some(10), back=Some(50)

sliding window (size 4):
  window=[1, 2, 3, 4], sum=10
  window=[2, 3, 4, 5], sum=14
  window=[3, 4, 5, 6], sum=18
  window=[4, 5, 6, 7], sum=22
  window=[5, 6, 7, 8], sum=26
  window=[6, 7, 8, 9], sum=30
  window=[7, 8, 9, 10], sum=34

as_slices: front=[2, 3, 4], back=[5]

make_contiguous: [-1, 0, 10, 20, 30, 40, 50]

round-trip Vec→VecDeque→Vec: [1, 2, 3, 4, 5]
uppercased: ['A', 'B', 'C']

All assertions passed.

LinkedList: Almost Never the Correct Selection

LinkedList<T> is a doubly-linked list. Each element is in its own heap allocation. A forward pointer and a backward pointer connect it to the adjacent elements. The API has the usual operations: push_front, push_back, pop_front, pop_back, front, back, contains, clear, and iteration.

Basic Operations

use std::collections::LinkedList;

let mut list: LinkedList<i32> = LinkedList::new();
list.push_back(1);
list.push_back(2);
list.push_back(3);
list.push_front(0);
// list is [0, 1, 2, 3]

list.pop_front(); // returns Some(0)
list.pop_back();  // returns Some(3)
// list is [1, 2]

You can also collect an iterator into a list:

let from_iter: LinkedList<&str> = ["hello", "world"].into_iter().collect();

As for VecDeque, Rust 1.95 added push_front_mut and push_back_mut to LinkedList. They insert a node and return &mut T to its element, so you can change the new element in place:

let mut log: LinkedList<String> = LinkedList::new();
let entry = log.push_back_mut(String::from("mid"));    // entry: &mut String
entry.push_str("dle");                                 // "mid" becomes "middle"
let first = log.push_front_mut(String::from("sta"));   // first: &mut String
first.push_str("rt");                                  // "sta" becomes "start"
// log is ["start", "middle"]

The One Operation Where LinkedList Is Best: O(1) Append

The append method moves all the elements of one list to the end of a different list in O(1) time. It only changes the head and tail pointers. It does not copy or reallocate elements:

let mut a: LinkedList<i32> = LinkedList::from([1, 2, 3]);
let mut b: LinkedList<i32> = LinkedList::from([4, 5, 6]);

a.append(&mut b);   // takes all the nodes of b
// a is [1, 2, 3, 4, 5, 6], b is []

For Vec, the equivalent extend operation is O(m), where m is the length of the second collection, because it must copy each element. The same is true for VecDeque. If your program mostly joins and divides large sequences, LinkedList could be the correct selection.

Why LinkedList Is Slower for Almost All Other Work

Modern CPUs use a hierarchy of caches. When you iterate a Vec or a VecDeque, the processor loads the subsequent elements before you use them, because they are contiguous in memory. One cache line (typically 64 bytes) can contain several elements. In a LinkedList, each node is a separate heap allocation that could be at any address. The access to the subsequent node through its pointer is almost always a cache miss.

The practical effect is large. For a sum of 1000 elements in a loop, Vec can be 10 to 50 times faster than LinkedList because of cache effects only. Each node also has an allocation overhead: two pointers and the allocator metadata.

The general rule:

If you need...	Use this
A queue (FIFO)	`VecDeque`
A stack (LIFO)	`Vec`
Fast insert/remove in the middle	`Vec::splice` or `VecDeque`
O(1) append/split of two large lists	`LinkedList`

Iteration and Mutation

LinkedList has iter(), iter_mut(), and into_iter(), so the standard iterator methods are available. But access by index is O(n), and there is no Index implementation. You cannot write list[2].

let mut list: LinkedList<i32> = LinkedList::from([1, 2, 3, 4, 5, 6]);
let sum: i32 = list.iter().sum();   // 21

// `&mut list` iterates by mutable reference: val is &mut i32.
for val in &mut list {
    *val *= 10;
}
// list is [10, 20, 30, 40, 50, 60]

04_13_linkedlist.rs prints:

list: [0, 1, 2, 3]
from_iter: ["hello", "world"]

push_*_mut: ["start", "middle"]

after popping both ends: [1, 2]

before append: a=[1, 2, 3], b=[4, 5, 6]
after append:  a=[1, 2, 3, 4, 5, 6], b=[]

sum = 21
after *10: [10, 20, 30, 40, 50, 60]

after clear: len=0

--- Performance comparison (conceptual) ---
• Vec/VecDeque: contiguous memory → CPU cache-friendly
• LinkedList: each node is a separate heap allocation
• For n=1000, iterating Vec can be 10–50x faster due to cache lines
• LinkedList wins only for O(1) append/split of two lists
• If you need a queue: use VecDeque
• If you need a stack: use Vec
• If you need fast insert/remove in the middle: consider Vec::splice

Both produce the same sum: 49995000

All assertions passed.

BinaryHeap: A Max-Heap Priority Queue

BinaryHeap<T> is a binary max-heap in a Vec<T>. It keeps the heap property: the largest element (by T: Ord) is always at the root. Use it for priority queues, top-k queries, and each task that needs the maximum element again and again.

Figure: BinaryHeap as a Max-Heap

BinaryHeap pyramid

Core Operations

Operation	Complexity	Description
`push(item)`	O(log n)	Inserts an element and moves it up to keep the heap property
`pop()`	O(log n)	Removes and returns the maximum, then restores the heap property
`peek()`	O(1)	Borrows the maximum and does not remove it
`into_sorted_vec()`	O(n log n)	Consumes the heap and returns the elements in ascending order

use std::collections::BinaryHeap;

let mut heap: BinaryHeap<i32> = BinaryHeap::new();
heap.push(5);
heap.push(1);
heap.push(10);
heap.push(3);

assert_eq!(heap.peek(), Some(&10));       // O(1): the maximum, not removed

assert_eq!(heap.pop(), Some(10));          // removes the maximum
assert_eq!(heap.pop(), Some(5));           // then the next largest
assert_eq!(heap.pop(), Some(3));
assert_eq!(heap.pop(), Some(1));
assert_eq!(heap.pop(), None);              // the heap is empty

The Debug output of a BinaryHeap shows the elements in the internal storage order, not in sorted order. Only peek and pop are sure to give you the maximum.

Building a Heap from a Collection

BinaryHeap implements From<Vec<T>>, and you can collect an iterator into it. The construction of a heap from n elements is O(n). n separate push calls are O(n log n):

// One O(n) heap construction from the Vec.
let heap: BinaryHeap<i32> = BinaryHeap::from(vec![8, 3, 6, 1, 9, 4]);
assert_eq!(heap.peek(), Some(&9));   // 9 is the largest value

into_sorted_vec: Heapsort in One Call

into_sorted_vec() consumes the heap and returns a Vec<T> in ascending order. This operation is, in effect, a heapsort:

let heap: BinaryHeap<i32> = BinaryHeap::from(vec![8, 3, 6, 1, 9, 4]);
let sorted = heap.into_sorted_vec();   // `heap` is moved and cannot be used again
assert_eq!(sorted, [1, 3, 4, 6, 8, 9]);

Min-Heap via Reverse

BinaryHeap is always a max-heap. For a min-heap, put each element in a std::cmp::Reverse wrapper:

use std::cmp::Reverse;

let mut min_heap: BinaryHeap<Reverse<i32>> = BinaryHeap::new();
min_heap.push(Reverse(5));
min_heap.push(Reverse(1));
min_heap.push(Reverse(10));
min_heap.push(Reverse(3));

// Reverse(1) compares as the largest element, so pop returns it first.
assert_eq!(min_heap.pop(), Some(Reverse(1)));  // the minimum
assert_eq!(min_heap.pop(), Some(Reverse(3)));  // the next smallest

Reverse<T> reverses the Ord implementation, so the "maximum" in the heap is the smallest value. The wrapper has no runtime cost. It compiles to the same code as a comparison that you reverse manually.

Custom Priority: Implementing Ord

For your own types, implement Ord to define "highest priority". The heap compares elements with cmp, so only the fields that your Ord implementation reads have an effect on the order:

#[derive(Debug, Eq, PartialEq)]
struct Task {
    priority: u32,
    name: String,
}

// Compare tasks by `priority` only. `name` has no effect on the order.
impl Ord for Task {
    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
        self.priority.cmp(&other.priority)
    }
}
// PartialOrd must agree with Ord, so it calls `cmp`.
impl PartialOrd for Task {
    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
        Some(self.cmp(other))
    }
}

let mut scheduler: BinaryHeap<Task> = BinaryHeap::new();
scheduler.push(Task { priority: 1,  name: "cleanup logs".into() });
scheduler.push(Task { priority: 10, name: "fix security bug".into() });
scheduler.push(Task { priority: 5,  name: "update docs".into() });
scheduler.push(Task { priority: 8,  name: "deploy hotfix".into() });

// `scheduler.pop()` returns the tasks in priority order: 10, 8, 5, 1

Capacity Management and Drain

As Vec does, BinaryHeap has with_capacity, reserve, and shrink_to_fit. The drain() method removes all the elements and returns an iterator. The iteration order is not sorted. The iterator gives the elements in the internal storage order:

let mut h = BinaryHeap::from([3, 1, 4, 1, 5]);
let drained: Vec<i32> = h.drain().collect();
// drained has all 5 elements, but in arbitrary order
assert!(h.is_empty());   // drain removed each element

If you need the elements in sorted order, use a while let Some(val) = heap.pop() loop.

Relaxed T: Ord Bounds in Rust 1.94

Before Rust 1.94, many BinaryHeap methods required T: Ord, although they do not compare elements. Examples are len(), is_empty(), capacity(), clear(), drain(), and into_vec(). Since Rust 1.94, these methods have relaxed bounds. They accept each T, with or without an Ord implementation. Thus you can use a BinaryHeap in more generic code, without trait bounds that are not necessary.

04_14_binaryheap.rs prints:

heap (Debug shows internal order, not sorted): [10, 3, 5, 1]
peek (max) = Some(10)

popped in descending order: 10, 5, 3, 1

from vec: peek = Some(9)
into_sorted_vec: [1, 3, 4, 6, 8, 9]

min-heap pops: 1, 3, ...

task processing order:
  [priority=10] fix security bug
  [priority=8] deploy hotfix
  [priority=5] update docs
  [priority=1] cleanup logs

after shrink_to_fit: len=10, cap=10

drained (arbitrary order): [5, 3, 4, 1, 1]

collected heap, max = Some(99)

All assertions passed.

Summary

Concept	Key point
`VecDeque`	Ring buffer. Push and pop are amortized O(1) at both ends
`as_slices`	Returns two `&[T]` slices that contain the buffer, which can wrap
`make_contiguous`	Moves the elements into one contiguous `&mut [T]`
`Vec` to `VecDeque`	`From<Vec<T>>` is O(1) and uses the same allocation
`*_mut` insertions (1.95)	`push_front_mut`/`push_back_mut`/`insert_mut` return `&mut` to the new element
`retain_back(n)` (1.99)	Keeps the last `n` elements. It is the front-side counterpart of `truncate`
`LinkedList`	Doubly-linked list. Push and pop are O(1) at both ends, and `append` is O(1)
Why not `LinkedList`	One heap allocation for each node removes cache locality. `Vec` and `VecDeque` are almost always faster
`BinaryHeap`	Max-heap priority queue. `push` and `pop` are O(log n), and `peek` is O(1)
`into_sorted_vec`	Consumes the heap and returns an ascending `Vec<T>` (heapsort)
Min-heap	Put the elements in `Reverse<T>`
Custom priority	Implement `Ord` on your type to control the order
Relaxed bounds (1.94)	`len`, `drain`, `clear`, and similar methods do not require `T: Ord`

Code Examples

File	Description
`04_12_vecdeque.rs`	Ring buffer creation, `push_front`/`push_back`, `push_*_mut`/`insert_mut` (1.95), sliding window, `as_slices`, `make_contiguous`, `Vec` conversion
`04_13_linkedlist.rs`	Basic linked list operations, `push_*_mut` (1.95), O(1) `append`, why `LinkedList` is rarely useful, performance comparison
`04_14_binaryheap.rs`	Max-heap `peek`/`push`/`pop`, `into_sorted_vec`, min-heap via `Reverse`, custom `Ord` priority queue, `drain`
`04_19_vecdeque_retain_back.rs`	`VecDeque::retain_back` (1.99): keep the last `n` elements, comparison with `truncate` and `drain`

4.5 · Slices and Arrays: The Foundation Types

Domain 4 — Collections Deep Dive Duration: ~15 minutes Library components: primitive [T] (slice), primitive [T; N] (array), std::slice, std::array

Introduction

Each contiguous collection in Rust has two primitive types as its base: slices ([T]) and arrays ([T; N]). A slice is a dynamically-sized view into a contiguous sequence of T values. An array is a fixed-size, stack-allocated sequence, and its length is part of its type.

You never own a bare [T]. You use it through a reference (&[T], &mut [T]) or a boxed pointer (Box<[T]>). Arrays are different, because they are values. [i32; 4] is a concrete type, and its value is in the location where you put it. An array is Copy if T is Copy.

Vec<T> implements Deref<Target = [T]>, so each slice method is automatically available on a vector. Thus the slice methods are not a specialized subject. They are the base of all work with sequential data in Rust. This is true for each backing store:

a Vec
an array
a Box<[T]>
a memory-mapped buffer

This tutorial shows:

How to sort a slice with sort, sort_by, and sort_unstable.
How to search a sorted slice with binary_search and partition_point.
Chunks, windows, and splits: chunks, chunks_exact, rchunks, windows, split, splitn, split_first, split_last, contains, starts_with.
array_windows and element_offset (stabilized in 1.94).
In-place manipulation: rotate_left, rotate_right, fill, fill_with, swap, swap_with_slice, reverse, copy_from_slice, clone_from_slice, repeat, concat, join.
Array utilities: array::from_fn, each_ref, each_mut, map, and const generic arrays.
array::try_from_fn (still unstable in 1.99) and the alternative that you use on stable Rust.

Slice indexing uses range syntax (a..b, a..=b, ..) frequently. Tutorial 14.2 describes the RangeBounds trait and the Bound type that this syntax uses. For advanced slice algorithms (partition-point, unstable select, group-by), see Tutorial 25.2.

Sorting and Searching

sort and sort_by

sort() puts the elements in ascending order. It requires T: Ord. It is a stable sort: equal elements keep their initial relative order. sort_by takes a comparator closure, so you fully control the order.

let mut scores = vec![85, 92, 78, 92, 88, 78, 95];
scores.sort();   // stable sort, ascending order
// scores is now [78, 78, 85, 88, 92, 92, 95]

A comparator is necessary for an order that is not the default order. These sorts all use sort_by:

a case-insensitive sort of strings
a sort in descending order
a sort of structs on more than one field

let mut words = vec!["Charlie", "alice", "Bob", "diana"];
// The comparator compares the lowercase forms, so the case has no effect on the order.
words.sort_by(|a, b| a.to_lowercase().cmp(&b.to_lowercase()));
// words is now ["alice", "Bob", "Charlie", "diana"]
// A plain sort() gives ["Bob", "Charlie", "alice", "diana"]: uppercase letters sort first.

let mut nums = vec![3, 1, 4, 1, 5, 9];
nums.sort_by(|a, b| b.cmp(a));   // b before a: descending order
// nums is now [9, 5, 4, 3, 1, 1]

For structs, sort_by lets you sort on one field. sort_by_key is a shorter form of the same sort: its closure returns the sort key. The two methods are stable. When two students have the same grade, they stay in their initial order:

// students: Vec<Student>. A Student has a `name: &'static str` and a `grade: u32`.
// Initial order: Alice 88, Bob 95, Carol 72, Dave 88.
students.sort_by(|a, b| a.grade.cmp(&b.grade));
// The same sort with a key function. `04_15_slice_sorting.rs` uses this form.
students.sort_by_key(|s| s.grade);
// Sorted order: Carol 72, Alice 88, Dave 88, Bob 95. Alice stays before Dave.

sort_unstable

sort_unstable() is typically faster than sort(). It does not allocate temporary storage, and it does not guarantee that equal elements keep their initial order. Use it when stability is not important. Also use it when the elements are simple scalars, for which the initial order of equal elements has no meaning.

For floating-point data, you cannot call sort or sort_unstable directly. f64 implements PartialOrd but not Ord, because of NaN. Use sort_by or sort_unstable_by with partial_cmp and unwrap:

let mut floats = vec![2.72, 1.41, 0.58, 1.73];
// partial_cmp returns None if an operand is NaN, and then unwrap panics. This data has no NaN.
floats.sort_unstable_by(|a, b| a.partial_cmp(b).unwrap());
// floats is now [0.58, 1.41, 1.73, 2.72]

After the sort, is_sorted() returns true. This method only examines the order. It does not sort the slice.

binary_search and partition_point

When a slice is in sorted order, binary_search finds an element in O(log n). If it finds the value, it returns Ok(index). If it does not find the value, it returns Err(index). That index is the position where you can insert the value and keep the sorted order:

let data = [2, 4, 6, 8, 10, 12, 14, 16];      // must be in sorted order
assert_eq!(data.binary_search(&10), Ok(4));   // 10 is at index 4
assert_eq!(data.binary_search(&7),  Err(3));  // no 7: its insertion point is index 3

partition_point is a related method. It takes a predicate and returns the index of the first element for which the predicate returns false. Use it to divide a sorted slice into two parts at a boundary value:

// `data` is the sorted array from the previous snippet.
let cut = data.partition_point(|&x| x < 10);   // 4
// data[..cut] is [2, 4, 6, 8]: each element is < 10
// data[cut..] is [10, 12, 14, 16]: each element is >= 10

04_15_slice_sorting.rs prints:

before sort: [85, 92, 78, 92, 88, 78, 95]
after sort:  [78, 78, 85, 88, 92, 92, 95]

case-insensitive sort: ["alice", "Bob", "Charlie", "diana"]
descending: [9, 5, 4, 3, 1, 1]

sorted by grade:
  Carol — 72
  Alice — 88
  Dave — 88
  Bob — 95

sort_unstable: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
sorted floats: [0.58, 1.41, 1.73, 2.72]

[0..9] is_sorted = true

binary_search(&10) = Ok(4)
binary_search(&7)  = Err(3)
  7 would be inserted at index 3

partition_point(< 10) = 4
  below: [2, 4, 6, 8]
  at/above: [10, 12, 14, 16]

All assertions passed.

Chunks, Windows, and Splitting

Figure: chunks(3) vs windows(3) vs split

04_5_slices_arrays_1

chunks and chunks_exact

chunks(n) divides a slice into non-overlapping sub-slices of length n. The last chunk may be shorter if the slice length is not a multiple of n. chunks_exact(n) is stricter. It yields only full-size chunks, and its remainder() method returns the elements that remain:

let data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];

for chunk in data.chunks(3) {   // chunk: &[i32]
    println!("  {chunk:?}");
}
// [1, 2, 3], [4, 5, 6], [7, 8, 9], [10]: the last chunk has only 1 element

let exact = data.chunks_exact(3);   // yields [1, 2, 3], [4, 5, 6], [7, 8, 9]
let remainder = exact.remainder();  // [10]: the element that does not fill a chunk

rchunks(n) starts at the right end of the slice. Thus the chunk at the left end may be short, and not the chunk at the right end. The iterator yields that chunk last.

windows

windows(n) yields overlapping sub-slices of length n. Each step moves forward by one element. Use it for sliding-window algorithms such as moving averages, pairwise comparisons, and pattern detection:

let prices = [100.0, 102.0, 98.0, 105.0, 103.0, 99.0, 107.0];
let averages: Vec<f64> = prices
    .windows(3)                             // w: &[f64], always 3 prices
    .map(|w| w.iter().sum::<f64>() / 3.0)   // the average of one window
    .collect();
// 7 prices give 5 windows. To 1 decimal place: [100.0, 101.7, 102.0, 102.3, 103.0]

windows is different from chunks: it never yields a partial slice. If the slice has fewer than n elements, the iterator is empty.

array_windows::<N>() (stabilized in 1.94) is the same operation with a window length that is part of the type. Each window is a &[T; N] array, not a &[T] slice. Thus a closure can destructure the window, and the compiler checks the number of elements:

let readings = [12, 15, 19, 18, 14, 16];

// windows(2): w is &[i32], so the closure uses indexes.
let with_windows: Vec<i32> = readings.windows(2).map(|w| w[1] - w[0]).collect();

// array_windows: each window is &[i32; 2]. The pattern [a, b] sets N = 2.
let deltas: Vec<i32> = readings.array_windows().map(|[a, b]| b - a).collect();
assert_eq!(deltas, [3, 4, -1, -4, 2]);
assert_eq!(deltas, with_windows);

element_offset (stabilized in 1.94) answers a related question: which index does this element reference have? It compares addresses, not values. It returns None for a reference that does not point to an element of the slice:

let duplicates = [7, 3, 7];
let last: &i32 = &duplicates[2];   // a reference to the third element

// position compares VALUES: the first element that is equal to 7 is at index 0.
assert_eq!(duplicates.iter().position(|value| value == last), Some(0));
// element_offset compares ADDRESSES: `last` points to index 2.
assert_eq!(duplicates.element_offset(last), Some(2));

let outside = 7;                   // an equal value that is not in the slice
assert_eq!(duplicates.element_offset(&outside), None);

04_21_array_windows_element_offset.rs prints:

deltas = [3, 4, -1, -4, 2]
peaks = 1
windows of 3 in a slice of 2: 0
hottest reading 19 is at index 2
an equal value outside the slice: None
position = Some(0), element_offset = Some(2)

All assertions passed.

Tutorial 25.2 uses array_windows again for pairwise processing.

split and splitn

split(predicate) divides a slice at each element that matches the predicate. str::split divides a string at a pattern in a similar way. The sub-slices do not include the elements that match. Two adjacent elements that match give an empty slice between them:

let data = [1, 0, 2, 3, 0, 0, 4, 5];
// Each 0 is a delimiter. The result does not contain the delimiters.
let segments: Vec<&[i32]> = data.split(|&x| x == 0).collect();
// segments is [[1], [2, 3], [], [4, 5]]: the two adjacent zeros give the empty slice

splitn(n, predicate) gives a maximum of n sub-slices. The last sub-slice contains the remainder of the slice, which the method does not split.

split_first, split_last, contains, starts_with

These convenience methods are small, but you use them frequently:

split_first() returns Some((first_element, rest_of_slice)), or None if the slice is empty.
split_last() returns Some((last_element, init_of_slice)), or None if the slice is empty.
contains(&value) does a linear scan for an element that is equal to value.
starts_with(&[T]) and ends_with(&[T]) tell you if a slice starts or ends with the given sub-slice.

04_16_slice_chunks_windows.rs prints:

chunks(3):
  [1, 2, 3]
  [4, 5, 6]
  [7, 8, 9]
  [10]

chunks_exact(3): [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
remainder: [10]

rchunks(3): [[8, 9, 10], [5, 6, 7], [2, 3, 4], [1]]

windows(3):
  [10, 20, 30]
  [20, 30, 40]
  [30, 40, 50]

3-period moving average: [100.0, 101.7, 102.0, 102.3, 103.0]

split on zeros:
  [1]
  [2, 3]
  []
  [4, 5]

splitn(3, 0): [[1], [2], [3, 0, 4]]

split_first: first=10, rest=[20, 30, 40]
split_last: last=40, init=[10, 20, 30]

data.contains(&5) = true
starts_with [1,2,3] = true

All assertions passed.

In-Place Slice Manipulation

rotate_left and rotate_right

rotate_left(n) moves each element n positions to the left. The first n elements go to the end of the slice. rotate_right(n) does the opposite operation. The two methods run in O(len) time with O(1) additional space:

let mut buf = [1, 2, 3, 4, 5];
buf.rotate_left(2);   // 1 and 2 go to the end
// buf is now [3, 4, 5, 1, 2]

Rotations are useful for ring buffers, cyclic permutations, and algorithms that reorder elements in place.

fill and fill_with

fill(value) sets each element of the slice to a clone of value. fill_with(f) calls a closure for each element. Use fill_with when each position needs a different value, or when the type is not Clone:

let mut buf = [0u8; 8];   // 8 bytes, each one is 0
buf.fill(0xFF);           // 0xFF is 255
// buf is now [255, 255, 255, 255, 255, 255, 255, 255]

let mut counter = 0;
let mut data = [0; 5];
// fill_with calls the closure one time for each element, from first to last.
data.fill_with(|| { counter += 10; counter });
// data is now [10, 20, 30, 40, 50]

swap and swap_with_slice

swap(i, j) exchanges two elements of the same slice. It panics if one of the indexes is out of bounds. swap_with_slice(other) exchanges the contents of two mutable slices of equal length. It does for slices what std::mem::swap does for single values:

let mut letters = ['a', 'b', 'c', 'd', 'e'];
letters.swap(1, 3);   // exchanges 'b' (index 1) and 'd' (index 3)
// letters is now ['a', 'd', 'c', 'b', 'e']

let mut left = [1, 2, 3];
let mut right = [7, 8, 9];
left.swap_with_slice(&mut right);   // panics if the two lengths are different
// left=[7, 8, 9], right=[1, 2, 3]

reverse

reverse() reverses the order of the elements in place. To reverse only a part of the slice, first take a mutable sub-slice:

let mut data = [1, 2, 3, 4, 5];
data.reverse();
// data is now [5, 4, 3, 2, 1]

let mut partial = [10, 20, 30, 40, 50];
partial[1..4].reverse();   // reverses only the elements at indexes 1, 2, and 3
// partial is now [10, 40, 30, 20, 50]

copy_from_slice and clone_from_slice

copy_from_slice(src) copies the elements of src into the slice. The two slices must have the same length. This method is the safe wrapper around memcpy for Copy types. clone_from_slice(src) does the same operation, but it calls clone() on each element. Thus it also works for types that are not Copy.

repeat, concat, and join

repeat(n) makes a new Vec that contains the elements of the slice n times. concat() and join(sep) operate on a slice of slices. concat flattens the inner slices into one Vec. join does the same and puts a separator between the inner slices:

let pattern = [1, 2, 3];
let repeated: Vec<i32> = pattern.repeat(3);     // [1, 2, 3, 1, 2, 3, 1, 2, 3]

// A slice of slices. The inner slices can have different lengths.
let slices: &[&[i32]] = &[&[1, 2], &[3, 4], &[5]];
let concatenated: Vec<i32> = slices.concat();   // [1, 2, 3, 4, 5]
let joined: Vec<i32> = slices.join(&0);         // [1, 2, 0, 3, 4, 0, 5]

04_17_slice_manipulation.rs prints:

before rotate_left(2): [1, 2, 3, 4, 5]
after  rotate_left(2): [3, 4, 5, 1, 2]

rotate_right(2): [4, 5, 1, 2, 3]

fill(0xFF): [255, 255, 255, 255, 255, 255, 255, 255]
fill_with(counter): [10, 20, 30, 40, 50]

before swap(1, 3): ['a', 'b', 'c', 'd', 'e']
after  swap(1, 3): ['a', 'd', 'c', 'b', 'e']

before swap_with_slice: left=[1, 2, 3], right=[7, 8, 9]
after  swap_with_slice: left=[7, 8, 9], right=[1, 2, 3]

before reverse: [1, 2, 3, 4, 5]
after  reverse: [5, 4, 3, 2, 1]

reverse [1..4]: [10, 40, 30, 20, 50]

copy_from_slice into [1..4]: [0, 100, 200, 300, 0]
clone_from_slice: ["a", "b"]

[1,2,3].repeat(3) = [1, 2, 3, 1, 2, 3, 1, 2, 3]
concat: [1, 2, 3, 4, 5]
join(&0): [1, 2, 0, 3, 4, 0, 5]

All assertions passed.

Array Utilities and Const Generics

array::from_fn and array::try_from_fn

std::array::from_fn makes an array of length N. It calls a closure one time for each index. The compiler infers the array length from the return type or from a turbofish annotation. This function is the idiomatic way to initialize an array with computed values:

// The closure gets each index `i` (a usize), from 0 to N - 1.
let squares: [i32; 10] = std::array::from_fn(|i| (i * i) as i32);
// [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

let powers: [u32; 8] = std::array::from_fn(|i| 1 << i);
// [1, 2, 4, 8, 16, 32, 64, 128]

// b'a' is the byte 97. The index adds 0 to 25, which gives the bytes of 'a' to 'z'.
let alphabet: [char; 26] = std::array::from_fn(|i| (b'a' + i as u8) as char);
// ['a', 'b', 'c', ..., 'z']

// The turbofish sets the length (5). No type annotation is necessary.
let ids = std::array::from_fn::<_, 5, _>(|i| format!("#{}", i + 1));
// ids: [String; 5] = ["#1", "#2", "#3", "#4", "#5"]

array::try_from_fn is the fallible counterpart. Its closure returns a Result (or an Option). If one call of the closure fails, the full construction fails. This function is still unstable in Rust 1.99 (nightly feature array_try_from_fn), so stable Rust rejects it.

On stable Rust, collect the elements into a Result<Vec<T>, E>. Then convert the Vec into an array with try_into:

let inputs = ["10", "20", "30", "40"];
// Nightly only: std::array::try_from_fn(|i| inputs[i].parse::<i32>())

// Stable: collect stops at the first Err and returns it.
let parsed: Result<Vec<i32>, _> = inputs.iter().map(|s| s.parse::<i32>()).collect();
// Vec<i32> to [i32; 4]: try_into fails if the Vec does not have exactly 4 elements.
let parsed: [i32; 4] = parsed.unwrap().try_into().unwrap();
// parsed is [10, 20, 30, 40]

let bad: Result<Vec<i32>, _> = ["1", "oops", "3"].iter().map(|s| s.parse::<i32>()).collect();
// bad is Err(ParseIntError { kind: InvalidDigit }): "oops" is not a number

Fallible construction is especially useful when you parse fixed-size input in which each element might be invalid.

each_ref and each_mut

each_ref() converts a &[T; N] into a [&T; N]: an array of references, one for each element. each_mut() does the same for mutable references. These methods are useful when you must pass references to single elements to an API that expects &T:

let values = [10, 20, 30];
let refs: [&i32; 3] = values.each_ref();   // borrows each element, does not move `values`
assert_eq!(*refs[1], 20);

let mut mutable = [1, 2, 3];
let [a, b, c] = mutable.each_mut();        // a, b, c: &mut i32, one for each element
*a *= 10;
*b *= 10;
*c *= 10;
assert_eq!(mutable, [10, 20, 30]);         // the writes changed the array itself

map on arrays

Arrays have a map method. It transforms each element by value and returns a new array of the same length. It is different from the iterator map: it is eager, and the result is still an array:

let names = ["Alice", "Bob", "Carol"];
let lengths: [usize; 3] = names.map(|n| n.len());   // one length for each name
// [5, 3, 5]

let ints = [1, 2, 3, 4, 5];
// map can change the element type (i32 to String). The length stays 5.
let strings: [String; 5] = ints.map(|n| format!("#{n}"));
// ["#1", "#2", "#3", "#4", "#5"]

Converting between slices and arrays

[T; N] implements TryFrom<&[T]> when T is Copy. Thus you can convert a slice reference to an array when the length matches. The conversion copies the elements. If the slice has the wrong length, you get an Err(TryFromSliceError):

// The annotation makes `slice` a real slice. Without it, the type is &[i32; 3].
let slice: &[i32] = &[10, 20, 30];
let arr: [i32; 3] = slice.try_into().unwrap();   // Ok: the slice has exactly 3 elements

let wrong: Result<[i32; 5], _> = slice.try_into();   // 3 elements, but the target needs 5
// wrong is Err(TryFromSliceError(()))

Const generic arrays

With const generics, you can write a function that is generic over the array length N. This gives type-safe, zero-cost abstractions over fixed-size data:

// N is a const generic parameter. The two arrays must have the same length N.
fn dot_product<const N: usize>(a: &[f64; N], b: &[f64; N]) -> f64 {
    let mut sum = 0.0;
    for i in 0..N {
        sum += a[i] * b[i];
    }
    sum
}

dot_product(&[1.0, 2.0, 3.0], &[4.0, 5.0, 6.0]);   // N = 3: 1*4 + 2*5 + 3*6 = 32.0
dot_product(&[1.0, 0.0], &[0.0, 1.0]);             // N = 2: 1*0 + 0*1 = 0.0
// dot_product(&[1.0, 2.0], &[1.0]);               // error[E0308]: mismatched types

The compiler monomorphizes the function for each array length that the call sites use. Thus there is no runtime dispatch, and the compiler knows the loop bounds at compile time.

You can also zip two arrays into one array of pairs. Use std::array::from_fn together with indexing:

let keys = ["x", "y", "z"];
let vals = [10, 20, 30];
// The closure reads index `i` of each array. The annotation sets the length to 3.
let zipped: [(&str, i32); 3] = std::array::from_fn(|i| (keys[i], vals[i]));
// zipped is [("x", 10), ("y", 20), ("z", 30)]

04_18_array_utilities.rs prints:

squares: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
powers of 2: [1, 2, 4, 8, 16, 32, 64, 128]
alphabet: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

parsed: [10, 20, 30, 40]
bad parse: Err(ParseIntError { kind: InvalidDigit })

each_ref: [10, 20, 30]
each_mut after *10: [10, 20, 30]

name lengths: [5, 3, 5]
mapped to strings: ["#1", "#2", "#3", "#4", "#5"]

slice→array: [10, 20, 30]
wrong size: Err(TryFromSliceError(()))

dot_product([1,2,3], [4,5,6]) = 32
dot_product([1,0], [0,1]) = 0

zipped: [("x", 10), ("y", 20), ("z", 30)]

All assertions passed.

Summary

Concept	Key APIs
Stable sort	`sort()`, `sort_by(cmp)`: equal elements keep their order
Unstable sort	`sort_unstable()`: typically faster, no allocation, no stability guarantee
Binary search	`binary_search(&val)`: `Ok(index)` or `Err(insert_point)` on a sorted slice
Partition point	`partition_point(pred)`: the index where the predicate changes from true to false
Chunking	`chunks(n)`, `chunks_exact(n)`, `rchunks(n)`: non-overlapping sub-slices
Windowing	`windows(n)`: overlapping sub-slices of length `n`. `array_windows::<N>()` (1.94): overlapping `&[T; N]` arrays
Element index	`element_offset(&elem)` (1.94): the index of a reference that points into the slice, by address
Splitting	`split(pred)`, `splitn(n, pred)`, `split_first`, `split_last`
Membership	`contains(&val)`, `starts_with(&[T])`, `ends_with(&[T])`
Rotation	`rotate_left(n)`, `rotate_right(n)`: cyclic shift in O(len)
Fill	`fill(val)`, `fill_with(f)`: set all the elements
Swap	`swap(i, j)`, `swap_with_slice(other)`: exchange elements
Reverse	`reverse()`: in-place reversal
Copy	`copy_from_slice`, `clone_from_slice`: bulk copy between slices of the same length
Repeat/join	`repeat(n)`, `concat()`, `join(sep)`: make a new `Vec` from slices
Array construction	`array::from_fn(f)`: make an array from a closure. `array::try_from_fn(f)` is the fallible form (unstable in 1.99)
Array transforms	`each_ref`, `each_mut`, `map`: element-wise operations that keep the array type
Slice to array	`<[T; N]>::try_from(&[T])`: fallible conversion, the lengths must match
Const generics	`fn foo<const N: usize>(arr: [T; N])`: generic over the array length

Code Examples

File	Description
`04_15_slice_sorting.rs`	`sort`, `sort_by`, `sort_by_key`, `sort_unstable`, `is_sorted`, `binary_search`, `partition_point`
`04_16_slice_chunks_windows.rs`	`chunks`, `chunks_exact`, `rchunks`, `windows`, `split`, `splitn`, `contains`
`04_17_slice_manipulation.rs`	`rotate_left/right`, `fill`, `swap`, `reverse`, `copy_from_slice`, `repeat`, `concat`, `join`
`04_18_array_utilities.rs`	`array::from_fn`, the stable alternative to `try_from_fn` (unstable in 1.99), `each_ref`, `each_mut`, `map`, slice-to-array `try_into`, const generics
`04_21_array_windows_element_offset.rs`	`array_windows` and `element_offset` (1.94): windows as arrays, and the index of an element reference

5.1 · Iterator Fundamentals: The Trait and Its Adapters

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::Iterator, adapter types in std::iter

Introduction

The Iterator trait is the most used abstraction in the Rust standard library. It has one required method, next(). That one method gives you 75 provided methods (adapters and consumers), and 60 of them are stable in Rust 1.99. Each for loop uses the trait. The compiled code is as fast as a loop that you write manually. Collections, ranges, strings, I/O lines, and command-line arguments all give you iterators.

This tutorial shows:

The Iterator trait, next(), and the pull model.
Laziness: why the construction of an adapter chain does no work, and how consumers pull items one at a time.
The core transformation adapters: map, filter, filter_map, flat_map, flatten, enumerate, zip, chain.
The selection adapters: take, skip, take_while, skip_while, step_by, cycle, fuse.
Why iterators are a zero-cost abstraction: the convenience has no runtime cost.

The Trait and the Pull Model

The trait has only one required method:

trait Iterator {
    type Item;                                  // the type of each item
    fn next(&mut self) -> Option<Self::Item>;   // the one required method
    // 75 provided methods call next() (60 of them are stable in Rust 1.99)
}

next() returns Some(item) until the sequence has no more items. Then it returns None. All the other operations (map, sum, collect, and the for loop) call next() again and again. This is a pull model: no stage makes an item until a later stage asks for it.

An iterator is a cursor, not a snapshot. Each call to next() moves the cursor forward permanently. Two calls never return the same element. For this reason, most adapter methods take self by value. When you wrap an iterator, the wrapper owns the cursor. Tutorial 5.3 shows how by_ref lends the cursor instead.

let mut it = [10, 20, 30].iter();   // it: slice::Iter<'_, i32>, each item is an &i32
assert_eq!(it.next(), Some(&10));   // each call moves the cursor one item forward
assert_eq!(it.next(), Some(&20));
assert_eq!(it.next(), Some(&30));
assert_eq!(it.next(), None);        // the sequence has no more items

Laziness: Nothing Happens Until Consumption

Adapters are lazy. map does not loop through the items. It returns a Map<I, F> struct that stores the previous iterator and the closure. filter, take, zip, and the other adapters do the same. Work occurs only when a consumer starts to pull. A consumer is a method that calls next(), such as collect or sum. A for loop is also a consumer.

Figure: A lazy pipeline — adapters store, consumers pull

05_1_iterator_fundamentals_1

05_01_lazy_pipeline.rs shows this with a counter in the map closure. The example searches a log for the first ERROR line. It stops at the first match, as ripgrep does:

// log: [&str; 6] holds six log lines. Lines 4 and 6 start with "ERROR".
// pulls counts the calls of the `map` closure.
// A Cell lets the closure change the counter through a shared reference.
let pulls = Cell::new(0_usize);
let mut errors = log
    .iter()
    .map(|line| { pulls.set(pulls.get() + 1); line.trim_start() })
    .filter(|line| line.starts_with("ERROR"));

assert_eq!(pulls.get(), 0);       // the pipeline exists, but no closure ran

let first = errors.next();        // pulls lines 1..=4 and stops at the first ERROR line
assert_eq!(pulls.get(), 4);       // the pipeline did not read lines 5 and 6

Remember these two results of laziness:

An early exit has no cost. Short-circuiting consumers (find, any, all, position) stop the pull when they have an answer.
A pipeline without a consumer does nothing. If you make data.iter().map(expensive) and never consume it, no code runs. The compiler gives a warning: iterators are lazy and do nothing unless consumed.

05_01_lazy_pipeline.rs prints:

after building the pipeline: 0 lines processed
first error found after processing 4 lines
second error found after processing 6 lines
`any` stopped after inspecting 2 lines
pipeline result == manual loop result: ["ERROR connection reset by peer", "ERROR disk quota exceeded"]

All assertions passed.

Transformation Adapters

These adapters change what goes through the pipeline. 05_02_transform_adapters.rs uses all of them on one realistic task. The task is to parse a key = value config file that has comments, blank lines, and list values. The example puts the file values on top of the defaults, as cargo does with its configuration layers.

Adapter	Shape	Typical use
`map(f)`	1 → 1	Transforms each item
`filter(p)`	1 → 0 or 1	Keeps the items that match
`filter_map(f)`	1 → 0 or 1	Transforms and filters in one step, through an `Option`
`flat_map(f)`	1 → many	Maps each item to an iterator and joins the iterators into one stream
`flatten()`	unnest	Removes one level of nesting (an `Option` is also a level)
`enumerate()`	1 → (index, item)	Adds a 0-based position to each item
`zip(other)`	pairwise	Steps through two sequences together and stops at the end of the shorter one
`chain(other)`	concat	Gives the items of one iterator, then the items of the other

These two are the most instructive:

// meaningful: Vec<&str> holds the trimmed config lines, with no blank lines and no comments.
// filter_map: split_once returns None for a line that has no '=', and filter_map drops it.
let entries: Vec<(&str, &str)> = meaningful
    .iter()
    .filter_map(|line| line.split_once('='))
    .map(|(k, v)| (k.trim(), v.trim()))   // ("replicas ", " 3") becomes ("replicas", "3")
    .collect();

// defaults: [(&str, &str); 2] is [("replicas", "1"), ("log_level", "info")].
// chain: the defaults come first and the file entries come second.
// In a map, a later duplicate key replaces the earlier value.
let merged: BTreeMap<&str, &str> =
    defaults.into_iter().chain(entries.iter().copied()).collect();
assert_eq!(merged["replicas"], "3");   // the file value replaced the default "1"

Option is iterable: it gives zero items or one item. Thus flatten() on an iterator of Options discards each None. You will see this pattern very frequently in code that parses text.

zip stops at the end of the shorter side. It does not panic, and it does not add padding. If you must detect a length mismatch, do a check afterwards, or use zip together with enumerate.

Selection Adapters

These adapters select which items go through, and they do not change the items. 05_03_slicing_adapters.rs shows a concrete use for each one:

skip(n) with take(n): pagination, as in results.skip(page * size).take(size).
take_while(p) and skip_while(p): these divide a message into header and body at the first blank line. filter makes a decision for each item, but these adapters make a decision once. take_while stops the stream permanently at the first item that fails the predicate. skip_while stops the skip at the first item that passes the predicate.
step_by(n): this downsamples a metrics stream. It keeps the first item, then each n-th item after it.
cycle(): this repeats a cloneable iterator forever. When you zip it with a finite task list, it gives a round-robin assignment. This is safe, because zip stops at the end of the task list.
fuse(): after its first None, a fused iterator never gives an item again. The basic Iterator contract permits an iterator to give items again after a None. Tutorial 5.4 explains the FusedIterator guarantee in detail.

Figure: take_while vs filter — one-time decision vs per-item decision

05_1_iterator_fundamentals_2

Be careful with take_while. It must consume the first item that fails the predicate, because it must test that item. That item is lost: take_while does not put it back. When you need the boundary item, use Peekable::next_if (Tutorial 5.3).

// message: [&str; 5] holds two header lines, one empty line "", and two body lines.
let mut it = message.iter().copied();
// by_ref lends `it` to take_while, so `it` stays usable after the collect (Tutorial 5.3).
let headers: Vec<&str> = it.by_ref().take_while(|l| !l.is_empty()).collect();
assert_eq!(it.next(), Some("body line 1"));  // take_while consumed the empty line ""

Why Iterators Are Zero-Cost

An adapter chain looks as if it allocates intermediate collections and calls through function pointers. It does not do these things:

No allocation. Each adapter is a plain struct that wraps the previous one. log.iter().map(f).filter(p) is one stack value. Its type is Filter<Map<slice::Iter<'_, &str>, F>, P>, and this type describes the full pipeline.
No virtual dispatch. Each closure has a unique anonymous type. Thus the compiler dispatches each next() call statically and can inline it.
Monomorphization. The compiler generates a specialized copy of the chain for these exact types. Then the optimizer merges the nested next() calls into one loop. The result is routinely the same assembly as the manual version, and it is sometimes better. The compiler can remove bounds checks, because it can prove that the iteration stays in bounds. Loops that use indexing do not get this advantage.

// log is the array of six log lines from the laziness example.
let by_pipeline: Vec<&str> = log.iter()
    .map(|line| line.trim_start())
    .filter(|line| line.starts_with("ERROR"))
    .collect();
// by_pipeline holds the two ERROR lines.
// The `for` loop version gives the same results, and routinely the same machine code.

The practical rule: choose between a loop and an iterator chain for readability, never for assumed performance.

Summary

Concept	Key point
`Iterator` trait	One required method, `next() -> Option<Item>`. The trait provides all the other methods
Pull model	Consumers control the flow. Each `next()` pulls one item through the full chain
Laziness	Adapters only wrap and store. No work occurs until a consumer pulls
Short-circuiting	`find`, `any`, and an early `next()` stop the pull when they have an answer
`map` / `filter` / `filter_map`	Transform, select, or do the two at once through an `Option`
`flat_map` / `flatten`	Join nested iterators into one stream. `flatten` drops each `None` from a stream of `Option`s
`enumerate` / `zip` / `chain`	Add indices, pair two streams (the shorter one sets the length), concatenate
`take` / `skip` (+`_while`)	Select a prefix or a suffix. The `_while` variants make a decision once, not for each item
`step_by` / `cycle` / `fuse`	Downsample, repeat forever, give no item after the first `None`
Zero-cost	Monomorphized structs and inlining give the performance of a manual loop

Code Examples

File	Description
`05_01_lazy_pipeline.rs`	A pull counter proves laziness. `next` and `any` stop early. A pipeline gives the same result as a manual loop
`05_02_transform_adapters.rs`	`map`, `filter`, `filter_map`, `flat_map`, `flatten`, `enumerate`, `zip`, and `chain` in a config parser
`05_03_slicing_adapters.rs`	`take`, `skip`, `take_while`, `skip_while`, `step_by`, `cycle`, and `fuse`. `take_while` consumes the boundary item

5.2 · Consumers, Collectors, and the FromIterator/Extend Traits

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::FromIterator, std::iter::Extend, std::iter::Sum, std::iter::Product

Introduction

Tutorial 5.1 showed how to make pipelines. This tutorial shows how to end them. Consumers (also known as terminal operations) are the methods that call next() until they have an answer. The answer can be a number, a boolean, a position, or a new collection. The most important consumer is collect. Two traits work with collect, and it helps to know their names:

FromIterator makes a collection from an iterator.
Extend appends an iterator to an existing collection.

This tutorial shows:

Reductions: fold, reduce, sum, product, count, for_each, unzip.
Searches: any, all, find, position, nth, last.
Extremes: min, max, min_by, max_by, min_by_key, max_by_key, and the rules for ties.
collect, FromIterator, and the collect::<Result<Vec<_>, _>>() fail-fast pattern.
Extend to append items, and when it is better than collect.

Choosing a Consumer

Each consumer answers a question. To select the correct consumer, you mostly need to state the question precisely:

Figure: Consumer decision map

05_2_consumers_collectors_1

All of these are provided methods. The implementation of each one calls next() in a loop (or try_fold, see Tutorial 5.6). This fact explains their behavior. count pulls each remaining item. any stops at the first true.

fold(init, f) passes an accumulator through each item. It is the universal reduction: you could write all the other reductions with it. 05_04_terminal_consumers.rs uses it to calculate latency statistics in one pass:

// requests: [(&str, u64); 6] holds (route, latency in ms) pairs.
// The accumulator is a (min, max, sum) tuple. The first argument of fold is its start value.
let (min, max, sum) = requests.iter().fold(
    (u64::MAX, u64::MIN, 0_u64),
    |(min, max, sum), &(_, ms)| (min.min(ms), max.max(ms), sum + ms),
);
// (min, max, sum) is (2, 340, 791)

reduce(f) is fold without an initial value. The first item becomes the initial accumulator, so the result is an Option. An empty stream has no first item, and then the result is None. Prefer reduce when there is no natural identity value. For example, a route name has no "zero".

sum() and product() use traits: std::iter::Sum and std::iter::Product. The standard library implements these traits for each numeric type, and for Option and Result of numeric types. For this reason, sum usually needs a type annotation:

// requests is the array of (route, latency in ms) pairs from the previous snippet.
let total: u64 = requests.iter().map(|(_, ms)| ms).sum();    // 791: the annotation selects u64
let compounded: f64 = [1.10, 0.95, 1.20].iter().product();  // 1.254: the three rates multiplied

for_each(f) runs a side effect for each item. It is equivalent to a for loop, but it fits at the end of a long pipeline. The rustc and cargo code uses this style convention: for for blocks of logic, and for_each for a one-line last step.

unzip() is the inverse of zip. It makes one pass through (A, B) pairs and gives two collections.

05_04_terminal_consumers.rs prints:

6 requests, 791 ms total, mean 131 ms
fold in one pass: min=2 max=340 sum=791
slowest route: Some(("/search", 340))
compounded throughput factor: 1.254
/search: 340 ms
/search: 295 ms
unzipped into 6 routes + 6 latencies

All assertions passed.

Searching: any, all, find, position, nth, last

05_05_searching_extremes.rs examines a server fleet during an incident:

// FLEET: [Server; 5]. A Server has the fields name, latency_ms, error_rate, and healthy.
// Two servers are not healthy: web-2 (index 1) and cache-1 (index 4).
FLEET.iter().any(|s| !s.healthy)              // -> bool (true), stops at the first true
FLEET.iter().all(|s| s.latency_ms < 1000)     // -> bool (true), stops at the first false
FLEET.iter().find(|s| !s.healthy)             // -> Option<&Server>, the ITEM (web-2)
FLEET.iter().position(|s| !s.healthy)         // -> Option<usize>, the INDEX (Some(1))

any and all replace the usual loop with a boolean flag. find and position replace the loop that counts an index. All four methods short-circuit.

Two positional consumers complete the set:

nth(n) skips n items and returns the next one. It consumes all the items up to and including that one. A call to nth(2) and then next() gives items 2 and 3, not items 2 and 0.
last() pulls the full iterator to get to the end. On a double-ended iterator, next_back() gets the last item in one step (Tutorial 5.4). Clippy tells you about this.

Extremes and Tie-Breaking

For items that have a natural order, min() and max() are sufficient. For structs, extract a key or supply a comparator:

// FLEET is the [Server; 5] array from the previous snippet.
// max_by_key compares the key but returns the item: Option<&Server>, here web-2 (340 ms).
let worst = FLEET.iter().max_by_key(|s| s.latency_ms);
// error_rate is an f64, and f64 is not Ord. total_cmp gives the comparator a total order.
let flakiest = FLEET.iter()
    .max_by(|a, b| a.error_rate.total_cmp(&b.error_rate));
// flakiest is cache-1. It ties with web-2 at 0.043, and max_by keeps the last maximum.

f64 does not implement Ord, because NaN prevents a total order. Thus max_by_key with a float key does not compile. The standard solution is f64::total_cmp in max_by.

Memorize the rule for ties: min* returns the first minimum, and max* returns the last maximum. In the example fleet, two servers have the same error rate. max_by selects the later server. min_by with a reversed comparator selects the earlier server. Thus, when ties exist, a reversed comparator is not equivalent to an exchange of min and max.

05_05_searching_extremes.rs prints:

fleet degraded: true
first unhealthy: web-2 at index 1
latency range: 5 ms .. 340 ms
flakiest (max_by, last tie wins): cache-1

All assertions passed.

collect and FromIterator

collect is one method with many possible results. It calls FromIterator::from_iter on the target type. Your type annotation (or turbofish) selects the implementation:

// words: &str is "the quick brown fox the lazy dog the end" (9 words, "the" occurs 3 times).
let list:   Vec<&str>           = words.split(' ').collect();   // 9 items
let unique: HashSet<&str>       = words.split(' ').collect();   // 7 items: no duplicates
// The first letter of each word: "tqbftldte"
let text:   String              = words.split(' ').filter_map(|w| w.chars().next()).collect();
// Word -> position. A later pair replaces an earlier pair, so index["the"] is 7.
let index:  BTreeMap<&str, usize> = words.split(' ').enumerate().map(|(i, w)| (w, i)).collect();

You can implement FromIterator for your own types. 05_06_collect_fromiterator.rs implements FromIterator<u64> for a LatencyStats struct. Thus a pipeline collects directly into the statistics, with no intermediate Vec and no second pass:

// LatencyStats has the fields count: usize, total: u64, min: u64, and max: u64.
impl FromIterator<u64> for LatencyStats {
    // collect() calls this function and gives it the pipeline as `iter`.
    fn from_iter<I: IntoIterator<Item = u64>>(iter: I) -> Self {
        let mut stats = LatencyStats { count: 0, total: 0, min: u64::MAX, max: 0 };
        for ms in iter { /* add ms to count and total, update min and max */ }
        stats
    }
}

// The annotation selects the implementation above.
let stats: LatencyStats = [12_u64, 340, 5, 7, 210].into_iter().collect();
// stats is LatencyStats { count: 5, total: 574, min: 5, max: 340 }

Figure: How collect dispatches

05_2_consumers_collectors_2

Collecting into Result and Option

The FromIterator implementation of Result in std is especially useful. When you collect an iterator of Result<T, E> into Result<Vec<T>, E>, the collection stops at the first Err and returns it. You get the vector only if each item is Ok. This is the standard pattern to parse all the items or report the item that failed:

let good = ["8080", "9090", "9091"];
// s.parse() returns Result<u16, ParseIntError>. The annotation on `ports` sets these types.
let ports: Result<Vec<u16>, ParseIntError> = good.iter().map(|s| s.parse()).collect();
assert_eq!(ports, Ok(vec![8080, 9090, 9091]));   // each item is Ok

let bad = ["8080", "not-a-port", "9091"];
let ports: Result<Vec<u16>, ParseIntError> = bad.iter().map(|s| s.parse()).collect();
assert!(ports.is_err());   // collect stops at "not-a-port" and does not parse "9091"

Option has the same behavior: one None makes the full result None. The two implementations stop early. The fail-fast check is in the loop of from_iter, so no work occurs after the first error. Internally, the short-circuit mechanism is ControlFlow (Tutorial 5.6).

Extend: Appending Instead of Rebuilding

collect always makes a new collection. When a collection already exists (an accumulation buffer, a cache, a config map), Extend appends an iterator to it:

let mut buffer: Vec<u32> = Vec::with_capacity(16);
buffer.extend([1, 2, 3]);                    // buffer is [1, 2, 3]
buffer.extend((10..13).map(|x| x * x));      // any IntoIterator is valid: adds 100, 121, 144

let mut settings = HashMap::from([("timeout", 30), ("retries", 3)]);
settings.extend([("retries", 5), ("workers", 8)]);  // a later key replaces the earlier value
assert_eq!(settings["retries"], 5);                 // 3 became 5, "timeout" is still 30

These details are important in practice (05_07_extend_trait.rs):

Extend reads size_hint to reserve space before it appends. A batch append is one reservation and N writes, not N possible reallocations (see Tutorial 5.4 for size_hint).
One collection can implement Extend<A> for several item types: String accepts char, &str, and String items.
If you implement Extend on your own type, callers can append to it from any iterator, and the type enforces its own invariants. The EventLog in the example has a size limit. It counts the events that it drops, and it does not grow without limit.
In a loop that accumulates batches, extend on one Vec uses the same allocation again. A collect for each batch allocates each time. For slices of Copy types specifically, extend_from_slice is the fastest method (Tutorial 4.1).

05_07_extend_trait.rs prints:

buffer after two batches: [1, 2, 3, 100, 121, 144]
len=14, capacity=16 (no reallocation)
hello, iterators
settings after override batch: 3 keys
event log kept 4 events, dropped 2

All assertions passed.

Summary

Concept	Key point
`fold(init, f)`	Universal reduction: accumulator + item → accumulator
`reduce(f)`	A fold that uses the first item as the initial accumulator. It returns `Option`
`sum` / `product`	Use the `Sum` and `Product` traits. Annotate the output type
`count` / `for_each`	Pull all the items. `count` counts them, and `for_each` runs side effects
`unzip`	Divides `(A, B)` pairs into two collections in one pass
`any` / `all`	Boolean questions that short-circuit
`find` / `position`	The first item that matches, or its index. The two methods stop early
`nth` / `last`	Positional access. `nth` consumes the prefix, and `last` consumes the full iterator
`min` / `max`	Ties: `min` keeps the first, and `max` keeps the last. Use `total_cmp` for floats
`collect`	Calls `FromIterator` on the target type
`FromIterator`	Implement it so that `collect` can make your own types
`Result`/`Option` collect	Fail-fast: the first `Err` or `None` stops the collection, and `collect` returns it
`Extend`	Appends to an existing collection. It reserves space with `size_hint`

Code Examples

File	Description
`05_04_terminal_consumers.rs`	`fold`, `reduce`, `sum`, `product`, `count`, `for_each`, and `unzip` on latency statistics
`05_05_searching_extremes.rs`	`any`, `all`, `find`, `position`, `nth`, `last`, the `min`/`max` family, the rule for ties, and `total_cmp`
`05_06_collect_fromiterator.rs`	`collect` into four collection types, a custom `FromIterator`, and fail-fast collection into `Result` and `Option`
`05_07_extend_trait.rs`	`Extend` on `Vec`, `String`, and `HashMap`, a custom `Extend`, and the allocations of `extend` compared with `collect`

5.3 · Advanced Iterators: Peekable, Chain, Scan, and Inspect

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::Peekable, std::iter::Scan, std::iter::Inspect, std::iter::Intersperse

Introduction

The adapters in Tutorial 5.1 are stateless. This tutorial adds the stateful and structural tools:

lookahead with Peekable
running state with scan
observation of a pipeline with inspect
consumption in stages with by_ref
cloned and copied, which change references into owned values

The last section shows two adapters that are still nightly-only on Rust 1.99: intersperse and array_chunks. They are useful to know before they become stable.

This tutorial shows:

Peekable: peek, peek_mut, next_if, next_if_eq, and next_if_map (stable since 1.94). The example is a lexer that you write manually.
scan for stateful transformations: running balances, circuit breakers, and prefix sums for sliding windows.
inspect to debug a pipeline without a change to its structure.
by_ref to consume an iterator in stages, and cloned compared with copied.
A nightly preview: intersperse/intersperse_with and array_chunks.

Peekable: One-Item Lookahead

iterator.peekable() wraps an iterator and adds a one-slot buffer. peek() pulls one item from the inner iterator into the slot and lends you a reference. The item stays available for the subsequent next(). That one slot is sufficient for a very large class of parsing problems. For example, a lexer uses it to make the decision between = and ==.

Figure: The peek slot — state machine

05_3_advanced_iterators_1

// it: Peekable<array::IntoIter<i32, 3>>
let mut it = [10, 20, 30].into_iter().peekable();
assert_eq!(it.peek(), Some(&10));   // peek returns a reference, and 10 stays in the slot
assert_eq!(it.peek(), Some(&10));   // a second peek gives the same item
assert_eq!(it.next(), Some(10));    // next takes the item from the slot, by value

peek_mut() lends the slot mutably. You can change the next item before a consumer takes it. The example increases the priority of the next task in place.

A peek does move the inner iterator forward. The item is no longer in the source: it is in the slot. If only your code uses the Peekable, you cannot see this difference. It is important only if you plan to use the inner iterator directly again.

Conditional Consumption: next_if, next_if_eq, next_if_map

The next_if family solves the take_while problem from Tutorial 5.1. take_while loses the first item that fails the predicate. These methods test the item and put it back:

next_if(pred) consumes the next item only if the predicate is true. If not, the item stays in the slot.
next_if_eq(&expected) consumes the next item only if it is equal to a value. The lexer uses it to find a second = after the first.
next_if_map(f) is stable since 1.94. The closure takes the item by value. It returns Ok(mapped) to consume and transform the item, or Err(item) to put the item back in the slot without a change. Thus it consumes and converts in one step:

// chars: Peekable<Chars>, the characters of the source text. The next char is a digit.
// to_digit(10) returns Some(digit) for '0'..='9', and ok_or(c) changes a None into Err(c).
// Ok(d) consumes the char. Err(c) puts the char back, and the loop stops.
let mut value = 0_u32;
while let Some(d) = chars.next_if_map(|c| c.to_digit(10).ok_or(c)) {
    value = value * 10 + d;   // for "42": value is 4, then 42
}

05_08_peekable_token_stream.rs uses all five methods. Four of them (peek, next_if, next_if_eq, next_if_map) make a lexer for total = 42 + 7. The lexer of rustc and many parser crates have the same structure:

// One arm of the lexer's `match` on the peeked char. tokens: Vec<Token>.
'=' => {
    chars.next();   // consume the first '='
    // next_if_eq consumes the second '=' only if it is the next char.
    if chars.next_if_eq(&'=').is_some() { tokens.push(Token::Eq) }   // the text was "=="
    else { tokens.push(Token::Assign) }                              // the text was "="
}

05_08_peekable_token_stream.rs prints:

consumed while small: [1, 2, 3]; 40 stayed in the stream
tokens: [Ident("total"), Assign, Number(42), Plus, Number(7)]
comparison: [Ident("count"), Eq, Number(10)]

All assertions passed.

scan: Stateful Transformation

scan(init, f) is the lazy form of fold. The state goes through the closure as in fold. But scan yields a value for each item and stays lazy. The closure can also stop the stream early: it returns None.

Figure: fold vs scan

05_3_advanced_iterators_2

05_09_scan_stateful.rs shows four uses of scan, from simple to complex. The first two use a transaction feed:

// Each fragment continues `transactions.iter()`.
// transactions: [i64; 6] is [500, -120, -80, 300, -900, 250].
// balance: &mut i64 is the state that scan keeps between items. It starts at 0.

// Running balance: a fold that yields each intermediate value.
.scan(0_i64, |balance, &tx| { *balance += tx; Some(*balance) })
// yields 500, 380, 300, 600, -300, -50

// Circuit breaker: the first overdraft STOPS the stream.
.scan(0_i64, |balance, &tx| {
    let next = *balance + tx;
    if next < 0 { None } else { *balance = next; Some(tx) }   // None stops the stream
})
// yields 500, -120, -80, 300. The -900 stops the stream, and scan does not read 250.

The third use is the sliding-window method from real metrics pipelines. One scan pass calculates prefix sums. After that, any window sum is prefix[i + w] - prefix[i]. You do not add the items of each window again. This method fits well with slice::windows (Tutorial 4.5).

The state and the yielded value do not need to have the same type. In the fourth use, a status feed has consecutive duplicates. The scan keeps the previous item as state and yields only the changes (ok → degraded → ok).

inspect: Debugging a Pipeline

When a chain gives the wrong output, you must find the stage that lost the item. inspect(f) passes each item through without a change and runs a closure on a reference to it. This is printf debugging that you can add to any pipeline, and the structure of the pipeline stays the same:

// readings: [&str; 4] is ["42", "-3", "oops", "17"].
let valid: Vec<u32> = readings.iter()
    .inspect(|raw| println!("  raw:      {raw:?}"))       // each item, before the parse
    .filter_map(|raw| raw.parse::<u32>().ok())            // drops "-3" and "oops"
    .inspect(|parsed| println!("  survived: {parsed}"))   // only the items that parsed
    .collect();
// valid is [42, 17]

05_10_inspect_by_ref.rs prints:

  raw:      "42"
  survived: 42
  raw:      "-3"
  raw:      "oops"
  raw:      "17"
  survived: 17
valid readings: [42, 17]
parsed 3 header lines + 1 body line(s)
cloned() performed 2 clones, lazily

All assertions passed.

The important point is the order of the lines. Items go through the chain one at a time. The pipeline inspects "42", parses it, and inspects it again before it reads "-3". This output makes laziness visible.

by_ref and cloned vs copied

by_ref. Adapters take self by value, so lines.take_while(...) moves lines. Then you cannot read the remainder of the document. by_ref() lends &mut I instead, and &mut I is also an Iterator. Thus a stage can consume only the items that it needs and then return the cursor:

// request: &str holds 3 header lines, one empty line, and the body line "run=nightly".
let mut lines = request.lines();
// by_ref lends `lines`. take_while stops at the empty line and consumes it.
let head: Vec<&str> = lines.by_ref().take_while(|l| !l.is_empty()).collect();   // 3 lines
let body: Vec<&str> = lines.collect();   // continues after the empty line: ["run=nightly"]

This is one pass with no index arithmetic. It is the standard structure for protocols with a header and a body, and for the batching pattern in Tutorial 5.5.

cloned vs copied. Slice iterators yield &T. These adapters change the references into owned values:

copied() makes a bitwise copy and compiles only for T: Copy. When you use it, the code documents and enforces that the operation is trivial.
cloned() calls clone() and accepts any T: Clone, at any cost. 05_10_inspect_by_ref.rs proves the cost and the laziness with a type that counts its clones. The count is zero after the construction of the pipeline, and exactly one for each item after consumption.

// names: [String; 1]. String is not Copy, so copied() is a compile error. This is intended:
//     let _: Vec<String> = names.iter().copied().collect();
// error[E0277]: the trait bound `String: Copy` is not satisfied

Nightly Preview: intersperse and array_chunks

These two adapters are not available on stable. On Rust 1.99, each one still requires nightly. Thus 05_11_intersperse_array_chunks.rs has required-features = ["nightly"] in the package manifest. Run it with this command:

cargo +nightly run -p domain-05-iterators --features nightly --bin 05_11_intersperse_array_chunks

intersperse(sep) (feature iter_intersperse, issue #79524) puts a separator between items. It is a lazy join for any iterator. intersperse_with(f) calculates each separator on demand. A method-resolution conflict prevents the stabilization, and this conflict is years old. The widely used itertools crate already has an intersperse method. A stable std method would change the method that existing code calls.
array_chunks::<N>() (feature iter_array_chunks, issue #100450) yields [T; N] by value, and the compiler checks N at compile time. slice::chunks (Tutorial 4.5) is different: it yields &[T], and the length of each slice is a runtime value. Destructuring needs no bounds checks. APIs such as u16::from_be_bytes accept the arrays directly. You can get the remaining items with into_remainder().

// stream: [u8; 5] is [0x12, 0x34, 0xAB, 0xCD, 0xEF].
let words: Vec<u16> = stream.into_iter()
    .array_chunks::<2>()           // yields [0x12, 0x34], then [0xAB, 0xCD]
    .map(u16::from_be_bytes)       // [u8; 2] -> u16, no try_into().unwrap()
    .collect();
// words is [0x1234, 0xABCD]. The last byte, 0xEF, is the remainder.

On stable, the alternative to intersperse is slice::join or manual separator logic. The alternative to array_chunks is slice::chunks_exact.

Summary

Concept	Key point
`peekable()`	One-slot lookahead buffer for any iterator
`peek` / `peek_mut`	Borrow (or change) the next item and do not consume it
`next_if` / `next_if_eq`	Consume the item only if a predicate or an equality test is true. If not, the item stays
`next_if_map` (1.94)	The closure gets the item by value: `Ok(mapped)` consumes it, and `Err(item)` puts it back
`scan(init, f)`	Lazy stateful map. It yields a value for each item. `None` from the closure stops the stream
fold vs scan	`fold` gives one eager answer. `scan` gives a lazy trace of the intermediate answers
`inspect(f)`	Lets you see each item as it goes through. It makes the pull order of the items visible
`by_ref()`	Lends the iterator to an adapter. You consume in stages and keep the cursor
`copied` / `cloned`	`&T` → `T`. `copied` requires (and proves) `Copy`, and `cloned` calls `clone()`
`intersperse` (nightly)	Lazy separators between items. A name conflict with itertools blocks its stabilization
`array_chunks` (nightly)	`[T; N]` blocks from any iterator, with `N` known at compile time. You can get the remainder

Code Examples

File	Description
`05_08_peekable_token_stream.rs`	`peek`, `peek_mut`, `next_if`, `next_if_eq`, and `next_if_map`, with a working lexer
`05_09_scan_stateful.rs`	A running balance, a circuit breaker that stops the stream, prefix sums, and edge detection with `scan`
`05_10_inspect_by_ref.rs`	A trace with `inspect`, a header and body split with `by_ref`, and `cloned` vs `copied` with a clone counter
`05_11_intersperse_array_chunks.rs`	(nightly) `intersperse`, `intersperse_with`, and `array_chunks` with `into_remainder`

5.4 · DoubleEndedIterator, ExactSizeIterator, and FusedIterator

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::DoubleEndedIterator, std::iter::ExactSizeIterator, std::iter::FusedIterator

Introduction

Iterator makes only one promise: next() yields items until it returns None. Three refinement traits add guarantees to that contract:

iteration from both ends
an exact remaining count
a permanent end

Because of these traits, rev() compiles, len() exists, and collect() allocates exactly one time. Each adapter keeps some of these traits and loses the others. When you know which traits an adapter keeps, you can explain the compile errors that say a method does not exist.

This tutorial shows:

DoubleEndedIterator and next_back, and the methods that start at the back: rev, rfind, rposition, rfold.
ExactSizeIterator, len(), and the size_hint contract that each iterator has.
How collect and extend use the size information to allocate only one time.
FusedIterator (the guarantee that None is permanent) and the cost of fuse() (usually nothing).
TrustedLen, the unsafe trait that is one step above and is still unstable.
How to implement the full trait family on a custom type.

The Trait Family Map

Figure: Iterator refinement traits and what each adds

05_4_iterator_trait_family_1

A slice iterator implements all three stable refinement traits. A std::io::Lines iterator implements none of them. It cannot know the line count in advance, and the text could in principle become longer. Ranges are between these two cases. 0..n implements all three (for example, when n is a usize). The unbounded 0.. is only fused, because it has no back end and no finite length.

DoubleEndedIterator: Two Cursors

A double-ended iterator keeps two cursors on one range of items. next() takes an item from the front, and next_back() takes an item from the back. The cursors move toward each other and never cross. After they meet, next() and next_back() both return None. This does not iterate the items two times. The iterator still yields each item exactly one time, from the end that asks first.

Figure: Front and back cursors converging on a slice iterator

DoubleEndedIterator two-cursor layout

// log: [&str; 6], six log lines from "boot" (first) to "shutdown" (last).
let mut it = log.iter();
assert_eq!(it.next(),      Some(&"boot"));               // front
assert_eq!(it.next_back(), Some(&"shutdown"));           // back
assert_eq!(it.next_back(), Some(&"ERROR timeout"));      // back again
assert_eq!(it.next(),      Some(&"ERROR disk warning")); // front
assert_eq!(it.count(), 2);                               // only the middle two remain

05_12_double_ended.rs uses the methods that come from next_back to analyze a log:

rev() exchanges the functions of the two cursors, so next() moves the back cursor. It does not buffer and does not allocate. log.iter().rev().take(3).rev() gives the same lines as tail -n 3, in chronological order.
rfind(p) and rposition(p) search from the back to the front. They find the most recent match, which is usually the match that you want in a log. rposition reports the index from the front. It needs ExactSizeIterator for that calculation, so the two traits operate together.
rfold(init, f) folds from the right. Use it when the operation is not commutative.
Algorithms with two cursors: a palindrome check reads chars() from both ends until the cursors meet in the middle.

05_12_double_ended.rs prints:

tail -n 3: ["request b", "ERROR timeout", "shutdown"]
most recent error at index 4: ERROR timeout
rfold nesting: usr(local(bin))
palindrome checks passed

All assertions passed.

size_hint: Every Iterator's Estimate

Each iterator has size_hint() -> (usize, Option<usize>). It returns a lower bound and an optional upper bound on the number of remaining items. It is a contract about bounds, not a promise of an exact count. Consumers may use it only for optimization. A wrong hint may waste memory, but it must never cause unsoundness in safe code.

Adapters change the hint together with the items. 05_13_exact_size_hints.rs verifies each row of this table:

Pipeline stage	`size_hint()`
`0..100`	`(100, Some(100))`: exact
`.map(f)`	`(100, Some(100))`: one output item for each input item, so no change
`.filter(p)`	`(0, Some(100))`: from no items to all items
`(0..100).chain(200..210)`	`(110, Some(110))`: the bounds add
`.take(5)` after a filter	`(0, Some(5))`: `take` limits the bounds
`iter::repeat(7)`	`(usize::MAX, None)`: no upper bound

ExactSizeIterator: len() and Its Loss

ExactSizeIterator is the promise that the hint is exact, and it adds one method. len() returns the remaining count. The count decreases as you consume items, so it is not the length of the source. In the usual case, the implementation adds no code. The default len() only reads size_hint().0.

The important point is which adapters keep the promise. map, rev, take, skip, and zip keep the exact count. filter, flat_map, and take_while cannot know their output count. Thus their result does not implement the trait and does not have the method:

// `filter` returns a `Filter`, which does not implement ExactSizeIterator.
// This line does not compile:
//     let n = (0..100).filter(|x| x % 3 == 0).len();
// error[E0599]: the method `len` exists for struct `Filter<...>`,
//               but its trait bounds were not satisfied

That compile error is correct information from the trait system: no one knows the length until the filter runs.

What collect() Does with the Hint

Vec::from_iter reads the hint at the start and reserves capacity for the lower bound. You can see the effect in the capacity:

// The hint is (1000, Some(1000)): collect reserves space for 1000 elements.
let v: Vec<u32> = (0..1000).collect();
assert_eq!(v.capacity(), 1000);        // one allocation, no unused capacity

// The hint is (0, Some(1000)): the Vec starts small and doubles its capacity.
let f: Vec<u32> = (0..1000).filter(|x| x % 2 == 0).collect();
assert_eq!(f.len(), 500);              // capacity: 512 today (implementation detail)

When you know the size better than the hint, give the size to the Vec. Use with_capacity and then extend:

// You know that 500 of the 1000 values are even.
let mut w: Vec<u32> = Vec::with_capacity(500);
w.extend((0..1000).filter(|x| x % 2 == 0));
assert_eq!(w.capacity(), 500);         // no growth steps, no unused capacity

05_13_exact_size_hints.rs prints:

after one next(): len() = 3
size hints verified for map/filter/chain/take/repeat
exact-size collect: len=1000, capacity=1000
filtered collect:  len=500, capacity=512   # (capacity varies by std impl)
with_capacity+extend: len=500, capacity=500

All assertions passed.

TrustedLen is one step above ExactSizeIterator: an unsafe marker trait that is still unstable. It means that unsafe code may rely on an exact hint. With it, collect also skips the capacity check for each item. You cannot implement it on stable 1.99, but the iterators of the standard library implement it for you. It is a large part of the reason why (0..1000).collect() is as fast as a loop in the style of memset.

FusedIterator: None Means None

The base contract permits an unusual behavior: after next() returns None, a later call may yield values again. Most iterators never do this, but generic code cannot assume that. Code that stores an iterator and polls it in more than one round (parsers, schedulers, mergers) would need defensive flags.

Two tools solve this problem:

fuse() is an adapter that you can put on any iterator. After the first None, the adapter always returns None.
FusedIterator is a marker trait. It declares that the iterator already behaves that way. Fuse has a specialization for it, so .fuse() on a fused iterator adds almost no overhead. Thus generic code can call .fuse() as a defensive step.

05_03_slicing_adapters.rs showed the Flicker iterator, which yields items again after None. 05_14_fused_trait_family.rs shows that fuse() costs almost nothing for an iterator that is already fused.

Adapters keep the marker when they can. Since Rust 1.99, StepBy implements FusedIterator when I implements it, so a function with a FusedIterator bound accepts a stepped iterator directly:

use std::iter::FusedIterator;

// The bound is the guarantee: after the first None, each poll returns None.
fn count_then_poll_again<I: FusedIterator>(mut iter: I) -> usize {
    let mut count = 0;
    while iter.next().is_some() {
        count += 1;
    }
    assert!(iter.next().is_none());   // an extra poll returns None again
    count
}

// 0, 3, 6, 9. Before Rust 1.99 this call needed `.fuse()` to compile.
assert_eq!(count_then_poll_again((0..10).step_by(3)), 4);

The implementation is conditional. iter::from_fn(..).step_by(2) is not fused, because FromFn is not fused. 05_22_step_by_fused.rs shows the two cases.

Implementing the Family

05_14_fused_trait_family.rs implements all four traits on a Countdown iterator. As a result, each generic method of the trait family becomes available:

// Countdown yields start, start-1, ..., 1. The example file has the method bodies.
impl Iterator for Countdown {
    type Item = u32;
    fn next(&mut self) -> Option<u32> { /* front cursor */ }
    fn size_hint(&self) -> (usize, Option<usize>) { /* exact */ }
}
impl DoubleEndedIterator for Countdown {
    fn next_back(&mut self) -> Option<u32> { /* back cursor */ }
}
impl ExactSizeIterator for Countdown {}   // promise: the hint is exact
impl FusedIterator for Countdown {}       // promise: None is final

// `starting_at(n)` makes a Countdown that yields n, n-1, ..., 1.
let liftoff: Vec<u32> = Countdown::starting_at(3).rev().collect(); // rev: needs DoubleEnded
// liftoff is [1, 2, 3]
assert_eq!(Countdown::starting_at(10).len(), 10);                  // len: needs ExactSize
let v: Vec<u32> = Countdown::starting_at(1000).collect();
assert_eq!(v.capacity(), 1000);                                    // collect uses the exact hint

The two empty impl blocks are promises, not code. Write them only when they are true. A wrong len(), or a fused iterator that yields items again after None, does not cause memory unsafety by itself. That would need TrustedLen. But a false promise causes wrong behavior in each consumer that relies on it.

05_14_fused_trait_family.rs prints:

countdown len after eating both ends: 8
collect() trusted our size_hint: capacity = 1000
fuse() on an already-fused iterator: zero-cost formality

All assertions passed.

Summary

Concept	Key point
`DoubleEndedIterator`	A second cursor at the back, which `next_back()` moves. The cursors move toward each other and never cross
`rev` / `rfind` / `rposition` / `rfold`	Methods that start at the back. They do not buffer or allocate
`size_hint()`	`(lower, Option<upper>)` on each iterator. The contract permits its use for optimization only
`ExactSizeIterator`	The hint is exact. The trait adds `len()` (the remaining count)
Trait preservation	`map`/`rev`/`take`/`zip` keep the exact count. `filter`/`flat_map` lose it
`collect` + hints	Reserves the lower bound at the start. An exact source gives one allocation
`FusedIterator`	Marker: after `None`, always `None`. A specialization makes `fuse()` almost free
`fuse()`	Makes `None` permanent on any iterator. A low-cost protection in generic code
`StepBy<I>: FusedIterator` (1.99)	`step_by` keeps the marker when the source iterator is fused
`TrustedLen` (nightly)	Unsafe marker. Unsafe code may omit checks because of the hint
Implementing the family	Empty impls are promises. Write them only when they are true

Code Examples

File	Description
`05_12_double_ended.rs`	`next_back`, `rev`, `rfind`, `rposition`, `rfold`, a palindrome check with two cursors
`05_13_exact_size_hints.rs`	`len()`, `size_hint` through adapters, how `collect` preallocates
`05_14_fused_trait_family.rs`	Full trait-family implementation on `Countdown`, the cost of `fuse()`, notes on `TrustedLen`
`05_22_step_by_fused.rs`	`FusedIterator` for `StepBy` (1.99): a `FusedIterator` bound accepts stepped iterators, and the implementation is conditional

5.5 · Creating Iterators: once, empty, repeat, successors, from_fn

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::once, std::iter::empty, std::iter::repeat, std::iter::successors, std::iter::from_fn, std::iter::IntoIterator

Introduction

Until now, each pipeline started from a collection. This tutorial explains how to create iterators without a collection: from one value, from a closure, or from a seed and a rule.

The std::iter factory functions make the small cases in one line each.
IntoIterator explains why for loops accept all of them.
Your own Iterator implementation is the last step. You write one next() method and get the full adapter library.

The topics of this tutorial are:

The factory functions: once, once_with, empty, repeat, repeat_with, repeat_n, successors, and from_fn. The RepeatN type of repeat_n implements Default since 1.97. A note covers the unstable from_coroutine.
IntoIterator: the desugaring of for, and the three forms (T, &T, &mut T).
How to implement Iterator and IntoIterator for your own types.

Choosing a Factory

Figure: Which factory function do you need?

05_5_creating_iterators_1

Each of these functions returns a real iterator. You can use it with all the adapters and consumers from Tutorials 5.1–5.4.

once, once_with, empty

05_15_once_empty_repeat.rs makes a CSV report from these three smallest sources:

// `iter` is `std::iter`. `rows` is the array ["alpha,3", "beta,7"].
// once: adds one value to a stream. Here it puts a header row before the data rows.
let report: Vec<&str> = iter::once("name,count").chain(rows).collect();
// report is ["name,count", "alpha,3", "beta,7"]

// once_with: the closure runs when a consumer takes the item, not at construction.
// `built` is a Cell<bool> that starts as false. It records the closure call.
let footer = iter::once_with(|| {
    built.set(true);
    "-- end of report --"
});
assert!(!built.get());   // the closure did not run yet
let full: Vec<&str> = report.iter().copied().chain(footer).collect();
assert!(built.get());    // collect consumed the footer and ran the closure

// empty: zero items of a specified type. It is the identity element for chain.
let extra_rows = iter::empty::<&str>();

Use once to put a sentinel value before or after a stream. You do not allocate a Vec of one element. once_with applies the lazy evaluation of Tutorial 5.1 to the creation of values, not only to their transformation. empty is useful in generic code where a branch has no items to add but the types must agree.

repeat, repeat_with, repeat_n

Three functions repeat a value:

repeat(v) clones v without end. The iterator is infinite, so always use it with take, zip, or a different adapter that sets a limit. A typical use fills a column to a constant width: cells.chain(iter::repeat("·")).take(width).
repeat_with(f) calls a closure for each item. It has no Clone bound. Thus it can make values that must be distinct (new buffers) or sequences with state (an ID counter that the closure captures).
repeat_n(v, n) yields exactly n copies, and its type implements more traits. RepeatN implements ExactSizeIterator and DoubleEndedIterator, which an infinite Repeat cannot implement. Thus len(), rev(), and a collect with one allocation are all available. For the last item, it moves the value out and does not clone it again.

New in 1.97: RepeatN implements Default, and the default value is an iterator that is already exhausted. It is the correct placeholder with no items for struct fields and for exchanges in the style of mem::take:

// The default RepeatN has no items to yield.
let drained: iter::RepeatN<&str> = iter::RepeatN::default();
assert_eq!(drained.len(), 0);

05_15_once_empty_repeat.rs prints:

report: ["name,count", "alpha,3", "beta,7"]
padded column: ["a", "bb", "·", "·"]
RepeatN::default() is an exhausted repeater (new in 1.97)

All assertions passed.

successors and from_fn

These two functions make the sequences that the factory functions with a predefined shape cannot make.

successors(seed, f): the closure calculates each value from the previous value, and None ends the sequence. The closure returns Option, so you can use an API that already returns Option with no adapter code. 05_16_successors_from_fn.rs has two examples that are similar to production code:

// Powers of two. checked_mul returns None on overflow, and None ends the sequence.
let powers: Vec<u32> = iter::successors(Some(1_u32), |&p| p.checked_mul(2)).collect();
assert_eq!(powers.len(), 32);   // 1, 2, 4, ..., 2^31

// A walk through the ancestor directories: cargo finds the nearest Cargo.toml this way.
// start: &Path = Path::new("/work/app/src/bin")
// dirs_with_manifest: [&Path; 1] contains "/work/app" (a simulated file lookup)
let project_root = iter::successors(Some(start), |dir| dir.parent())
    .find(|dir| dirs_with_manifest.contains(dir));
assert_eq!(project_root, Some(Path::new("/work/app")));

from_fn(f): the closure is the iterator, and its captured variables are the state. A Fibonacci sequence needs no struct. from_fn together with by_ref from Tutorial 5.3 gives the batching pattern (database inserts in chunks):

// `source` is the mutable iterator (1..=8).peekable().
let batches: Vec<Vec<u32>> = iter::from_fn(|| {
    // by_ref borrows `source`, so take(3) removes only the next three items.
    let batch: Vec<u32> = source.by_ref().take(3).collect();
    // An empty batch means that `source` has no more items: end the sequence.
    if batch.is_empty() { None } else { Some(batch) }
}).collect();
// batches is [[1, 2, 3], [4, 5, 6], [7, 8]]

General rule: if the next value is a function of the previous value, use successors (the state is visible in the stream). If the state is auxiliary (counters, buffers, handles), use from_fn, and the closure owns the state. On nightly, iter::from_coroutine would let a coroutine that uses yield do this task. On stable 1.99, use from_fn.

IntoIterator: Why for Loops Work

for is not a special mechanism. It is a trait call. for job in queue { ... } desugars to approximately this code:

// into_iter converts `queue` into an iterator one time, before the loop.
let mut it = IntoIterator::into_iter(queue);
// The loop calls next() until next() returns None.
while let Some(job) = it.next() { ... }

05_17_into_iterator_for_loops.rs runs the desugared form manually and verifies that the two forms give the same result. Three facts follow from the desugaring:

The expression after in selects the impl. Vec<T> implements IntoIterator three times:
- for x in v moves and consumes the Vec (Item = T).
- for x in &v borrows (Item = &T).
- for x in &mut v lets you change the elements in place (Item = &mut T).
The methods into_iter(), iter(), and iter_mut() correspond to these three forms, in this sequence.
Arrays iterate by value since Rust 1.53: for code in [200_u32, 404, 500] yields u32, not &u32. The method call array.into_iter() yields values since edition 2021.
Every Iterator is IntoIterator (the blanket impl returns self). Thus for x in data.iter().filter(...) compiles directly. You can also pass a partial pipeline to each API that has an IntoIterator bound.

That bound is the most flexible signature for a function that takes a group of items:

// I can be any type that converts into an iterator of u32.
fn total_cost<I: IntoIterator<Item = u32>>(jobs: I) -> u32 {
    jobs.into_iter().sum()
}
total_cost(vec![3, 4, 5]);            // Vec: returns 12
total_cost([3, 4, 5]);                // array: returns 12
total_cost(1..=4);                    // range: returns 10
total_cost((1..=4).filter(|n| n % 2 == 0)); // pipeline: returns 6

Implementing Iterator for Custom Types

The trait has one required method, and that method is all that you must write. 05_18_custom_iterator.rs makes a Deltas iterator that moves a window along a price series. It yields the change between consecutive samples. Its only cursor state is a slice that becomes shorter:

// Deltas<'a> has one field, `remaining: &'a [i64]`: the prices that it did not consume yet.
impl Iterator for Deltas<'_> {
    type Item = i64;
    fn next(&mut self) -> Option<i64> {
        // The slice pattern needs two prices. With fewer than two, the iteration ends.
        let [a, b, ..] = self.remaining else { return None };
        let delta = b - a;
        self.remaining = &self.remaining[1..];   // b becomes the `a` of the next pair
        Some(delta)
    }
    fn size_hint(&self) -> (usize, Option<usize>) { /* exact, see Tutorial 5.4 */ }
}

Immediately, the provided methods are available. The trait has about 75 of them, and 60 are stable in 1.99. Examples are series.deltas().max(), .filter(...).count(), and .scan(...) for cumulative moves.

Add impl IntoIterator for &PriceSeries (it returns Deltas), and a for loop accepts the type itself: for delta in &series. Each API with an IntoIterator bound from the previous slide also accepts it. Most custom collections start with the impl for &T, which borrows. Vec has the same impl.

05_18_custom_iterator.rs prints:

deltas: [25, -15, 60, -30]
cumulative moves: [25, 10, 70, 40]
down moves: 2
net change: 40 cents

All assertions passed.

Summary

Concept	Key point
`once(v)` / `once_with(f)`	One item. `once_with` makes the item only when a consumer takes it
`empty()`	Zero items with a specified type. The identity element for `chain`
`repeat(v)`	Infinite clones. Always set a limit with `take`/`zip`
`repeat_with(f)`	A closure call for each item. No `Clone` bound. Sequences with state
`repeat_n(v, n)`	Exactly n items. `ExactSize` + `DoubleEnded`. `RepeatN: Default` since 1.97
`successors(seed, f)`	Next value from the previous value. An API that returns `Option` fits directly
`from_fn(f)`	The closure is the iterator. The captured variables are the state
`for` desugaring	`IntoIterator::into_iter` + `while let Some(...) = it.next()`
Three forms	`v` / `&v` / `&mut v` → `Item = T` / `&T` / `&mut T`
`IntoIterator` bound	The most flexible signature for an API that takes items
Custom `Iterator`	Implement `next()` (and a correct `size_hint`). The trait provides all the other methods

Code Examples

File	Description
`05_15_once_empty_repeat.rs`	`once`, `once_with`, `empty`, `repeat`, `repeat_with`, `repeat_n`, `RepeatN::default()`
`05_16_successors_from_fn.rs`	Powers that stop at overflow, a walk through ancestor directories, Fibonacci, batching with `from_fn` + `by_ref`
`05_17_into_iterator_for_loops.rs`	`for` desugaring, the three forms, arrays by value, `IntoIterator` bounds
`05_18_custom_iterator.rs`	Custom `Deltas` iterator + `IntoIterator` for `&PriceSeries`

5.6 · ControlFlow: Short-Circuiting Iterator Operations

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::ops::ControlFlow, std::iter::Iterator::try_for_each, std::iter::Iterator::try_fold

Introduction

break and continue are keywords. They operate only in the body of a loop. When an early exit must cross a function boundary (a callback, a visitor, a fold closure), you cannot use the keywords. Then programs use improvised alternatives:

boolean flags
sentinel values
Result values that are not really errors

std::ops::ControlFlow<B, C> is the solution of the standard library: break and continue as a value.

This tutorial shows:

The ControlFlow<B, C> enum, its methods, and why it is not a Result with a different name.
try_fold and try_for_each: iteration that can stop early and continue later.
How ControlFlow relates to the Try mechanism behind ? (and to FromResidual, Tutorial 2.1).
The visitor pattern: recursive traversals that the caller can stop, as the AST visitors of rustc do.
A small convenience: is_break and is_continue are const-stable since 1.95.

The Enum: Break and Continue as Values

// The definition in std::ops. The default type of C is ().
enum ControlFlow<B, C = ()> {
    Continue(C),   // continue, with the value C (usually ())
    Break(B),      // stop now, with the reason or the result B
}

The most important design point: neither variant is an error. Result::Err means failure. ControlFlow::Break means completed early: a search found its target, a budget became full, or a rule matched. When you use the correct type, ?, logging, and the error-handling conventions keep their usual meaning.

Figure: Choosing among Option, Result, and ControlFlow

05_6_controlflow_short_circuit_1

The API has methods that are similar to the methods of Result. 05_19_controlflow_basics.rs uses each of them:

// Verdict is a Copy enum from the example. Its variants are Allow and Deny.
let settled: ControlFlow<Verdict> = ControlFlow::Break(Verdict::Deny);
settled.is_break();                 // true (callable in const contexts since 1.95)
settled.break_value();              // Some(Verdict::Deny)
settled.continue_value();           // None
settled.map_break(|v| format!("verdict: {v:?}"));  // Break("verdict: Deny"): changes one side

The example has a chain of firewall rules. The usual version needs two flags: decided: bool and a verdict with a false default value. In the example, each rule returns ControlFlow<Verdict>. Thus the difference between "settled" and "undecided" moves from mutable state into the type.

try_fold: Fold That Can Stop

fold cannot stop, but try_fold can. The closure returns ControlFlow. Continue(new_acc) continues the fold. Break(payload) stops immediately with a value that gives the reason:

// record_sizes: [u64; 7] = [30, 25, 40, 20, 60, 10, 35]. FRAME_LIMIT: u64 = 100 (bytes).
// `bytes` is the accumulator: the total size of the records in the frame.
let packed = record_sizes.iter().try_fold(0_u64, |bytes, &size| {
    let next = bytes + size;
    if next > FRAME_LIMIT {
        ControlFlow::Break((bytes, size))   // frame full: the total and the next record
    } else {
        ControlFlow::Continue(next)
    }
});
// 30 + 25 + 40 = 95 is in the limit. 95 + 20 = 115 is not, so the fold stops.
assert_eq!(packed, ControlFlow::Break((95, 20)));

Compare the alternatives. A plain fold would keep a "full" flag through all the remaining items. A Result would label a full frame as an Err, and a full frame is not an error. ControlFlow states exactly what occurred: a normal, expected early exit, with the evidence.

Figure: try_fold short-circuit flow

05_6_controlflow_short_circuit_2

A fold that never breaks returns Continue(final_acc). Thus the return type shows whether the fold reached the end of the items or stopped early. The result of a plain fold cannot show this difference.

Resumability and try_for_each

One feature is not well known: try_fold takes &mut self and consumes items only up to the break. The iterator stays usable, and its position is immediately after the item that caused the break. Call try_fold again, and the processing continues. 05_20_try_fold_try_for_each.rs packs a stream of records into frames of a constant size. It uses a plain loop around try_fold, so batch processing that can continue needs no more code.

One caveat is important: the closure already consumed the item that caused the break. If you need that item, put it in the Break payload (as the example does), or use a Peekable source (Tutorial 5.3).

try_for_each is the form for side effects. It is a for_each that can stop, and it returns the evidence, not only a bool:

// names: [&str; 4] = ["metrics.log", "trace.log", "core dump", "audit.log"]
let first_invalid = names.iter().try_for_each(|name| {
    // A name with a space is invalid: stop and return that name.
    if name.contains(' ') { ControlFlow::Break(*name) }
    else { ControlFlow::Continue(()) }
});
// The closure did not examine "audit.log".
assert_eq!(first_invalid, ControlFlow::Break("core dump"));

05_20_try_fold_try_for_each.rs prints:

frame 1: Break((95, 20))
frames: [95, 20, 70, 35]
first invalid name: Break("core dump")
checked sums: Err("overflow") / Ok(6)

All assertions passed.

ControlFlow and the Try Machinery

The real signature of try_fold accepts each type that implements the (unstable) Try trait. ControlFlow, Result, and Option all implement it. With Result, try_fold becomes a fallible fold. A typical example is a sum with an overflow check:

// sizes: [u64; 3] = [u64::MAX / 2, u64::MAX / 2, 3]
// checked_add returns None on overflow, and ok_or converts None to Err("overflow").
let sum: Result<u64, &str> =
    sizes.iter().try_fold(0_u64, |acc, &x| acc.checked_add(x).ok_or("overflow"));
// sum is Err("overflow"): the third addition overflows

? operates on ControlFlow too. It unwraps Continue, and it returns early on Break. The use of the Try impl is stable. Only an implementation of Try for your own types is not stable.

try_fold is also important for performance, because it is the mechanism of internal iteration. The default implementations of find, any, all, and position all call try_fold. Adapters such as Chain override it. Their version runs a tight loop for each segment and does not do the state checks of next() for each item. Tutorial 2.1 explains how Try uses residuals and what FromResidual does in ? conversions.

An override of try_fold must name Try as a bound. Thus custom iterators cannot override it on stable 1.99. You get the benefit through the iterators of the standard library.

The Visitor Pattern: Stoppable Traversals

ControlFlow is most useful in recursive traversal, where boolean flags multiply. 05_21_controlflow_visitor.rs traverses a file tree that it keeps in memory. At each file, the visitor closure decides whether to continue. ? propagates a Break through every level of recursion with one character:

// A method of `enum Node { File { name, size }, Dir { name, children: Vec<Node> } }`.
// `f` is the visitor. It returns Break(node) to stop, or Continue(()) to continue.
fn visit_files<'a, F>(&'a self, f: &mut F) -> ControlFlow<&'a Node>
where
    F: FnMut(&'a Node) -> ControlFlow<&'a Node>,
{
    match self {
        Node::File { .. } => f(self),    // a file: the visitor decides
        Node::Dir { children, .. } => {
            for child in children {
                child.visit_files(f)?;   // a Break returns from each level of the recursion
            }
            ControlFlow::Continue(())    // no file in this directory caused a Break
        }
    }
}

The example finds the first file that is larger than the limit. It also proves the early exit: the traversal never visits the files after the match. With a visitor that always returns Continue, the same method does a full traversal. One traversal implementation is sufficient for searches and for full passes.

The alternatives have disadvantages. A visitor that returns bool needs if !child.visit(f) { return false; } at each level. It also does not tell the caller where the traversal stopped. A visitor that returns Result passes the found node through Err, which gives a wrong meaning to the error path.

rustc uses the same design. The methods of rustc_ast::visit::Visitor can return ControlFlow, so an analysis can stop an AST traversal early.

05_21_controlflow_visitor.rs prints:

first oversized file: Some("video.bin")
files visited before stopping: ["README.md", "main.rs", "video.bin"]
total size (full walk): 13049

All assertions passed.

Summary

Concept	Key point
`ControlFlow<B, C>`	`break`/`continue` as a value. `Break(B)` holds the payload of the early exit
Not an error	`Break` is a normal result. Keep `Result` for real failures
`is_break` / `is_continue`	Predicates. Callable in const contexts since 1.95
`break_value` / `continue_value`	Get one side as an `Option`
`map_break` / `map_continue`	Change one side and keep the other side
`try_fold`	A fold that stops at `Break`. It returns `Continue(acc)` if it never breaks
Resumability	The iterator stays usable after a break. Call `try_fold` again to continue
Consumed item	The closure consumed the item that caused the break. Put it in the payload if you need it
`try_for_each`	Side effects with an early exit and with evidence, not only a `bool`
`Try` under `?`	The use of `?` on `ControlFlow` is stable. An implementation of `Try` is not (see 2.1)
Internal iteration	`find`/`any`/`all`/`position` use `try_fold`. Adapters optimize it
Visitor pattern	`ControlFlow` + `?` = recursion that can stop, as in the visitors of rustc

Code Examples

File	Description
`05_19_controlflow_basics.rs`	The enum, its methods, const `is_break`, a rule chain with `?` and no flags
`05_20_try_fold_try_for_each.rs`	Frame packing with `try_fold`, continuation after `Break`, `Result` folds, notes on internal iteration
`05_21_controlflow_visitor.rs`	Recursive tree traversal that can stop, `Break` propagation with `?`, comparison with `bool` and `Result`

6.1 · Threads: Spawning, Joining, and Thread-Local Storage

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::thread, std::thread::spawn, std::thread::Builder, std::thread::JoinHandle, std::thread::scope, thread_local!, std::thread::LocalKey

Introduction

Rust threads are real OS threads. The standard library has no green-thread runtime and no scheduler. Rust adds its type system to the OS primitive. The Send and Sync traits (Tutorial 6.8) let the compiler reject data races at the thread::spawn call site, before your program runs.

This tutorial shows:

thread::spawn and JoinHandle: how to start threads, get their return values, and contain their panics.
thread::Builder: how to name threads and set stack sizes, and why its spawn returns io::Result.
thread::park/unpark: the low-level blocking primitive and its token semantics.
thread::scope: how to borrow stack data across threads without Arc or 'static bounds.
thread_local! and LocalKey<T>: per-thread state that needs no synchronization.

spawn and JoinHandle

thread::spawn(f) runs the closure f on a new OS thread and immediately returns a JoinHandle<T>. T is the return type of the closure. join() blocks until the thread finishes. Then it returns Result<T, Box<dyn Any + Send>>: Ok with the return value, or Err with the panic payload.

Figure: Fork/Join Lifecycle

06_1_threads_spawn_scope_1

// The closure runs on a new OS thread and returns 42.
let handle: thread::JoinHandle<u64> = thread::spawn(|| 6 * 7);
// `join` blocks until the thread finishes. `expect` unwraps the Ok value.
let answer = handle.join().expect("worker should not panic");
assert_eq!(answer, 42);

A spawned thread may continue to run after the function that created it returns. Thus its closure must be 'static: the closure must own all the data that it uses. A move closure does this. Move owned chunks into the threads. Get the results through join:

// dataset: Vec<u64> — the values 1 to 80.
// partial_checksum(&[u64]) -> u64 returns the sum of x * x for a slice.
// Split the dataset into 4 owned chunks of 20 values, one for each worker.
let chunks: Vec<Vec<u64>> = dataset.chunks(20).map(<[u64]>::to_vec).collect();

let mut handles = Vec::new();
for (worker_id, chunk) in chunks.into_iter().enumerate() {
    // `move` transfers `chunk` and `worker_id` into the closure.
    handles.push(thread::spawn(move || (worker_id, partial_checksum(&chunk))));
}

// Join all the workers, then add the partial results.
let mut total = 0;
for handle in handles {
    let (_, sum) = handle.join().expect("worker should not panic");
    total += sum;
}
// total is 173880, the same value as partial_checksum(&dataset)

The join is the synchronization point. After join returns, all the work of the worker happens-before all your subsequent work. Thus the sum of the partial results after all the joins is deterministic. A read of shared state while the workers run is not deterministic. Each example in this domain obeys this rule: assert on values that are final after the joins, never on the interleaving order.

Panics Stay in Their Thread

A panic in a spawned thread does not stop the process. It unwinds only that thread. join reports the panic as Err with the panic payload, a Box<dyn Any + Send>. The payload is a &'static str for a panic with a literal message, and a String for a formatted message:

// The closure panics. The panic unwinds only the spawned thread.
// The default panic hook also prints a message to stderr.
// `06_01_spawn_join.rs` replaces the hook to keep that message out of its output.
let doomed = thread::spawn(|| panic!("sensor offline"));
let outcome = doomed.join();   // Err(Box<dyn Any + Send>)
assert!(outcome.is_err());

let payload = outcome.unwrap_err();
// A literal panic message is a &'static str. `downcast_ref` returns None for other types.
let message = payload
    .downcast_ref::<&str>()
    .copied()
    .unwrap_or("<non-string panic payload>");
assert_eq!(message, "sensor offline");

Thus join().expect(...) is the standard pattern. It panics in the parent thread when a worker panicked, so a failure propagates and does not disappear.

Each thread has a Thread handle, which thread::current() returns. The handle has an id() and an optional name(). The name of the main thread is always "main". Threads from plain spawn have no name.

thread::Builder: Names, Stack Sizes, and Fallible Spawning

thread::Builder configures a thread before the thread starts:

// This code is in a function that returns io::Result, so `?` can propagate the error.
// `06_02_builder_park.rs` calls `expect` in place of `?`.
let handle = thread::Builder::new()
    .name("metrics-flusher".to_string())   // thread::current().name() returns this name
    .stack_size(512 * 1024)                // 512 KiB (the default is 2 MiB)
    .spawn(|| { /* ... */ })?;             // spawn returns io::Result<JoinHandle<T>>

Names appear in panic messages, debuggers, and profilers. Give a name to each long-lived thread.
Stack size is important for workers with deep recursion (larger) and for large numbers of small threads (smaller).
spawn returns io::Result because the creation of an OS thread can fail. The process may reach its thread limit, or the OS may refuse the stack allocation. Plain thread::spawn unwraps this result internally, so it panics where Builder::spawn lets you recover. Services that spawn threads on demand should use the builder.

park and unpark: The Primitive Underneath

thread::park() blocks the current thread. Thread::unpark() wakes a specific thread. Each thread owns one wake-up token. unpark makes the token available. park consumes the token, and it returns immediately if the token was already available. Thus an early unpark still has its effect, and the order of park and unpark is not critical.

Two rules make park-based code correct:

park may wake spuriously. Never assume that a return from park means that a different thread called unpark.
Check a real condition in a loop. Call park between the checks:

// go: Arc<AtomicBool>, initially false. The worker and the coordinator share it.
// worker: the JoinHandle of the worker thread.

// Worker: park until `go` is true. The loop also handles spurious wakeups.
while !go.load(Ordering::Acquire) {
    thread::park();
}

// Coordinator: publish the condition FIRST, then wake.
go.store(true, Ordering::Release);
worker.thread().unpark();   // `JoinHandle::thread` returns the Thread handle

Condvar requires the same order: set the condition, then wake (Tutorial 6.2). Channels use park/unpark internally when they must block, and on some platforms Once does too. Use park/unpark directly only when you build such primitives yourself.

Scoped Threads: Borrowing Without Arc

thread::spawn requires 'static, so its closure must own its data. thread::scope removes this requirement by its structure. It guarantees that it joins each thread spawned in the scope before it returns. Thus scoped threads can borrow local variables, even mutably.

Figure: Scope Guarantees Joins Before Borrows End

06_1_threads_spawn_scope_2

let readings: Vec<f64> = vec![22.1, 23.4, 24.0, 22.8, 21.9, 23.1, 24.6, 22.2];

let (min, max, sum) = thread::scope(|s| {
    // Each closure borrows `readings` by shared reference.
    let min_handle = s.spawn(|| readings.iter().copied().fold(f64::INFINITY, f64::min));
    let max_handle = s.spawn(|| readings.iter().copied().fold(f64::NEG_INFINITY, f64::max));
    let sum_handle = s.spawn(|| readings.iter().sum::<f64>());

    // `ScopedJoinHandle::join` works the same as `JoinHandle::join`.
    (
        min_handle.join().expect("min worker should not panic"),
        max_handle.join().expect("max worker should not panic"),
        sum_handle.join().expect("sum worker should not panic"),
    )
    // `thread::scope` returns the value of its closure.
});
// min is 21.9, max is 24.6, sum is approximately 184.1

The code has no move, no Arc, and no clone. The borrow checker can see that no thread continues after the scope. Mutable access is also possible if the borrows are disjoint:

let mut samples: Vec<u32> = (1..=8).collect();   // [1, 2, 3, 4, 5, 6, 7, 8]
// `split_at_mut` returns two mutable slices that do not overlap.
let (front, back) = samples.split_at_mut(4);

thread::scope(|s| {
    s.spawn(|| {
        for value in front.iter_mut() {
            *value *= 10;
        }
    });
    s.spawn(|| {
        for value in back.iter_mut() {
            *value *= 100;
        }
    });
});   // no explicit joins: the scope joins the two threads here
assert_eq!(samples, [10, 20, 30, 40, 500, 600, 700, 800]);

At its end, the scope automatically joins the threads that you did not join, and it propagates their panics. thread::scope returns the value that its closure returns. Spawned threads can also use the Scope handle to spawn more threads in the same scope.

Thread-Local Storage: thread_local! and LocalKey

thread_local! declares a static for which each thread gets its own independent copy. It needs no locks and no atomics, and it has no contention:

thread_local! {
    // A `const { … }` initializer is the fast path: no lazy-init check on each access.
    static OPS_COUNTER: Cell<u64> = const { Cell::new(0) };
    // A non-const initializer runs lazily, one time for each thread, on the first access.
    static SCRATCH: RefCell<Vec<u8>> = RefCell::new(Vec::with_capacity(64));
}

The type of the declared name is LocalKey<T>. To get access, call with. It gives the copy of the current thread to a closure. Keys with a Cell also have the direct helpers get, set, take, and replace. Since Rust 1.99, they also have update:

// OPS_COUNTER and SCRATCH are the thread-locals from the previous snippet.

// Each of these two lines adds 1 to the counter of the current thread.
OPS_COUNTER.set(OPS_COUNTER.get() + 1);   // direct helpers of a Cell key, no `with` closure
OPS_COUNTER.update(|ops| ops + 1);        // 1.99: the same read-modify-write in one call

/// Encodes a message in the scratch buffer of the current thread.
fn encode(message: &str) -> usize {
    SCRATCH.with(|scratch| {
        // scratch: &RefCell<Vec<u8>> — the copy of the current thread
        let mut buf = scratch.borrow_mut();
        buf.clear();   // removes the old bytes and keeps the allocation
        buf.extend_from_slice(message.as_bytes());
        buf.push(b'\n');
        buf.len()      // encode("telemetry: cpu=42%") returns 19
    })
}

with gives only a shared reference, so you need interior mutability (Cell, RefCell). No other thread can get access to this copy, so the single-threaded cell types are sufficient. The typical use is a scratch buffer for each thread: each thread allocates the buffer one time and uses it again with no synchronization. The destructor of a thread-local runs when the thread exits. The copy of the main thread is fully independent of the copies of the workers.

06_04_thread_local.rs prints:

worker 1 counted 10 ops in its own thread-local
worker 2 counted 20 ops in its own thread-local
worker 3 counted 30 ops in its own thread-local
worker 4 counted 40 ops in its own thread-local
main thread's copy: 0
encoded 19 bytes via thread-local scratch

All assertions passed.

Summary

Concept	Key point
`thread::spawn`	Starts a new OS thread. The closure must be `'static` (it owns its data through `move`).
`JoinHandle::join`	Blocks until the thread finishes. Returns `Ok(value)` or `Err(panic payload)`.
Joins synchronize	Assert on state after the joins, never on the interleaving order.
Worker panics	A panic stays in its thread. `join` returns it as `Err`.
`thread::Builder`	Sets the name and the stack size. `spawn` returns `io::Result`, so you can recover from a failure.
`park` / `unpark`	Token-based blocking. Always check a condition again in a loop.
`thread::scope`	Threads borrow stack data (even `&mut`). The scope joins them at its end.
`thread_local!`	Each thread has an independent copy. A `const { … }` initializer is the fast path.
`LocalKey<Cell<T>>`	Direct `get`/`set`/`take` without a `with` closure. `update` since 1.99.

Code Examples

File	Description
`06_01_spawn_join.rs`	`spawn`, `join`, `move` closures, panic payloads, thread identity
`06_02_builder_park.rs`	`Builder` name and stack size, fallible spawn, `park`/`unpark` handshake
`06_03_scoped_threads.rs`	`thread::scope`: shared and disjoint-mutable borrows, nested spawns
`06_04_thread_local.rs`	`thread_local!`, `LocalKey` accessors (`update` since 1.99), per-thread scratch buffer

6.2 · Mutex, RwLock, and Poison Recovery

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::Mutex, std::sync::MutexGuard, std::sync::RwLock, std::sync::RwLockReadGuard, std::sync::RwLockWriteGuard, std::sync::Condvar, std::sync::PoisonError, std::sync::TryLockError

Introduction

In most languages, only a convention connects a mutex to the data that it protects. If you forget to lock the mutex, you have a data race. The Mutex<T> of Rust owns the data. The only path to the T is through lock(). Thus access without a lock is a compile error, not a failure at runtime.

This tutorial shows:

Mutex<T>: lock, try_lock, the MutexGuard RAII pattern, and into_inner.
RwLock<T>: many concurrent readers or one writer, and when that trade-off is an advantage.
Poisoning: what it means, why lock() returns a Result, and how to recover with PoisonError::into_inner and clear_poison.
Condvar: how to block until a condition is true, without a busy-wait loop.

`Mutex<T>`: The Lock Owns the Data

Figure: Mutex Lock Lifecycle

06_2_mutex_rwlock_poison_1

The usual example is a counter that several threads share. This version is safe:

// The mutex owns the u64. `Arc` gives each thread shared ownership of the mutex.
let counter = Arc::new(Mutex::new(0_u64));

let mut handles = Vec::new();
for _ in 0..4 {
    let counter = Arc::clone(&counter);   // one Arc handle for each thread
    handles.push(thread::spawn(move || {
        for _ in 0..1_000 {
            // `lock` blocks until the mutex is free. The guard derefs to &mut u64.
            let mut guard = counter.lock().unwrap();
            *guard += 1;
        }  // the guard drops here and unlocks the mutex, even on panic or early return
    }));
}
for handle in handles {
    handle.join().expect("worker should not panic");
}

assert_eq!(*counter.lock().unwrap(), 4_000);   // deterministic after the joins

Three points are important:

Arc<Mutex<T>> is the standard pair. Arc shares ownership across threads, and Mutex serializes access. Tutorial 6.8 explains why you need the two types.
MutexGuard is RAII. The unlock is in Drop, so no code path can omit it. To release the lock early, before slow work that does not need the lock, call drop(guard) explicitly.
lock() returns Result because of poisoning (slide 5). .unwrap() is the idiomatic default, not a shortcut: the program refuses to run on data that is possibly corrupt.

When the sharing ends, you can recover the data without a lock. Arc::try_unwrap(arc) returns the Mutex when only one owner remains. mutex.into_inner() returns the T. Exclusive ownership proves that no other thread can race with you.

try_lock: Refusing to Wait

try_lock() never blocks. It immediately returns Ok(guard) or Err(TryLockError::WouldBlock). Use it for opportunistic work. For example: flush the cache if no thread uses it, or else skip this cycle.

let busy = Mutex::new(-1_i32);
let held = busy.lock().unwrap();   // this thread holds the lock

// The mutex is not re-entrant: a second attempt from the same thread fails.
match busy.try_lock() {
    Err(TryLockError::WouldBlock) => { /* skip, try again in the next cycle */ }
    Err(TryLockError::Poisoned(_)) => unreachable!("nothing has panicked"),
    Ok(_) => unreachable!("mutex is already held"),
}
drop(held);                         // release the lock
assert!(busy.try_lock().is_ok());   // now `try_lock` succeeds immediately

The Mutex of std is not re-entrant. A second lock from the same thread never returns: the documentation says that it might panic or deadlock. A second try_lock fails. Thus the example above is deterministic without a second thread. The same property is a risk in real programs. Do not call a function that locks a mutex from inside a critical section that holds the same lock.

`RwLock<T>`: Many Readers or One Writer

RwLock divides access into read() and write(). read() is shared: any number of read guards can exist concurrently. write() is exclusive. RwLock is a good selection for read-mostly data. An example is a service configuration that each request reads and only a redeploy rewrites.

let telemetry = RwLock::new(vec![10_u32, 20, 30]);

// Two read guards exist at the same time, in the same thread.
let reader_a = telemetry.read().unwrap();
let reader_b = telemetry.read().unwrap();
assert_eq!(reader_a.iter().sum::<u32>(), 60);
assert_eq!(reader_b.len(), 3);

// While a reader exists, a writer cannot enter. `try_write` fails immediately.
assert!(matches!(telemetry.try_write(), Err(TryLockError::WouldBlock)));

drop(reader_a);
drop(reader_b);
// No readers remain, so the writer gets exclusive access.
telemetry.write().unwrap().push(40);
assert_eq!(*telemetry.read().unwrap(), [10, 20, 30, 40]);

// `06_06_rwlock.rs` then shares an Arc<RwLock<Config>> between 4 reader threads
// and 1 writer thread. The readers never block each other.

The reader/writer rule is the rule of the borrow checker (&T xor &mut T), which RwLock applies at runtime. RwLock is the thread-safe equivalent of RefCell: it blocks where RefCell panics. Tutorial 7.3 describes the remainder of that family.

When RwLock is not better than Mutex: If writes are frequent, or critical sections are very small, the extra bookkeeping costs more than it saves. This bookkeeping includes the reader count and writer preference. Std also makes no fairness guarantee. It inherits the policy of the OS lock, so a continuous flow of readers can starve writers on some platforms. Use Mutex by default. Use RwLock when a profile shows that reads are much more frequent than writes and the read sections do real work.

Poisoning: The Signal of a Panic

A mutex becomes poisoned when a thread panics while it holds the guard. The reason is that the panic may have interrupted a multi-step update, which leaves the protected data with a broken invariant. Poisoning makes that suspicion visible: each subsequent lock() returns Err(PoisonError).

Figure: Poison Recovery Flow

06_2_mutex_rwlock_poison_2

Poisoning is advisory, not destructive. The PoisonError contains the guard that you requested:

// ledger: Arc<Mutex<Ledger>>, initially Ledger { enqueued: 10, processed: 8 }.
// A worker added 5 to `enqueued`, then panicked while it held the guard.
let mut guard = match ledger.lock() {
    Ok(guard) => guard,
    Err(poisoned) => poisoned.into_inner(),   // get the guard from the PoisonError
};
guard.enqueued -= 5;             // repair the half-applied update: 15 becomes 10 again
drop(guard);

assert!(ledger.is_poisoned());   // the recovery does NOT clear the flag
ledger.clear_poison();           // this call clears the flag
assert!(ledger.lock().is_ok());

RwLock becomes poisoned in the same way, with one difference: only a writer that panics poisons it. A reader that panics could not corrupt the data, so it does not poison the lock.

06_07_poison_recovery.rs prints:

mutex poisoned after worker panic: true
recovering: taking the guard out of the PoisonError
state at recovery: enqueued=15 processed=8
after clear_poison: enqueued=10 processed=8

All assertions passed.

Condvar: Sleeping Until a Condition Holds

A Condvar works together with a Mutex that protects the actual condition state. A waiter atomically releases the lock and sleeps. A notifier changes the state and wakes the waiters. Use wait_while: it checks the predicate again on each wakeup. Thus spurious wakeups (the OS can produce them) are harmless:

// jobs: Mutex<VecDeque<u32>>, ready: Condvar. The two threads share the pair in an Arc.

// Consumer: sleep while the queue is empty.
let mut guard = jobs.lock().unwrap();
// `wait_while` releases the lock while it sleeps. It returns the guard when the
// predicate is false.
guard = ready.wait_while(guard, |q| q.is_empty()).unwrap();
let job = guard.pop_front().expect("predicate guarantees non-empty");
drop(guard);   // release the lock BEFORE the slow work

// Producer: change the state, THEN notify. Never use the opposite order.
// job: u32 — the number of the next job
jobs.lock().unwrap().push_back(job);
ready.notify_one();

notify_one wakes one waiter (work queues). notify_all wakes all the waiters (startup gates, shutdown broadcasts).
wait_timeout limits the sleep. It returns the guard and a WaitTimeoutResult, and the timed_out() method tells you why the thread woke.
The order "change the state, then notify" is important. If you notify first, the waiter may check the predicate again, find the old state, and sleep. Then it misses the only wakeup that it could get.

Condvar is the most general tool for a wait. Channels (Tutorial 6.5) and Barrier (Tutorial 6.7) package the same mechanism for their specific cases, and they are easier to use.

Choosing: Mutex, RwLock, or Something Else

Situation	Use this
Any shared mutable state (the default selection)	`Mutex<T>`
Read-mostly data, where the read sections do real work	`RwLock<T>`
One value, one operation (counter, flag)	`Atomic*` (Tutorial 6.3)
Data that flows in one direction between threads	channels (Tutorial 6.5)
A wait on an arbitrary predicate	`Condvar`
One-time initialization	`OnceLock`/`LazyLock` (Tutorial 6.6)

Two habits prevent most lock bugs:

Keep critical sections small: compute outside, change the data inside.
Never call unknown code (callbacks, Display implementations, loggers that allocate) while you hold a guard.

Summary

Concept	Key point
`Mutex<T>` owns its data	Access is only through `lock()`. Access without a lock does not compile.
`MutexGuard`	RAII: `Drop` unlocks the mutex on each code path.
`lock().unwrap()`	Idiomatic: it propagates poisoning and does not run on corrupt data.
`try_lock`	Does not block. Returns `WouldBlock` when a thread holds the lock. Use it for opportunistic work.
Not re-entrant	A second `lock` from the same thread never returns. Do not lock the same mutex again in a critical section.
`RwLock`	Concurrent readers xor one writer. Best for read-mostly data.
Writer starvation	No fairness guarantee. The behavior depends on the platform.
Poisoning	A panic while a thread holds the guard causes it. It is an advisory flag, not data loss.
`PoisonError::into_inner`	Returns the guard. Repair the invariants, then call `clear_poison()`.
`Condvar::wait_while`	Checks the predicate again on each wakeup, so spurious wakeups are harmless.
Change the state, then notify	The one ordering rule of `Condvar` protocols.

Code Examples

File	Description
`06_05_mutex_basics.rs`	`lock`, guard RAII, `try_lock`/`WouldBlock`, `into_inner`
`06_06_rwlock.rs`	Concurrent readers, `try_write`, read-mostly config workload
`06_07_poison_recovery.rs`	Deliberate poisoning, `PoisonError::into_inner`, `clear_poison`
`06_08_condvar.rs`	`wait_while` queue, `notify_all` gate, `wait_timeout`

6.3 · Atomic Types and Their Operations

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::atomic::AtomicBool, std::sync::atomic::AtomicUsize, std::sync::atomic::AtomicPtr, std::sync::atomic::Ordering

Introduction

Atomics are the smallest concurrency primitive. An atomic is a single value whose operations complete indivisibly, as dedicated CPU instructions. It needs no lock, no syscall, and no blocking. Each higher-level tool in this domain (Mutex, channels, OnceLock) uses atomics at its lowest level.

This tutorial shows:

The atomic type family and its portability limits.
load, store, swap: flags and one-winner claims.
The fetch_* read-modify-write (RMW) family: fetch_add, fetch_sub, fetch_or, fetch_and, fetch_xor.
Closure-based RMW: update/try_update (stabilized in Rust 1.95), and the legacy fetch_update (deprecated in Rust 1.99).
from_mut / from_mut_slice / get_mut_slice (1.98): exclusive &mut integers as atomics.
When an atomic is better than a Mutex, and when it is not.

Memory orderings appear here only as a general rule: Relaxed for standalone counters, SeqCst when you are not sure. Tutorial 6.4 gives the full model.

The Atomic Type Family

std::sync::atomic provides AtomicBool, AtomicI8/AtomicU8 through AtomicI64/AtomicU64, AtomicIsize/AtomicUsize, and AtomicPtr<T>. Three properties define them:

Interior mutability: all operations take &self. An atomic in a static needs no unsafe, no lazy initialization, and no mut:

// `AtomicU64::new` is a const fn, so it can initialize a static.
static REQUESTS_SERVED: AtomicU64 = AtomicU64::new(0);

Sync by design: the purpose of an atomic is that threads share a &AtomicU64. The hardware serializes concurrent operations.
Portability limit: not every target has every width (some 32-bit platforms have no 64-bit atomics). AtomicUsize and AtomicBool are the safest defaults. Each type exists only where the platform supports lock-free operations on it.

Each operation takes an Ordering parameter. Until Tutorial 6.4, use this rule. A counter that needs only its own consistency can use Relaxed. Code that publishes other memory should use SeqCst, the default when you are not sure.

A unique &mut u32 proves that no other thread can observe the value. Thus, since 1.98, you can view those bytes as an atomic with AtomicU32::from_mut(&mut n). from_mut_slice does the same for a full array. get_mut_slice is the inverse: it gives an exclusive &mut [AtomicU32] as an ordinary &mut [u32] for single-threaded initialization. The exclusive borrow does the work of a lock.

load, store, swap: Flags and Claims

load and store are atomic reads and writes. The usual pattern is a shutdown flag:

let shutdown = AtomicBool::new(false);
// The coordinator sets the flag. In a real program, it does this while the worker runs.
// `06_09_atomic_counter.rs` sets the flag BEFORE the worker starts, for a deterministic result.
shutdown.store(true, Ordering::Relaxed);

let iterations = thread::scope(|s| {
    s.spawn(|| {
        let mut work_done = 0_u32;
        // The worker checks the flag before each unit of work.
        while !shutdown.load(Ordering::Relaxed) {
            work_done += 1;
        }
        work_done
    })
    .join()
    .expect("worker should not panic")
});
assert_eq!(iterations, 0);   // the flag was already true at the first check

swap writes a new value and returns the old value in one indivisible step. This gives two common patterns:

One-winner claim: N threads race, and exactly one wins:

// claimed: AtomicBool, initially false. 8 threads run this code.
let already_claimed = claimed.swap(true, Ordering::Relaxed);   // returns the OLD value
if !already_claimed {
    // Only one thread gets `false`. That thread runs the job.
}

Take-and-reset: drain a dirty-flags word atomically. A flag that a different thread raises concurrently goes into this batch or the subsequent batch. The program never loses a flag and never counts a flag twice:
```
let dirty = AtomicU8::new(0b0110);
let batch = dirty.swap(0, Ordering::Relaxed);   // read AND clear in one step
// batch is 0b0110, and dirty is now 0
```

Which thread wins a claim is different on each run. That exactly one thread wins is deterministic. These tutorials assert on this type of property.

fetch_add: The Canonical Lock-Free Counter

A non-atomic count += 1 from two threads is a lost-update bug: each thread reads 5, and each thread writes 6. fetch_add does the read-modify-write as one step:

// REQUESTS_SERVED is the static AtomicU64 from the first snippet.
// 4 threads each run this line 10_000 times:
REQUESTS_SERVED.fetch_add(1, Ordering::Relaxed);

// After the join of all the threads, the value is exactly 40_000 on each run:
assert_eq!(REQUESTS_SERVED.load(Ordering::Relaxed), 40_000);

Each fetch_* operation returns the previous value. Frequently, that value is the result that you need. For example, a lock-free ID allocator is only next_id.fetch_add(1, ...), because each caller gets a different previous value.

The bitwise family operates on flag words:

fetch_or sets bits. Each worker sets its own "done" bit. OR is commutative, and that is why no lock is necessary.
fetch_and clears bits.
fetch_xor toggles bits.

fetch_max and fetch_min also exist, for a running maximum or minimum.

06_09_atomic_counter.rs prints:

requests served: 40000
allocated ids: [100, 101, 102, 103]
worker exited after 0 iterations (flag was pre-set)

All assertions passed.

fetch_update, update, try_update: Arbitrary RMW

There is no fetch_clamp or fetch_saturating_mul. For an operation that is not built in, you supply a closure. Rust 1.95 stabilized the two current methods:

update(set_order, fetch_order, f) applies f: T -> T. It repeats an internal compare-exchange loop until its write succeeds. Then it returns the previous value:

// peak_latency_us: AtomicU32, initially 0. Several threads share it.
// sample: u32, one latency measurement.
// Keep the maximum of all the samples from all the threads:
peak_latency_us.update(Ordering::Relaxed, Ordering::Relaxed, |peak| peak.max(sample));

try_update(set_order, fetch_order, f) takes f: T -> Option<T>. If the closure returns None, the call stops and does not write. The result is Ok(previous) after a write, and Err(current) after a stop. A token bucket uses it to "decrement unless zero" in one atomic step:
```
// tokens: AtomicU32, initially 3. 8 threads run this code.
let outcome = tokens.try_update(Ordering::Relaxed, Ordering::Relaxed, |t| {
 t.checked_sub(1) // None when t == 0: stop, no write
});
match outcome {
 Ok(_previous) => { /* this thread got a token: the previous value was 1 or more */ }
 Err(_current) => { /* the pool is empty: the current value is 0 */ }
}
```
With 3 tokens and 8 threads, which threads get a token is different on each run. That exactly 3 succeed and 5 fail is deterministic.
fetch_update is the original form with an Option, and its behavior is identical to try_update. It appears in much existing code. Rust 1.99 deprecates it ("renamed to try_update"), so the compiler gives a warning for each use. Write try_update in new code.

One rule applies: the closure may run more than one time, because the method tries again when a different thread wins the race. Keep the closure pure: no side effects, no I/O.

When an Atomic Is Better Than a Mutex, and When It Is Not

Figure: Choosing Between Atomics and Locks

06_3_atomic_types_operations_1

Atomics are better for single independent values. They have no blocking, no syscalls, and no poisoning, and the hot path is one CPU instruction.

Atomics are worse when an invariant includes more than one value. Each of two separate atomics can be consistent while a reader sees the pair in an inconsistent state. Atomicity does not compose across variables. Mutex<Stats> makes the full struct one critical section. Two AtomicU64 values make two independent guarantees. When you are not sure, start with Mutex. Use an atomic only when a profiler shows the need.

Summary

Concept	Key point
Atomic types	`AtomicBool`, integer widths, `AtomicPtr`. Operations take `&self`. `new` is a `const fn`.
Statics	`static N: AtomicU64 = AtomicU64::new(0)` needs no `unsafe` and no lazy initialization.
`load` / `store`	Atomic read and write, for shutdown flags and published state.
`swap`	Writes a value and returns the old value: one-winner claims, take-and-reset.
`fetch_add` etc.	Indivisible RMW. Returns the previous value (ID allocator).
`fetch_or/and/xor`	Lock-free bit operations. Commutative operations make a lock unnecessary.
`update` (1.95)	Closure `T -> T`, repeated compare-exchange loop, returns the previous value.
`try_update` (1.95)	Closure `T -> Option<T>`. `None` stops the call without a write.
`fetch_update`	Legacy name of `try_update`, deprecated since 1.99.
`from_mut` / `from_mut_slice` / `get_mut_slice` (1.98)	Exclusive `&mut` integers ↔ atomics. The exclusive borrow does the work of a lock.
Closure rule	The closure may run more than one time. Keep it pure.
Atomic vs Mutex	Atomic for one value. Mutex for invariants across values.

Code Examples

File	Description
`06_09_atomic_counter.rs`	`fetch_add` counters, static atomics, ID allocation, shutdown flag
`06_10_atomic_bitflags.rs`	`fetch_or`/`and`/`xor`, `swap` take-and-reset, one-winner claim
`06_11_atomic_update.rs`	`update`/`try_update` (1.95), token bucket, `fetch_update` (deprecated in 1.99)
`06_28_atomic_from_mut.rs`	`from_mut` / `from_mut_slice` / `get_mut_slice` (1.98)

6.4 · Memory Ordering: Relaxed, Acquire/Release, and SeqCst

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::atomic::Ordering, std::sync::atomic::fence, std::sync::atomic::compiler_fence

Introduction

Compilers reorder instructions, and CPUs reorder memory operations. They do this aggressively and invisibly, and the result is correct for single-threaded code. When a second thread observes your memory, the order of events is no longer clear. Each atomic operation takes an Ordering argument. With this argument, you request exactly as much cross-thread ordering as you need.

This tutorial shows:

The guarantee of each Ordering: Relaxed, Acquire, Release, AcqRel, SeqCst.
The happens-before relation, with the message-passing pattern (a payload and a ready flag) as the example.
The difference between compare_exchange and compare_exchange_weak, and the retry loop.
fence and compiler_fence.
The common bugs: the assumption that Relaxed orders unrelated memory, and orderings that do not match between the store side and the load side.

What Each Ordering Guarantees

Each atomic operation is itself indivisible with every ordering. The orderings differ in the constraint that the operation puts on the other memory accesses near it:

Ordering	For	Guarantee
`Relaxed`	loads and stores	Atomicity only. No cross-thread ordering of other memory.
`Release`	stores	The compiler and the CPU cannot move a write from before this store to after it.
`Acquire`	loads	The compiler and the CPU cannot move a read or a write from after this load to before it.
`AcqRel`	RMW (read-modify-write) operations	The two halves: `Acquire` for the read and `Release` for the write.
`SeqCst`	all operations	`Acquire`/`Release`, plus one global order of all `SeqCst` operations.

The pairing rule: a Release store synchronizes-with an Acquire load of the same atomic that observes the stored value. This edge and the program order in each thread together make the happens-before relation. The happens-before relation decides what a thread is guaranteed to see.

Relaxed is correct for independent counters (Tutorial 6.3), because a counter needs only its own consistency. When a flag publishes other memory, Relaxed on that flag is a bug.

The Message-Passing Pattern

The producer thread writes data and then sets a flag. The consumer thread waits for the flag and then reads the data. Channels, OnceLock, and every hand-off protocol use this pattern.

Figure: Release/Acquire Happens-Before Edge

Release/Acquire happens-before timeline

static PAYLOAD: AtomicU64 = AtomicU64::new(0);       // the data
static READY: AtomicBool = AtomicBool::new(false);   // the flag that publishes the data

// Producer thread:
PAYLOAD.store(42, Ordering::Relaxed);     // (1) write the data
READY.store(true, Ordering::Release);     // (2) then publish it

// Consumer thread:
while !READY.load(Ordering::Acquire) {    // (3) spin until the flag is true
    hint::spin_loop();                    // tells the CPU that this is a spin-wait
}
assert_eq!(PAYLOAD.load(Ordering::Relaxed), 42);   // (4) always sees the 42 from (1)

The chain (1) → (2) → (3) → (4) has no gap:

Release keeps (1) before (2).
The synchronizes-with edge connects (2) to the load (3) that reads true.
Acquire keeps (4) after (3).

The payload accesses can be Relaxed, because the flag supplies the ordering.

The example binary shows two engineering practices:

Bounded spinning: spin for a short time with hint::spin_loop(), which emits the pause instruction of the CPU. Then use thread::yield_now(), so that the producer is sure to get scheduler time, even on one core. Never spin at full speed without a limit.
Correct by construction, not by chance: on x86, a Relaxed flag would usually pass, because the hardware is strongly ordered. On ARM and POWER, a Relaxed flag can fail in practice. The example runs 200 rounds, but its purpose is not to test the race. The rounds use a guarantee that holds in each round and on each architecture.

06_12_message_passing.rs prints:

message passing: 200 rounds, payload visible every time

All assertions passed.

compare_exchange: Ordering-Aware CAS

compare_exchange(expected, new, success, failure) is a compare-and-swap (CAS) operation. It atomically replaces the value only if the value equals expected. On success, it returns Ok(previous). On failure, it returns Err(actual) and writes nothing.

It takes two orderings: one for success and one for failure. A failed CAS does no store, so the failure ordering cannot be Release or AcqRel.

A one-shot claim, for example leader election:

// leader: AtomicU32, starts as IDLE (0). my_id: u32, the id of this thread (not 0).
// Success ordering: AcqRel. Failure ordering: Acquire.
match leader.compare_exchange(IDLE, my_id, Ordering::AcqRel, Ordering::Acquire) {
    Ok(_) => { /* exactly one thread gets Ok: it is the leader */ }
    Err(current_leader) => { /* each other thread gets the id of the leader */ }
}

The weak retry loop

compare_exchange_weak can fail spuriously: it can report a failure even when the value matched. On LL/SC architectures (ARM, RISC-V), the strong version hides spurious failures with its own internal loop. That internal loop is unnecessary work if your code already retries. The standard pattern is a retry loop around the weak version. This example implements fetch_mul, an operation that the standard library does not have:

// Multiplies the atomic by `factor` and returns the previous value.
fn fetch_mul(atom: &AtomicU32, factor: u32) -> u32 {
    let mut current = atom.load(Ordering::Relaxed);   // the first value to try
    loop {
        let new = current * factor;
        match atom.compare_exchange_weak(current, new, Ordering::Relaxed, Ordering::Relaxed) {
            Ok(previous) => return previous,   // the CAS stored `new`
            // Err holds the value that is in the atomic now. Try again with it.
            Err(actual) => current = actual,
        }
    }
}

General rule: in a loop, use the weak version. For one attempt, use the strong version. The update and try_update methods from Tutorial 6.3 contain this same loop. So does fetch_update, which Rust 1.99 deprecates in favor of try_update.

06_13_compare_exchange.rs prints:

leader election: winner id=1 (varies), winners=1 losers=3
fetch_mul x4 threads: 3 * 2^4 = 48

All assertions passed.

fence and compiler_fence

atomic::fence(ordering) separates the ordering from a specific atomic operation. The producer calls a Release fence and then does any store. The consumer does a load that observes this store and then calls an Acquire fence. This sequence creates the same synchronizes-with edge. The result is one ordering point for a full batch:

// BATCH: [AtomicU64; 4] and PUBLISHED: AtomicBool are statics. values is [11, 22, 33, 44].
// Producer: N Relaxed stores, ONE Release fence, then a Relaxed flag store.
for (slot, value) in BATCH.iter().zip(values) {
    slot.store(value, Ordering::Relaxed);
}
fence(Ordering::Release);                   // orders all the stores above before the flag store
PUBLISHED.store(true, Ordering::Relaxed);   // the publication point

// Consumer: a Relaxed flag loop, then ONE Acquire fence.
while !PUBLISHED.load(Ordering::Relaxed) { hint::spin_loop(); }
fence(Ordering::Acquire);
// All the BATCH slots are now visible: [11, 22, 33, 44].

A fence has a cost one time for each batch. An ordering on each operation has a cost for each operation. Most code never needs a fence. Use a fence when a profile shows many stores before one publication point.

compiler_fence is a different type of tool. It emits no CPU instruction. It only makes sure that the compiler does not reorder memory accesses across it. Its use is ordering on the same core: signal handlers, interrupt handlers, and memory-mapped I/O. It is never a replacement for fence in cross-thread code.

06_14_fences.rs prints:

batch after publication: [11, 22, 33, 44]

All assertions passed.

SeqCst, and the Bugs to Avoid

Acquire/Release creates edges between pairs of operations. SeqCst also puts each SeqCst operation into one total order that all threads agree on. The standard case that needs SeqCst is store buffering (Dekker's algorithm). Two threads each store their own flag and then load the flag of the other thread. With only Acquire/Release, the two threads can both load false. With SeqCst, at least one thread must see true.

If your protocol depends on more than one independent atomic, start with SeqCst.

Figure: Choosing an Ordering

06_4_memory_ordering_1

The common bugs:

Relaxed used to order unrelated memory. Relaxed gives a guarantee for the atomic itself and for nothing near it. A payload behind a Relaxed flag is broken in practice on weakly-ordered CPUs. On x86, the bug is not visible until you run the program on ARM.
Mismatched sides. A Release store synchronizes only with an Acquire load of the same atomic. Release on one side and Relaxed on the other side create no edge. Examine the two sides together.
Ordering used to excuse a data race. No ordering makes a non-atomic data race defined behavior. Fences order atomics. They do not make a race on a static mut sound.

The practice: use the weakest ordering that you can justify in a comment. Use SeqCst when the justification is not precise. Make the code correct first. Then weaken the ordering only with evidence.

Summary

Concept	Key point
`Relaxed`	Atomicity only. Correct for independent counters.
`Release` store	Earlier writes cannot move after it. It is the publish side.
`Acquire` load	Later accesses cannot move before it. It is the consume side.
synchronizes-with	A `Release` store and an `Acquire` load of the same atomic that observes the value
happens-before	Program order and synchronizes-with edges, combined
`AcqRel`	For RMW operations that read and also publish
`SeqCst`	Adds one global total order. For protocols with more than one atomic. The safe default.
`compare_exchange`	Two orderings. The failure ordering cannot be `Release` or `AcqRel`.
`compare_exchange_weak`	It can fail spuriously. Use it in retry loops.
`fence`	Ordering for a batch, separate from a single operation
`compiler_fence`	Compiler only. For signal handlers and MMIO, not for cross-thread synchronization.

Code Examples

File	Description
`06_12_message_passing.rs`	Release/Acquire hand-off, bounded spin loop, 200 verified rounds
`06_13_compare_exchange.rs`	One-shot CAS claim, weak-CAS retry loop (`fetch_mul`)
`06_14_fences.rs`	Batch publication with `fence`, `compiler_fence`, `SeqCst` notes

6.5 · Channels: mpsc and mpmc Message Passing

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::mpsc, std::sync::mpmc, Sender, Receiver, SyncSender, RecvError, SendError, TryRecvError, TrySendError

Introduction

"Do not communicate by sharing memory. Instead, share memory by communicating." Channels move the ownership of values between threads. After send(value), the producer cannot use the value. The compiler does more than prevent the data race: the program cannot express it.

This tutorial shows:

mpsc::channel (unbounded): multiple producers, one consumer, and iteration on the receiver.
mpsc::sync_channel (bounded): backpressure, try_send, and the rendezvous channel with a bound of 0.
The error types: SendError, TrySendError, TryRecvError, RecvTimeoutError. Also the shutdown pattern in which you drop the sender.
The unstable std::sync::mpmc module (nightly as of 1.99): cloneable receivers and the worker pool that they make possible.
How to select between channels and shared state.

mpsc::channel: Multi-Producer, Single-Consumer

Figure: mpsc Topology

06_5_channels_mpsc_mpmc_1

channel() returns a (Sender<T>, Receiver<T>) pair with an unbounded buffer, so send never blocks. Sender is Clone: this is the "mp" (multi-producer) in the name. Receiver is not Clone: this is the "sc" (single-consumer).

// LogRecord is a struct with the fields producer: u32, seq: u32, and message: String.
let (tx, rx) = mpsc::channel::<LogRecord>();

for producer in 1..=3 {
    let tx = tx.clone();   // each producer thread gets its own Sender
    thread::spawn(move || {
        for seq in 0..4 {
            let message = format!("worker {producer} event {seq}");
            // send moves the record into the channel.
            tx.send(LogRecord { producer, seq, message }).expect("receiver alive");
        }
    });   // the clone drops when the producer thread finishes
}
drop(tx);   // drop the ORIGINAL Sender too (the list below gives the reason)

let mut records: Vec<LogRecord> = Vec::new();
for record in rx {   // blocks for each message, ends when the channel closes
    records.push(record);
}
// records now holds all 12 records (3 producers, 4 records each)

Three details are important:

send moves the value. The ownership goes through the channel. There is no clone, no lock, and no aliasing.
The receiver loop ends when every Sender is gone. If the coordinator does not drop the original tx, the result is a common deadlock. The loop waits forever for a sender that will never send and will never disconnect.
The order is FIFO for each sender. Messages from one sender arrive in the order of the sends. The interleaving between senders depends on the scheduler. Assert the order of one sender. Never assert the interleaving between senders.

06_15_mpsc_basics.rs prints:

collected 12 records from 3 producers
per-producer FIFO order verified
total message bytes: 192
recv got: single shot

All assertions passed.

sync_channel: Bounds and Backpressure

sync_channel(bound) sets a limit on the buffer of messages in transit. When the buffer is full, send blocks until the consumer receives a message. This block is backpressure. Because of backpressure, the memory use of a bounded channel does not grow when a fast producer supplies a slow consumer:

let (tx, rx) = mpsc::sync_channel::<u32>(2);   // the buffer holds 2 messages
tx.send(1).unwrap();          // buffered
tx.send(2).unwrap();          // buffered, and the buffer is now full
match tx.try_send(3) {        // does not block: it returns an error immediately
    Err(TrySendError::Full(rejected)) => assert_eq!(rejected, 3),  // Full returns the value
    _ => unreachable!(),      // rx is alive and the buffer is full
}

try_send returns the rejected value in the error. Nothing disappears silently. Its second failure mode is TrySendError::Disconnected(value): the receiver is gone.

The rendezvous channel

sync_channel(0) has no buffer. Each send blocks until a recv is in progress at the same time. Thus the message transfer is a synchronization point. It is a handshake between two threads that needs no other mechanism:

let (tx_ready, rx_ready) = mpsc::sync_channel::<&str>(0);   // bound 0: no buffer
// The worker thread owns tx_ready, and the main thread owns rx_ready.
// worker:  tx_ready.send("initialized")   blocks until main calls recv
// main:    rx_ready.recv()                returns Ok("initialized")

How to select a bound:

0: a handshake in which the two threads move in step.
A small bound (approximately the CPU count): the buffer absorbs bursts, and memory use has a limit.
Unbounded channel(): only when a different part of the program already limits the rate of the producer.

06_16_sync_channel.rs prints:

try_send on full buffer: rejected value 3 handed back
backpressured transfer: 20 items, in order
rendezvous handshake: initialized
try_send after receiver dropped: Disconnected(99)

All assertions passed.

Channel Errors: Each Way a Channel Operation Fails

Each error type answers one precise question. Empty and Disconnected need opposite reactions:

API	Error	Meaning	Reaction
`send`	`SendError(value)`	The receiver is gone	Log or requeue the returned value
`try_send`	`TrySendError::Full(value)`	The buffer is full	Apply a backoff policy or a drop policy
`try_send`	`TrySendError::Disconnected(value)`	The receiver is gone	Stop the producer
`try_recv`	`TryRecvError::Empty`	No message at this time, and senders are alive	Poll again later
`try_recv`	`TryRecvError::Disconnected`	All senders are gone, and the queue is empty	Exit the loop
`recv`	`RecvError`	All senders are gone	Exit the loop
`recv_timeout`	`RecvTimeoutError::Timeout`	No message during the timeout	Do housekeeping, then retry
`recv_timeout`	`RecvTimeoutError::Disconnected`	All senders are gone (it returns immediately)	Exit the loop

The shutdown pattern is a result of this design. Do not send a special "STOP" sentinel value. Drop the senders. The for job in rx loop of the consumer then ends. It ends even if a producer panicked, because the Sender of that producer drops during unwinding:

// tx and rx are the two ends of mpsc::channel::<u32>().
// The worker adds the jobs until the channel closes, then returns the sum.
let worker = thread::spawn(move || rx.into_iter().sum::<u32>());
for job in 1..=5 { tx.send(job).unwrap(); }   // sends 1, 2, 3, 4, 5
drop(tx);                                  // no sender remains: this IS the shutdown signal
assert_eq!(worker.join().unwrap(), 15);    // 1 + 2 + 3 + 4 + 5

06_17_channel_errors.rs prints:

try_recv: Empty (senders alive, nothing queued)
try_recv: Disconnected (all senders dropped)
send failed, value recovered: "audit-event-4711"
recv_timeout with queued message: Ok("prompt delivery")
recv_timeout on silent channel: Timeout after ~10ms
recv_timeout on closed channel: Disconnected (immediate)
worker processed sum 15, then shut down cleanly

All assertions passed.

std::sync::mpmc: Cloneable Receivers (Nightly)

The mpmc module (multi-producer, multi-consumer) is still unstable as of Rust 1.99. Its feature is mpmc_channel, and its tracking issue is #126840. The main difference from mpsc is that Receiver is Clone. Thus a pool of workers can receive from one shared queue. The channel delivers each message to exactly one consumer: it is a job queue, not a broadcast.

Figure: mpmc Worker Pool

06_5_channels_mpsc_mpmc_2

#![feature(mpmc_channel)]   // nightly only
use std::sync::mpmc;

let (tx, rx) = mpmc::channel::<u64>();
for _ in 0..3 {
    let rx = rx.clone();                    // impossible with mpsc::Receiver!
    // Each worker takes jobs from the same queue. Two workers never get the same job.
    // `process` is a placeholder for the work on one job.
    thread::spawn(move || for job in rx { process(job); });
}

By design, the API matches the mpsc API: channel, sync_channel, and the same method names. A migration will be mostly a change of the use line. Until mpmc is stable, stable Rust has these alternatives:

Arc<Mutex<mpsc::Receiver<T>>>.
One channel for each worker, with round-robin dispatch.
The crossbeam-channel crate, which was the model for this API.

The example binary is nightly-gated. Run it with this command:

cargo +nightly run -p domain-06-concurrency --features nightly --bin 06_18_mpmc_channel

06_18_mpmc_channel.rs prints this output. The number of jobs for each worker changes between runs:

worker drained 2 jobs (varies)
worker drained 15 jobs (varies)
worker drained 13 jobs (varies)
total jobs: 30, total sum: 465

All assertions passed.

Choosing: Channels or Shared State

Shape of the problem	Tool
Data flows in one direction, and the stages make a pipeline	Channels
Work items go to a pool of workers	mpmc (nightly) or `Arc<Mutex<Receiver>>`
Many threads read and update one long-lived structure	`Arc<Mutex<T>>` or `Arc<RwLock<T>>` (6.2)
One value with single-step updates	Atomics (6.3)
All threads must reach a checkpoint together	`Barrier` (6.7)

Channels are the correct tool when the transfer of ownership agrees with the problem domain: jobs, log records, results. They remove aliasing by construction. The cost is the move of the data and a queue overhead for each message.

Shared state is the correct tool when the threads really use one common structure, for example a cache or a metrics registry. For such a structure, a copy through a channel would be artificial.

You can use the two together. A common architecture uses channels to distribute the work and one Arc<Mutex<_>> for the final aggregate.

Summary

Concept	Key point
`mpsc::channel`	Unbounded. `send` never blocks. `Sender` is `Clone`, and `Receiver` is not.
`send(value)`	Moves the ownership. The program cannot express a data race on the value.
Receiver iteration	`for msg in rx` ends when every sender is gone
Drop the original `tx`	If you do not, the consumer loop never ends
FIFO for each sender	Guaranteed. The interleaving between senders is not guaranteed.
`sync_channel(n)`	Bounded. A full buffer blocks `send`: this is backpressure.
`sync_channel(0)`	Rendezvous: each send waits for a matching recv
`try_send` / `try_recv`	They do not block. A `try_send` error returns the value. A `try_recv` error is `Empty` or `Disconnected`.
`recv_timeout`	Blocks for a limited time. `Disconnected` returns immediately.
Shutdown pattern	Drop the senders. Do not send sentinel messages.
`std::sync::mpmc`	Nightly (1.99): cloneable receivers, and the channel delivers each message exactly once

Code Examples

File	Description
`06_15_mpsc_basics.rs`	Cloned senders, ownership transfer, receiver iteration, FIFO checks
`06_16_sync_channel.rs`	Bounds, `try_send`/`Full`, backpressure, rendezvous channel
`06_17_channel_errors.rs`	All the error types and the drop-the-sender shutdown pattern
`06_18_mpmc_channel.rs`	(nightly) Cloneable receivers: 2 producers and a pool of 3 consumers

6.6 · Once, OnceLock, LazyLock, LazyCell: One-Time Initialization

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::Once, std::sync::OnceLock, std::sync::LazyLock, std::cell::OnceCell, std::cell::LazyCell

Introduction

One of the most common needs in concurrent programs is to initialize a value exactly once, whichever thread asks first. Examples are global configurations, compiled regexes, lookup tables, and loggers. In the past, this needed external crates (lazy_static, once_cell). The standard library now has a family of five types that does all of this work.

This tutorial shows:

Once: runs a side effect exactly once (call_once, is_completed).
OnceLock: stores a value exactly once and is thread-safe (get_or_init, set, get).
LazyLock: a value together with its initializer closure, thread-safe. It has get/get_mut/force_mut (1.94) and From<T> (1.96).
OnceCell and LazyCell: the single-threaded counterparts in std::cell.
The decision matrix with four cells, and how Lazy poisoning differs from Mutex poisoning.

Once: Exactly One Side Effect

Once stores no value. It records only whether its closure ran. Each thread calls call_once, and the closure body runs exactly once. Callers that arrive during the execution block until it finishes. Thus the code after call_once can always rely on a complete setup:

static LOGGER_SETUP: Once = Once::new();

fn init_logging() {
    LOGGER_SETUP.call_once(|| {
        // Install the log sinks and open the files here.
        // This body runs exactly once in the process.
    });
    // At this point the setup is ALWAYS complete, in each thread.
}

is_completed() tells you the state and starts nothing. Once is the correct tool when the thing that must occur once is an effect, not a value that you want to get. Examples are the registration of a callback and the initialization of a C library. When you do want a value, the other types of the family are better.

OnceLock: Exactly One Value

OnceLock<T> is a thread-safe slot that you can write once and read without limit. It is the standard library solution for a static that the program computes at runtime, in safe code:

// Config is a struct with the fields worker_threads: u32 and verbose: bool.
static CONFIG: OnceLock<Config> = OnceLock::new();   // empty at program start

fn config() -> &'static Config {
    // The first call runs the closure and stores the value. Each later call returns it.
    CONFIG.get_or_init(|| Config { worker_threads: 4, verbose: false })
}

If several threads call get_or_init at the same time, one thread runs the closure and the other threads block for a short time. Then all the threads get a reference to the same value. The example binary asserts that the initialization counter is exactly 1 after 6 threads race.

The value can also come from outside, with no closure:

let chosen_port: OnceLock<u16> = OnceLock::new();
assert_eq!(chosen_port.set(8080), Ok(()));      // the first set stores the value
assert_eq!(chosen_port.set(9090), Err(9090));   // too late: set returns the rejected value
assert_eq!(chosen_port.get(), Some(&8080));     // get never initializes

set makes the result of the race explicit for the thread that succeeds and for the threads that fail. get reads the state and does not initialize the value. OnceLock can get its value at a later time, and this is the difference between OnceLock and LazyLock.

06_19_once_and_oncelock.rs prints:

Once closure ran 1 time(s)
OnceLock closure ran 1 time(s)
global config: Config { worker_threads: 4, verbose: false }
port locked in: Some(8080)
get() on empty OnceLock: None

All assertions passed.

LazyLock: The Closure Is Part of the Type

LazyLock<T> contains the value and its initializer. The call sites only deref it. They cannot see that an initialization occurs:

// The closure is part of the static. Nothing runs until the first use.
static KEYWORDS: LazyLock<HashMap<&'static str, u32>> = LazyLock::new(|| {
    HashMap::from([("fn", 1), ("let", 2), ("impl", 3), ("match", 4)])
});

// In any place and in any thread: the first deref builds the map, exactly once.
// `get` here is HashMap::get, which the deref makes available.
assert_eq!(KEYWORDS.get("match"), Some(&4));

The 1.94 and 1.96 additions complete the explicit control. All of these are associated functions, called as LazyLock::f(&value), so that they do not conflict with the methods of the deref target:

LazyLock::get(&l) (1.94) returns Option<&T>. It reads the state and does not force the initialization.
LazyLock::force(&l) (1.80) initializes the value now and returns &T. A deref does this implicitly.
LazyLock::force_mut(&mut l) and get_mut (1.94) let you change the value in place. The &mut proves exclusive access, so no synchronization occurs.
LazyLock::from(value) (1.96) makes a pre-initialized Lazy, and its closure never runs. Use it for tests and dependency injection: give fixture data to a field that is usually lazy.

Poisoning is different from Mutex poisoning. If the initializer panics, the Lazy stays poisoned, and each subsequent access panics too. The first access consumed the FnOnce closure, so the Lazy cannot run it again. Mutex has a PoisonError recovery API (compare Tutorial 6.2), but a Lazy has no recovery API.

Keep Lazy initializers infallible. If the initialization can fail, use OnceLock::get_or_init, which you can retry with a new closure. A pattern in the style of get_or_try_init is a second alternative (the get_or_try_init method itself is still unstable in 1.99).

06_20_lazylock_lazycell.rs prints:

before first use: builds = 0
after 4 racing readers: builds = 1
get -> None before force, Some after
force_mut grew the cache in place: ["seed", "grown"]
LazyLock::from: pre-initialized, no closure involved
LazyCell forced via deref: "--------"
poisoned LazyLock: first access panicked, second panics too

All assertions passed.

OnceCell and LazyCell: The Single-Thread Variants

std::cell::OnceCell and std::cell::LazyCell have the same API as their sync counterparts, but they are !Sync. The compiler rejects code that shares them between threads. In exchange, they do no synchronization: no atomics and no blocking, only a flag check.

The typical use is memoization in a struct field, where the initializer needs &self. A closure that you store at construction cannot borrow the document that it belongs to. Thus the field is a OnceCell, not a LazyCell:

struct Document {
    text: String,
    word_count: OnceCell<usize>,   // computed at most once, on demand
}

impl Document {
    fn word_count(&self) -> usize {
        // The first call counts the words and stores the result. Each later call reads it.
        // The closure borrows self.text, and it needs only &self.
        *self.word_count.get_or_init(|| self.text.split_whitespace().count())
    }
}

LazyCell is the correct type when you do know the closure at construction. An example is a lookup table of a parser that only some inputs need:

let ascii_upper: LazyCell<Vec<char>> = LazyCell::new(|| ('A'..='Z').collect());
// The first use builds the table. If the program never uses it, the closure never runs.

The two types have get and get_mut. OnceCell also has set and take. LazyCell also has force_mut (1.94) and From<T> (1.96). These are the same operations that the sync types have, without the thread safety and without its cost.

The Four-Cell Decision Matrix

Two questions select the type:

Figure: Choosing an Init Type

06_6_once_lazy_init_1

	Value provided later	Closure at construction
Single-threaded (`std::cell`)	`OnceCell<T>`	`LazyCell<T>`
Thread-safe (`std::sync`)	`OnceLock<T>`	`LazyLock<T>`

Practical defaults:

A global that each thread reads: static X: LazyLock<T>.
A global that gets its value from main (CLI arguments, a loaded configuration): static X: OnceLock<T> and set.
A memoized field for each instance: a OnceCell in the struct.
A lazily-built helper that one scope owns: LazyCell.
An effect, not a value: Once.

06_21_cell_matrix.rs has one scenario for each cell of the matrix. It prints:

A OnceCell: word count memoized = 9
B LazyCell: table built on demand, len = 26
C OnceLock: session id = alpha (varies)
D LazyLock: filtered tokens = ["art", "war", "way", "zen"]

All assertions passed.

Summary

Concept	Key point
`Once::call_once`	Exactly one execution. Concurrent callers block until it is complete.
`OnceLock::get_or_init`	One thread computes the value. All threads get the same `&T`.
`OnceLock::set`	Initialization from outside. `Err(value)` returns the rejected value.
`get` (all types)	Reads the state and does not initialize. `None` means that no initialization occurred.
`LazyLock`	The type stores the closure. The call sites only deref.
`force` (1.80), `force_mut` and `get_mut` (1.94)	Explicit initialization through `&`, and mutation with no lock through `&mut`
`From<T>` (1.96)	A pre-initialized `LazyCell` or `LazyLock`. The closure never runs.
Lazy poisoning	A panic in the initializer makes all later accesses panic. There is no recovery, which is different from `Mutex`.
`OnceCell`/`LazyCell`	The same API for one thread. `!Sync`, with no synchronization cost.
The matrix	Two questions (threads, closure at construction) select one of four types. `Once` is for effects.

Code Examples

File	Description
`06_19_once_and_oncelock.rs`	`call_once`, a race on `get_or_init`, `set`/`get`, exactly-once assertions
`06_20_lazylock_lazycell.rs`	Lazy statics, `force_mut`/`get_mut`, `From<T>`, poisoning behavior
`06_21_cell_matrix.rs`	All four quadrants in one program, from a memoized field to a global table

6.7 · Barrier and Rendezvous Synchronization

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::Barrier, std::sync::BarrierWaitResult

Introduction

A lock controls which thread may access the data. A channel controls where the data goes. A Barrier does a third job: it makes sure that all the threads are ready. A barrier is a checkpoint for a fixed party of threads. Each call to wait blocks until the last member of the party arrives. Then the barrier releases all the threads together.

This tutorial shows:

Barrier::new(n) and wait: the rendezvous where each thread waits for all the other threads.
BarrierWaitResult::is_leader: how to select exactly one thread for the work that follows a rendezvous.
Phased algorithms: how to use one barrier again in each round (map, barrier, reduce, barrier, repeat).
Barrier compared with Condvar (6.2) and with channels (6.5): the same rendezvous in three versions, and when to use each one.

Barrier::wait: All Wait for All

Figure: Four Threads Meet at a Barrier

06_7_barrier_rendezvous_1

You create a Barrier for a fixed party size n. Each thread calls wait(). The first n - 1 calls block. The n-th call releases all the threads:

const WORKERS: usize = 4;

// The party size is fixed: WORKERS threads must call `wait`.
let barrier = Arc::new(Barrier::new(WORKERS));
// A shared counter of the workers that completed phase 1. It starts at 0.
let setups_done = Arc::new(AtomicUsize::new(0));

// Each worker thread gets a clone of the two Arcs and runs these lines:
setups_done.fetch_add(1, Ordering::Relaxed);                // phase 1: the setup
barrier.wait();                                             // returns when the 4th worker arrives
assert_eq!(setups_done.load(Ordering::Relaxed), WORKERS);   // phase 2: NEVER fails

That assertion shows the guarantee of a barrier. The OS can schedule the threads in any order, and each phase-1 write is still complete before a thread runs phase-2 code.

wait also synchronizes memory. All the operations of each thread before its wait happen-before all the operations of each thread after its wait. This is the Release/Acquire mechanism of Tutorial 6.4, and the barrier includes it.

A typical use is a load-test harness. Each worker must complete its setup before a worker starts to send requests. If not, the workers that start early measure a system that has no load.

is_leader: One Thread for the Follow-Up Work

wait returns a BarrierWaitResult. For each rendezvous, is_leader() returns true for exactly one of the threads that wait. That thread is the leader. The leader is not necessarily the first thread or the last thread that arrives:

// Each worker runs this code. `barrier` is the Barrier from the previous snippet.
let result = barrier.wait();   // result: BarrierWaitResult
if result.is_leader() {
    // Only one thread for each rendezvous runs this block.
    // Do the work that must occur one time for each rendezvous here:
    // aggregate the results, print the banner of the round, reset shared scratch state.
}

Some work must occur only one time for each rendezvous. is_leader selects the thread that does this work. You do not need to elect a coordinator thread, and the threads do not race on a flag. The example binary 06_22_barrier_rendezvous counts the leaders of one rendezvous of 4 threads. It asserts that the count is exactly 1.

Two notes about usage:

A Barrier::new(1) releases immediately, and its only member is always the leader. This is useful in tests and in degenerate configurations.
After the barrier releases a full party, it resets automatically. It can then serve the next rendezvous of the same size. You do not create a new barrier. Phased algorithms depend on this property.

Phased Algorithms: One Barrier, Many Rounds

Iterative parallel computations (simulation steps, iterative solvers, map-reduce rounds) repeat one sequence:

Compute in parallel.
Synchronize.
Combine the results.
Repeat.

The standard structure uses two waits per round:

// barrier: Barrier for WORKERS (4) threads.
// partials: Vec<AtomicUsize>, one slot for each worker.
// round_totals: Mutex<Vec<usize>>, one total for each round.
// Each worker runs this loop. worker_id is 0, 1, 2, or 3. ROUNDS is 5.
for round in 0..ROUNDS {
    // MAP: each worker writes only its own slot, so there is no contention.
    let contribution = (worker_id + 1) * (round + 1);
    partials[worker_id].store(contribution, Ordering::Relaxed);

    // Barrier 1: each slot holds the value of THIS round before a thread reduces.
    let map_done = barrier.wait();

    // REDUCE: only the leader reads all the slots and records the total.
    if map_done.is_leader() {
        let total: usize = partials.iter().map(|p| p.load(Ordering::Relaxed)).sum();
        round_totals.lock().unwrap().push(total);   // round 0 pushes 1 + 2 + 3 + 4 = 10
    }

    // Barrier 2: no worker starts round R+1 (and overwrites a slot)
    // while the leader still reads the slots.
    barrier.wait();
}

If you omit barrier 1, the leader can add partials of round R to partials of round R-1. If you omit barrier 2, a fast worker can overwrite a slot during the reduction. With the two barriers, the total of each round is deterministic. The example asserts [10, 20, 30, 40, 50] for the five rounds, in each run.

06_23_barrier_phases.rs prints:

per-round reduced totals: [10, 20, 30, 40, 50]
one Barrier, 10 rendezvous, zero re-allocation

All assertions passed.

The Same Rendezvous, Three Ways

The example binary 06_24_barrier_vs_channels solves one problem in three versions. The problem is: all the workers must complete phase A before a worker starts phase B.

Approach	What the code needs	Properties
`Barrier`	1 shared object, and one `wait()` call in each worker	Symmetric and reusable. The code shows the intent.
Channels	N "done" sends to a coordinator, and N "go" channels (one for each worker) for the reply	Asymmetric. The broadcast to the workers needs one channel for each worker.
`Mutex` + `Condvar` latch	A manual count, `notify_all`, and a `wait_while` predicate	`Barrier` does this internally. A subtle error is easy to make.

General rules:

Barrier: use it for N symmetric peers that all wait for each other, possibly in repeated rounds. If you plan to make a "countdown latch" from a Condvar, a barrier is probably the latch that you need.
Channels (6.5): use them when data moves between threads, or when the roles are asymmetric (coordinator and worker, producer and consumer). A sync_channel(0) rendezvous is the special case for two parties.
Condvar (6.2): use it when the release condition is a custom predicate that no standard primitive expresses ("queue non-empty and not paused").

06_24_barrier_vs_channels.rs prints:

barrier version: ok (1 sync object, 1 line per worker)
channel version: ok (N+1 channels + a coordinator loop)
condvar latch version: ok (manual counting + wakeup logic)

All assertions passed.

The Thread Count: The One Way to Misuse a Barrier

The party size is a contract, not a hint. If fewer than n threads call wait, each thread that called wait blocks forever. This is a barrier deadlock: there is no error, no timeout, and no poisoning. In practice, this deadlock has three causes:

Two numbers that do not agree. The barrier is Barrier::new(4), but an edit changed the spawn loop to 0..5. The wait of the fifth thread starts a new rendezvous that never completes. Derive the two numbers from one const WORKERS.
A worker that exits early. A ? or return path does not reach the wait. Structure each worker so that each exit path reaches the barrier or stops the full party.
A worker that panics. A Mutex poisons, but a Barrier does not. The thread that panicked never arrives, and the other threads wait forever. If a worker can panic in the middle of a phase, a protocol that uses channels handles the failure better. With channels, the other threads can detect a dropped Sender (Tutorial 6.5).

The examples in this domain prevent the three causes, and production code should do the same:

They use one shared constant.
Each worker has only one path, and that path goes directly through the wait.
They join the threads at the end, so the parent thread sees a panic.

Summary

Concept	Key point
`Barrier::new(n)`	A checkpoint for a fixed party of `n` threads
`wait()`	Blocks until the `n`-th thread arrives. Then the barrier releases all the threads together.
Memory effect	The operations before each `wait` happen-before the operations after all the waits
`BarrierWaitResult::is_leader`	`true` for exactly one thread in each rendezvous. Use it for work that occurs one time in each round.
Reuse	The barrier resets automatically after each full release. A phased loop needs only one barrier.
Two waits per round	One wait after the map phase makes the reduction safe. One wait after the reduce phase makes the overwrite safe.
Compared with channels	Use a barrier when symmetric peers all wait for each other. Use channels when data moves between threads.
Compared with `Condvar`	A barrier is a special case of a latch. It is ready-made and hard to misuse.
Deadlock risk	The party size must be equal to the actual number of threads that wait. Share one constant.
No poisoning	A worker that panicked never arrives, and the other workers wait forever. Join the threads and propagate the panic.

Code Examples

File	Description
`06_22_barrier_rendezvous.rs`	`wait` semantics, a deterministic phase assertion, and exactly one leader
`06_23_barrier_phases.rs`	Map-reduce rounds: one barrier, two waits per round, and exact totals
`06_24_barrier_vs_channels.rs`	The same rendezvous with a `Barrier`, with channels, and with a `Condvar` latch

6.8 · Send, Sync, and the Marker Trait Contracts

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::marker::Send, std::marker::Sync, std::marker::Unpin, std::marker::PhantomData, std::marker::PhantomPinned

Introduction

Two marker traits with no methods are the base of all the topics in this domain. They explain the signature of thread::spawn, why Arc exists together with Rc, and why a Mutex makes data shareable. Send and Sync are the implementation of the "fearless concurrency" of Rust. Thread safety is a type property: the compiler derives it automatically and checks it at compile time.

This tutorial shows:

The contracts: what Send and Sync promise, and why they are unsafe auto traits.
What breaks Send (Rc<T>), what breaks Sync (Cell<T>, RefCell<T>), and the substitution that repairs each one.
How Arc<Mutex<T>> satisfies the two contracts, layer by layer.
How to opt out with PhantomData and opt in with unsafe impl, and the proof that each one requires.
How Unpin and PhantomPinned relate to the other marker traits.

Two Contracts, Zero Methods

Send: ownership of the value may move to another thread.
Sync: threads may share a &T at the same time.

One rule connects the two traits: T: Sync if and only if &T: Send. To share a value is to send references to it. The two traits are two views of one idea.

The two traits are auto traits: the compiler derives them from the structure of a type. A struct is Send when each field is Send, and Sync when each field is Sync. You write no annotation and no derive:

struct Inventory {          // Send + Sync automatically, because each field is:
    items: Vec<String>,     //   Vec<String> is Send + Sync
    total: usize,           //   usize is Send + Sync
}

The two traits are also unsafe traits: a manual implementation is a promise that the compiler cannot check. This divides the work. The compiler derives the traits from structure for 99% of the types. A person takes written responsibility for the other 1% (slide 6).

The examples use these compile-time probes:

// A call to a probe compiles only if the type `T` satisfies the bound.
// The bodies are empty, so the probes do no work at run time.
fn assert_send<T: Send>() {}
fn assert_sync<T: Sync>() {}

assert_send::<String>();          // compiles: String is Send
// assert_send::<Rc<String>>();   // does not compile: Rc<String> is not Send
// error[E0277]: `Rc<String>` cannot be sent between threads safely

What Breaks Send: `Rc<T>`

The reference count of an Rc is a plain integer that is not atomic. That is the performance advantage of Rc. If two threads clone or drop the same allocation, they race on the count. The race can have these results:

The count loses increments.
The program frees the value while a thread still uses it.
The value leaks forever.

Thus Rc is neither Send nor Sync, and the compiler rejects the code at the call site:

let rc = Rc::new(String::from("local"));   // rc: Rc<String>
// The closure captures `rc` and moves it to the new thread.
// This line does not compile:
// thread::spawn(move || rc.len());
// error[E0277]: `Rc<String>` cannot be sent between threads safely
//   note: required because it's used within this closure
//   note: required by a bound in `spawn`

Read that error as design feedback, not as an obstacle: it names the exact type that you must replace. The substitution is Arc. It has an atomic count, the same API, and a small cost:

let arc = Arc::new(String::from("shared"));   // arc: Arc<String>
let for_worker = Arc::clone(&arc);            // a second handle to the same String
thread::spawn(move || for_worker.len());      // compiles and is correct: Arc<String> is Send

This is the effect of the bound in the signature of thread::spawn: F: Send + 'static. The closure moves to the new thread, so each value that it captures must be Send. One captured Rc makes the full closure not Send. Auto traits propagate through closures exactly as they propagate through structs.

What Breaks Sync: Cell and RefCell

Cell::set writes through &self with no synchronization. If two threads do that at the same time, the result is a data race by definition. Note the asymmetry: Cell<T: Send> is Send, because a move of the full cell to one other thread is harmless. A Cell only refuses to be shared:

let shared = Arc::new(Cell::new(0_u64));   // shared: Arc<Cell<u64>>
let clone = Arc::clone(&shared);           // a second handle to the same Cell
// The closure moves `clone` to a new thread, so two threads share one Cell.
// This line does not compile:
// thread::spawn(move || clone.set(1));
// error[E0277]: `Cell<u64>` cannot be shared between threads safely
//   note: required for `Arc<Cell<u64>>` to implement `Send`

Arc<T> is Send only when T: Send + Sync. The only job of an Arc is to share its payload, so it cannot hold a payload that is not shareable.

RefCell fails for the same reason. Its borrow counter is not atomic, so its runtime borrow checks (Tutorial 7.3) are sound only on one thread.

Figure: The Substitution Map

06_8_send_sync_markers_1

The compiler enforces each row of the figure. The types on the left side cannot cross a thread boundary. Thus the upgrade is never optional, and you cannot forget it.

How `Arc<Mutex<T>>` Satisfies Both

Examine this common combination from the inner layer to the outer layer:

T: Send: the data can move between threads. For example, the thread that holds the last handle drops the data.
Mutex<T>: Send + Sync when T: Send: the lock supplies the exclusion that T does not have. Even a RefCell becomes shareable in a Mutex. Only one thread can access it at a time, so no race on its non-atomic internals can occur.
Arc<Mutex<T>>: Send + Sync: Arc requires a payload that is Send + Sync, and the Mutex layer guarantees that. Arc adds shared ownership with an atomic count.

Each layer adds exactly one capability: Mutex makes the data shareable, and Arc makes the data shared. The bounds are conditional, so the compiler checks the full stack for each instantiation. Arc<Mutex<Rc<u8>>> still fails, because Rc<u8> is not Send. A clone of the Arc on another thread could drop the Rc<u8> on that thread.

One more fact is useful. MutexGuard is deliberately !Send, because some OS mutexes require that the thread that locks them also unlocks them. But MutexGuard is Sync when T: Sync. Scoped threads can borrow a guard, but the thread that locked the mutex must drop the guard.

PhantomData to Opt Out, unsafe impl to Opt In

To opt out. A usize handle can represent a session of a C library. To the compiler, the handle looks Send + Sync, because its structure is only an integer. If the threading rules of the C library do not agree, a zero-sized PhantomData field encodes the policy:

struct FfiSession {
    raw_handle: usize,   // represents a C pointer or a C token
    // `*const ()` is neither Send nor Sync, so this field removes the two
    // auto traits from the struct. The field has a size of zero bytes.
    _not_thread_safe: PhantomData<*const ()>,
}
// assert_send::<FfiSession>();   // does not compile
// error[E0277]: `*const ()` cannot be sent between threads safely

Each downstream user now gets a compile error, not a runtime bug that is hard to reproduce. PhantomData<Rc<()>> also works. The raw-pointer form is the convention for an FFI-related type that is not thread-safe.

To opt in. Raw pointers are !Send and !Sync because of suspicion, not because of necessity: the compiler cannot know what they point to. When you can prove exclusive ownership, you may override the compiler:

// `ptr` is the only pointer to a heap allocation of `len` bytes.
// The raw pointer field makes the struct !Send and !Sync by default.
struct ExclusiveBuffer { ptr: *mut u8, len: usize }

// SAFETY: ExclusiveBuffer owns its allocation exclusively. No other pointer
// to the allocation exists, and the API never gives one to a caller. Thus a
// move of the struct to another thread moves the buffer as it moves any owned
// value. Nothing stays on the first thread that can race with the buffer.
unsafe impl Send for ExclusiveBuffer {}

The requirement for an unsafe impl is a written argument about aliasing and synchronization. "The tests pass" is not such an argument. If the struct ever gives a second pointer to other code, the impl becomes unsound, and the compiler never gives a warning.

Note that the example claims only Send, not Sync. Claim the minimum that you need. Prefer to compose std types that are already thread-safe. Write manual impls only at FFI boundaries.

06_27_phantom_unsafe_impl.rs prints:

FfiSession usable locally; !Send enforced at compile time
ExclusiveBuffer summed on another thread: 192

All assertions passed.

Unpin and PhantomPinned: The Other Marker Pair

Unpin is another auto marker trait, with a different contract. Send and Sync control where a value may go (threads). Unpin controls whether a value may move after you pin it (addresses). Almost all types are Unpin. For example, a move of a String or a Vec never invalidates it:

// A compile-time probe of the same form as `assert_send`.
fn assert_unpin<T: Unpin>() {}

assert_unpin::<String>();             // compiles: String is Unpin
// FfiSession is the !Send + !Sync struct from the previous section. It is Unpin.
assert_unpin::<Vec<FfiSession>>();    // compiles

Self-referential types (the state machines of an async fn, intrusive linked nodes) store pointers into themselves. A move of such a value corrupts those pointers. These types opt out with PhantomPinned, exactly as FfiSession opts out of Send and Sync with PhantomData<*const ()>:

// PhantomPinned is a zero-sized type that is not Unpin.
// Thus a struct that contains it is not Unpin.
struct SelfReferential { _pin: PhantomPinned }
// assert_unpin::<SelfReferential>();   // does not compile
// error[E0277]: `PhantomPinned` cannot be unpinned

For now, the important point is the symmetry: a zero-sized marker field changes an invariant into a compile-time property. Domain 15 continues with Pin and the async mechanisms that use it.

Summary

Concept	Key point
`Send`	Ownership may move to another thread
`Sync`	Threads may share a `&T`. `T: Sync` ⇔ `&T: Send`.
Auto traits	The compiler derives them from structure. One field without the property removes it from the full type.
`unsafe` traits	A manual impl is a promise that a person checks, not the compiler
`Rc` breaks `Send`	The reference count is not atomic. Use `Arc`.
`Cell`/`RefCell` break `Sync`	The interior mutability is not atomic. Use `Atomic*`, `Mutex`, or `RwLock`.
`Arc<Mutex<T>>`	`Mutex` makes the data shareable, and `Arc` makes the data shared. The combination needs only `T: Send`.
`MutexGuard`	`!Send` (the thread that locks must also unlock), but `Sync` when `T: Sync`
`PhantomData<*const ()>`	A zero-sized field that opts a handle type out of `Send` and `Sync`
`unsafe impl Send/Sync`	Requires a proof about aliasing and synchronization. Claim the minimum.
`Unpin` / `PhantomPinned`	The same marker mechanism controls whether a pinned value may move (Domain 15)

Code Examples

File	Description
`06_25_send_sync_auto.rs`	Compile-time probes, automatic propagation, the relation between `&T` and `Sync`, and `Arc<Mutex<T>>`
`06_26_not_send_not_sync.rs`	The failures of `Rc`, `Cell`, and `RefCell` with the exact errors, and the substitution table
`06_27_phantom_unsafe_impl.rs`	An opt-out with `PhantomData`, a justified `unsafe impl Send`, and `PhantomPinned`

7.1 · Box<T>: The Simplest Heap Allocation

Domain 7 — Smart Pointers and Heap Allocation Duration: ~15 minutes Library components: std::boxed::Box

Introduction

Box<T> is the simplest smart pointer in Rust. It is an owning pointer to one value on the heap. It has no reference count, no locks, and no runtime checks. It allocates when you create it, and it deallocates when it drops.

This simplicity is the reason why Box is very common. Use Box when you only need to put a value on the heap and own it. Each other smart pointer in this domain is easiest to understand as Box plus one more capability.

This tutorial shows:

Box::new and placement: the program builds the value on the stack, then moves it to the heap.
The Deref and DerefMut implementations that make a box transparent, and the deref-move (*b) that consumes the box. Box::into_inner stays nightly-only.
Boxing of recursive types: the indirection that gives an enum or a linked structure a finite size.
Boxing of trait objects: Box<dyn Trait> and Box<dyn Fn>, and thin pointers compared with fat pointers.
Deliberate exits from RAII. Box::leak is for values that live as long as the process. The Box::into_raw / Box::from_raw round trip is for API boundaries that pass raw pointers. Rust 1.99 adds the NonNull variants.

Box::new and Placement Semantics

Box::new(v) allocates space on the heap and moves v into that space. The sequence is important. First, the program evaluates the expression that makes v, on the stack. Then the program copies the completed value to the heap. Rust does not guarantee in-place construction (placement new). For very large values, the temporary copy on the stack is real, and it is occasionally important.

After construction, the box itself is one machine word on the stack: the pointer. The ownership semantics are the same as for any other value. When the Box goes out of scope, its Drop implementation frees the heap allocation. A move of the Box moves only the pointer, never the payload.

Figure: Box<T> — Stack Handle, Heap Payload

Box stack and heap layout

// The 512-byte array goes to the heap. `on_heap` is the pointer on the stack.
let on_heap: Box<[u8; 512]> = Box::new([0u8; 512]);
assert_eq!(size_of::<Box<[u8; 512]>>(), size_of::<usize>()); // handle: 1 word

let big = Box::new([7u8; 4096]);              // Box<[u8; 4096]>
let payload_addr: *const u8 = big.as_ptr();   // address of the payload on the heap
let moved = big;                              // moves 8 bytes, not 4096
assert!(std::ptr::eq(payload_addr, moved.as_ptr()));   // the payload did not move

Remember these two facts about size:

Thin vs. fat: Box<T> for a sized T is one word. A box of an unsized payload needs a second word of metadata: a length for Box<[T]>, or a vtable pointer for Box<dyn Trait>.
Niche optimization: a box is never null. Thus Option<Box<T>> uses the null bit pattern for None and has the size of one pointer. For this reason, Option<Box<Node>> is the idiomatic nullable owning pointer in Rust.

use std::fmt::Display;

let word = size_of::<usize>();                        // 8 bytes on a 64-bit target
assert_eq!(size_of::<Box<i32>>(), word);              // thin
assert_eq!(size_of::<Box<[i32]>>(), 2 * word);        // ptr + len
assert_eq!(size_of::<Box<dyn Display>>(), 2 * word);  // ptr + vtable
assert_eq!(size_of::<Option<Box<i32>>>(), word);      // niche: null = None

Deref, DerefMut, and Moving Out

Box<T> implements Deref<Target = T> and DerefMut, so a box is transparent in almost every position:

*b reads or writes the payload.
Method calls auto-deref to the methods of T.
&Box<String> coerces to &str when a function needs a &str.

The same mechanism gives Vec<T> the slice methods (tutorial 4.1).

let mut counter = Box::new(41_i32);
*counter += 1;                       // DerefMut: writes through the box
assert_eq!(*counter, 42);            // Deref: reads through the box

let message = Box::new(String::from("hello, heap"));   // Box<String>
assert_eq!(message.len(), 11);       // String::len via auto-deref

fn shout(s: &str) -> String { s.to_uppercase() }
// Deref coercion: &Box<String> → &String → &str
assert_eq!(shout(&message), "HELLO, HEAP");

Deref-move: `*b` in value position

Box is the only smart pointer that lets you move the payload out through a dereference. let v = *b; consumes the box, frees the heap allocation, and gives you the payload by value. The compiler has built-in support for this operation, and the Deref trait cannot express it. For this reason, a move through *rc on an Rc does not compile.

let boxed_config = Box::new(String::from("mode=fast"));   // Box<String>
let config: String = *boxed_config;   // deref-move: consumes boxed_config
assert_eq!(config, "mode=fast");
// A use of boxed_config here is error[E0382]: the box no longer exists.

The explicit, named form is Box::into_inner(b). It is still nightly-only (feature box_into_inner, tracking issue #80437). It exists because a named function is clearer in generic code, and because *b is easy to read incorrectly as a plain deref. Example 07_05_box_into_inner.rs shows it behind the nightly feature gate of this repository.

07_01_box_fundamentals.rs prints (the sizes are for a 64-bit target):

size_of::<[u8; 512]>()      = 512 bytes (the payload)
size_of::<Box<[u8; 512]>>() = 8 bytes (the handle)

Box<i32>         = 8 bytes (thin)
Box<[i32]>       = 16 bytes (ptr + len)
Box<dyn Display> = 16 bytes (ptr + vtable)
Option<Box<i32>> = 8 bytes (niche: null means None)

shout(&message) = "HELLO, HEAP"
payload stayed put across the move: true
moved out of the box: "mode=fast"

All assertions passed.

Boxing Recursive Types

The compiler cannot give a layout to a type that contains itself directly, because the size would be infinite:

// This enum does not compile.
enum Expr {
    Num(i64),
    Add(Expr, Expr),   // error[E0072]: recursive type `Expr` has infinite size
}

The suggestion of the compiler is the solution: insert some indirection. A Box<Expr> is always one word, whatever it points to. Thus each variant has a fixed size, and the compiler can give the enum a layout. The recursion then occurs on the heap.

Figure: Expression Tree — Every Edge Is a Box

07_1_box_heap_allocation_1

enum Expr {
    Num(i64),
    Add(Box<Expr>, Box<Expr>),   // each child is one pointer to a node on the heap
    Sub(Box<Expr>, Box<Expr>),
    Mul(Box<Expr>, Box<Expr>),
    Neg(Box<Expr>),
}

fn eval(expr: &Expr) -> i64 {
    match expr {
        Expr::Num(n) => *n,
        // `a` and `b` are &Box<Expr>. Deref coercion passes them to eval as &Expr.
        Expr::Add(a, b) => eval(a) + eval(b),
        Expr::Sub(a, b) => eval(a) - eval(b),
        Expr::Mul(a, b) => eval(a) * eval(b),
        Expr::Neg(e) => -eval(e),
    }
}

// num, add, sub, mul, and neg are helper functions. Each one returns a Box<Expr>.
// For example: fn num(n: i64) -> Box<Expr> { Box::new(Expr::Num(n)) }
// The tree for (2 + 3) * -(4 - 6):
let expr = mul(add(num(2), num(3)), neg(sub(num(4), num(6))));
assert_eq!(eval(&expr), 10);   // 5 * 2

The evaluation goes through the boxes with plain & borrows. A match on &Expr gives a reference to each boxed child, and that reference auto-derefs to &Expr.

The same indirection also makes linked structures possible. With Option<Box<Node>> as the next pointer, you can make a singly linked stack. Its push and pop operations are an Option::take and a new link.

07_02_box_recursive_types.rs prints:

expr  = ((2 + 3) * -(4 - 6))
value = 10
size_of::<Expr>() = 24 bytes

stack popped in LIFO order: 30, 20, 10

All assertions passed.

Box<dyn Trait>: Owned Trait Objects

dyn Trait is unsized, because different implementors have different sizes. Thus a dyn Trait value must be behind a pointer. Box<dyn Trait> is the owning pointer: a fat pointer of two words (data pointer and vtable pointer). With it, you can store values of different types together, and you can select those types at runtime.

trait Stage {
    fn name(&self) -> &'static str;
    fn apply(&self, input: String) -> String;
}

// Trim, Redact, and Uppercase are three structs that implement Stage.
// A factory: the `spec` string selects the concrete type at runtime.
fn stage_for(spec: &str) -> Option<Box<dyn Stage>> {
    match spec {
        "trim"   => Some(Box::new(Trim)),   // Box<Trim> coerces to Box<dyn Stage>
        "redact" => Some(Box::new(Redact { secret: "1234" })),
        "upper"  => Some(Box::new(Uppercase)),
        _ => None,                          // an unknown name gives no stage
    }
}

// A heterogeneous pipeline: three different types, one element type.
// specs: [&str; 3] = ["trim", "redact", "upper"]
let pipeline: Vec<Box<dyn Stage>> =
    specs.iter().filter_map(|spec| stage_for(spec)).collect();
// text: String, starts as "  the launch code is 1234  "
for stage in &pipeline {
    text = stage.apply(text);   // dynamic dispatch through the vtable
}
// text is now "THE LAUNCH CODE IS [REDACTED]"

Each call goes through the vtable at runtime. That indirection is the cost of dynamic dispatch. Type erasure is its benefit.

The same technique applies to closures, which have unique types that you cannot name. Box<dyn Fn(i64) -> i64> lets you store closures in fields and collections. Box<dyn Error> (tutorial 2.2) applies this pattern to error handling. Domain 24 examines trait objects in full.

07_03_box_trait_objects.rs prints:

input: "  the launch code is 1234  "
after   trim: "the launch code is 1234"
after redact: "the launch code is [REDACTED]"
after  upper: "THE LAUNCH CODE IS [REDACTED]"

size_of::<Box<dyn Stage>>() = 16 bytes (ptr + vtable)

start: 3
after double: 6
after square: 36
after negate: -36

All assertions passed.

Box::leak and the Raw Pointer Round Trip

Sometimes you must deliberately go out of RAII. Box gives two ways to do this. Each one transfers the responsibility for the allocation, not the allocation itself.

Box::leak — a deliberate, one-time leak

Box::leak(b) consumes the box and returns &'static mut T. The destructor never runs, so the reference is valid for the remainder of the program. This is the standard method to make a &'static reference from a value that the program computes at runtime. For example, CLI tools leak their parsed arguments, and loggers leak their configuration. The OS reclaims the memory when the process exits.

General rule: leak only values that must live as long as the process. Box::leak in a loop is an ordinary memory leak, which is a defect.

fn make_static_config(region: &str, verbose: bool) -> &'static str {
    let config = format!("region={region};verbose={verbose}");   // a String made at runtime
    // String → Box<str> → &'static mut str, which coerces to the &'static str return type.
    Box::leak(config.into_boxed_str())
}
// make_static_config("eu-west-1", true) returns "region=eu-west-1;verbose=true".

Box::into_raw / Box::from_raw — ownership as a raw pointer

Box::into_raw(b) disables the destructor and gives you the raw pointer to the allocation. Box::from_raw(ptr) is the inverse: it attaches ownership and the destructor again. With this pair, owned values go across API boundaries that pass raw pointers. C FFI handles are the primary example (domain 20).

The safety contract of the round trip:

The pointer must come from Box::into_raw (same layout, same allocator).
You must call Box::from_raw at most once for each into_raw. A second call is a double free.
Between the two calls, no other code may free the pointer. No reference into the allocation may stay alive across the reconstruction.

struct Session { user: String, hits: u32 }

let session = Box::new(Session { user: String::from("ada"), hits: 0 });
let raw: *mut Session = Box::into_raw(session);   // ownership → raw pointer
// No Box owns the allocation now. This code is responsible for it.

// SAFETY: `raw` comes from Box::into_raw, and no code freed it. No other
// pointer or reference to the allocation exists.
unsafe {
    (*raw).hits += 1;
    (*raw).hits += 1;
}

// SAFETY: `raw` comes from Box::into_raw. This is the only from_raw call,
// and no code uses `raw` after it.
let session: Box<Session> = unsafe { Box::from_raw(raw) };
assert_eq!(session.hits, 2);   // `session` drops normally: no leak, no double free

07_04_box_leak_raw.rs prints:

static config: region=eu-west-1;verbose=true
session round-tripped through a raw pointer: user=ada, hits=2

All assertions passed.

Rust 1.99 adds the same round trip with a NonNull<T> pointer: Box::into_non_null and Box::from_non_null. A Box pointer is never null. *mut T does not record that fact, and NonNull<T> does. The safety contract is the same as for into_raw / from_raw.

Use this pair when you store the pointer. For example, a list can store each link as an Option<NonNull<Node>>, which is one pointer wide (tutorial 16.3):

use std::ptr::NonNull;

let boxed: Box<String> = Box::new(String::from("heap value"));
let ptr: NonNull<String> = Box::into_non_null(boxed);   // ownership → NonNull pointer
// No Box owns the allocation now. The String stays alive.

// SAFETY: `ptr` comes from Box::into_non_null, and this is the only conversion back.
let restored: Box<String> = unsafe { Box::from_non_null(ptr) };
assert_eq!(*restored, "heap value");                    // `restored` drops normally

07_15_box_non_null.rs also makes a two-node list with these functions, and then frees it. It prints:

round trip: heap value
Option<NonNull<Node>> = 8 bytes, *mut Node = 8 bytes
sum of the linked nodes: 3
freed 2 nodes

All assertions passed.

Summary

Concept	Key point
`Box::new(v)`	Allocates on the heap and moves `v` in. The program builds `v` on the stack first.
Layout	One word (thin pointer) for a sized `T`. Two words for `Box<[T]>` and `Box<dyn Trait>`.
`Option<Box<T>>`	Has the size of one pointer because of the niche optimization. It is the idiomatic nullable owning pointer.
`Deref` / `DerefMut`	The box is transparent: `*b`, method calls with auto-deref, coercion to `&T`.
Deref-move `*b`	Consumes the box and returns the payload. Only `Box` has this compiler support.
`Box::into_inner`	Named form of deref-move. It is still nightly (`box_into_inner`).
Recursive types	`Box` gives recursive enums and linked structures a finite size.
`Box<dyn Trait>`	Owned trait object. The fat pointer is a data pointer and a vtable pointer. Dispatch is at runtime.
`Box<dyn Fn…>`	Erases closure types that you cannot name. You can then store closures in fields and collections.
`Box::leak`	Exchanges the destructor for `&'static mut T`. Use it only for values that live as long as the process.
`Box::into_raw` / `from_raw`	Ownership across API boundaries that pass raw pointers. Call `from_raw` exactly one time for each `into_raw`.
`Box::into_non_null` / `from_non_null` (1.99)	The same round trip with `NonNull<T>`. The type records that the pointer is not null.

Code Examples

File	Description
`07_01_box_fundamentals.rs`	`Box::new`, `Deref`/`DerefMut`, deref-move, thin vs. fat pointers, niche optimization
`07_02_box_recursive_types.rs`	Expression tree and linked stack: `Box` gives recursive types a finite size
`07_03_box_trait_objects.rs`	`Box<dyn Trait>` pipeline, factory function, `Box<dyn Fn>` closures
`07_04_box_leak_raw.rs`	`Box::leak` for a `&'static` config, and the `into_raw`/`from_raw` round trip with safety notes
`07_05_box_into_inner.rs`	`Box::into_inner` vs. deref-move (nightly, `--features nightly`)
`07_15_box_non_null.rs`	`Box::into_non_null` / `Box::from_non_null` (1.99): ownership round trip with `NonNull`, and a two-node list with `Option<NonNull<Node>>` links

7.2 · Rc<T> and Arc<T>: Reference Counting Strategies

Domain 7 — Smart Pointers and Heap Allocation Duration: ~15 minutes Library components: std::rc::Rc, std::rc::Weak, std::sync::Arc, std::sync::Weak

Introduction

Box<T> (tutorial 7.1) gives a value exactly one owner. But some data has several owners. Examples are a configuration that each subsystem shares, and a node that more than one parent can reach. You cannot predict which of the owners lives longest.

Reference counting is the solution. The value is on the heap, adjacent to a counter. Each new handle increments the counter. When the last handle drops, the value drops too.

The standard library has two reference-counted pointers. They differ in exactly one property:

Rc<T> has plain integer counters. It is fast, and it is for one thread only (!Send).
Arc<T> has atomic counters. It has a slightly higher cost, and it is safe to share across threads.

This tutorial shows:

Rc::clone and strong_count.
How to take the value back with try_unwrap.
How to break cycles with Weak<T>.
Self-referential construction with Rc::new_cyclic.
Clone-on-write mutation with make_mut.
Why the standard library has both Rc and Arc.

Rc<T>: Shared Ownership

An Rc<T> handle points to one heap block that holds three items: the strong count, the weak count, and the value. Rc::clone copies the pointer and increments the strong count. It does not touch the value. When the strong count becomes zero, Rc drops the value.

Figure: One Allocation, Many Handles — Rc Heap Layout

Rc heap layout with strong and weak counts

use std::rc::Rc;

// Config is a struct with two fields: theme (String) and retries (u32).
let config = Rc::new(Config { theme: String::from("dark"), retries: 3 });   // Rc<Config>
assert_eq!(Rc::strong_count(&config), 1);

// Logger and Fetcher are structs with one field each: config: Rc<Config>.
let logger  = Logger  { config: Rc::clone(&config) };   // count: 1 → 2
let fetcher = Fetcher { config: Rc::clone(&config) };   // count: 2 → 3
assert_eq!(Rc::strong_count(&config), 3);

assert!(Rc::ptr_eq(&config, &logger.config));   // same allocation
assert_eq!(logger.config.retries, 3);           // Deref, like Box

Remember these two rules:

Write Rc::clone(&x), not x.clone(). The associated-function syntax tells the reader that the call is a cheap copy of the handle, not a deep clone. The same convention applies to Rc::strong_count, Rc::ptr_eq, and similar functions. The standard library defines them as associated functions, so that they can never conflict with methods of T.
Rc has no DerefMut. Shared data is immutable through Rc. logger.config.retries = 5 fails with error[E0594]: cannot assign to data in an Rc. The two alternatives are Rc::make_mut (slide 6) and RefCell (tutorial 7.3).

Reclaiming the Value: try_unwrap

Shared ownership does not have to be permanent. Rc::try_unwrap(rc) moves the value out if the strong count is exactly 1. If not, it returns the Rc in the Err variant, so you lose nothing:

// `config`, `logger`, and `fetcher` are from the previous snippet. The count is 3.
drop(logger);                                    // count: 3 → 2
// try_unwrap fails and returns the same Rc in Err. The pattern binds it to `config` again.
let Err(config) = Rc::try_unwrap(config) else {  // fetcher still shares it
    unreachable!()
};

drop(fetcher);                                   // count: 2 → 1
// The count is 1, so try_unwrap returns Ok(Config), and expect gives the Config.
let owned: Config = Rc::try_unwrap(config).expect("last handle standing");

The successful unwrap costs no clone. try_unwrap moves the Config out and deallocates the counter block. Sometimes you want a move when the handle is unique and a clone when it is not. Rc::unwrap_or_clone does exactly that in one call.

07_06_rc_shared_ownership.rs prints:

created:              strong_count = 1
+ logger, + fetcher:  strong_count = 3
- logger:             strong_count = 2
try_unwrap while shared: Err (count still 2)
try_unwrap as sole owner: Ok -> Config { theme: "dark", retries: 3 }

All assertions passed.

Weak<T>: Breaking Cycles

Reference counting has one known weakness: cycles. If A holds an Rc to B and B holds an Rc to A, their counts can never become zero. Neither destructor can run. The borrow checker cannot help, because nothing is wrong with the borrows. The usual example is a parent↔child tree: parents own their children, and children point back to their parents.

Weak<T> is the solution. It is a non-owning handle that does not keep the value alive. Rc counts the weak handles separately (weak_count). You cannot dereference a Weak. Instead, you call upgrade(), which returns Option<Rc<T>>: Some while the value is alive, and None after the value drops. Thus the design itself prevents dangling pointers and use-after-free.

Figure: Ownership Down, Weak Up — the Cycle Broken

07_2_rc_arc_reference_counting_1

use std::cell::RefCell;
use std::rc::{Rc, Weak};

// RefCell (tutorial 7.3) lets the code change the links through a shared Rc.
struct TreeNode {
    parent: RefCell<Weak<TreeNode>>,      // non-owning back-pointer
    children: RefCell<Vec<Rc<TreeNode>>>, // owning forward pointers
}

// `/* … */` replaces `children: RefCell::new(Vec::new())`.
// Weak::new() is an empty Weak: the root has no parent.
let root = Rc::new(TreeNode { parent: RefCell::new(Weak::new()), /* … */ });
// Rc::downgrade makes a Weak from an Rc. It increments only the weak count.
let leaf = Rc::new(TreeNode { parent: RefCell::new(Rc::downgrade(&root)), /* … */ });
root.children.borrow_mut().push(Rc::clone(&leaf));   // the root owns the leaf

assert_eq!(Rc::strong_count(&root), 1);   // the Weak does not keep root alive
assert_eq!(Rc::weak_count(&root), 1);     // the Weak in leaf.parent

// The root is alive, so upgrade returns Some(Rc<TreeNode>).
let parent = leaf.parent.borrow().upgrade().expect("parent alive");

The design rule: ownership goes in one direction, and each back-edge is Weak. Example 07_07_rc_weak_cycles.rs uses a drop counter to make the difference visible. The version with the strong cycle never runs a destructor. The version with Weak drops the two nodes when the scope ends. The example prints:

strong cycle built: root count = 2, leaf count = 2
scope ended — destructors run: 0 (leaked!)

weak design: root strong = 1, weak = 1; leaf strong = 2
scope ended — destructors run: 2 (no leak)
observer.upgrade() after death: None

All assertions passed.

Rc::new_cyclic: Self-Referential Construction

Sometimes a value needs a Weak reference to itself. Examples are a job that gives other code handles to itself, and a node that can give its own address to registries. This is a circular problem. The Weak comes from Rc::downgrade, but the Rc exists only after you build the value.

Rc::new_cyclic solves the problem in one call:

It allocates the heap block.
It passes a Weak to your constructor closure. The value is not alive at this time.
It moves the value that the closure returns into the allocation.

struct Job { id: u32, me: Weak<Job> }   // `me` is the weak self-reference

impl Job {
    fn new(id: u32) -> Rc<Self> {
        // `me` is a &Weak<Job> that points to the new allocation.
        Rc::new_cyclic(|me| {
            assert!(me.upgrade().is_none());   // the Job is not constructed yet
            Job { id, me: me.clone() }         // keep a copy of the Weak in the Job
        })
    }
    // Gives a non-owning handle to this job.
    fn handle(&self) -> Weak<Job> { self.me.clone() }
}

let job = Job::new(7);                         // Rc<Job>
assert_eq!(Rc::strong_count(&job), 1);
assert_eq!(Rc::weak_count(&job), 1);           // the self-reference

Inside the closure, upgrade() returns None. This guarantee makes the API sound, because you cannot read a value that is not complete. The self-reference is Weak for the same reason as the back-pointers. A strong Rc<Self> field would be a cycle of one node that nothing can break.

Note what occurs at the end. When the last strong handle drops, Rc destroys the value immediately. But the counter block stays allocated until the last Weak drops. upgrade() reads that counter block.

07_08_rc_new_cyclic.rs prints:

job 7 created: strong = 1, weak = 1
two handles out:  strong = 1, weak = 3
job dropped: handles now upgrade to None

All assertions passed.

make_mut: Clone-on-Write

Rc::make_mut(&mut rc) returns &mut T, and first it makes sure that the mutation is safe. It is the copy-on-write operation for shared ownership (compare Cow<'_, T>, tutorial 1.2). Its behavior depends on the other handles to the allocation:

State of the allocation	What `make_mut` does
One strong owner, no weak handles	Mutates in place, with no copy
Other strong handles exist	Clones the value. Your handle then points to the copy.
Only weak handles exist	Moves the value to a new allocation. It disassociates the weak handles.

// doc: Rc<Vec<String>>, a document of two lines
let mut doc = Rc::new(vec![String::from("# Notes"), String::from("- buy coffee")]);
let snapshot = Rc::clone(&doc);                 // cheap snapshot: the strong count is 2

// `doc` is shared, so make_mut clones the Vec first. The push goes to the copy.
Rc::make_mut(&mut doc).push(String::from("- water plants"));

assert_eq!(Rc::strong_count(&doc), 1);          // each handle is now the only owner
assert_eq!(Rc::strong_count(&snapshot), 1);     // of its own allocation
assert!(!Rc::ptr_eq(&doc, &snapshot));          // two different allocations
assert_eq!(snapshot.len(), 2);                  // the snapshot did not change
assert_eq!(doc.len(), 3);                       // `doc` has the new line

A snapshot is O(1). You pay for a real copy only at the first write that another handle could observe. The rule for weak handles is less obvious, but it protects in the same way. After make_mut, the old Weak handles upgrade to None, and they do not see your mutation.

07_09_rc_make_mut_cow.rs prints:

before edit: strong_count = 2, shared = true
after edit:  doc has 3 lines, snapshot still 2 (diverged = true)
sole-owner edit: in place (allocation unchanged = true)
weak-only edit: weak refs disassociated (upgrade -> None)
unwrap_or_clone: recovered Vec with 5 lines

All assertions passed.

Arc<T>: Crossing Threads — and Why Rc Cannot

The counters of Rc are plain integers. If two threads clone at the same time, each could read 2 and each could write 3. Then the count misses one reference, and eventually a use-after-free occurs. Rust does not rely on you to prevent this. The type system makes it impossible: Rc<T> is !Send, so code that gives an Rc to a thread does not compile:

// This code does not compile.
let rc = std::rc::Rc::new(vec![1, 2, 3]);
thread::spawn(move || rc.len());   // the closure moves `rc` to the new thread
// error[E0277]: `Rc<Vec<i32>>` cannot be sent between threads safely

Arc<T> (atomically reference counted) replaces the plain counters with atomic operations. That is the only difference, and the only cost. The API is almost the same as the API of Rc, method for method: Arc::clone, strong_count, try_unwrap, make_mut, Arc::downgrade to std::sync::Weak, and Arc::new_cyclic.

use std::sync::Arc;
use std::thread;

let readings: Arc<Vec<i64>> = Arc::new((1..=1000).collect());   // the values 1 to 1000

let mut handles = Vec::new();                   // one JoinHandle<i64> for each worker
for worker in 0..4 {
    let data = Arc::clone(&readings);           // atomic increment of the count
    // `move` gives the clone to the thread. The thread drops it when the closure ends.
    handles.push(thread::spawn(move || {
        // Each worker adds its own quarter of the data: 250 elements.
        data[worker * 250..(worker + 1) * 250].iter().sum::<i64>()
    }));
}
let total: i64 = handles.into_iter().map(|h| h.join().unwrap()).sum();
assert_eq!(total, 500_500);                     // the sum of 1..=1000

assert_eq!(Arc::strong_count(&readings), 1);    // deterministic: join waited for each worker
let owned = Arc::try_unwrap(readings).expect("sole owner");   // Vec<i64>, moved out

Arc alone gives shared read access. For shared mutation, use it together with the synchronization types: Arc<Mutex<T>>, Arc<RwLock<T>> (tutorial 6.2), or atomics (6.3). Select Rc when the sharing provably stays on one thread. The compiler tells you as soon as that is no longer true.

07_10_arc_threads.rs prints:

sum computed by 4 workers: 500500
after joins: strong_count = 1
reclaimed the Vec by value: len = 1000
sync::Weak after drop: upgrade -> None

All assertions passed.

Summary

Concept	Key point
`Rc<T>`	Shared ownership with plain counters, for one thread only (`!Send`)
`Arc<T>`	The same API with atomic counters. It is the thread-safe type.
`Rc::clone(&x)`	Copies the handle and increments `strong_count`. It does not touch the value.
`Rc::ptr_eq`	Tells you if two handles share one allocation
No `DerefMut`	Shared data is immutable. To mutate, use `make_mut` or `RefCell` (7.3).
`Rc::try_unwrap`	Moves the value out when the count is 1. If not, `Err(rc)` returns the handle.
`Rc::unwrap_or_clone`	Moves the value if the handle is unique, and clones it if not
`Weak<T>`	Non-owning handle with a separate count. `upgrade()` returns `Option<Rc<T>>`.
Cycle rule	Ownership goes in one direction, and each back-edge is `Weak`
`Rc::new_cyclic`	Builds a value that holds a `Weak` to itself. `upgrade` returns `None` during construction.
`Rc::make_mut`	Clone-on-write `&mut T`: in place if unique, clone if shared. It disassociates weak handles.
Value vs. allocation	The value drops when the strong count is 0. The counter block stays until the weak count is also 0.

Code Examples

File	Description
`07_06_rc_shared_ownership.rs`	`Rc::new`/`clone`, `strong_count`, `ptr_eq`, `try_unwrap` with a shared config
`07_07_rc_weak_cycles.rs`	Parent↔child tree: a strong cycle that leaks, and the same tree with `Weak` back-pointers and no leak
`07_08_rc_new_cyclic.rs`	`Rc::new_cyclic` for a job that gives other code `Weak` handles to itself
`07_09_rc_make_mut_cow.rs`	`make_mut` clone-on-write: the shared, unique, and weak-only cases, and `unwrap_or_clone`
`07_10_arc_threads.rs`	`Arc` fan-out to worker threads, why `Rc` is `!Send`, `Arc::try_unwrap`, `sync::Weak`

7.3 · Cell<T> and RefCell<T>: Interior Mutability

Domain 7 — Smart Pointers and Heap Allocation Duration: ~15 minutes Library components: std::cell::Cell, std::cell::RefCell, std::cell::Ref, std::cell::RefMut, std::cell::UnsafeCell

Introduction

The aliasing rule of Rust is: shared (&T) XOR mutable (&mut T). The compiler usually enforces this rule at compile time. Interior mutability is the approved exception. It is a set of types that permit mutation through a shared reference. Each type replaces the compile-time check with a different runtime discipline.

Two reasons to use interior mutability occur frequently:

Bookkeeping behind an API that is logically read-only: hit counters, caches, memoization. lookup(&self) should look like a query, although it updates statistics.
Shared ownership that needs mutation: Rc<T> (tutorial 7.2) has no DerefMut. Rc<RefCell<T>> is the usual single-threaded combination for shared mutable data. Example 07_07 used RefCell fields in Rc nodes to connect the links of a tree.

This tutorial shows:

Cell<T>: values move in and out, with zero overhead.
RefCell<T>: borrows that the Ref/RefMut guards check at runtime. A conflict gives BorrowError or BorrowMutError.
The conversions between a cell of an array and an array of cells, with the AsRef impls stabilized in Rust 1.95.
UnsafeCell<T>: the primitive below all of these types.

Cell<T>: Values In, Values Out

The discipline of Cell<T> is very simple: no reference to the contents ever exists. You copy or move whole values in and out with set, get, replace, take, and swap. Thus there is nothing to alias and nothing to check at runtime. A Cell<T> has exactly the same cost as a bare T.

use std::cell::Cell;

struct PriceCache {
    prices: Vec<(&'static str, u32)>,  // (SKU, price in cents), immutable after construction
    hits: Cell<u32>,                   // mutated through &self
    misses: Cell<u32>,
}

impl PriceCache {
    fn lookup(&self, sku: &str) -> Option<u32> {   // &self, not &mut self
        // … find `sku` in self.prices …
        // On a hit: get copies the count out, and set moves the new count in.
        self.hits.set(self.hits.get() + 1);
        // … on a miss, self.misses counts in the same way …
    }
}

Why `get` requires `Copy`

get gives you the value while the cell keeps it too. Only types that you can duplicate (Copy types) permit that. For a type that is not Copy, Cell<String>::get fails with error[E0599]: … trait bounds were not satisfied: String: Copy. You can still use contents that are not Copy in full. You move the values and do not copy them:

let label: Cell<String> = Cell::new(String::from("draft"));

let old = label.replace(String::from("review"));  // moves "review" in, returns "draft"
assert_eq!(old, "draft");

let current = label.take();                        // moves "review" out, leaves String::default()
assert_eq!(current, "review");
assert_eq!(label.take(), "");                      // the default (empty) String was in the cell

swap exchanges the contents of two cells in place. into_inner consumes the cell and returns the value. Its by-value self proves that no other code can observe the cell after the call. Like each type in this tutorial, Cell is !Sync: it is for one thread only.

07_11_cell_basics.rs prints:

lookups done: hits = 3, misses = 1
non-Copy Cell: replace -> "draft", take -> "review"
after swap: front = "secondary", backup = "primary"

All assertions passed.

RefCell<T>: The Borrow Checker at Runtime

A Cell is not convenient when you want to call methods on the contents. Examples are a push to a Vec and the mutation of a field. RefCell<T> keeps the data in place and lends real references. It enforces the aliasing rule with a runtime counter. The counter permits any number of Ref guards (from borrow()), XOR one RefMut guard (from borrow_mut()).

Figure: RefCell Borrow States

07_3_cell_refcell_interior_mutability_1

The guards are the most important part. Each Ref or RefMut holds its borrow until it drops. Thus the lifetimes of the guards, not the statement boundaries, control which borrows conflict:

use std::cell::RefCell;

struct EventSink { events: RefCell<Vec<String>> }

impl EventSink {
    fn record(&self, event: &str) {   // &self, but the method mutates
        // borrow_mut returns a RefMut<Vec<String>> guard. The guard lives only
        // for this expression, so it cannot overlap another borrow.
        self.events.borrow_mut().push(event.to_owned());
    }
}

Ref::map projects a guard onto a component (Ref<Vec<String>> → Ref<str>). It lends part of the data, and the borrow of the whole value stays active. RefMut::map does the same for a RefMut.

try_borrow and try_borrow_mut: Conflicts as Values

A borrow() or borrow_mut() that conflicts panics (for example, with the message RefCell already borrowed). A panic is the right result for a real bug. But sometimes a conflict is a legitimate runtime condition (reentrant callbacks, observers, recursive walks). The try_ variants report the conflict as a Result:

// `sink` is an EventSink that holds three events.
let reader_a = sink.events.borrow();          // Ref<Vec<String>>
let reader_b = sink.events.borrow();          // many readers at the same time: permitted

let denied = sink.events.try_borrow_mut();    // a writer while the readers are alive
assert!(denied.is_err());                     // Err(BorrowMutError), no panic

drop(reader_a);
drop(reader_b);                               // no guard is alive now
assert!(sink.events.try_borrow_mut().is_ok()); // this temporary RefMut drops immediately

let writer = sink.events.borrow_mut();        // RefMut<Vec<String>>
assert!(sink.events.try_borrow().is_err());   // Err(BorrowError): the writer is alive

General rule: use borrow/borrow_mut inside short methods where no guard can overlap. Use try_* at boundaries where reentrancy is possible. The equivalent type for more than one thread is RwLock (tutorial 6.2), which blocks and does not panic.

07_12_refcell_runtime_borrow.rs prints:

events recorded: 3
writer while 2 readers live: Err(RefCell already borrowed)
reader while writer live:    Err(RefCell already mutably borrowed)
Ref::map projection: first event = "connected"

All assertions passed.

Arrays and Slices of Cells

Cell<T> has the same memory layout as T. Thus a set of conversions between one cell of many values and many cells is possible at no cost:

Cell::from_mut(&mut T) -> &Cell<T>: you hold exclusive access, and you exchange it for a cell view that you can share.
Cell<[T]>::as_slice_of_cells(&self) -> &[Cell<T>]: converts a cell of a slice into a slice of cells.
Stabilized in Rust 1.95: AsRef impls for arrays. .as_ref() converts Cell<[T; N]> directly to &[Cell<T>; N], and Cell<[T]> to &[Cell<T>].

This is important because &[Cell<T>] is a shared view that still permits writes to the elements. Different parts of the code can hold views that overlap at the same time. &mut [T] cannot do that:

// scale and clip are functions that take a shared view, for example:
//     fn scale(samples: &[Cell<i32>], factor: i32)
let mut raw = [8, 40, -3, 25, 7];
// &mut [i32] → &Cell<[i32]> → &[Cell<i32>]
let samples: &[Cell<i32>] = Cell::from_mut(&mut raw[..]).as_slice_of_cells();

scale(samples, 2);   // multiplies each element by 2
clip(samples, 50);   // clamps each element to the range -50..=50
assert_eq!(raw, [16, 50, -6, 50, 14]);   // `raw` is usable again: the cell view ended

let buffer: Cell<[i32; 6]> = Cell::new([1, -2, 3, -4, 5, -6]);
let cells: &[Cell<i32>; 6] = buffer.as_ref();   // Rust 1.95 AsRef
cells[0].set(10);                               // writes element 0 through the shared view

The best example is windows that overlap. Each iteration reads the adjacent element and writes the current element. iter_mut can never permit that:

// cells: &[Cell<i32>], a view of the array [1, 2, 4, 8, 16]
for window in cells.windows(2) {   // window: &[Cell<i32>] of length 2
    // Add the next element to the current element.
    window[0].set(window[0].get() + window[1].get());
}
// The array is now [3, 6, 12, 24, 16].

07_13_cell_arrays_slices.rs prints:

scaled + clipped: [16, 50, -6, 50, 14]
array-of-cells writes: [10, 20, 3, -4, 5, -6]
pairwise-summed in place: [3, 6, 12, 24, 16]

All assertions passed.

Choosing a Cell

std::cell also contains OnceCell<T> (write-once) and LazyCell<T> (initialized on first access). These are single-assignment disciplines. Tutorial 6.6 describes them in full, together with their thread-safe equivalents OnceLock and LazyLock. When you know all these types, the selection follows a fixed procedure:

Figure: Interior Mutability Decision Tree

07_3_cell_refcell_interior_mutability_2

Prefer Cell when it is sufficient to move whole values. It cannot panic, and it has no cost. Use RefCell when you need real references into the data. Accept that incorrect use then shows as a runtime error, not as a compile error.

UnsafeCell<T>: The Primitive Underneath

Each interior mutability type (Cell, RefCell, Mutex, RwLock, the atomics) wraps the same primitive: UnsafeCell<T>. It is the only approved way to mutate through &T. The compiler assumes that no code writes behind a & reference. It exempts &UnsafeCell<T> from this assumption, and it never marks &UnsafeCell<T> as noalias for the optimizer. A cast of &T to &mut T in any other way is undefined behavior, also inside unsafe. An unsafe block does not make that cast valid.

Its API is very small. The primary method is get(&self) -> *mut T. It is safe to call, and only the dereference of the pointer is unsafe. Each wrapper adds a discipline that makes the dereference sound:

UnsafeCell<T>   raw permission to mutate behind &T          (unsafe)
Cell<T>         + values move in/out, no references escape  → safe
RefCell<T>      + counted borrows, checked at runtime       → safe
Mutex<T>        + OS locking, opts back into Sync           → safe + threads

A miniature Cell shows the form of such a safety argument (example 07_14):

use std::cell::UnsafeCell;

struct MiniCell<T> { value: UnsafeCell<T> }

impl<T> MiniCell<T> {
    fn set(&self, value: T) {
        // self.value.get() returns *mut T. The call is safe, but the write is not.
        // SAFETY: no reference to the contents ever leaves MiniCell, and
        // UnsafeCell is !Sync, so no other thread can hold &self. Thus the
        // write cannot alias anything.
        unsafe { *self.value.get() = value; }
    }
}

The example also measures the costs. UnsafeCell<u64> and Cell<u64> are exactly 8 bytes, which is zero overhead. RefCell<u64> is 16 bytes: one more word for the borrow counter. !Sync propagates automatically from UnsafeCell. For this reason, each type in this tutorial is for one thread only. The exception is a type that explicitly becomes Sync again with locking (unsafe impl Sync, as Mutex does).

07_14_unsafecell_foundation.rs prints:

MiniCell works: counter = 2, title flipped from "draft"
sizes: u64 = 8, Cell<u64> = 8, RefCell<u64> = 16

All assertions passed.

Summary

Concept	Key point
Interior mutability	Mutation through `&T`. A runtime discipline enforces the aliasing rule.
`Cell<T>`	Values move in and out (`set`/`get`/`replace`/`take`/`swap`). It has zero overhead and cannot panic.
`get` needs `Copy`	The cell keeps the value and also gives you one. Only types that you can duplicate permit that.
`RefCell<T>`	Real references through counted guards: many `Ref` XOR one `RefMut`
`Ref` / `RefMut`	A guard holds its borrow until it drops. `Ref::map` projects a guard onto a component.
`try_borrow` / `try_borrow_mut`	A conflict gives `Err(BorrowError)` / `Err(BorrowMutError)`, not a panic
`Cell::from_mut`	`&mut T` → `&Cell<T>`: exchanges exclusive access for a view that you can share
`as_slice_of_cells`	`&Cell<[T]>` → `&[Cell<T>]`. The `AsRef` forms for arrays are stable since Rust 1.95.
`OnceCell` / `LazyCell`	Write-once cells and lazy-init cells. Tutorial 6.6 describes them in full.
`UnsafeCell<T>`	The primitive: the only legal channel for mutation from `&T` to `*mut T`
`!Sync`	All of `std::cell` is for one thread only. The thread-safe equivalents are in `std::sync`.

Code Examples

File	Description
`07_11_cell_basics.rs`	`Cell` counters behind `&self`, `get`/`set`/`replace`/`take`/`swap`, and the `Copy` bound
`07_12_refcell_runtime_borrow.rs`	`RefCell` guards, `Ref::map`, and `BorrowError`/`BorrowMutError` results from `try_*`
`07_13_cell_arrays_slices.rs`	`Cell::from_mut`, `as_slice_of_cells`, and the `AsRef` conversions for arrays (Rust 1.95)
`07_14_unsafecell_foundation.rs`	A `MiniCell` built on `UnsafeCell`, safety arguments, size comparisons

8.1 · Read, Write, BufRead: The I/O Trait Hierarchy

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::Read, std::io::Write, std::io::BufRead, std::io::BufReader, std::io::BufWriter, std::io::LineWriter, std::io::copy

Introduction

The Rust I/O system has three basic traits:

Read gets bytes from a source.
Write sends bytes to a destination.
BufRead adds line-oriented and delimiter-oriented methods to a buffered Read.

Files, sockets, standard streams, in-memory buffers, and compression wrappers all use these three traits. Thus a function that takes impl Read operates on all of them without changes.

This tutorial shows:

The trait hierarchy and the types that implement each trait.
Read: partial reads, read_to_end, read_to_string, read_exact, and why Ok(0) means EOF.
Write: write and write_all, flush, and the difference between io::Write and fmt::Write.
BufRead and BufReader: read_line, lines, split, and the fill_buf/consume primitives.
BufWriter and LineWriter: when the bytes that you write arrive at their destination.
How to compose readers with chain and take, and how to stream with io::copy.

The Trait Hierarchy

There are three small traits and one supertrait relationship: BufRead: Read. All other items in std::io are implementors or adapters of these traits.

Figure: The core I/O traits and their main implementors

08_1_read_write_bufread_1

(In the figure, ByteSlice is the &[u8] primitive and ByteVec is Vec<u8>. A dotted arrow means "implements".)

Two design decisions are important:

Each operation can fail. Each operation returns io::Result (tutorial 8.4), because a disk can become full, a pipe can break, and a connection can reset. The lines() iterator also gives io::Result<String> items.
Byte counts are contracts. read and write return the number of bytes that they moved. That number can be less than the number that you requested. The *_all and *_exact variants do the loop for you.

The write! macro operates with two different traits. io::Write moves bytes, and fmt::Write moves string slices. The write! and writeln! macros use the two traits through separate trait implementations. Tutorial 3.2 gives the details of fmt::Write.

Read: Pulling Bytes

Read has one required method: fn read(&mut self, buf: &mut [u8]) -> io::Result<usize>. This method defines the full contract:

It fills a maximum of buf.len() bytes.
It returns the number of bytes that it filled.
It returns Ok(0) for a non-empty buffer to signal EOF.

A short read is not an error and not EOF. A caller that obeys the contract calls read in a loop.

// These lines are in a function that returns io::Result<()>, so `?` is valid.
let mut source: &[u8] = b"HELLO WORLD";  // &[u8] implements Read

let mut chunk = [0u8; 4];                // a 4-byte buffer on the stack
let n = source.read(&mut chunk)?;        // n can be less than 4 (here n == 4)
assert_eq!(&chunk[..n], b"HELL");

let n = source.read(&mut chunk)?;        // the first read moved the slice forward
assert_eq!(&chunk[..n], b"O WO");        // the second read gives the next 4 bytes

The provided methods contain the loops that you would otherwise write manually:

read_to_end(&mut Vec<u8>) reads all bytes until EOF.
read_to_string(&mut String) does the same and also validates UTF-8. Bytes that are not valid UTF-8 cause an InvalidData error.
read_exact(&mut [u8]) fills the whole buffer or fails with UnexpectedEof. It is the usual method for fixed-size fields in binary formats (tutorial 8.2).

// A 5-byte source: one big-endian u32, then one more byte.
let mut source: &[u8] = &[0x12, 0x34, 0x56, 0x78, 0x9A];
let mut word = [0u8; 4];
source.read_exact(&mut word)?;                         // fills all 4 bytes
assert_eq!(u32::from_be_bytes(word), 0x1234_5678);

let err = source.read_exact(&mut word).unwrap_err();   // only 1 byte left
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);

All these methods also retry internally after an ErrorKind::Interrupted error. Tutorial 8.4 tells you more about that special error.

Write: Pushing Bytes

Write is the opposite of Read and has the same type of contract. write may accept only a prefix of your data, and it returns the number of bytes that it accepted. write_all calls write in a loop until it writes all the data. Application code almost always needs write_all. If you ignore a short write, the result is silent data corruption.

let mut frame: Vec<u8> = Vec::new();     // Vec<u8> implements Write: each write appends
frame.write_all(b"LEN=11;")?;
frame.write_all(b"hello world")?;        // frame is now b"LEN=11;hello world"

// A &mut [u8] writer has a FIXED capacity. It has two failure modes:
let mut fixed = [0u8; 4];
let mut slot: &mut [u8] = &mut fixed;
let accepted = slot.write(b"toolong")?;  // Ok(4): partial write, NO error
assert_eq!(accepted, 4);                 // fixed is now b"tool"

let mut slot: &mut [u8] = &mut fixed;    // a new writer on the same 4 bytes
slot.write_all(b"cool")?;                // fits exactly, the writer is now full
let err = slot.write_all(b"x").unwrap_err();
assert_eq!(err.kind(), std::io::ErrorKind::WriteZero);   // write accepted 0 bytes

flush sends all buffered bytes to their destination. On a bare Vec, flush does nothing. Call it anyway. Then your code still operates as intended if you later replace the Vec with a BufWriter<File> or a socket. A missing flush causes problems with buffered writers (see the next slides).

BufRead and BufReader

Without a buffer, each read call on a File is a system call. If you read a File one line at a time, you call the OS for each small group of bytes. BufReader wraps any Read and keeps an internal buffer (8 KiB by default). It implements BufRead, which adds the text-oriented methods that most programs use.

Figure: How BufReader turns many small reads into few large ones

08_1_read_write_bufread_2

The most important methods are:

read_line(&mut String) appends one line, including the trailing \n. It returns 0 at EOF. Call clear() on the string between calls.
lines() returns an iterator of io::Result<String> items without the newline. It is the most common method to read text.
split(byte) is the equivalent of lines() for any delimiter byte. It gives Vec<u8> chunks, each in an io::Result.
read_until and skip_until do one step at a time. read_until keeps the bytes, and skip_until discards them.
fill_buf() and consume(n) are the two primitives that all the methods above use. fill_buf() shows you the buffered bytes. consume(n) tells the reader how many bytes you used.

In memory, &[u8] already implements BufRead, so tests do not need a wrapper. Be careful with method resolution on a bare &[u8]: the inherent split method of the slice shadows BufRead::split. Put the bytes in a Cursor (tutorial 8.2) to get the BufRead version.

// path: PathBuf, the path of a text file that exists.
// File implements Read but not BufRead. BufReader<File> implements BufRead.
for line in BufReader::new(File::open(&path)?).lines() {
    let line = line?;                    // an error can occur in the middle of the iteration
    // ... use `line`: a String without the trailing newline
}

BufWriter and LineWriter

BufWriter is the equivalent wrapper for output. It collects small writes in memory and sends them to the destination in large blocks. The example uses a counting sink as a replacement for a file, which makes the effect visible. 100 writes arrive at the destination as one call:

// CountingSink is a Write type that the example defines. It appends the size
// of each write call that it receives to its `write_calls: Vec<usize>` field.
let mut writer = BufWriter::with_capacity(4096, CountingSink::default());
for i in 0..100u32 {
    write!(writer, "{i:03},")?;                     // 4 bytes: "000,", "001,", ...
}
// get_ref() borrows the wrapped sink. All 400 bytes are still in the buffer.
assert_eq!(writer.get_ref().write_calls.len(), 0);
writer.flush()?;
assert_eq!(writer.get_ref().write_calls.len(), 1);  // one write call of 400 bytes

Remember these three behaviors:

The flush on drop hides errors. The Drop implementation of BufWriter tries to flush, but it cannot report a failure. End with an explicit flush() or into_inner(), so that you get the errors. into_inner() flushes and returns the wrapped writer.
Large writes bypass the buffer. A payload that is larger than the buffer goes directly to the destination as one call. BufWriter does not copy it into the buffer.
LineWriter flushes on \n. It buffers as BufWriter does. When a newline arrives, it immediately sends all bytes up to and including the newline. io::stdout() uses the same policy (tutorial 8.5).

08_03_bufwriter_linewriter.rs prints:

unbuffered: 100 writes reached the sink as 100 calls
BufWriter: 100 writes reached the sink as 1 call(s)
BufWriter: 1024-byte write bypassed the 64-byte buffer ([1024])
LineWriter: flushed through the newline, held "trailing" until into_inner

All assertions passed.

Composing Streams: chain, take, io::copy

You can compose readers as you compose iterators:

a.chain(b) reads all of a, then all of b. This is the concatenation of two streams.
r.take(n) limits a reader to n bytes. It is necessary when a length prefix gives the size of a field. It is also a low-cost protection against input that has no limit. Use r.by_ref().take(n) if you need the underlying reader after the call.
io::copy(&mut reader, &mut writer) moves bytes from any Read into any Write and returns the total count. It reuses one internal buffer, so memory use stays constant for a stream of any size.

// header: &[u8] = b"PREVIEW:" (8 bytes). body: a Vec<u8> of 1000 '#' bytes.
const PREVIEW_LIMIT: u64 = 64;
let mut preview = Vec::new();
let n = io::copy(
    // Read the header, then the body, and stop after 64 bytes in total.
    &mut header.chain(&body[..]).take(PREVIEW_LIMIT),
    &mut preview,
)?;
assert_eq!(n, PREVIEW_LIMIT);            // preview holds "PREVIEW:" and 56 '#' bytes

io::copy has an internal specialization for a source or a sink that is already buffered. It uses that buffer again and does not allocate a second one. As of Rust 1.99, std has no public io::copy_buf function that gives this optimization as a separate API. Use io::copy.

Summary

Concept	Key point
`Read::read`	It may return fewer bytes than you requested. `Ok(0)` means EOF.
`read_to_end` / `read_to_string`	They read all bytes until EOF. The string version requires valid UTF-8 (`InvalidData`).
`read_exact`	It fills the whole buffer or fails. Short input gives `UnexpectedEof`.
`Write::write`	It may accept only a prefix. It returns the count.
`write_all`	It loops until it writes all the bytes. It fails with `WriteZero` if no progress is possible.
`flush`	It sends buffered bytes to the destination. Call it before you rely on the data.
`io::Write` vs `fmt::Write`	Bytes and `str`. The `write!` macros are the same, but the traits are different (see 3.2).
`BufRead`	`read_line`, `lines`, `split`, which use `fill_buf` + `consume`
`BufReader`	Adds an 8 KiB read-ahead buffer to any `Read`
`BufWriter`	It batches small writes. The flush on drop hides errors, so use `into_inner`.
`LineWriter`	Buffered, but it flushes at each `\n` (the policy of stdout)
`chain` / `take`	Concatenate readers / limit a reader
`io::copy`	It streams `Read` → `Write` in constant memory. Internally, it reuses a buffer that already exists.

Code Examples

File	Description
`08_01_read_write_fundamentals.rs`	Partial reads, `read_to_end`/`read_to_string`/`read_exact`, `write` and `write_all`, `flush`, and `io::Write` compared with `fmt::Write`
`08_02_bufread_lines_split.rs`	`read_line`, `lines`, `split`, `read_until`/`skip_until`, `fill_buf`+`consume`, and `BufReader` on a real temporary file
`08_03_bufwriter_linewriter.rs`	A counting sink shows the batching of `BufWriter`, the buffer bypass, and the newline flush of `LineWriter`
`08_04_chain_take_copy.rs`	`chain`, `take`, `by_ref`, and `io::copy` in one streaming pipeline with a byte limit

8.2 · Cursor, Sink, Empty, Repeat: In-Memory I/O and Binary Parsing

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::Cursor, std::io::Empty, std::io::Sink, std::io::Repeat, std::io::empty, std::io::sink, std::io::repeat

Introduction

The traits from tutorial 8.1 give their full benefit only if you can use them without real devices. std::io has four in-memory types for this purpose:

Cursor<T> makes a byte buffer into a full Read + Write + Seek stream.
io::empty() is a reader that is always at EOF.
io::sink() is a writer that discards all bytes.
io::repeat(byte) is an infinite reader of one byte.

For the I/O traits, these types do the work of /dev/null and /dev/zero. They are also the base of deterministic I/O tests.

This tutorial shows:

Cursor<T>, and how the backing store (&[u8], Vec<u8>, &mut [u8]) sets its capabilities.
empty, sink, and repeat as test doubles and data generators.
The primary use case, a parser for a binary file:
- It parses a header with read_exact and from_le_bytes.
- It uses Seek to go through an offset table.
- It uses Cursor<Vec<u8>> to write patches.

Cursor: A Position Over a Buffer

A Cursor<T> has only two fields: the wrapped buffer and a u64 position. A bare &[u8] reader does not have that position. A slice reader can only move forward: the slice is its own position, and it becomes shorter with each read. A cursor can go to any position and then return.

Figure: Cursor position and offset-table navigation over a 52-byte container

Cursor over the RIMG container: header fields, offset table, and chunks with seek arrows

// These lines are in a function that returns io::Result<()>, so `?` is valid.
let mut cur = Cursor::new(&b"ABCDEFGHIJ"[..]);   // Cursor<&[u8]>, position 0

let mut chunk = [0u8; 3];
cur.read_exact(&mut chunk)?;        // reads "ABC" and moves the position forward
assert_eq!(cur.position(), 3);

cur.set_position(0);                // back to the start: a bare &[u8] cannot do this
cur.read_exact(&mut chunk)?;        // reads the same 3 bytes again
assert_eq!(&chunk, b"ABC");

cur.seek(SeekFrom::End(-2))?;       // position 8, 2 bytes before the end (tutorial 8.3)

Cursor also has these methods:

position() and set_position() read and set the position at low cost. They do not return an io::Result, and they do not need a trait import.
get_ref() and get_mut() give access to the whole underlying buffer, independently of the position.
into_inner() returns the buffer when you finish with the stream.

Choosing the Backing Store

Cursor<T> is a reader for any T: AsRef<[u8]>. Only the owned and mutable variants implement Write. The selection of T is the API design decision:

Figure: Which Cursor backing store fits the job

08_2_cursor_in_memory_io_1

Two behaviors of the variants that can write are not obvious:

Cursor<Vec<u8>> fills gaps with zeros. If you seek past the end and then write, the bytes in the gap become zeros. This is the in-memory equivalent of a sparse file:

let mut cur = Cursor::new(Vec::new());   // Cursor<Vec<u8>>, empty
cur.write_all(b"header")?;               // 6 bytes, the position is now 6
cur.set_position(9);                     // 3 bytes past the end
cur.write_all(b"!")?;                    // the Vec grows to 10 bytes
assert_eq!(cur.get_ref(), b"header\0\0\0!");   // bytes 6, 7, and 8 are zeros

Cursor<&mut [u8]> cannot grow. A write at the end accepts zero bytes, so write_all reports WriteZero. Thus this variant is the appropriate selection to patch fixed-size regions such as packet headers.

The pattern that results is this: write functions that are generic over Read + Seek (or Write). Pass a File in production and a Cursor in tests. The code is the same, the tests use no disk, and the tests are fully deterministic.

empty, sink, and repeat

Three zero-cost utility streams complete the set of in-memory types:

io::empty(): each read returns Ok(0). It is the standard "no input" value for code that requires impl Read. It also implements BufRead, so empty().lines() gives no items.
io::sink(): it accepts and discards any number of bytes. Use it as a test double when you only need to know that serialization runs. You can also use it for a mandatory output that you want to ignore.
io::repeat(b): an infinite reader of one byte. Always limit it with take or a fixed-size read_exact. A read_to_end call on the bare reader cannot complete: std returns an OutOfMemory error immediately.

When you compose them with io::copy (tutorial 8.1), one line does the work that otherwise needs a loop:

// reader: any value that implements Read.
// Measure the size of a stream and store nothing:
let size = io::copy(&mut reader, &mut io::sink())?;   // size: u64, the byte count

// Make a 16-byte 0xAB fixture without a loop:
let mut fixture = Vec::new();
io::repeat(0xAB).take(16).read_to_end(&mut fixture)?;   // fixture == [0xAB; 16]

// Send 10 KiB of synthetic input through a pipeline, with no allocation:
let n = io::copy(&mut io::repeat(b'x').take(10 * 1024), &mut io::sink())?;   // n == 10240

Binary Parsing: Header and Offset Table

The primary use case of Cursor is a parser for a binary container. The example defines a 52-byte "RIMG" format as a const array. The array contains a magic number, little-endian header fields, and then an offset table that points to tagged chunks (see the figure on slide 2). Real formats (BMP, PNG, WAV, TTF) have this same structure.

The usual pattern for a fixed-size field is read_exact into a stack array, then from_le_bytes. Tutorial 18.1 shows the endian methods:

// Reads 4 bytes and decodes them as a little-endian u32.
fn read_u32_le<R: Read>(r: &mut R) -> io::Result<u32> {
    let mut buf = [0u8; 4];
    r.read_exact(&mut buf)?;        // UnexpectedEof if fewer than 4 bytes remain
    Ok(u32::from_le_bytes(buf))
}

// Header is a struct that the example defines, with the four fields below.
// read_u16_le is the same as read_u32_le, with a 2-byte buffer and u16.
fn parse_header<R: Read>(r: &mut R) -> io::Result<Header> {
    let mut magic = [0u8; 4];
    r.read_exact(&mut magic)?;
    if &magic != b"RIMG" {
        // The input is corrupt: return an error, do not panic.
        return Err(io::Error::new(io::ErrorKind::InvalidData, "bad magic"));
    }
    // Rust evaluates the field expressions in the order written, which is the file order.
    Ok(Header {
        version: read_u16_le(r)?,       // bytes 4 and 5
        chunk_count: read_u16_le(r)?,   // bytes 6 and 7
        width: read_u32_le(r)?,         // bytes 8 to 11
        height: read_u32_le(r)?,        // bytes 12 to 15
    })
}

Obey this rule: malformed input is data corruption, not a bug. A parser returns InvalidData or UnexpectedEof errors (tutorial 8.4). It never panics. You need no more code for truncated input, because read_exact already reports UnexpectedEof.

After the sequential parse of the header, the offset table makes random access possible. With Read + Seek, a Cursor can do what a plain slice reader cannot:

// Chunk is a struct that the example defines: a 4-byte tag and a u32 value.
fn read_chunk<S: Read + Seek>(s: &mut S, offset: u32) -> io::Result<Chunk> {
    s.seek(SeekFrom::Start(u64::from(offset)))?;   // go to the absolute offset of the chunk
    let mut tag = [0u8; 4];
    s.read_exact(&mut tag)?;
    Ok(Chunk { tag, value: read_u32_le(s)? })
}

// cur: a Cursor<&[u8]> on the RIMG bytes. offsets: Vec<u32> == [28, 36, 44], from the table.
let dpi_y = read_chunk(&mut cur, offsets[2])?;   // read the LAST chunk first

Writing Back: Patching Bytes In Place

A write uses the same offsets. Do these steps:

Copy the container into a Cursor<Vec<u8>>.
Seek to the absolute position of a field.
Write exactly as many bytes as the field has.
Parse the container again to verify the patch.

// RIMG is the 52-byte const array. HEIGHT_FIELD_OFFSET is 12.
let mut cur = Cursor::new(RIMG.to_vec());          // a mutable copy: Cursor<Vec<u8>>

cur.seek(SeekFrom::Start(HEIGHT_FIELD_OFFSET))?;   // byte 12
cur.write_all(&720u32.to_le_bytes())?;             // 480 -> 720

cur.seek(SeekFrom::Start(28 + 4))?;                // GAMA chunk at 28, value after the 4-byte tag
cur.write_all(&2400u32.to_le_bytes())?;            // 2200 -> 2400

cur.rewind()?;                                     // back to byte 0, to parse again
let header = parse_header(&mut cur)?;
assert_eq!(header.height, 720);                    // the patch is in the bytes
assert_eq!(header.width, 640);                     // the adjacent field did not change
assert_eq!(cur.get_ref().len(), RIMG.len());       // the length is the same: no append

All bytes of the input are in a const array, so each step of the cycle (parse, patch, parse again) is reproducible. 08_07_binary_header_parsing.rs prints:

parsed: 640x480 v1, chunks at [28, 36, 44], GAMA=2200 DPIY=300
rejected: bad magic = InvalidData, truncated = UnexpectedEof
patched: height=720 GAMA=2400 (re-parsed from the rewritten bytes)

All assertions passed.

Summary

Concept	Key point
`Cursor<T>`	A buffer and a `u64` position give full `Read + Write + Seek` in memory
`Cursor<&[u8]>`	Borrowed, read and seek. Unlike a bare `&[u8]`, it can go backward.
`Cursor<Vec<u8>>`	Writes make the `Vec` grow. A write past EOF fills the gap with zeros.
`Cursor<&mut [u8]>`	It patches a fixed-size buffer in place. An overflow gives `WriteZero`.
`position` / `set_position`	They read and set the position at low cost, without an `io::Result`.
`get_ref` / `get_mut` / `into_inner`	Examine or recover the underlying buffer
`io::empty()`	A reader (and `BufRead`) that is always at EOF: the "no input" value
`io::sink()`	A writer that discards all bytes. Use it with `io::copy` to measure streams.
`io::repeat(b)`	An infinite reader of one byte. Always limit it with `take`.
Binary parsing	`read_exact` + `from_le_bytes` for fields. `Seek` for offset tables.
Error discipline	Bad magic → `InvalidData`. Truncation → `UnexpectedEof`. Never panic.

Code Examples

File	Description
`08_05_cursor_read_write_seek.rs`	The three backing stores, the position, the zero-filled gap, patches in place, and functions generic over `Read + Seek`
`08_06_empty_sink_repeat.rs`	`empty`/`sink`/`repeat` as test doubles, the size of a stream with `copy` → `sink`, and fixture generation
`08_07_binary_header_parsing.rs`	The full RIMG container: `const` bytes, header parse, navigation with the offset table, rejection of corrupt input, and patches written after a seek

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::Seek, std::io::SeekFrom

Introduction

Read and Write move through a stream in one direction, from the start to the end. Seek lets you go to any position in the stream. One trait method, seek(SeekFrom) -> io::Result<u64>, and the three anchors of the SeekFrom enum are sufficient to move through any random-access stream. Such streams are files, cursors, and all types that wrap them.

This tutorial shows:

SeekFrom::Start, SeekFrom::Current, SeekFrom::End, and the convenience methods stream_position and rewind.
The edge cases: a seek to before byte 0 (an error) and a seek past EOF (legal, with special write semantics).
The stale-buffer problem when you use Seek together with BufReader, and seek_relative as the efficient solution.
The offset-table use case on a real file: fixed-size records, trailers at a known distance from the end, and patches in place.

SeekFrom: Three Anchors

You give each seek relative to one of three anchors. Start takes a u64, because no position exists before the start. Current and End take an i64, because a seek from these anchors must be able to go backward.

Figure: The three SeekFrom anchors over a 26-byte stream

08_3_seek_navigation_1

seek returns the new absolute position. Use that return value, not a second call that queries the position:

// These lines are in a function that returns io::Result<()>, so `?` is valid.
// A Cursor<&[u8]> on the 26 letters: 'A' is at position 0, 'Z' is at position 25.
let mut cur = Cursor::new(b"ABCDEFGHIJKLMNOPQRSTUVWXYZ".as_slice());

let pos = cur.seek(SeekFrom::Start(9))?;    // pos == 9, next read: 'J'
let pos = cur.seek(SeekFrom::Current(5))?;  // pos == 14 (9 + 5), next read: 'O'
let pos = cur.seek(SeekFrom::End(-1))?;     // pos == 25, next read: 'Z'

SeekFrom::End is the usual method to read trailers. ZIP central directories, database footers, and checksum blocks are all at known distances from EOF. You can get to them when you do not know the length of the file.

Two convenience methods are short forms of seek: stream_position() is seek(SeekFrom::Current(0)), and rewind() is seek(SeekFrom::Start(0)). No stable stream_len exists. The portable procedure has three steps:

Save the position.
Call seek(End(0)), which returns the length.
Seek to the saved position.

The Edges: Before Zero and Past EOF

The behavior at the two limits is not symmetrical, and each behavior is important in parser code:

A seek to before byte 0 is an error. A seek that gives a negative position fails with ErrorKind::InvalidInput. The position does not change, which is important:

// cur: the Cursor on the 26 letters from the previous snippet.
cur.seek(SeekFrom::Start(10))?;
let err = cur.seek(SeekFrom::Current(-11)).unwrap_err();   // 10 - 11 = -1
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
assert_eq!(cur.stream_position()?, 10);   // the failed seek did not move the position

A seek past EOF is legal. A seek only sets a number. It does not compare the number with the stream length. A read at that position returns EOF (Ok(0)) immediately. A write at that position, on a target that can grow, fills the gap with zeros. Sparse files behave this way on most filesystems, and Cursor<Vec<u8>> does the same:
```
let mut cur = Cursor::new(b"1234".to_vec()); // Cursor<Vec<u8>>, 4 bytes
cur.seek(SeekFrom::Start(8))?; // 4 bytes past EOF: no error
cur.write_all(b"XY")?; // the Vec grows to 10 bytes
assert_eq!(cur.get_ref(), b"1234\0\0\0\0XY"); // bytes 4 to 7 are zeros
```

The BufReader Stale-Buffer Problem

BufReader (tutorial 8.1) reads ahead of the bytes that you consume. If you request 4 bytes, it may read 8 KiB from the source. From that moment, two positions exist:

The position of the underlying stream, which is far ahead.
The logical position, which is the point that your code consumed to.

Each Seek bug with a buffered reader occurs because code uses one position as the other.

Figure: Why seeking the inner reader serves stale bytes

08_3_seek_navigation_2

The example uses assertions to show the wrong bytes:

// data: a Vec<u8> with the 26 letters 'A' to 'Z'. four: [u8; 4], the read buffer.
let mut reader = BufReader::with_capacity(8, Cursor::new(data));   // an 8-byte buffer
reader.read_exact(&mut four)?;             // ABCD consumed, EFGH buffered

reader.get_mut().seek(SeekFrom::Start(0))?; // WRONG: a seek directly on the inner reader
reader.read_exact(&mut four)?;
assert_eq!(&four, b"EFGH");                // stale buffer, wrong data!
assert_eq!(reader.stream_position()?, 0);  // the reported position is also wrong

The rule: after you wrap a reader, do all navigation through the wrapper. The Seek implementation of BufReader keeps the two positions consistent, but it has a cost. Each seek discards the buffer, and the next read must get the bytes from the source again. This is true even for a 2-byte move that the buffer could serve.

seek_relative: Keeping the Buffer

BufReader::seek_relative(i64) removes this cost. It behaves as seek(SeekFrom::Current(n)) does, with one difference. When the target is inside the buffered bytes, it only changes the buffer index. It does not discard the buffer, and it does not fill the buffer again:

// data and four: the same as in the previous snippet.
let mut reader = BufReader::with_capacity(8, Cursor::new(data));
reader.read_exact(&mut four)?;         // ABCD consumed, EFGH buffered
assert_eq!(reader.buffer().len(), 4);  // buffer() returns the bytes not consumed yet

reader.seek_relative(2)?;              // skip E and F: the target is in the buffer
assert_eq!(reader.buffer().len(), 2);  // the buffer is KEPT, only its index moved

reader.read_exact(&mut four)?;
assert_eq!(&four, b"GHIJ");            // correct bytes: G and H come from the kept buffer

A move to a target outside the buffered range becomes a real seek. Use seek_relative when you skip small, known amounts in a buffered stream:

padding bytes
fixed-size fields that you do not need
alignment gaps

Then each skip is an O(1) index change. Without seek_relative, each skip discards an 8 KiB buffer.

08_09_bufreader_seek_gotcha.rs prints:

consumed 4, source at 8, buffer holds 4 — logical pos = 4
GOTCHA: after get_mut().seek(0), next read was EFGH, not ABCD
seek through BufReader: correct ABCD, but the buffer was dropped
seek_relative(2): buffer kept (2 left), read GHIJ; long hop -> U

All assertions passed.

Use Case: A Fixed-Size Record Store

The offset-table pattern from tutorial 8.2 also applies to real files. With fixed-size records, arithmetic is the offset table: record i starts at header_len + i * record_len. The example writes a small store to disk, in a unique temporary directory that it removes at the end. Then it uses the three seek patterns:

// HEADER_LEN is 8 and RECORD_LEN is 12. Both are u64 constants.
fn record_offset(index: u32) -> u64 {
    HEADER_LEN + u64::from(index) * RECORD_LEN   // record 3 starts at byte 44
}

// file: a File open for reading. The store has 5 records (68 bytes).
// O(1) random access, independent of the number of records before this one:
file.seek(SeekFrom::Start(record_offset(3)))?;

// The last record, when you do not know the count (12 bytes before EOF):
file.seek(SeekFrom::End(-i64::try_from(RECORD_LEN).expect("fits i64")))?;

// A patch in place: a File is Read + Write + Seek when you open it for read and write.
let mut rw = OpenOptions::new().read(true).write(true).open(&path)?;
let mut rec1 = read_record(&mut rw, 1)?;       // example function: seek, then read 12 bytes
rec1.score += 990;                             // 10 -> 1000
rw.seek(SeekFrom::Start(record_offset(1)))?;   // back to the start of record 1
rw.write_all(&rec1.to_bytes())?;               // 12 bytes: the file length does not change

Database pages, game asset packs, and font tables use this access pattern. All the code is generic over Read + Write + Seek, so the same functions operate on a Cursor<Vec<u8>> in tests. Domain 9 (tutorial 9.1) gives the details of file creation and OpenOptions.

Summary

Concept	Key point
`SeekFrom::Start(u64)`	An absolute position. Negative positions do not exist.
`SeekFrom::Current(i64)`	A signed move from the current position
`SeekFrom::End(i64)`	A position relative to the end: the usual method for trailers and footers
`seek` return value	The new absolute position. Use it, and do not query the position again.
`stream_position` / `rewind`	Short forms of `Current(0)` / `Start(0)`
Stream length	No stable `stream_len` exists. Save the position, seek to `End(0)`, then seek to the saved position.
Seek before 0	`InvalidInput`, and the position does not change
Seek past EOF	Legal. Reads return EOF, and writes fill the gap with zeros (as in a sparse file).
`get_mut()` + seek	The stale-buffer problem: the buffer serves wrong bytes, and the reported position is wrong
Seek through `BufReader`	The bytes and the position are consistent, but each seek discards the whole buffer
`seek_relative`	A move inside the buffer keeps the buffer. It is the efficient skip.
Fixed-size records	The arithmetic `header + i × size` replaces a stored offset table

Code Examples

File	Description
`08_08_seek_seekfrom.rs`	The three anchors, `stream_position`/`rewind`, the stream length procedure, errors for negative positions, reads past EOF, and zero-filled writes
`08_09_bufreader_seek_gotcha.rs`	The two positions of a read-ahead buffer and the stale-buffer bug with observable wrong bytes. A seek through the wrapper compared with `seek_relative`, which keeps the buffer
`08_10_offset_table_records.rs`	A store of fixed-size records in a real temporary file: O(1) record access, the last record with `SeekFrom::End`, and a patch in place

8.4 · std::io::Error: Anatomy of I/O Errors

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::Error, std::io::ErrorKind, std::io::Result

Introduction

Each fallible operation in Domain 8 returns io::Result<T>, which is an alias for Result<T, io::Error>. That one error type must represent many different failures. Two examples are a missing file (an OS errno) and a checksum mismatch in your own parser (an application payload). The type must also be cheap to construct on hot paths.

To write code that handles I/O failures, you must know the internal representations of the type, its ErrorKind taxonomy, and its payload mechanism. Without this knowledge, code can only propagate strings.

This tutorial shows:

The three internal representations of io::Error and the constructors that make them: Error::new, Error::other, From<ErrorKind>, Error::from_raw_os_error.
The ErrorKind enum: which operations produce which kinds, and how to match on kind() to select a recovery strategy.
Why ErrorKind::Interrupted is special.
Custom error payloads: get_ref, into_inner, and Error::downcast.

Three Representations, Four Constructors

Internally, an io::Error stores one of three representations, and each constructor makes one of them. You can observe the difference with raw_os_error() and get_ref():

Figure: How each constructor shapes the error

08_4_io_error_anatomy_1

use std::io;
use std::io::ErrorKind;

let simple: io::Error = ErrorKind::TimedOut.into();      // kind only: no allocation
let os     = io::Error::from_raw_os_error(2);            // errno or Win32 code
let custom = io::Error::new(ErrorKind::TimedOut, "gateway");  // kind + boxed payload

assert_eq!(simple.raw_os_error(), None);   // a kind-only error has no OS code
assert!(simple.get_ref().is_none());       // and it has no payload
assert_eq!(os.raw_os_error(), Some(2));    // an OS error keeps its code
assert!(custom.get_ref().is_some());       // the payload "gateway"

Practical notes:

From<ErrorKind> is allocation-free. Prefer it when you have no payload to add.
Error::other(payload) (stable since 1.74) is the short form of Error::new(ErrorKind::Other, payload). It is the usual pattern to put an application error into an io::Error when no specific kind applies. The standard library never uses Other for OS failures, so a match on Other finds only your errors.
OS error code 2 maps to NotFound on Unix (ENOENT) and on Windows (ERROR_FILE_NOT_FOUND). But each platform has its own Display text. Match on kind(), never on message strings.

ErrorKind: The Taxonomy

Operations produce the kinds. You do not invent them. Example 08_12 produces the first four kinds of this table with real operations, and Interrupted with a mock reader:

Operation	Kind
`File::open` on a missing path	`NotFound`
`read_exact` on a stream that is too short	`UnexpectedEof`
`read_to_string` on invalid UTF-8	`InvalidData`
`write_all` into a full buffer of fixed size	`WriteZero`
A syscall that a signal stops	`Interrupted`
An invalid seek (to before byte 0, tutorial 8.3)	`InvalidInput`

Two kinds have similar names. InvalidInput means that your arguments were incorrect. InvalidData means that the contents of the stream were malformed.

The purpose of the taxonomy is to let you select a strategy for each category:

// Returns the name of the strategy for an error.
fn policy(err: &io::Error) -> &'static str {
    match err.kind() {
        // A missing file is frequently an expected case, not a failure.
        ErrorKind::NotFound => "create it and continue",
        // Temporary conditions: try again a limited number of times.
        ErrorKind::Interrupted | ErrorKind::TimedOut | ErrorKind::WouldBlock => "retry",
        // A retry does not correct a permission problem.
        ErrorKind::PermissionDenied => "report and abort",
        // Truncated or corrupt input: do not trust partial results.
        ErrorKind::UnexpectedEof | ErrorKind::InvalidData => "reject the input",
        // All other kinds, and each kind that a future release adds.
        _ => "propagate",
    }
}

// `ErrorKind::X.into()` makes a kind-only io::Error (see the previous section).
assert_eq!(policy(&ErrorKind::TimedOut.into()), "retry");
assert_eq!(policy(&ErrorKind::AddrInUse.into()), "propagate");   // the `_` arm

The wildcard arm is mandatory, because ErrorKind is #[non_exhaustive]. The standard library continues to add kinds. For example, 1.83 stabilized StorageFull, NotADirectory, IsADirectory, and more. Then 1.85 stabilized QuotaExceeded and CrossesDevices. Your match must continue to compile when a release adds a kind.

Interrupted: The Kind You Retry

ErrorKind::Interrupted means that the operation did nothing: call it again. On Unix, it corresponds to EINTR. A signal arrived during a syscall, and the system stopped the operation before it moved any bytes. Interrupted is the one kind that is always safe to retry immediately. The helper methods of the standard library do this retry. A manual loop does it as follows:

// flaky: a mock reader. It returns Interrupted two times, then it reads "payload".
// buf: [u8; 7], the destination buffer.
loop {
    let n = match flaky.read(&mut buf) {
        Ok(n) => n,                                                 // n bytes are in buf
        Err(e) if e.kind() == ErrorKind::Interrupted => continue,   // no bytes moved: call again
        Err(e) => return Err(e),                                    // each other kind is fatal here
    };
    // ... use the n bytes ...
    break;
}

The responsibility has two parts. A manual read loop must handle Interrupted itself. But read_to_end, read_exact, write_all, and io::copy each retry it internally. Example 08_12 proves the two parts with a mock reader that injects two interruptions. The manual loop makes three attempts. read_to_end hides the interruptions fully.

08_12_errorkind_matching.rs prints:

produced: NotFound, UnexpectedEof, InvalidData, WriteZero
policy: NotFound->create, TimedOut->retry, Eof->reject, other->propagate
Interrupted: manual loop took 3 attempts; read_to_end hid them

All assertions passed.

Custom Payloads: get_ref, into_inner, downcast

Error::new accepts any payload that implements Error + Send + Sync + 'static, not only strings. Thus an io::Error can carry a typed value. Generic callers see a kind and a Display message. Callers that know the payload type can recover the structured value.

// A structured payload. PartialEq lets you compare a recovered payload by value.
#[derive(Debug, PartialEq, Eq)]
struct ChecksumMismatch { expected: u32, actual: u32 }
// ... Display + Error impls ...
// Display prints "checksum mismatch: expected deadbeef, got 0badf00d" for the value below.

// A simulated block read: block 7 fails, each other block succeeds.
fn read_block(block: u32) -> io::Result<Vec<u8>> {
    if block == 7 {
        return Err(io::Error::new(
            ErrorKind::InvalidData,                                           // the kind
            ChecksumMismatch { expected: 0xDEAD_BEEF, actual: 0x0BAD_F00D },  // the payload
        ));
    }
    Ok(vec![0u8; 512])
}

Three methods recover the payload. They differ in ownership:

get_ref() borrows the payload as &dyn Error. Use it with downcast_ref::<T>() to read the fields and not consume the error.
into_inner() consumes the error and returns the boxed payload. It returns None for kind-only errors and for OS errors.
Error::downcast::<T>() (stable since 1.79) does the two steps in one call. If the type matches, it returns the payload by value. If the type does not match, it returns the original error with no change, so a failed attempt has no cost:

// err: io::Error, for example from read_block(7). downcast consumes it.
match err.downcast::<ChecksumMismatch>() {
    // mismatch: ChecksumMismatch, by value. This arm prints "expected deadbeef".
    Ok(mismatch) => println!("expected {:08x}", mismatch.expected),
    // A different payload type: downcast returns the same io::Error, with no change.
    Err(err) => return Err(err),
}

Code that follows the source() chain (tutorial 2.2) must know one detail. The payload is part of the io::Error. It is not a separate link behind it. The Display output of the io::Error is the message of the payload, and its source() returns the source of the payload. Thus, if you put the error from read_block in an application error, the chain has 2 links, not 3. To get the payload as a value, use downcast.

io::Result and Propagation Patterns

io::Result<T> is only a type alias. It adds no new semantics. It makes signatures shorter, and these signatures occur everywhere in I/O code:

fn parse_port(input: &str) -> io::Result<u16> {
    input.trim().parse()   // Result<u16, ParseIntError>
        // Convert the ParseIntError: select a kind and attach a String payload.
        .map_err(|e| io::Error::new(ErrorKind::InvalidInput, format!("bad port: {e}")))
}

// parse_port("8080") returns Ok(8080).
// parse_port("http") returns an Err with the kind InvalidInput and the message
// "bad port: invalid digit found in string".

Import std::io and write the qualified name (io::Result). Then the alias does not shadow the Result of the prelude in the remainder of the module.

How to select a propagation pattern:

In modules that do much I/O: return io::Result and use ? freely. At the module boundary, wrap errors of other types with Error::new or Error::other. Select the most specific ErrorKind that you can justify.
At application boundaries: convert the io::Error into your domain error (Domain 2 patterns). Keep the io::Error as the source(), so that you do not lose context.
In main: fn main() -> io::Result<()> is valid because io::Error implements the necessary traits. Most examples in this domain use this signature.

Summary

Concept	Key point
`io::Result<T>`	Alias for `Result<T, io::Error>`. Use the qualified name.
Three representations	Kind only, OS code, or kind + boxed payload
`From<ErrorKind>`	Allocation-free. The `Display` text is the standard text of the kind.
`Error::new(kind, payload)`	Attaches any payload that is `Error + Send + Sync + 'static`
`Error::other(payload)`	Short form of `new(ErrorKind::Other, ...)` (1.74). `Other` never comes from the OS.
`Error::from_raw_os_error`	Wraps an errno. `raw_os_error()` returns `Some(code)`.
`kind()` matching	One strategy for each category. Never match on `Display` strings.
`#[non_exhaustive]`	Always keep a `_` arm, because releases continue to stabilize new kinds
`InvalidInput` vs `InvalidData`	Incorrect arguments vs. malformed stream contents
`Interrupted`	Retry immediately. The std helpers retry it for you.
`get_ref` / `into_inner` / `downcast`	Borrow, extract, or recover the typed payload (`downcast`: 1.79)
`source()` detail	The payload is part of the error, not a separate chain link

Code Examples

File	Description
`08_11_error_construction.rs`	The four constructors, `io::Result`, `raw_os_error`, and assertions that show the three internal representations
`08_12_errorkind_matching.rs`	Real operations that produce real kinds, a recovery policy with `match`, and the retry of `Interrupted` with a mock reader
`08_13_custom_payloads_downcast.rs`	Structured payloads: `get_ref` with `downcast_ref`, `into_inner`, `Error::downcast` on a match and on a mismatch, and the 2-link `source()` chain

8.5 · stdin, stdout, stderr: Standard Streams

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::stdin, std::io::stdout, std::io::stderr, std::io::Stdin, std::io::Stdout, std::io::Stderr, print!, println!, eprint!, eprintln!

Introduction

Each process starts with three streams. Rust gives each stream a global, thread-safe handle: io::stdin(), io::stdout(), and io::stderr(). The handles look simple, and the first println! of each Rust programmer uses one. But the handles contain three design decisions:

A locking model: each access synchronizes on a global lock.
A buffering policy: stdout is line-buffered, and stderr is unbuffered.
A failure mode: println! panics when stdout is a closed pipe.

You must know all three to write a CLI tool that behaves correctly.

This tutorial shows:

The three handles, their lock types (StdinLock, StdoutLock), and why it is important to hold a lock during a burst of writes.
print!/println!/eprint!/eprintln! and their contract: they panic on failure.
The buffering differences between stdout and stderr, and when to call flush.
Broken pipes: why println! panics under | head, and the graceful-exit pattern.
The testable-CLI pattern: how to write the program core against the BufRead and Write traits.

Three Global Streams and Their Locks

Each handle gives access to a resource that is global to the process, and a lock protects that resource. Stdout also wraps its output in a LineWriter (tutorial 8.1), which is the cause of the line buffering. Since Rust 1.61, the lock guards are 'static. Thus io::stdout().lock() is valid as one expression, and you do not need a separate binding for the handle.

Figure: The standard streams' locking and buffering model

08_5_standard_streams_1

The macros use the same mechanism. On each call, println! acquires the stdout lock, formats, writes, and releases the lock. For a burst of output, this has two disadvantages: an overhead on each call, and a risk of interleaved output. A line from a different thread can appear between any two of your calls. The usual pattern is to hold the lock:

use std::io;
use std::io::Write;   // necessary for writeln! on the lock

// `?` needs a function that returns io::Result.
let mut out = io::stdout().lock();          // StdoutLock<'static>
for row in 1..=3u32 {
    // No other thread can write to stdout between these lines.
    writeln!(out, "report row {row}: value={}", row * 100)?;
}
// `out` drops at the end of its scope, and the drop releases the lock.

Two more details are important. First, the stdout lock is reentrant: a println! from the same thread while you hold the lock does not deadlock. (A different thread blocks.) Second, StdinLock implements BufRead, so you can call read_line and lines (tutorial 8.1) directly on it.

The Print Macros and Their Contract

The four macros write to the two output streams:

Macro	Stream	Newline
`print!`	stdout	no
`println!`	stdout	yes
`eprint!`	stderr	no
`eprintln!`	stderr	yes

The macros are convenient, but they have a disadvantage. They return (), so they cannot report a failed write. Instead, they panic with a message such as failed printing to stdout: Broken pipe (os error 32). Some programs have their output as their product (each program that you can use in a pipeline). For those programs, that panic is a real category of bug. Slide 5 shows how to handle it.

The general rule to select a stream has two parts. stdout is for the payload: the data that the user requested, and that the subsequent pipeline stage reads. stderr is for commentary: progress, warnings, and diagnostics. cargo obeys this rule. Its build messages go to stderr, so cargo run --quiet | jq sends only the program output to jq.

A read from stdin blocks until input arrives, so the examples in this domain never read it. Comments in the examples show the connection (io::stdin().lock() as a BufRead). The pattern on slide 6 lets you test the read code with no keyboard.

Buffering: stdout vs. stderr

The two output streams make opposite trade-offs between latency and throughput:

stdout is line-buffered. Bytes collect in an internal LineWriter. The LineWriter writes them to the OS when a \n arrives, on flush, or when the buffer is full. This is good for throughput, but the result can surprise you when you print a partial line.
stderr is unbuffered. Each write goes directly to the OS. Diagnostics arrive even if the process crashes immediately after the write. An error channel needs this behavior.

The typical symptom is a prompt that does not appear.

// This snippet needs the same imports as the previous snippet.
let mut out = io::stdout().lock();
write!(out, "progress: ")?;   // no newline: the text stays in the buffer
out.flush()?;                 // now the text is visible
for step in ["25%", "50%", "75%", "100%"] {
    write!(out, "{step} ")?;
    out.flush()?;             // makes each increment visible immediately
}
writeln!(out)?;               // the newline ends the line and flushes it

The strict form of the rule is as follows. The documentation promises line buffering only when stdout is a terminal. The standard library may use block buffering for pipes in a future release. Code that needs a partial line to be visible immediately must call flush. That call is correct with each buffering policy.

08_14_stdout_stderr_locking.rs prints these lines. The three [diag] lines go to stderr:

one println!, one lock/unlock round-trip
a second println!, a second round-trip
report row 1: value=100
report row 2: value=200
report row 3: value=300
println! while holding the lock: fine on the same thread
progress: 25% 50% 75% 100%
[diag] this line goes to stderr, unbuffered   # (stderr: the position relative to stdout can vary)
[diag] stderr lock held for a two-line burst   # (stderr)
[diag] no flush needed — stderr writes go straight out   # (stderr)
stdin().lock() is BufRead — reading skipped to stay non-interactive

All assertions passed.

Broken Pipes: Why println! Panics

Run yourtool | head -3. head exits after three lines, and its exit closes the read end of the pipe. The subsequent write to stdout fails with ErrorKind::BrokenPipe. If that write came from println!, the process panics.

On Unix, the startup code of Rust sets SIGPIPE to ignored. Thus the OS does not kill the process silently, which is the default for shell tools. Instead, the failure arrives as a regular io::Error. println! cannot return the error, so it panics.

A closed pipe in the middle of a pipeline is not an error. It means that the consumer has all the data that it wants. The solution has two parts: write through fallible calls, and filter the one harmless kind:

// Writes `total` lines to `out`. Returns the number of lines that it wrote.
fn stream_lines<W: Write>(out: &mut W, total: u32) -> io::Result<u32> {
    let mut written = 0u32;
    for n in 1..=total {
        writeln!(out, "result line {n}")?;   // a failed write is an Err value, not a panic
        written += 1;
    }
    Ok(written)
}

// The filter: BrokenPipe is a normal end, and each other error is a failure.
fn run<W: Write>(out: &mut W) -> io::Result<()> {
    match stream_lines(out, 100) {
        Ok(_) => Ok(()),
        Err(e) if e.kind() == ErrorKind::BrokenPipe => Ok(()),   // the consumer closed the pipe
        Err(e) => Err(e),                                        // a real failure: propagate it
    }
}

// With real output, pass the locked stdout: run(&mut io::stdout().lock())

A real broken pipe needs a second process. Thus example 08_16 reproduces the failure deterministically with a mock writer. The mock writer accepts 45 bytes, and then each write returns BrokenPipe. The example proves two facts. The failure occurs on the fourth writeln! call. The filter accepts only BrokenPipe: a StorageFull error continues to propagate.

CLI tools in production, such as ripgrep, use this same graceful-exit pattern.

The Testable-CLI Pattern

All the topics of this domain lead to one structural decision. The core of the program should take impl BufRead and impl Write parameters. It should not use the global streams directly.

// Copies the lines of `input` to `output` and puts a number before each non-empty line.
// Returns the count of numbered lines.
fn number_lines<R: BufRead, W: Write>(input: R, mut output: W) -> io::Result<u32> {
    let mut numbered = 0u32;
    for line in input.lines() {
        let line = line?;                 // line: String, with no newline at the end
        if line.is_empty() {
            writeln!(output)?;            // an empty line gets no number
        } else {
            numbered += 1;
            writeln!(output, "{numbered:>4}  {line}")?;   // for example "   1  alpha"
        }
    }
    output.flush()?;                      // sends buffered bytes to the destination
    Ok(numbered)
}

The same function then operates in each configuration:

Tests: call number_lines(&b"alpha\n\nbeta\n"[..], &mut Vec::new()). You can assert the exact bytes, and you do not start a process. The in-memory types from tutorial 8.2 are valid arguments, and io::empty() and io::sink() cover edge cases.
Production: call number_lines(stdin.lock(), stdout.lock()). The locks are the trait implementations that you need. Because you hold them for the full run, you also get the burst-lock advantage from slide 2 at no cost.

This pattern is not a feature of the standard library. It is the design pattern that the trait hierarchy exists to make possible. It is also the reason why tutorials 8.1–8.4 told you to write code against traits.

Summary

Concept	Key point
`stdin()` / `stdout()` / `stderr()`	Global, thread-safe handles. Repeated calls are cheap.
`lock()`	`'static` guards (1.61). Lock one time for a burst: no interleaved output, less overhead.
Reentrancy	A `println!` on the same thread is safe while you hold the stdout lock
`StdinLock`	Implements `BufRead`: call `read_line` and `lines` directly
`print!` family	The macros panic on a write failure, because they cannot return errors
stdout buffering	Line-buffered through a `LineWriter`. Call `flush` for partial lines.
stderr buffering	None. Diagnostics arrive even if the process crashes, and they do not go into piped stdout.
stdout vs stderr	Payload vs. commentary. Keep commentary out of pipelines.
`BrokenPipe`	Normal under `\| head`. Map it to `Ok(())` and propagate all other errors.
SIGPIPE on Unix	The Rust startup code ignores it. Failures arrive as `io::Error`, and the OS does not kill the process.
Testable-CLI pattern	The core takes `impl BufRead` + `impl Write`: locks in production, buffers in tests

Code Examples

File	Description
`08_14_stdout_stderr_locking.rs`	Output with a lock on each call and with a held lock, reentrancy, `flush` of partial lines, unbuffered stderr, `StdinLock` as `BufRead`
`08_15_testable_stdio_pattern.rs`	A program core that is generic over traits: a test in memory, edge cases, and a connection to a real `StdoutLock`
`08_16_broken_pipe_handling.rs`	A deterministic `BrokenPipe` from a mock pipe, the graceful-exit filter, and proof that other errors continue to propagate

9.1 · Reading, Writing, and Creating Files

Domain 9 — Filesystem Operations Duration: ~15 minutes Library components: std::fs::File, std::fs::OpenOptions, std::fs::read_to_string, std::fs::read, std::fs::write

Introduction

Each file API in std::fs is a thin wrapper around the same OS primitive: open a file with a set of flags. Rust gives you that primitive at three levels of convenience. Most of the skill is to select the level that fits the task:

One-line functions: fs::read_to_string, fs::read, fs::write. One call opens the file, transfers the data, and closes the file.
Constructors: File::open, File::create, File::create_new. These are named short forms for the three most common flag sets.
The builder: OpenOptions. You write each flag explicitly, for all other cases.

This tutorial shows:

The three levels.
What you get because File implements Read + Write + Seek: random-access I/O with one handle.
How to set permissions at creation time.

Each example writes only in a unique scratch directory in std::env::temp_dir(), and removes that directory on exit. A Drop guard does the removal, so the cleanup occurs even if an assertion panics. You can use this RAII pattern in your own tests.

Figure: Three levels of file-opening convenience

09_1_files_read_write_create_1

The Convenience Layer: fs::write, fs::read_to_string, fs::read

When you need the whole file and do not need to keep it open, the free functions in std::fs are the idiomatic selection. Each function opens the file, transfers the data, and closes the file in one call. You do not manage a handle:

// config: PathBuf, the path of `app.conf` in the scratch directory.
fs::write(&config, "retries = 3\nverbose = true\n")?; // creates or truncates, then writes 27 bytes

let text = fs::read_to_string(&config)?;   // whole file → String (validates UTF-8)
let bytes = fs::read(&config)?;            // whole file → Vec<u8> (raw bytes, no validation)
assert_eq!(bytes, text.as_bytes());        // the two reads give the same 27 bytes

The difference between the two read functions is important. read_to_string requires valid UTF-8, and fails with ErrorKind::InvalidData if the content is not valid UTF-8. fs::read returns the bytes of the file as they are:

// blob: PathBuf, the path of `blob.bin` in the scratch directory.
fs::write(&blob, [0xFF, 0xFE, 0x00, 0x42])?;   // 4 bytes that are not valid UTF-8
let err = fs::read_to_string(&blob).expect_err("0xFF is never valid UTF-8");
assert_eq!(err.kind(), ErrorKind::InvalidData);
assert_eq!(fs::read(&blob)?.len(), 4);         // the raw read succeeds

These one-line functions keep the full content in memory. When a file is large, or when you need only the start of the file, use the streaming APIs (BufReader and Read::read in tutorial 8.1).

File::open, File::create, File::create_new

The three constructors express the three intents that you have 95% of the time:

Constructor	Access	If the file is missing	If the file is present
`File::open`	read	`NotFound`	opens the file
`File::create`	write	creates the file	truncates the file to 0 bytes
`File::create_new`	write	creates the file	`AlreadyExists`

Two behaviors are important. First, File::create truncates the file at open time. The old contents are gone before you write the first byte:

// `config` still holds the 27 bytes from the first snippet.
let mut file = File::create(&config)?;               // opens for write and truncates
assert_eq!(fs::metadata(&config)?.len(), 0);         // the file is empty before the first write
file.write_all(b"retries = 5\n")?;                   // writes 12 bytes

Second, File::create_new checks that the file does not exist and creates it as one atomic operation. A manual if !path.exists() { File::create(path) } has a time-of-check-to-time-of-use (TOCTOU) race. A different process can create the file between your check and your creation. With create_new, the OS does the two steps as one. Thus create_new is the tool to use for lock files and "first run" markers:

// `config` exists, so the exclusive creation fails. The file does not change.
let err = File::create_new(&config).expect_err("config already exists");
assert_eq!(err.kind(), ErrorKind::AlreadyExists);

09_01_file_open_create.rs prints:

read_to_string: 27 bytes
read:           27 bytes (same content, unvalidated)
read_to_string on binary blob: InvalidData
File::open on a missing file:  NotFound
File::create truncated the old config, then wrote 12 bytes
File::create_new on existing:  AlreadyExists
File::create_new created first-run.marker

All assertions passed.

The OpenOptions Builder

Some tasks do not fit the constructors:

an append to a log
a change of bytes in place
a handle with read and write access

For these tasks, write the flags explicitly with OpenOptions. The figure shows the decision tree:

Figure: Choosing OpenOptions flags

09_1_files_read_write_create_2

append(true) implies write access. Each write first moves to the end of the file, and the move and the write are one atomic operation. Thus the mode is safe even when several processes append to the same log:

// log: PathBuf, the path of `app.log`. `create(true)` creates the file on the first use.
let mut file = OpenOptions::new().append(true).create(true).open(&log)?;
writeln!(file, "boot: ok")?;   // adds one line at the end of the file

A common error is write(true) without truncate(true). The writes replace bytes in place from position 0, and the bytes that you do not replace stay in the file:

// state: PathBuf, the path of `state.txt`.
fs::write(&state, "HELLO WORLD")?;                              // 11 bytes
let mut file = OpenOptions::new().write(true).open(&state)?;    // no truncate: the content stays
file.write_all(b"bye")?;                                        // replaces bytes 0..3 only
assert_eq!(fs::read_to_string(&state)?, "byeLO WORLD");         // the old tail "LO WORLD" stays

The standard library rejects flag combinations that have no meaning. It returns ErrorKind::InvalidInput before it calls the OS. The example shows four such combinations:

truncate without write access
append together with truncate
create without write or append access
no access mode

Clippy's nonsensical_open_options lint also finds the first two combinations at compile time.

09_02_openoptions_builder.rs prints:

app.log after two appends:
boot: ok
listen: 127.0.0.1:8080

write(true) over "HELLO WORLD":  "byeLO WORLD"
write(true).truncate(true):      "bye"
second create_new on instance.lock: AlreadyExists
4 nonsensical flag combinations all rejected: InvalidInput

All assertions passed.

File as Read + Write + Seek

File implements all three I/O traits (tutorials 8.1 and 8.3). Thus one handle that you open with .read(true).write(true) can do random-access I/O. Random-access I/O is the base of each database file, index, and archive format. With fixed-width records, slot N always starts at byte N * RECORD_SIZE:

// RECORD_SIZE is 8: each record is a u32 id and a u32 score, as little-endian bytes.
fn write_record(file: &mut File, slot: u64, id: u32, score: u32) -> io::Result<()> {
    file.seek(SeekFrom::Start(slot * RECORD_SIZE))?;   // moves the cursor to the start of the slot
    file.write_all(&id.to_le_bytes())?;                // 4 bytes
    file.write_all(&score.to_le_bytes())               // 4 bytes. This result is the return value.
}

SeekFrom has three variants:

Start(u64): an absolute position.
Current(i64): a position relative to the cursor.
End(i64): a position relative to the end of the file. For example, SeekFrom::End(-8) goes to the last 8-byte record, and you do not need to know the number of records.

stream_position() returns the position of the cursor. rewind() is a short form of seek(SeekFrom::Start(0)).

You can seek past the end of the file. The file grows on the subsequent write, and the gap that you did not write reads as zeros. On many filesystems, the gap is a sparse "hole" that uses no disk space:

// `file` holds 3 records (24 bytes). `read_record(&mut file, slot)` returns (id, score).
write_record(&mut file, 9, 1010, 999)?;         // slots 3..=8 were never written
assert_eq!(file.metadata()?.len(), 80);         // 10 slots of 8 bytes
assert_eq!(read_record(&mut file, 5)?, (0, 0)); // the hole reads as zeros

09_03_read_write_seek.rs prints:

wrote 3 records, cursor at byte 24
slot 1 updated in place: (1002, 500)
SeekFrom::End(-8) reads the last record: id 1003
slot 9 written; file is now 80 bytes; unwritten slot 5 reads as (0, 0)

All assertions passed.

Setting Permissions on Open

On Unix, OpenOptionsExt::mode sets the permission bits of the file atomically at creation. This is important for secrets. If you create the file and then call chmod, the file has the default permissions for a short time. During that time, a different process can open the file. .mode(0o600) removes that interval:

use std::os::unix::fs::OpenOptionsExt;   // adds `mode` to OpenOptions (Unix only)

// secrets: PathBuf, the path of `credentials.toml`.
let mut file = OpenOptions::new()
    .write(true)
    .create_new(true)     // fails if the file exists
    .mode(0o600)          // owner: read and write. Group and other: no access.
    .open(&secrets)?;

The process umask filters the mode that you request. A umask can only clear bits. Thus a 0o600 request guarantees that group and other get no access, in any environment. The example shows two more details:

If you create a file with .mode(0o444) (read-only), you can still write to it through the handle that created it. The OS checks the permissions at open time, not for each write. The OS refuses a new open for write access (the root user is an exception).
To unlink a file, you need write permission on the directory, not on the file. Thus read-only files do not prevent the cleanup of the scratch directory on Unix.

Windows has no mode bitmask, because it controls access with ACLs. Its std::os::windows::fs::OpenOptionsExt has access_mode and attributes as the alternative. On Windows, the example prints a skip message.

On Unix, 09_04_permissions_on_open.rs prints:

credentials.toml mode: 0o600 (requested 0o600)
receipt.txt readonly:  true
re-open for write:     PermissionDenied   # (varies: the root user gets "allowed")

All assertions passed.

Summary

Concept	Key point
`fs::write` / `fs::read_to_string` / `fs::read`	One-line functions for a whole file. There is no handle to manage.
`read_to_string` vs `read`	`read_to_string` requires valid UTF-8 (`InvalidData` on failure). `read` returns raw bytes.
`File::open`	Read-only access. The call returns `NotFound` if the file is missing.
`File::create`	Write access. The call creates the file or truncates it at open time.
`File::create_new`	Exclusive creation: one atomic check and creation, with no TOCTOU race
`OpenOptions`	The full flag builder that all three constructors use
`append(true)`	The flag implies write access. Each write first moves to the end of the file.
`write` without `truncate`	Overwrite in place. Old bytes stay after the bytes that you wrote.
Invalid flag combinations	The standard library rejects them with `ErrorKind::InvalidInput` before it calls the OS.
`File: Read + Write + Seek`	One handle with random access. `SeekFrom::{Start, Current, End}` specifies the position.
Seek past EOF	Legal. The gap reads as zeros (a sparse hole).
`OpenOptionsExt::mode` (Unix)	The method sets the permissions atomically at creation. The umask filters the mode.

Code Examples

File	Description
`09_01_file_open_create.rs`	`File::open`/`create`/`create_new`, `fs::write`/`read_to_string`/`read`, UTF-8 vs raw reads
`09_02_openoptions_builder.rs`	`OpenOptions` flags, append-only logs, the write-without-truncate error, rejected combinations
`09_03_read_write_seek.rs`	A fixed-width record store: random access with one `Read + Write + Seek` handle
`09_04_permissions_on_open.rs`	Unix `OpenOptionsExt::mode`: private files from the moment of creation (the example skips on Windows)

9.2 · Directory Operations and Metadata

Domain 9 — Filesystem Operations Duration: ~15 minutes Library components: std::fs, std::fs::DirEntry, std::fs::Metadata, std::fs::Permissions, std::fs::FileType, std::fs::FileTimes

Introduction

Filesystem code is less simple when it uses directories:

A creation can fail in the middle of a chain of missing directories.
A listing arrives in unspecified order.
Metadata has fields that some platforms do not have.

This tutorial gives you the mental model that you need to write correct portable code:

The four directory functions (create_dir, create_dir_all, remove_dir, remove_dir_all), and the exact ErrorKind that each one returns on failure.
fs::read_dir and the DirEntry iterator, with a manual recursive walk.
The Metadata struct: type checks, sizes, the three timestamps, and Permissions.
Why timestamp code must assert sanity properties and never exact values, and why you must handle the Result that created() returns.
How to set timestamps by path with fs::set_times and fs::set_times_nofollow (stabilized in 1.99).

Creating Directories

fs::create_dir creates exactly one level. The parent must exist, and the directory itself must not exist. Each of the two failures has an exact error kind:

// tmp: the path of an existing scratch directory. src: the path `tmp.join("src")`.
fs::create_dir(&src)?;                                    // ok: the parent exists
let err = fs::create_dir(&src).expect_err("src already exists");
assert_eq!(err.kind(), ErrorKind::AlreadyExists);         // a second call is an error

let deep = tmp.join("target/debug/deps");                 // `target` does not exist
let err = fs::create_dir(&deep).expect_err("parents missing");
assert_eq!(err.kind(), ErrorKind::NotFound);              // it does not create the parents

fs::create_dir_all is the equivalent of mkdir -p. It creates each missing ancestor. It also succeeds if the directory already exists, and this is important. Because the call is idempotent, you can call it unconditionally at startup:

// `deep` is the path target/debug/deps from the previous snippet.
fs::create_dir_all(&deep)?;   // creates target/, target/debug/, target/debug/deps
fs::create_dir_all(&deep)?;   // second call: also Ok, because the directory exists

Removing Directories

Removal is not symmetric with creation, and the difference is a safety feature. fs::remove_dir refuses a directory that is not empty, with ErrorKind::DirectoryNotEmpty:

// `src` is the directory from the first snippet. It is empty at this point.
fs::write(src.join("main.rs"), "fn main() {}\n")?;       // puts one file in src/
let err = fs::remove_dir(&src).expect_err("src has a file in it");
assert_eq!(err.kind(), ErrorKind::DirectoryNotEmpty);

fs::remove_dir_all is the rm -rf of the standard library. It removes the contents first and then the directory. It does not ask for confirmation, and it does not use a trash can. Remember two things:

create_dir_all accepts a directory that exists, but remove_dir_all does not accept a target that is missing. The removal of a path that does not exist returns NotFound.
On Windows, the removal can fail on a read-only file on some filesystems (FAT32, for example). Restore the write permission before you remove a scratch tree (see the section "Permissions and fs::set_permissions").

09_05_create_remove_dirs.rs prints:

create_dir src/:                ok
create_dir src/ again:          AlreadyExists
create_dir target/debug/deps:   NotFound
create_dir_all (twice):         ok — idempotent
remove_dir on non-empty src/:   DirectoryNotEmpty
remove_dir on empty deps/:      ok
remove_dir_all src/:            ok (file inside deleted too)
remove_dir_all src/ again:      NotFound

All assertions passed.

read_dir and the DirEntry Iterator

fs::read_dir returns an iterator of io::Result<DirEntry>. Each entry can fail independently (a file may disappear during the listing), and thus the item type is a Result. The iterator never includes . and .., which POSIX readdir does include.

The most important fact about read_dir is that the iteration order is unspecified. The order differs between platforms, between filesystems, and even between runs. Code that asserts or displays a listing must first collect the entries and sort them:

// root: &Path, a directory that holds notes.txt, readme.md, and src/.
let mut names = Vec::new();
for entry in fs::read_dir(root)? {
    let entry = entry?;   // each item is an io::Result<DirEntry>
    // file_name() returns an OsString. This line converts it to a String.
    names.push(entry.file_name().to_string_lossy().into_owned());
}
names.sort(); // without the sort, the comparison can fail on some runs
assert_eq!(names, ["notes.txt", "readme.md", "src"]);

DirEntry gives you three methods:

path() returns the root joined with the name.
file_name() returns only the final component.
file_type() returns the type of the entry. On most platforms, it is cheaper than a full metadata() call, because the OS returns the type together with the name.

A recursive walk needs only an explicit stack. An explicit stack has no recursion-depth limit, and the walkdir crate uses the same approach internally:

// root: &Path. found: Vec<PathBuf>, empty at the start.
let mut stack = vec![root.to_path_buf()];        // the directories that the walk must read
while let Some(dir) = stack.pop() {
    for entry in fs::read_dir(&dir)? {
        let entry = entry?;
        if entry.file_type()?.is_dir() {
            stack.push(entry.path());            // the loop reads this subdirectory later
        } else if entry.path().extension() == Some(OsStr::new("rs")) {
            found.push(entry.path());            // keeps each file with the extension "rs"
        }
    }
}
found.sort();                                    // gives the result a stable order

09_06_read_dir_walk.rs prints:

top level (sorted): ["notes.txt", "readme.md", "src"]
files: 2, dirs: 1

.rs files found by the walk:
  src/lib.rs                    # (Windows prints backslashes)
  src/main.rs
  src/util/helpers.rs

All assertions passed.

The Metadata Struct

fs::metadata(path) calls stat. It returns all the data that the OS has about the path, in one snapshot. fs::symlink_metadata is the variant that does not follow a symlink. metadata follows a symlink and describes the target. symlink_metadata describes the link itself (tutorial 9.4 gives the details of links).

Figure: Metadata and its satellite types

09_2_directories_metadata_1

// file: PathBuf, the path of `data.log`, which holds the 10 bytes "0123456789".
let meta = fs::metadata(&file)?;            // one snapshot: type, size, timestamps, permissions
assert!(meta.is_file() && !meta.is_dir() && !meta.is_symlink());
assert_eq!(meta.len(), 10);                 // for a file, len() is the exact byte count

Be careful with one case: the len() of a directory is platform-defined (block sizes, entry counts, or other data that the filesystem stores). It is never the "total size of contents". To compute the size of a directory, walk its contents.

Timestamps: Sanity, Not Exactness

Metadata gives three timestamps, each as an io::Result<SystemTime>. The Result has a purpose:

modified(): available on all the major platforms. This is the timestamp that you can rely on.
accessed(): production systems frequently disable it or update it in batches (noatime/relatime mounts). Use it only as information.
created(): may legitimately fail with ErrorKind::Unsupported, because many filesystems do not record the creation time. Portable code uses match on the result and does not unwrap it:

// meta: the Metadata of `data.log`. modified: SystemTime, from `meta.modified()?`.
match meta.created() {
    Ok(created) => assert!(created <= modified),   // the write came after the creation
    // The kind is ErrorKind::Unsupported on a filesystem that has no creation time.
    Err(e) => println!("created() unsupported on this filesystem: {:?}", e.kind()),
}

Never assert exact timestamp values. Assert properties:

ordering (created <= modified)
approximate recency (modified().elapsed()? < 1 hour)
monotonicity after writes

For monotonicity, use >= and not >. Filesystems round timestamps (some to whole seconds), so two writes that are close in time can have the same timestamp:

// Before this line, the example appends one byte to `file`. `modified` is the earlier timestamp.
let modified_after = fs::metadata(&file)?.modified()?;
assert!(modified_after >= modified);   // >= because the two timestamps can be equal

09_07_metadata_timestamps.rs prints:

data.log: is_file=true, len=10 bytes
scratch dir: is_dir=true
modified 0s ago (sanity: < 1 hour)                    # (the number of seconds can vary)
created() supported here; created <= modified holds   # (varies with the filesystem)
after append: len=11, modified moved forward (or equal): true

link.txt → target.txt:                                # (Unix only: Windows prints a skip line)
  metadata():         is_file=true, len=7
  symlink_metadata(): is_symlink=true

All assertions passed.

The rule "never assert exact values" has one exception: a timestamp that your program set. Since Rust 1.99, fs::set_times sets the access time and the modification time of a path. File::set_times (stable since 1.75) needs an open file handle. The path function also works for a directory. The FileTimes builder holds the values, and a timestamp that you do not set stays as it is. Archive extractors and sync tools use this function to restore a recorded modification time:

use std::fs::{self, FileTimes};
use std::time::{Duration, SystemTime};

// 2001-09-09 01:46:40 UTC, as seconds after the Unix epoch.
let archived = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000_000);
let times = FileTimes::new().set_accessed(archived).set_modified(archived);
fs::set_times(&report, times)?;                     // `report` is a path, not an open File

// Compare whole seconds: each filesystem stores a different precision.
let modified = fs::metadata(&report)?.modified()?;
assert_eq!(modified.duration_since(SystemTime::UNIX_EPOCH).unwrap().as_secs(), 1_000_000_000);

fs::set_times follows a symbolic link and changes the target. fs::set_times_nofollow changes the link itself, in the same way that symlink_metadata reads the link itself. 09_15_set_times.rs prints:

set_times: report.txt mtime = 1000000000 (seconds after the epoch)
set_times: archive/ mtime   = 1000000000
set_times on a missing path: NotFound
set_times_nofollow: link mtime = 1600000000, target mtime = 1000000000   # (Unix only)

All assertions passed.

Permissions and fs::set_permissions

Permissions is a snapshot value, not a live handle. A mutation changes only your copy in memory. The disk does not change until fs::set_permissions writes the value:

// report: PathBuf, the path of a writable file.
let mut perms = fs::metadata(&report)?.permissions();       // a copy of the permissions
perms.set_readonly(true);                                   // changes only the copy in memory
assert!(!fs::metadata(&report)?.permissions().readonly());  // the file on disk is still writable
fs::set_permissions(&report, perms)?;                       // writes the copy to the disk
assert!(fs::metadata(&report)?.permissions().readonly());   // now the file is read-only

The portable API is small by design: readonly() and set_readonly. It has only what Windows and Unix have in common. Two platform notes:

The set_readonly(false) hazard: on Unix, the call sets the write bits for owner, group, and others, so the file becomes world-writable. Clippy reports the call (permissions_set_readonly_false). On Unix, use explicit bits with PermissionsExt::from_mode(0o644). On Windows, only the read-only attribute exists, so set_readonly(false) is the correct call.
chmod semantics: fs::set_permissions applies exactly the bits that you give it, and the umask has no effect. This is different from OpenOptions::mode at creation (tutorial 9.1).

09_08_set_permissions.rs prints:

set_readonly(true) alone:      disk still writable
after fs::set_permissions:     readonly = true
write to read-only file:       PermissionDenied   # (varies: the root user can write)
restored writable:             readonly = false

deploy.sh mode: 0o755 (rwxr-xr-x) — executable bit set   # (Unix only)

All assertions passed.

Summary

Concept	Key point
`create_dir`	One level. `AlreadyExists` on a second call, `NotFound` when a parent is missing.
`create_dir_all`	The call creates the full chain. It is idempotent, so you can call it unconditionally.
`remove_dir`	Empty directories only. `DirectoryNotEmpty` for all others.
`remove_dir_all`	Recursive, as `rm -rf`. The call returns `NotFound` for a missing target.
`read_dir`	Iterator of `io::Result<DirEntry>`. The order is unspecified: sort before you assert.
`DirEntry::file_type()`	Cheaper than `metadata()`. Sufficient for the decisions of a walk.
`Metadata`	One `stat` snapshot: type, `len`, timestamps, permissions
`metadata` vs `symlink_metadata`	`metadata` follows the symlink. `symlink_metadata` describes the link itself.
`len()` on directories	A platform-defined value. Never the "size of contents".
`modified` / `accessed` / `created`	Reliable / informational / possibly `Unsupported`
Timestamp assertions	Sanity properties only: ordering, recency, `>=` monotonicity
`fs::set_times` / `set_times_nofollow` (1.99)	The functions set the access time and the modification time by path. The `nofollow` form changes a symlink itself.
`Permissions`	A snapshot value. Changes apply only through `fs::set_permissions`.
`set_readonly(false)`	World-writable on Unix. Use `PermissionsExt` mode bits there.

Code Examples

File	Description
`09_05_create_remove_dirs.rs`	The four directory functions and the exact `ErrorKind` of each failure mode
`09_06_read_dir_walk.rs`	`read_dir`, `DirEntry`, a sort for the unspecified order, a recursive walk with a stack
`09_07_metadata_timestamps.rs`	`Metadata` fields, sanity assertions for timestamps, a `created()` call that can fail, `symlink_metadata`
`09_08_set_permissions.rs`	`Permissions` as a snapshot, a read-only round trip, Unix mode bits with `PermissionsExt`
`09_15_set_times.rs`	`fs::set_times` / `fs::set_times_nofollow` (1.99) with `FileTimes`: timestamps by path for files, directories, and symlinks

9.3 · Path and PathBuf: Cross-Platform Path Manipulation

Domain 9 — Filesystem Operations Duration: ~15 minutes Library components: std::path::Path, std::path::PathBuf, std::path::Component, std::path::Prefix, std::path::MAIN_SEPARATOR

Introduction

A path looks like a string, but it is not a string. A path has platform-specific separators, and it may contain non-UTF-8 bytes (it wraps OsStr, see tutorial 3.4). Rust compares paths component by component, not character by character. std::path gives you two types and a parser that handle these properties:

Path and PathBuf are the borrowed type and the owned type. They correspond exactly to str and String.
The builder methods are join, push, pop, set_file_name, and set_extension. The anatomy accessors are parent, file_name, file_stem, and extension.
components() and ancestors() give the parsed, structural view.
canonicalize resolves a path through the filesystem, and std::path::absolute resolves it lexically. display() prints a path.

Almost all of these operations are lexical: they change the path in memory and never access the disk. The exceptions are canonicalize, exists(), and try_exists(), which ask the filesystem. The platform-specific problems occur with canonicalize (slide 6).

Path vs PathBuf: Borrowed vs Owned

Path is an unsized view. PathBuf owns a heap buffer and is mutable. The relationship is the same as for str and String, and that includes the conversion methods:

Figure: The Path/PathBuf ownership pair

09_3_path_pathbuf_1

use std::ffi::OsStr;
use std::path::Path;
use std::path::PathBuf;

let borrowed: &Path = Path::new("src/main.rs");     // a view of the literal, no allocation
let owned: PathBuf = PathBuf::from("src/main.rs");  // owns a copy on the heap
assert_eq!(borrowed, owned.as_path());              // as_path borrows the PathBuf as &Path
// PathBuf derefs to Path, so each Path method works on a PathBuf.
assert_eq!(owned.extension(), Some(OsStr::new("rs")));

The API design rule for &str and String (tutorial 3.1) applies here too. A function should take &Path, and it should return PathBuf when it builds a new path. To give callers the most flexibility, accept impl AsRef<Path>. The free functions of std::fs declare their path parameters with the same AsRef<Path> bound. Thus you can pass &str, String, &Path, or PathBuf to fs::read:

// `classify` is a helper of the example. It takes &Path and returns a label for the extension.
fn describe(path: impl AsRef<Path>) -> String {
    let path = path.as_ref(); // convert to &Path one time, then use only &Path
    format!("{:<12} → {}", path.display(), classify(path))
}

// One function accepts four argument types, and the caller does no conversion.
describe("src/lib.rs");               // &str
describe(String::from("Cargo.toml")); // String
describe(Path::new("README.md"));     // &Path
describe(PathBuf::from("build.log")); // PathBuf

Building Paths: join, push, pop

join borrows the path and returns a new PathBuf. push mutates the PathBuf in place. The same division exists between + and push_str on strings. The two methods insert the platform separator for you:

let manifest = Path::new("workspace").join("Cargo.toml"); // new PathBuf: workspace/Cargo.toml

let mut nested = PathBuf::from("reports");
nested.push("2026");          // reports/2026
nested.push("q3.md");         // reports/2026/q3.md
assert!(nested.pop());        // removes q3.md and returns true: reports/2026

Each program that handles paths must take one behavior into account: a join of an absolute path replaces the full base.

let hijacked = Path::new("safe/base").join("/etc/passwd");
assert_eq!(hijacked, Path::new("/etc/passwd"));   // join discards "safe/base" and gives no error
// push with an absolute path replaces the full path in the same way.

The documentation of join specifies this behavior. If a program joins user input to a sandbox directory, this behavior becomes a path-traversal vulnerability. Clippy's join_absolute_paths lint reports the cases where the argument is a literal. Validate untrusted input before you join it.

09_09_path_pathbuf_basics.rs prints the output below. The next slide explains the set_extension line:

Path::new / PathBuf::from agree: src/main.rs
built by push:  reports/2026/q3.md
join("/etc/passwd") hijacks the base: /etc/passwd
data.tar + set_extension("gz") = backup/data.gz
src/lib.rs   → Rust source
Cargo.toml   → manifest
README.md    → docs
build.log    → other

All assertions passed.

The Anatomy Accessors

Each path divides into named parts. Each accessor returns an Option, because not every path has every part:

Figure: Anatomy of a path

09_3_path_pathbuf_2

file_stem and extension divide the file name at the last dot only. Thus app-v2.4.1.tar.gz has the stem app-v2.4.1.tar and the extension gz, not tar.gz. A leading dot never starts an extension, so the full name of a dotfile is its stem:

let dotfile = Path::new(".gitignore");
assert_eq!(dotfile.file_stem(), Some(OsStr::new(".gitignore")));   // the full name is the stem
assert_eq!(dotfile.extension(), None);                             // there is no extension

The methods that mutate a path use the same last-dot rule. set_extension replaces all the text after the last dot, which can be an unexpected result:

let mut archive = PathBuf::from("backup/data.tar");
archive.set_extension("gz");                        // replaces "tar", does not append
assert_eq!(archive, Path::new("backup/data.gz"));   // not data.tar.gz
// To get data.tar.gz, call set_extension("tar.gz") on backup/data.

set_file_name replaces the full final component. with_extension and with_file_name are the variants that do not mutate: each returns a new PathBuf.

components() and ancestors()

components() gives the parsed view: an iterator of Component values. The variants are Prefix (Windows only), RootDir, CurDir, ParentDir, and Normal. The parser normalizes the path a little. It removes repeated separators and each interior ., but it keeps a leading . and each ..:

let messy = Path::new("./src//utils/./mod.rs");
// messy.components() yields 4 values:
// [CurDir, Normal("src"), Normal("utils"), Normal("mod.rs")]
// The parser removed the second "/" of "//" and the interior ".".

The parser keeps .. because it cannot resolve .. lexically. If a is a symlink to /x/y, then a/../b names /x/b, not the sibling b. Only canonicalize may remove .., because it reads the real filesystem. This one design decision explains most of the difference between the lexical APIs and the filesystem-backed APIs.

ancestors() yields the path and then each parent, to the top. It is the standard tool to go up the tree until you find Cargo.toml:

let chain: Vec<&Path> = Path::new("/var/log/syslog").ancestors().collect();
// chain is ["/var/log/syslog", "/var/log", "/var", "/"]

Comparisons also use components. starts_with and ends_with match full components, never substrings. strip_prefix is the inverse of join:

let config = Path::new("/etc/nginx/nginx.conf");
assert!(config.starts_with("/etc"));
assert!(!config.starts_with("/et"));    // "/et" is not a full component
assert!(!config.ends_with("conf"));     // "conf" is the extension, not a full component
// strip_prefix removes the base and returns the relative remainder as &Path.
assert_eq!(config.strip_prefix("/etc").unwrap(), Path::new("nginx/nginx.conf"));

On Windows, an absolute path starts with a Prefix component (C:, a UNC share) before RootDir. Unix does not have this concept. For this reason, the answer to "is this path absolute?" depends on the platform.

MAIN_SEPARATOR is / on Unix and \ on Windows, but you rarely need it. join and push insert the separator for you, and the parser accepts / on each platform.

09_10_path_components.rs prints:

path:      builds/2026-07/app-v2.4.1.tar.gz
  parent:    builds/2026-07
  file_name: app-v2.4.1.tar.gz
  file_stem: app-v2.4.1.tar
  extension: gz
.gitignore: stem=.gitignore, extension=None

"./src//utils/./mod.rs" parses to 4 components
ancestors of /var/log/syslog: 4 entries
starts_with("/etc")=true, starts_with("/et")=false (component-wise)

MAIN_SEPARATOR on this platform: '/'   # (Unix. Windows prints '\\')
(no Prefix components on Unix — absolute paths start at RootDir '/')   # (Unix. Windows prints "prefix: drive C")

All assertions passed.

canonicalize, std::path::absolute, and display

canonicalize asks the filesystem to resolve ., .., and symlinks. It returns an absolute path to the real file. Thus the path must exist: if it does not, the call fails with NotFound. canonicalize is the correct way to test if two lexically different paths name the same file:

// direct: PathBuf, <tmp>/data/metrics.csv (the file exists)
// messy:  PathBuf, <tmp>/data/../data/./metrics.csv (a different spelling of the same file)
// The code is in a function that returns io::Result, so `?` is permitted.
assert_ne!(direct, messy);                                  // the path values are different
assert_eq!(direct.canonicalize()?, messy.canonicalize()?);  // the two paths name one file

One problem occurs in practice on macOS. There, env::temp_dir() is below /var, and /var is a symlink to /private/var. Thus a canonicalized temp path does not start with the original temp_dir() string. Code that compares raw prefixes fails there. The portable rules are:

Assert the tail of a canonical path (canon.ends_with("data/metrics.csv")), never its prefix.
To compare two paths, canonicalize both sides first.

std::path::absolute is the lexical alternative. It uses the current directory to make a path absolute, but it resolves no symlinks and accesses no inodes. Thus it works on paths that do not exist yet, such as output locations and files that you will create.

Path does not implement Display, because a path may not be valid UTF-8. To print a path, call display(): it returns an adapter that always works but is lossy. When you need the strict answer, call to_str(), which returns None for a non-UTF-8 name.

exists() converts each I/O error to false. try_exists() returns the error to the caller. For a dangling symlink, try_exists() returns Ok(false) (tutorial 9.4).

09_11_canonicalize_absolute.rs prints the output below. In the first line, the program itself prints the … in place of the temp directory:

lexical form:  …/data/../data/./metrics.csv
canonical ends with data/metrics.csv: true
lexically different, canonically equal: true
temp root canonicalized to a DIFFERENT path (symlinked ancestor, e.g. /var)   # (macOS. With no symlinked ancestor, the line says "already canonical")
canonicalize on a missing path: NotFound
path::absolute on the same path: ok (works without touching disk)
display(): /var/folders/yk/rj3_nqx94_35jbstjmxqgzl80000gn/T/rust-tut-09-11-49609/data/metrics.csv   # (varies: temp directory and process id)
exists: file=true, ghost=false

All assertions passed.

Summary

Concept	Key point
`Path` / `PathBuf`	Borrowed view and owned buffer, the same pair as `str` and `String`
Function signatures	Take `&Path` (or `impl AsRef<Path>`), return `PathBuf`
`join` / `push`	`join` borrows and builds a new path. `push` mutates in place. Each inserts the separator for you
Absolute-join problem	`base.join("/abs")` discards the base. Validate untrusted input
`file_stem` / `extension`	They divide the name at the last dot. A leading dot never starts an extension
`set_extension`	It replaces the text after the last dot: `data.tar` + `gz` → `data.gz`
`components()`	Parsed view. It removes `//` and each interior `.`, and it keeps `..` (symlink semantics)
`ancestors()`	The path and each parent, for a search upward to the nearest marker file
`starts_with` / `ends_with`	Full components, never substrings
`canonicalize`	Filesystem-backed. The path must exist. It resolves symlinks
macOS temp problem	`/var` → `/private/var`: assert canonical tails, compare canonical forms
`std::path::absolute`	Lexical absolute path. It works on paths that do not exist yet
`display()`	Lossy printing for paths that are possibly not UTF-8

Code Examples

File	Description
`09_09_path_pathbuf_basics.rs`	Borrowed and owned paths, `join`/`push`/`pop`, `set_file_name`/`set_extension`, the `AsRef<Path>` pattern
`09_10_path_components.rs`	Anatomy accessors, `components()`, `ancestors()`, component-wise comparisons, `MAIN_SEPARATOR`
`09_11_canonicalize_absolute.rs`	`canonicalize` and `path::absolute`, the macOS problem with the symlinked temp directory, `display`, `try_exists`

9.4 · Symbolic Links, Hard Links, and File Copying

Domain 9 — Filesystem Operations Duration: ~15 minutes Library components: std::fs::copy, std::fs::rename, std::fs::hard_link, std::fs::read_link, std::os::unix::fs, std::os::windows::fs

Introduction

It seems easy to move data in a filesystem, but the details are important:

rename is atomic, but only in one filesystem.
copy duplicates permission bits that you possibly do not want.
A hard link is the file, but a symlink only names a path to a file.

This tutorial explains the model behind these operations:

fs::copy and fs::rename, and the pattern behind each safe config update: write to a temporary file, then rename.
Cross-device moves: why rename fails with ErrorKind::CrossesDevices, and which alternative to use.
fs::hard_link: two directory entries, one inode.
Platform-specific symlink creation, fs::read_link, and dangling links.

Figure: Hard links vs symbolic links

09_4_links_and_copying_1

fs::copy

fs::copy(from, to) reads from and writes its contents to to. It returns the number of bytes that it copied:

// original: PathBuf of a file that contains 4096 bytes.
// backup: PathBuf of the destination.
// The code is in a function that returns io::Result, so `?` is permitted.
let copied = fs::copy(&original, &backup)?;    // u64: the number of bytes copied
assert_eq!(copied, 4096);
assert!(original.exists() && backup.exists()); // copy does not change or delete the source

Remember these three behaviors:

It overwrites an existing destination and does not ask. There is no "no-clobber" flag. If that is important, first check with Path::try_exists, or open the destination yourself with create_new.
It copies the permission bits together with the contents (it is cp -p, not cp). A read-only source gives a read-only copy. Remember this before you try to change the copy. On Windows, fs::remove_file first uses DeleteFile, which refuses a read-only file. Then it tries a second method that ignores the read-only attribute.
It follows symlinks: a copy of a symlink copies the contents of the target.

Internally, fs::copy uses fast OS mechanisms where they are available: copy_file_range on Linux, and fclonefileat or fcopyfile on macOS. Thus it is usually faster than a manual read/write loop through user space.

fs::rename and Atomic Replacement

fs::rename(from, to) moves a file or a full directory tree in one step. If to is an existing file, rename atomically replaces it. Readers see the old file or the new file, never a partially written intermediate state. This guarantee is the base of the standard pattern for a safe update:

// live: PathBuf of the config file in use. It contains {"version": 1}.
// staged: PathBuf of a temporary name in the same directory.
// Step 1: write the complete new version to the temporary name.
fs::write(&staged, "{\"version\": 2}")?;
// Step 2: rename the temporary file to the live name. This step replaces the live file atomically.
fs::rename(&staged, &live)?;
// `staged` does not exist now, and `live` contains {"version": 2}.

cargo, rustup, and each editor with a "safe save" function use this pattern. A crash at any point leaves the intact old file or the intact new file, never half of each. Compare this with a direct fs::write(&live, …): a crash during the write leaves a truncated, half-written file.

The atomicity guarantee comes from the semantics of POSIX rename(2) and Windows MoveFileExW. It applies only in one filesystem, and the next slide explains how to handle that limit.

The platforms differ when the destination is an existing directory:

On Unix, rename of a directory can also replace an empty directory.
On Windows 10 version 1607 and later, the behavior is the same if the filesystem supports FileRenameInfoEx.
On other Windows systems, the destination must not be a directory.

09_12_copy_rename.rs prints the output below. The next slide explains the move_file line:

fs::copy: 4096 bytes; source and backup both exist
fs::copy onto existing file: silently replaced
rename over live config: atomic replace, version 2 visible
rename on a directory: moved with contents
move_file within one filesystem used: rename

All assertions passed.

Cross-Device Moves: rename Fails, copy + delete Is the Alternative

A rename never moves bytes. It changes the directory entries that point at an inode, and an inode number has a meaning only in its own filesystem. If you move a file between mount points, the OS refuses with EXDEV. Two examples are a move from a tmpfs /tmp to your home partition, and a move from a container volume to a bind mount. Rust reports EXDEV as ErrorKind::CrossesDevices (stable since Rust 1.85).

Figure: The move_file fallback strategy

09_4_links_and_copying_2

// Moves a file and returns the name of the strategy that it used.
fn move_file(from: &Path, to: &Path) -> io::Result<&'static str> {
    match fs::rename(from, to) {
        Ok(()) => Ok("rename"),
        // CrossesDevices is EXDEV: `from` and `to` are on different filesystems.
        Err(e) if e.kind() == ErrorKind::CrossesDevices => {
            fs::copy(from, to)?;
            fs::remove_file(from)?; // not atomic: two steps that other processes can see
            Ok("copy+delete")
        }
        Err(e) => Err(e), // each other error goes to the caller unchanged
    }
}

mv does the same internally. In 09_12_copy_rename.rs, the two paths are in one filesystem, so move_file returns "rename". The alternative path (copy, then delete) loses these guarantees:

Between the copy and the remove_file, the two files exist.
A crash at that time leaves the two files.
The copy is not atomic: a different process can see it when it is only half-written.

If atomicity is important, copy to a temporary name on the destination filesystem. Then use rename for the last step.

Hard Links: Two Names, One Inode

fs::hard_link(original, link) creates a second directory entry for the same inode. The entry is not a pointer and not a copy. The function works on each platform, NTFS included. The two names are fully equal:

// original: PathBuf of a file that contains "id,value\n1,10\n" (14 bytes).
// link: PathBuf of a name that does not exist yet.
fs::hard_link(&original, &link)?;
// A write through one name is visible through the other name.
// The example appends "2,20\n" through `link`. Then `original` contains 19 bytes.

If you delete one name, you do not delete the data. remove_file only unlinks a name, and the inode stays until its last name (and its last open handle) is gone. For this reason, the deletion of a hard-linked backup releases no space. For the same reason, backup tools that deduplicate data (such as rsync --link-dest) and the on-disk caches of cargo use hard links:

fs::remove_file(&original)?;   // deletes the name `original`, not the data
assert_eq!(fs::read_to_string(&link)?, "id,value\n1,10\n2,20\n"); // `link` still has the data

On Unix, MetadataExt::nlink gives the link count: 1 before the link, 2 after it, and 1 again after the unlink. Hard links have two limits, and the two are by design:

A hard link cannot span filesystems, because inode numbers are local to a filesystem.
A hard link cannot point at a directory, because that would permit cycles in the directory tree.

09_13_hard_links.rs prints:

before hard_link: nlink = 1          # (nlink lines are Unix-only)
after hard_link:  nlink = 2
appended via link: original sees 19 bytes
original removed: data still intact via the remaining link
hard_link to a directory: PermissionDenied   # (exact kind varies by OS)

All assertions passed.

Symbolic Links

A symlink stores a path. The OS resolves the path on each access, and it resolves a relative path against the parent directory of the link. Windows distinguishes file symlinks from directory symlinks, and historically it restricted who may create them. For this reason, the creation functions are in platform modules, not in the portable std::fs:

Unix: std::os::unix::fs::symlink(target, link) is one function for all targets.
Windows: std::os::windows::fs::symlink_file and symlink_dir. They require administrator rights or Developer Mode, so the example prints a skip message on Windows.

The functions that read a symlink are portable. fs::read_link returns the stored target verbatim, with no resolution. canonicalize (tutorial 9.3) follows the chain of links to the real file:

use std::os::unix::fs::symlink;

// target: PathBuf of the existing file <tmp>/app-2026-07-10.log (14 bytes).
// link: PathBuf of <tmp>/latest.log, the symlink that this code creates.
symlink("app-2026-07-10.log", &link)?;   // a relative target (log rotation uses this form)
assert_eq!(fs::read_link(&link)?, PathBuf::from("app-2026-07-10.log")); // the stored text
assert_eq!(link.canonicalize()?, target.canonicalize()?);               // the same real file

The difference between "follow" and "do not follow" from tutorial 9.2 is the important concept here. fs::metadata follows the link and describes the target (is_file, the len of the target). fs::symlink_metadata describes the link itself (is_symlink). A dangling link makes the difference visible:

// dangling: PathBuf of a new symlink. Its target, no-such-file.txt, does not exist.
symlink("no-such-file.txt", &dangling)?;                // succeeds: no check of the target
assert!(!dangling.exists());                            // exists() follows the link: false
// try_exists() also follows the link. It returns Ok(false).
assert!(fs::symlink_metadata(&dangling)?.is_symlink()); // but the link itself is there

As with hard links, fs::remove_file(link) deletes the link, never the target.

On Unix, 09_14_symlinks.rs prints:

read_link(latest.log) = app-2026-07-10.log
metadata: file of 14 bytes; symlink_metadata: is_symlink=true
canonicalize(link) == canonicalize(target): true
dangling link: exists()=false, but symlink_metadata finds it
remove_file(link): target survives
read_dir through a dir symlink: 1 entry

All assertions passed.

Summary

Concept	Key point
`fs::copy`	It returns the number of bytes copied, copies the permission bits, and overwrites an existing destination
`fs::rename`	Atomic move and replace in one filesystem. It also works on directories
Write-temp-then-rename	The standard pattern for crash-safe file updates
`ErrorKind::CrossesDevices`	`EXDEV`: a rename across mount points is impossible by design
copy + delete alternative	The method of `mv`, but with two visible, non-atomic steps
`fs::hard_link`	Second name for the same inode. Portable, NTFS included
Hard-link deletion	`remove_file` unlinks a name. The data stays until the last name is gone
Hard-link limits	No cross-filesystem links, no directory links
Symlink creation	`std::os::unix::fs::symlink`. On Windows: `symlink_file`/`symlink_dir` and privileges
`fs::read_link`	It returns the stored target verbatim, with no resolution
Dangling symlinks	`exists()` follows the link and returns `false`. `symlink_metadata` still finds the link
`remove_file` on a symlink	It deletes the link, never the target

Code Examples

File	Description
`09_12_copy_rename.rs`	`fs::copy` semantics, atomic rename and replace, the helper for the `CrossesDevices` case
`09_13_hard_links.rs`	`fs::hard_link`, writes visible through the two names, `nlink`, deletion semantics
`09_14_symlinks.rs`	Unix symlinks, `read_link`, dangling links, directory symlinks (the example prints a skip message on Windows)

10.1 · TCP: TcpListener and TcpStream

Domain 10 — Networking Duration: ~15 minutes Library components: std::net::TcpListener, std::net::TcpStream, std::net::Shutdown

Introduction

std::net gives you TCP with two types. TcpListener waits for connections. TcpStream is a connected, bidirectional byte stream. It implements the Read and Write traits that you know from Domain 8.

There is no event loop, no callback, and no builder. The API is a blocking socket API, and each call maps one-to-one to an operation of the OS. This API is sufficient for many programs: cargo itself, test harnesses, and many internal tools use exactly this API.

This tutorial shows:

How to bind to an ephemeral port (127.0.0.1:0) and read the real address with local_addr(). How to accept connections with accept() and incoming().
How to connect with TcpStream::connect and connect_timeout, move bytes with Read/Write, and examine queued data with peek.
How to share one socket between two roles with try_clone, and how to half-close a connection with shutdown. After a half-close, a read that returns Ok(0) is the protocol signal, not an error.
The socket settings: set_read_timeout, set_write_timeout, set_nonblocking, and set_nodelay.

Each example runs fully on loopback in one process. One thread accepts the connection (the "server"), and a second thread connects (the "client"). This pattern (bind port 0, then connect from a second thread) is also how you write integration tests for real network code.

The examples also show one production practice. Sandboxes and restricted CI runners can forbid sockets. If the first bind fails, each example prints a clear skip message and exits normally. It does not crash. Network availability is a property of the environment, not a bug.

Binding and Accepting

TcpListener::bind takes a value that implements the ToSocketAddrs trait (Tutorial 10.3) and returns a listening socket. In practice, the most useful address is 127.0.0.1:0. Port 0 tells the OS to select a free port. Thus there is no hard-coded port and no collision between parallel test runs. local_addr() returns the port that the OS assigned.

Figure: Bind → accept/connect → echo between two threads of one process

10_1_tcp_listener_stream_1

// Port 0 tells the OS to select a free ephemeral port.
let listener = match TcpListener::bind("127.0.0.1:0") {
    Ok(listener) => listener,
    Err(e) => {
        // The environment forbids sockets: print a skip message and stop.
        println!("skipping: sockets unavailable in this environment ({e})");
        return;
    }
};
// addr: SocketAddr, the real address of the listener.
let addr = listener.local_addr().expect("bound listener has a local address");
assert_ne!(addr.port(), 0); // the OS replaced 0 with a real port

accept() blocks until a client connects. It returns (TcpStream, SocketAddr): the connected stream and the address of the peer. That address contains the ephemeral source port of the client. Together with the server address, it makes the 4-tuple (src IP, src port, dst IP, dst port). In the kernel, the 4-tuple identifies each TCP connection uniquely.

accept blocks, so a different thread must start the connection. For this reason, the example spawns a client thread. thread::scope (Tutorial 6.1) joins that thread automatically.

Connecting, Reading, Writing, Peeking

On the client side, TcpStream::connect(addr) does the three-way handshake and returns a connected stream. The related function connect_timeout(&addr, dur) limits the time that the connection attempt can take. A plain connect to an unreachable host can block for the OS default, which is often more than one minute.

The signature of connect_timeout is different. It takes one &SocketAddr, not impl ToSocketAddrs. The timeout applies to exactly one connection attempt, not to a full list of resolver candidates.

// server: SocketAddr, the address of the listener (from `local_addr()`).
let mut stream = TcpStream::connect_timeout(&server, Duration::from_secs(2))?;
let local = stream.local_addr()?;  // the ephemeral address of this client
let peer  = stream.peer_addr()?;   // the address of the listener: equal to `server`

After the connection, a TcpStream is a Read + Write value. All the rules of Domain 8 apply:

write can accept fewer bytes than you give it. Use write_all, which calls write in a loop until the kernel accepts all the bytes.
read can return fewer bytes than you request. Use read_exact for frames of a fixed size.
TCP is a byte stream with no message boundaries. Two write_all calls can arrive as one segment, and one write can arrive as three. Each protocol adds its own framing: fixed sizes, length prefixes, or delimiters.

peek(&mut buf) reads queued bytes and does not consume them. The subsequent read returns the same data again. As read does, peek blocks until a minimum of one byte is available. A typical use is to examine the first bytes and select a parser ("TLS or plaintext?") before you consume the data.

// stream: TcpStream, the accepted connection. The client sent 16 bytes.
let mut sniff = [0u8; 16];
let n = stream.peek(&mut sniff)?;   // n can be less than 16, the data stays queued
let mut request = [0u8; 16];
stream.read_exact(&mut request)?;   // the same bytes, now consumed
assert_eq!(request[..n], sniff[..n]);

10_01_tcp_echo_roundtrip.rs prints:

listener bound on 127.0.0.1 (port is ephemeral)
[server] accepted a connection from loopback
[server] peeked 16 bytes without consuming: "health-check-001"
[server] read the same 16 bytes: "health-check-001"
[server] echoed them back
[client] connected within its 2s budget; echo matches the probe
round trip complete: bind -> accept/connect -> write -> peek -> read -> echo

All assertions passed.

Serving Multiple Clients with `incoming()`

listener.incoming() is an iterator over io::Result<TcpStream>. It is a short form of loop { listener.accept() }. The iterator is infinite: it never returns None, so a plain for loop over it never ends. This is the usual structure of a server. For examples and tests, .take(n) stops the loop after n connections:

// The loop accepts exactly 3 connections, then it ends.
for stream in listener.incoming().take(3) {
    let stream = stream?;           // each item is a new connection, or an accept error
    handle_client(&stream);         // handle_client is in the next snippet
}

One detail is useful in real code: &TcpStream also implements Read and Write, not only TcpStream. Thus a BufReader can borrow the stream to read lines with a buffer. At the same time, you can write replies through a second shared reference to the same socket. No clone is necessary:

// `answer(&line)` is a placeholder for the code that makes the reply text.
fn handle_client(stream: &TcpStream) {
    let mut reader = BufReader::new(stream);   // borrows the stream, reads through &TcpStream
    let mut line = String::new();
    reader.read_line(&mut line).expect("read request");   // reads to the first '\n'
    // `writeln!` needs a mutable binding, so copy the shared reference.
    let mut writer = stream;                   // a second &TcpStream to the same socket
    writeln!(writer, "{}", answer(&line)).expect("send reply");
}

Example 10_02 makes a small request/response service from these parts. Each exchange has four steps:

The client connects.
The client sends one command line (ADD 40 2).
The client receives one reply line (42).
The client disconnects.

Redis and SMTP have the same structure. Here, the full service is approximately one screen of code. A real server gives each accepted stream to a thread or to an async task, and does not serve the stream inline. The accept loop itself is the same.

10_02_tcp_incoming_requests.rs prints:

listener bound on loopback (ephemeral port)
[client 1] sent "ADD 2 3" -> got 5
[client 2] sent "ADD 40 2" -> got 42
[client 3] sent "ADD -8 15" -> got 7

server log (in order served):
  ADD 2 3 -> 5
  ADD 40 2 -> 42
  ADD -8 15 -> 7
incoming() yielded exactly 3 connections, then we stopped taking

All assertions passed.

`try_clone`: Two Handles, One Socket

TcpStream is not Clone. The duplication of a socket is an OS operation that can fail, so the method is try_clone() -> io::Result<TcpStream>. The clone is a second handle to the same socket: the same file descriptor family, the same TCP 4-tuple, the same byte stream. Bytes that you write through one handle or the other go to the same connection, in order.

// stream: TcpStream, a connected client stream.
let mut clone = stream.try_clone()?;   // clone: TcpStream, a second handle to the socket
stream.write_all(b"chunk-1 ")?;   // through the original
clone.write_all(b"chunk-2")?;     // through the clone: the same byte stream
// The peer receives "chunk-1 chunk-2".
assert_eq!(stream.peer_addr()?, clone.peer_addr()?);

The reason for try_clone is that plain std sockets have no API that splits a stream into an owned read half and an owned write half. Two threads cannot each own the same TcpStream value. try_clone is the standard solution: the reader thread owns one handle, and the writer thread owns the other handle. (The into_split method of Tokio solves the same problem for async streams.)

A consequence: an operation at the socket level through one handle has an effect on all handles. A shutdown of the write direction through the original also forbids writes through the clone. The next section shows this.

`shutdown`: Half-Close and EOF

When a TcpStream drops, it closes the socket. Before that, shutdown(how) lets you close each direction separately:

Shutdown::Write sends a FIN: "I will write nothing more." The read of the peer returns Ok(0) after the peer consumes the buffered data. Your read direction stays open.
Shutdown::Read stops reception. The OS discards or refuses more inbound data. The details are platform-dependent, so rely on Write.
Shutdown::Both does the two operations above.

Figure: Half-close life cycle of a TCP connection

10_1_tcp_listener_stream_2

The half-close is a real protocol tool, not a minor detail. Exchanges in the style of HTTP/1.0 use it in three steps:

Send a request body of unknown length.
Call shutdown(Shutdown::Write) to mark the end of the body.
Read the response on the same connection.

On the receive side, read_to_end runs exactly until that FIN. Without the FIN, the call would block forever. A byte stream has no boundaries, so EOF is the only built-in signal that the sender sent all its data:

// Client side. stream: TcpStream, body: &[u8], the request body.
stream.write_all(body)?;
stream.shutdown(Shutdown::Write)?;        // sends FIN: the body is complete
let mut response = String::new();
stream.read_to_string(&mut response)?;    // the read direction is still open

// Server side. stream: the accepted TcpStream, body: an empty Vec<u8>, reply: &[u8].
let n = stream.read_to_end(&mut body)?;   // returns at the FIN of the client, n: the body length
stream.write_all(reply)?;
// The server stream drops after the reply. Then `read_to_string` of the client gets EOF.

Remember the EOF convention: a read that returns Ok(0) means that the peer will write no more data. It is not an error, and it is permanent: each subsequent read returns Ok(0) again. A read loop that treats Ok(0) as "try again" never ends. This is a common bug in read loops that you write manually.

10_03_tcp_try_clone_shutdown.rs prints:

listener bound on loopback (ephemeral port)
two handles, one socket: wrote "chunk-1 " then "chunk-2" via the clone
shutdown(Shutdown::Write): FIN sent — our outbound half is closed
write via the clone after shutdown failed as expected (kind: BrokenPipe)   # (varies by platform)
response still readable after write-shutdown: "received 15 bytes"
read after server close: Ok(0) — clean EOF
server report: saw EOF after 15 bytes: "chunk-1 chunk-2"

All assertions passed.

Timeouts, Nonblocking Mode, and TCP_NODELAY

By default, each read and each write blocks with no time limit. A stalled peer can block your thread forever. Three settings change that behavior:

Read and write timeouts

set_read_timeout(Some(dur)) puts a deadline on each blocking read. set_write_timeout is symmetric. A write timeout occurs only when the kernel send buffer is full, which means that the reader stalled. Two special cases apply, and example 10_04 asserts each one:

The setter rejects Some(Duration::ZERO) with InvalidInput. To set no timeout, pass None, not zero.
The error kind at expiry is platform-specific: WouldBlock on Unix, TimedOut on Windows. Portable code matches the two kinds:

// stream: TcpStream, buf: a byte buffer. `process` and `retry_later` are placeholders.
stream.set_read_timeout(Some(Duration::from_millis(150)))?;
match stream.read(&mut buf) {
    Ok(n) => process(&buf[..n]),   // n bytes arrived before the deadline
    // The deadline passed with no data: WouldBlock on Unix, TimedOut on Windows.
    Err(e) if matches!(e.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut) => retry_later(),
    Err(e) => return Err(e),       // each other error is a real failure
}

Nonblocking mode

set_nonblocking(true) makes each operation return immediately. When there is no data to read, the error is ErrorKind::WouldBlock on all platforms. The accept of a TcpListener behaves in the same way.

Each event loop uses this primitive. In essence, mio (and tokio, which uses mio) polls many nonblocking sockets and sleeps until one is ready. Example 10_04 has a small version: on WouldBlock, it sleeps for a short time and then tries again.

TCP_NODELAY

Nagle's algorithm collects small writes into fewer packets. That is good for throughput and bad for latency. Some request/response protocols send a small message and then wait (RPC, multiplayer games, REPLs). Those protocols want set_nodelay(true), and for this reason most RPC frameworks set it by default. Each setter has a related getter (read_timeout(), nodelay(), …), so you can check the configuration.

10_04_tcp_timeouts_nonblocking.rs prints:

listener bound on loopback (ephemeral port)
default read_timeout: None (reads block forever)
read_timeout set to Some(150ms)
set_read_timeout(Some(ZERO)) is rejected: InvalidInput
blocking read gave up at the deadline: kind=WouldBlock   # (varies: TimedOut on Windows)
read completed once data arrived: "pong"
write_timeout set to Some(2s)
nodelay enabled: small writes go out immediately
nonblocking read with no data: WouldBlock immediately
poll loop received "done"

All assertions passed.

Summary

Concept	Key point
`TcpListener::bind("127.0.0.1:0")`	Port 0 tells the OS to select a free ephemeral port. Read the port with `local_addr()`.
`accept()`	Blocks. Returns `(TcpStream, SocketAddr)`: the stream and the peer half of the 4-tuple.
`incoming()`	An infinite iterator, the short form of `loop { accept() }`. It never returns `None`.
`TcpStream::connect` / `connect_timeout`	Does the handshake. The timeout variant limits one attempt on one `&SocketAddr`.
`Read`/`Write` on `&TcpStream`	Shared references are sufficient: a `BufReader` can borrow the stream while you write.
Byte stream	No message boundaries. Add framing with fixed sizes, length prefixes, or delimiters.
`peek`	Reads queued bytes and does not consume them.
`try_clone`	A second handle to the same socket. A reader thread and a writer thread can each own one handle.
`shutdown(Shutdown::Write)`	Half-close: sends a FIN, the peer reads `Ok(0)`, and your read direction stays open.
`read` → `Ok(0)`	The EOF signal, not an error. It is permanent.
`set_read_timeout`	A deadline for each read. The expiry kind is `WouldBlock` (Unix) or `TimedOut` (Windows). `Some(ZERO)` gives `InvalidInput`.
`set_nonblocking(true)`	Returns `WouldBlock` immediately and does not block. It is the primitive of event loops.
`set_nodelay(true)`	Disables Nagle batching. It is the default selection for request/response protocols.
Unavailable sockets	A failure of the first `bind` is a report about the environment: print a message and skip, do not crash.

Code Examples

File	Description
`10_01_tcp_echo_roundtrip.rs`	Bind to `127.0.0.1:0`, `local_addr`, `accept`, `connect_timeout`, `peek`, echo round trip between two threads
`10_02_tcp_incoming_requests.rs`	`incoming()` accept loop that serves three sequential clients, `Read`/`Write` on `&TcpStream` with `BufReader`
`10_03_tcp_try_clone_shutdown.rs`	`try_clone` (two handles, one socket), `shutdown(Shutdown::Write)` half-close, `Ok(0)` as permanent EOF
`10_04_tcp_timeouts_nonblocking.rs`	`set_read_timeout`/`set_write_timeout`, rejection of a zero duration, `set_nonblocking` poll loop, `set_nodelay`

10.2 · UDP: Connectionless Communication

Domain 10 — Networking Duration: ~15 minutes Library components: std::net::UdpSocket

Introduction

TCP gives you a reliable, ordered byte stream between exactly two endpoints. UDP gives you something much more primitive: separate, self-contained messages with the name datagrams. A datagram can arrive out of order, arrive twice, or not arrive at all. In exchange, UDP gives you:

no setup cost,
hard message boundaries,
the ability to communicate with any number of peers (or a full network segment) through one socket.

The full API is one type: UdpSocket. There is no division into listener and stream, because there are no connections to accept. A bound UDP socket can immediately receive from any peer and send to any peer.

This tutorial shows:

When to use UDP, and when its trade-offs are wrong.
The connectionless operations: bind, send_to, recv_from, and peek_from. The reply-to-sender pattern, which makes stateless servers possible.
Datagram boundaries: each send is one message, and UDP never merges or splits messages.
Connected UDP: connect with send/recv. It gives a default destination and a source filter in the kernel, and it sends nothing on the network.
Socket options: set_broadcast, set_ttl, and the multicast methods (set_multicast_loop_v4, set_multicast_ttl_v4, join_multicast_v4, leave_multicast_v4).

As in Tutorial 10.1, all the examples run on loopback (127.0.0.1:0) in one process. If the environment forbids sockets, each example prints a skip message and exits normally. Sandboxes commonly refuse multicast joins. Thus the multicast example tolerates a refusal: it reports the refusal and never crashes.

Choosing TCP or UDP

The question is never which protocol is better. The question is what the application does when the network loses a packet. If all work stops until the retransmission of the packet, TCP already does that for you. If the lost packet is not important because a newer packet is already in transit, the retransmissions of TCP cause harm. You wait for stale data.

Figure: Decision tree — TCP or UDP?

10_2_udp_connectionless_1

Each typical UDP protocol agrees with one of those branches:

DNS: one small question and one small answer. A retry costs less than a handshake.
Game servers: position updates, where only the newest update is important.
mDNS/SSDP discovery: one query goes to the full network at the same time.
QUIC/HTTP-3: it builds reliability again in userspace on top of UDP, to remove the head-of-line blocking of TCP.

A consequence: if you select UDP, reliability becomes the task of your application (sequence numbers, acknowledgements, retries). If you build all of TCP again, use TCP.

The Connectionless Core: `bind`, `send_to`, `recv_from`

The two ends of a UDP exchange are the same. Each end calls UdpSocket::bind on an address (here, loopback with an ephemeral port). A send names the destination for each datagram. A receive reports the source for each datagram:

// Two sockets on loopback. Port 0 tells the OS to select a free port for each socket.
let collector = UdpSocket::bind("127.0.0.1:0")?;
let sensor    = UdpSocket::bind("127.0.0.1:0")?;

// send_to returns the number of bytes that it sent.
let sent = sensor.send_to(b"temp=21.5", collector.local_addr()?)?;
assert_eq!(sent, 9);          // a datagram is all-or-nothing: no loop for partial sends

let mut buf = [0u8; 64];
let (n, src) = collector.recv_from(&mut buf)?;   // n: 9, src: the address of `sensor`
assert_eq!(&buf[..n], b"temp=21.5");
collector.send_to(b"ack", src)?;   // reply to the sender: no session state

Three things are different from the TCP code in Tutorial 10.1:

No accept. The first datagram arrives with no connection step. The socket serves any number of peers.
No partial writes. send_to returns the full length or an error, because the kernel takes a full datagram or nothing. In contrast, you must call the write of TCP in a loop.
recv_from gives you the address of the sender. That one value makes stateless servers possible. A DNS server has no connection table: it answers each datagram to its source and then forgets the datagram. This reply-to-sender pattern is the core of each UDP service.

peek_from is the datagram equivalent of TcpStream::peek. It returns the next queued datagram (and its source) and does not consume it. The subsequent recv_from returns the same full message.

Datagram Boundaries and Buffer Sizing

TCP is a byte stream. It does not keep the boundaries between writes, so the receiver must find them again. UDP is the opposite: the datagram is the boundary. Three sends are always three messages. One recv_from consumes exactly one datagram, never one and a half:

// sensor, collector: the UdpSockets from the previous snippet.
// collector_addr: SocketAddr, the result of `collector.local_addr()`.
for payload in [b"a".as_slice(), b"bb", b"ccc"] {   // payloads of 1, 2, and 3 bytes
    sensor.send_to(payload, collector_addr)?;
}
// Three recv_from calls return the sizes [1, 2, 3]. UDP never merges them into "abbccc".

For this reason, simple protocols frequently use UDP. They need no length prefixes and no scan for delimiters. Each packet is one message, and the parser starts at byte zero.

The difficulty is the buffer size. If your buffer is smaller than the datagram that arrives, Unix discards the excess bytes, and recv_from returns an error on Windows. The excess does not stay in the queue for a second read. Make each buffer as large as the maximum message of your protocol.

The absolute maximum for an IPv4 UDP payload is 65,507 bytes. But well-behaved protocols stay below the Ethernet MTU of approximately 1,500 bytes, to prevent IP fragmentation. The network loses a fragmented datagram if it loses one of the fragments.

10_05_udp_send_recv.rs prints:

collector and sensor bound on loopback (ephemeral ports)
sensor sent 1 datagram of 9 bytes
collector peeked 9 bytes without consuming: "temp=21.5"
collector received 9 bytes from the sensor's address
sensor got the ack back from the collector
3 sends -> 3 recvs of sizes [1, 2, 3] (boundaries preserved, never coalesced)

All assertions passed.

Connected UDP: `connect` + `send`/`recv`

UdpSocket::connect(addr) uses a TCP term, but the operation is fully local: it sends no packet, and there is no handshake. It has exactly two effects:

send/recv (which have no address argument) now use addr implicitly.
The kernel filters inbound datagrams: it silently drops each datagram whose source is not addr, before your code sees the datagram.

Figure: The connected-socket filter

10_2_udp_connectionless_2

That filter is a real (but small) protection for clients that have one peer. Stray datagrams from other hosts cannot confuse a game client that is connected to its server. The implicit destination also removes the address argument from each send. Before connect, the socket has no default peer: peer_addr() returns an error of kind ErrorKind::NotConnected, and plain send fails.

connect puts nothing on the network, so it is cheap and you can call it again. A second call only changes the default destination and the filter. Example 10_06 shows the full sequence with assertions:

A stranger socket sends a datagram to the client. The kernel filters it, and the recv of the client gets a timeout.
The client calls connect(stranger_addr).
The client now accepts the datagrams of the same sender.

10_06_udp_connected.rs prints:

client, server, and stranger bound on loopback (ephemeral ports)
peer_addr() before connect: Err(NotConnected)
send() before connect fails: kind=Uncategorized   # (varies by platform)
client connected to the server: send/recv now need no address
round trip via send()/recv(): "state:ok"
datagram from a non-peer source: filtered by the kernel (recv timed out)
after connect(stranger): its datagrams are accepted

All assertions passed.

Broadcast and TTL

Two socket options control how far a datagram can travel:

set_broadcast(true) gives permission to send to broadcast addresses such as 255.255.255.255 ("each host on this segment"). The option is off by default as a protection against accidents, and the OS refuses broadcast sends until you enable it. DHCP discovery is the standard user: a client that has no IP address sends a request to the full segment. broadcast() returns the current value.
set_ttl(n) sets the IP time-to-live: the number of router hops before a router drops the packet. Each setter has a getter (ttl()), so you can check the configuration. The TCP settings in Tutorial 10.1 have the same symmetry of setter and getter.

// socket: a bound UdpSocket.
socket.set_ttl(64)?;               // the hop limit of outbound datagrams is now 64
assert_eq!(socket.ttl()?, 64);     // the getter returns the value that you set
assert!(!socket.broadcast()?);     // broadcast is off by default
socket.set_broadcast(true)?;
assert!(socket.broadcast()?);      // the OS now permits broadcast sends

set_nonblocking(true) operates exactly as it does for TCP. An empty receive queue reports ErrorKind::WouldBlock immediately, on each platform. Thus a discovery beacon can poll its socket between other tasks, and does not block a thread in recv_from.

Multicast Group Membership

Broadcast reaches each host, also the hosts that do not want the data. Multicast delivers only to the sockets that joined a group. A group is an address in 224.0.0.0/4 (IPv4). For example, mDNS uses 224.0.0.251 and SSDP uses 239.255.255.250. Membership is per interface:

// socket: a bound UdpSocket. GROUP is an example group address.
const GROUP: Ipv4Addr = Ipv4Addr::new(224, 0, 0, 123);
socket.set_multicast_loop_v4(true)?;   // deliver the group sends of this socket to this host too
socket.set_multicast_ttl_v4(1)?;       // 1 is the default: stay on the local network
// The second argument is the address of the local interface: here, loopback.
match socket.join_multicast_v4(&GROUP, &Ipv4Addr::LOCALHOST) {
    Ok(()) => {
        // The OS now delivers the datagrams of the group to this socket.
        socket.leave_multicast_v4(&GROUP, &Ipv4Addr::LOCALHOST)?;
    }
    // The environment refuses the join: report the refusal and continue.
    Err(e) => println!("join not permitted here ({e}) — skipping"),
}

The options, in order:

multicast_loop_v4 controls whether the OS sends your own group datagrams back to local sockets. This is useful in tests, and it is usually on.
multicast_ttl_v4 has the default 1, so multicast never crosses a router unless you increase the value deliberately.
join_multicast_v4(group, interface) and leave_multicast_v4 control the membership itself.

Note the tolerant match around the join. Group membership needs cooperation from the OS and the network stack. Sandboxes, containers, and CI runners frequently refuse it. The setters are plain socket options and operate in each environment. The join is the privileged step. Report the refusal and continue: all the assertions of the example are on the operations that always succeed.

10_07_udp_socket_options.rs prints:

socket bound on loopback (ephemeral port)
ttl set and read back: 64
broadcast: default false -> enabled true
multicast_loop_v4=true, multicast_ttl_v4=1 (stay on the local link)
joined multicast group 224.0.0.123 on the loopback interface   # (varies: may print a skip instead)
left multicast group 224.0.0.123   # (absent if the OS refuses the join)
nonblocking recv_from on an empty socket: WouldBlock immediately

All assertions passed.

Summary

Concept	Key point
Datagram model	Self-contained messages. The network can lose, duplicate, or reorder them, but never merges or splits them.
`UdpSocket::bind`	One type for "client" and "server". No accept and no connection setup.
`send_to`	A destination for each datagram. All-or-nothing (no loop for partial sends).
`recv_from`	Returns `(len, source)`. One call consumes exactly one datagram.
Reply-to-sender	Answer to the `recv_from` source. The server is stateless and has no session table.
`peek_from`	Returns the next datagram and does not consume it.
Buffer sizing	If the buffer is too small, Unix discards the excess and Windows returns an error. Stay below the MTU.
`connect` + `send`/`recv`	A default destination and a kernel source filter. It sends nothing on the network, and you can call it again.
`set_broadcast`	Off by default. Enable it before you send to broadcast addresses.
`set_ttl` / `set_multicast_ttl_v4`	Hop limits. The multicast default is 1 (local network only).
`join_multicast_v4` / `leave_multicast_v4`	Group delivery by request, per interface. The OS can refuse the join: tolerate the refusal.
`set_nonblocking`	An empty queue gives `WouldBlock` immediately, on all platforms.
Reliability	It is the task of your application: sequence numbers, acks, retries. The alternative is TCP.

Code Examples

File	Description
`10_05_udp_send_recv.rs`	`bind`, `send_to`, `recv_from`, `peek_from`, reply-to-sender, datagram boundaries
`10_06_udp_connected.rs`	`connect` + `send`/`recv`, `peer_addr`, kernel filter for non-peer datagrams, second `connect` call
`10_07_udp_socket_options.rs`	`set_broadcast`, `set_ttl`, multicast loop/TTL/join/leave (tolerant), nonblocking `recv_from`

10.3 · IP Addresses and Socket Addresses

Domain 10 — Networking Duration: ~15 minutes Library components: std::net::IpAddr, std::net::Ipv4Addr, std::net::Ipv6Addr, std::net::SocketAddr, std::net::SocketAddrV4, std::net::SocketAddrV6, std::net::ToSocketAddrs, std::net::AddrParseError

Introduction

Each bind and connect in Tutorials 10.1 and 10.2 uses a small family of address types. Unlike the socket types, the address types are pure data. Construction, parsing, and classification do no I/O at all. Thus this is the part of std::net that is the easiest to test.

The design has two parallel enums:

IpAddr contains a host address of one of the two families (Ipv4Addr or Ipv6Addr).
SocketAddr adds a port to an address, again for each family (SocketAddrV4 or SocketAddrV6). SocketAddrV6 has two more fields that only IPv6 uses.

FromStr (with AddrParseError) converts strings to these types. The ToSocketAddrs trait converts all these forms to the input of the socket APIs. Because of this trait, TcpStream::connect("localhost:6379") accepts a string, a tuple, or a complete address.

Figure: The std::net address type family

10_3_ip_socket_addrs_1

This tutorial describes:

parsing and construction,
the canonical text form,
the classification predicates (and the predicates that are still unstable),
the IPv4-mapped trap in IPv6,
the SocketAddr family,
the reason why ToSocketAddrs can block.

Parsing and Constructing

The ordinary parse() method converts a string to an address. Ipv4Addr and Ipv6Addr each parse their own family. IpAddr accepts the two families and records which family it got. Thus IpAddr is the correct type for configuration values, where the operator can write 10.0.0.1 today and ::1 tomorrow:

// A parse failure gives an AddrParseError.
let v4: Ipv4Addr = "192.168.1.10".parse()?;   // only IPv4 text is valid here
let either: IpAddr = "::1".parse()?;          // IPv4 text or IPv6 text is valid here
assert!(matches!(either, IpAddr::V6(_)));     // the value records its family

A failure gives an AddrParseError. This error is deliberately opaque (no position, no variants): its function is yes-or-no validation, not diagnostics. The parser is strict on purpose:

"256.0.0.1" fails, because octets are u8.
"1::2::3" fails, because :: can occur only one time.
"192.168.01.1" fails, because the parser rejects leading zeros.

The last case is the most interesting one. The C function inet_aton reads 010 as octal 8. Parsers that disagree about that exact ambiguity caused real bypasses of SSRF filters. Rust refuses to guess.

To construct an address in code, you have several options:

Ipv4Addr::new(192, 168, 1, 10) and Ipv6Addr::new(0x2001, 0xdb8, …) take each component.
From<[u8; 4]> / From<[u16; 8]> / From<[u8; 16]> convert from raw bytes or segments. octets() and segments() do the opposite conversion.
from_bits/to_bits use the full address as one big-endian integer (u32 / u128, stable since 1.80). With these functions, CIDR arithmetic is one line:

// v4 is 192.168.1.10 from the previous snippet. `v4.to_bits()` is 0xC0A8_010A.
let net = Ipv4Addr::from_bits(v4.to_bits() & 0xFFFF_FF00);  // apply a /24 mask
assert_eq!(net, Ipv4Addr::new(192, 168, 1, 0));             // the network address

new, from_bits, and to_bits are const fn. The From conversions are not usable in a const item on stable Rust. There, use from_octets or from_segments (stable since 1.91).

The associated constants replace magic numbers:

Ipv4Addr::LOCALHOST (127.0.0.1)
Ipv4Addr::UNSPECIFIED (0.0.0.0, which means "all interfaces" in a bind)
Ipv4Addr::BROADCAST (255.255.255.255)
Ipv6Addr::LOCALHOST (::1) and Ipv6Addr::UNSPECIFIED (::)

One Address, Many Spellings: Canonical Display

An IPv6 address has many valid spellings: 0:0:0:0:0:0:0:1, 0000::0001, and ::1 are the same address. Display always writes the RFC 5952 canonical form: lowercase hex, with the longest run of zeros compressed one time:

// The input has uppercase hex digits and all its zeros.
let verbose: Ipv6Addr = "2001:0DB8:0000:0000:0000:0000:0000:0001".parse()?;
assert_eq!(verbose.to_string(), "2001:db8::1");   // lowercase, the zeros compressed

The general rule: never compare addresses as strings. Two strings that are not equal can name one address. Parse first, and then compare the typed values. For example, the value that you parse from 0:0:0:0:0:0:0:1 is equal to Ipv6Addr::LOCALHOST, although the two spellings are different. The sequence parse → display → parse loses no data, so the canonical text is safe to store and to log.

10_08_ip_parsing_construction.rs prints:

parsed: 192.168.1.10 (v4), 2001:db8::7334 (v6), 10.0.0.1 and ::1 via IpAddr
rejected inputs report: "invalid IPv4 address syntax"
bit-level: 192.168.1.10 & /24 mask = 192.168.1.0
constants: LOCALHOST, UNSPECIFIED, BROADCAST spare you magic numbers
canonical display: "2001:0DB8:...:0001" -> "2001:db8::1"

All assertions passed.

Classifying Addresses

Each classification predicate tests if an address is in a range that IANA or an RFC assigned. The stable IPv4 set:

Predicate	Range	RFC
`is_loopback`	127.0.0.0/8	1122
`is_private`	10/8, 172.16/12, 192.168/16	1918
`is_link_local`	169.254.0.0/16	3927
`is_multicast`	224.0.0.0/4	5771
`is_broadcast`	255.255.255.255	919
`is_documentation`	192.0.2/24, 198.51.100/24, 203.0.113/24	5737
`is_unspecified`	0.0.0.0	1122

IPv6 has different categories. There is no broadcast, because multicast does that function. "Private" divides into two predicates:

is_unique_local (fc00::/7, the equivalent of RFC 1918)
is_unicast_link_local (fe80::/10, which each interface configures automatically)

is_loopback (::1), is_multicast (ff00::/8), and is_unspecified (::) also exist for IPv6. IpAddr forwards the common predicates to the variant that it contains. Thus code for the two families, such as "is this peer loopback?", needs no match.

Know one limit before you use a classifier: is_global() ("is this address publicly routable?") is still unstable as of Rust 1.99. It is behind #![feature(ip)] (tracking issue #27709), together with is_shared, is_benchmarking, and Ipv6Addr::multicast_scope. The definition continues to change with the updates of the IANA registry. On stable, compose the test that you need from the stable predicates above.

The IPv4-Mapped Problem and `to_canonical`

Dual-stack servers frequently see IPv4 clients through IPv6 sockets. The sockets report these clients as IPv4-mapped addresses: ::ffff:a.b.c.d. This causes a trap. As an IPv6 value, the mapped loopback is not ::1. Thus each IPv6 predicate answers for the mapped range, not for the embedded IPv4 address:

let mapped: Ipv6Addr = Ipv4Addr::LOCALHOST.to_ipv6_mapped();  // ::ffff:127.0.0.1
assert!(!mapped.is_loopback());          // false: as IPv6, this value is not ::1

let canonical: IpAddr = mapped.to_canonical();  // IpAddr::V4(127.0.0.1)
assert!(canonical.is_loopback());        // true: classify the canonical value

Think of an admin-panel guard that calls peer.ip().is_loopback() and does not canonicalize the address. It accepts or rejects the same client, and the result depends on the socket family of the connection. The solution is one call: to_canonical() converts IPv4-mapped addresses to IpAddr::V4 and does not change real IPv6 addresses.

The related method to_ipv4_mapped() is precise: it returns Some only for the ::ffff:0:0/96 range. Prefer it to the less strict to_ipv4, which also converts the deprecated "IPv4-compatible" addresses.

10_09_ip_classification.rs prints:

IPv4 classification:
  address          loopback private link_local multicast broadcast doc
  127.0.0.1        true     false   false      false     false     false
  10.0.0.1         false    true    false      false     false     false
  192.168.1.10     false    true    false      false     false     false
  169.254.7.7      false    false   true       false     false     false
  224.0.0.123      false    false   false      true      false     false
  255.255.255.255  false    false   false      false     true      false
  192.0.2.88       false    false   false      false     false     true

IPv6 classification:
  ::1 loopback, fe80::/10 unicast_link_local, fc00::/7 unique_local, ff00::/8 multicast

IPv4-mapped trap:
  (::ffff:127.0.0.1).is_loopback()                = false  <- surprise!
  (::ffff:127.0.0.1).to_canonical() = 127.0.0.1, .is_loopback() = true

IpAddr forwards predicates: 2 of 3 peers are loopback

All assertions passed.

The SocketAddr Family

A host address alone is not sufficient for a connection: a socket needs a port. SocketAddrV4 is an Ipv4Addr and a u16 port. SocketAddrV6 has two more fields that the IPv6 wire format requires:

scope_id tells which interface a link-local address is on. fe80::1 exists on each interface, so the address needs an interface index. The text form is [fe80::1%3]:443.
flowinfo is the IPv6 flow label. Programs rarely use it, and it is usually 0.

The socket APIs use the SocketAddr enum, which is valid for the two families. It has ip(), port(), and is_ipv4()/is_ipv6(). It also has the in-place setters set_ip/set_port. With them, you get "same host, different port" and do not construct a new value.

The text form has one special rule. IPv6 uses : in the address, so the port needs brackets: [2001:db8::1]:443. The text "::1:9000" with no brackets is a valid IPv6 address. For that reason, it must fail to parse as a SocketAddr:

let ok: SocketAddr = "[::1]:9000".parse()?;          // address ::1, port 9000
// With no brackets, the text is the IPv6 address ::1:9000, and it has no port.
assert!("::1:9000".parse::<SocketAddr>().is_err());  // IPv6 needs [addr]:port

`ToSocketAddrs`: Why `connect("localhost:...")` Can Block

Each bind/connect in this domain is generic over ToSocketAddrs. This trait converts an address-like value to an iterator of SocketAddr. The result is an iterator because one name can resolve to several candidates (IPv4 and IPv6, or multiple records). TcpStream::connect tries each candidate in order until one succeeds.

Figure: Which ToSocketAddrs impls block?

10_3_ip_socket_addrs_2

The division is important in operation. Inputs that have the form of an address convert with no syscall: SocketAddr, (Ipv4Addr, u16), and strings that contain IP literals. A hostname goes through the platform resolver, getaddrinfo. That call blocks and has no timeout parameter. With a slow or broken DNS configuration, it can stall for seconds. This has three consequences:

Async runtimes move name resolution to a pool of blocking threads (spawn_blocking in tokio).
Services that are sensitive to latency resolve names one time at startup, not for each request.
The examples of this tutorial resolve only "localhost". They also tolerate a failure and print a skip message, because a sandbox can have no resolver at all.

To use the same pattern in your code, you need one signature. TcpStream::connect, TcpListener::bind, and UdpSocket::bind have exactly this form:

// `spec` can be a SocketAddr, a (host, port) tuple, or a string such as "localhost:80".
fn first_addr(spec: impl ToSocketAddrs) -> io::Result<SocketAddr> {
    spec.to_socket_addrs()?   // can block: a hostname goes through the resolver
        .next()               // the first candidate: Option<SocketAddr>
        // An empty iterator becomes a NotFound error.
        .ok_or_else(|| io::Error::new(io::ErrorKind::NotFound, "resolved to no addresses"))
}

10_10_socket_addr_resolution.rs prints:

SocketAddrV4: 192.168.1.10:8080
SocketAddrV6: [2001:db8::1]:443 and link-local [fe80::1%3]:443
SocketAddr after set_port/set_ip: 127.0.0.1:9090
parsed 127.0.0.1:8080 and [::1]:9000; unbracketed v6 correctly rejected
non-resolving impls: SocketAddr, (IpAddr, u16), IP-literal strings
"localhost:8080" resolved to 2 candidate(s); all loopback: true   # (varies by OS resolver config)

All assertions passed.

Summary

Concept	Key point
`IpAddr` = `V4 \| V6`	A host address of one of the two families. It forwards the common predicates with no `match`.
`parse()` / `AddrParseError`	Strict validation: octets in range, one `::`, no leading zeros (octal ambiguity).
`octets` / `segments` / `to_bits`	Raw views. With `from_bits`/`to_bits`, a CIDR mask is one line.
Constants	`LOCALHOST`, `UNSPECIFIED`, `BROADCAST`: no magic numbers.
`Display`	RFC 5952 canonical form. Never compare addresses as strings.
Classification	`is_loopback`, `is_private`, `is_multicast`, `is_link_local`, …: RFC ranges, all stable.
`is_global` and related predicates	Still unstable (`feature(ip)`, #27709). Compose the test from stable predicates.
IPv4-mapped trap	`::ffff:127.0.0.1` fails the IPv6 predicates. Call `to_canonical()` before you classify.
`SocketAddrV4`	`Ipv4Addr` + port
`SocketAddrV6`	Adds `flowinfo` and `scope_id` (`[fe80::1%3]:443` tells which interface the link-local address is on).
`SocketAddr` text form	IPv6 needs brackets: `[::1]:9000`. Forms with no brackets are ambiguous, and the parser rejects them.
`ToSocketAddrs`	An iterator of candidates. The impls for address forms never block.
Hostname resolution	`getaddrinfo` blocks and has no timeout. Resolve names at startup, or on a blocking thread in async code.

Code Examples

File	Description
`10_08_ip_parsing_construction.rs`	Parsing, `AddrParseError` rejections (for example, leading zeros), constructors, `to_bits` CIDR math, constants, RFC 5952 display
`10_09_ip_classification.rs`	`is_*` classification for the two families (a table for IPv4), the IPv4-mapped trap, `to_canonical`
`10_10_socket_addr_resolution.rs`	`SocketAddrV4`/`V6` (scope_id, flowinfo), bracket parsing, `ToSocketAddrs` impls, tolerant `localhost` resolution

11.1 · Instant, SystemTime, and Duration

Domain 11 — Time and Duration Duration: ~15 minutes Library components: std::time::Instant, std::time::SystemTime, std::time::Duration, std::time::UNIX_EPOCH, std::time::SystemTimeError

Introduction

std::time is small by design. It has one type for a span of time (Duration) and two types for a point in time (Instant and SystemTime). There are two point types because your machine has two fundamentally different clocks. This is the most important idea in this tutorial. The first clock is the monotonic clock: it only moves forward, as a stopwatch does. The second clock is the wall clock: it shows calendar time, and the OS, NTP, and the user can move it in each direction.

Most time bugs occur when a program uses the wall clock where it needs the monotonic clock.

This tutorial shows:

Duration construction: a constructor for each unit (from_secs through from_nanos, and from_mins/from_hours).
The Duration accessors that truncate, and the float conversions (as_secs_f64, from_secs_f64, try_from_secs_f64).
Duration arithmetic: why the operators panic on overflow, and when to use checked_* or saturating_*.
Instant: the opaque, monotonic clock for benchmarks, timeouts, and deadlines.
SystemTime and UNIX_EPOCH: wall-clock time, and conversions to and from Unix timestamps.
Why the subtraction of two wall-clock readings returns a Result.
SystemTimeError: an error that contains useful data. Its duration() method returns the magnitude of a backward difference.

Tutorial 11.2 uses these types for sleep, timeouts, and rate limiting.

Duration: An Unsigned Span of Time

A Duration is a length of time. It contains a u64 of whole seconds and a u32 of subsecond nanoseconds. Two properties come from that representation, and both are important in the remainder of this domain:

It is unsigned. There is no negative Duration. For this reason, a subtraction of timestamps can fail, and the arithmetic API handles each boundary explicitly (see the next slide).
It is exact. It has no floats internally and no precision loss. It has nanosecond resolution across half a trillion years.

There is a constructor for each common unit. Since Rust 1.91, these include from_mins and from_hours. from_days is still available only on nightly:

use std::time::Duration;

let request_timeout = Duration::from_mins(2);       // 120 s
let retry_delay     = Duration::from_millis(250);   // 0.25 s
let tick            = Duration::from_micros(100);   // 0.0001 s

// Duration::new(secs, nanos) normalizes: nanos >= 1 billion carry into seconds.
let carried = Duration::new(1, 1_500_000_000);      // 1 s + 1.5 s = 2.5 s
assert_eq!(carried, Duration::from_millis(2_500));

There are two families of accessors. A common bug is to use one family where you need the other:

Whole-unit accessors truncate: as_secs() on 2.5 s returns 2, never 3. The subsec_millis, subsec_micros, and subsec_nanos accessors return the fraction.
Total-unit accessors return u128: as_millis, as_micros, and as_nanos return the full span in that unit. The type is u128 because nanoseconds in a u64 reach only ~584 years.

For log output and ratio calculations, there are float conversions:

as_secs_f64() returns the full span, which includes the fraction.
from_secs_f64 makes a duration from a float. It panics on a negative value or on NaN.
try_from_secs_f64 returns a Result. Use it when the float comes from configuration or user input.
mul_f64 scales a span (for example, for backoff jitter).
div_duration_f64 returns how many times one span fits in a different span.

// `carried` (2.5 s), `request_timeout` (120 s), and `retry_delay` (250 ms)
// are the durations from the previous snippet.
assert!((carried.as_secs_f64() - 2.5).abs() < f64::EPSILON); // as_secs_f64() is 2.5
assert!(Duration::try_from_secs_f64(-1.0).is_err());         // no negative spans

let ratio = request_timeout.div_duration_f64(retry_delay);   // 120 s / 0.25 s
assert!((ratio - 480.0).abs() < f64::EPSILON);               // 480 retries fit in 2 min

The type also has Duration::ZERO (with is_zero()), Duration::MAX, and abs_diff. abs_diff returns the magnitude of a difference. Use it when you do not know which operand is larger.

11_01_duration_construction.rs prints:

request_timeout = 120s
retry_delay     = 250ms
dns_ttl         = 3600s
tick            = 100µs
one_ns          = 1ns

Duration::new(1, 1_500_000_000) = 2.5s
as_secs=2 subsec_millis=500 subsec_nanos=500000000
as_secs_f64 = 2.5
try_from_secs_f64(-1.0) -> Err(TryFromFloatSecsError { kind: Negative })
120s / 250ms = 480 retries fit
abs_diff(300ms, 750ms) = 450ms

All assertions passed.

Duration Arithmetic: Checked, Saturating, or Panic

Duration supports +, -, * u32, and / u32. These operators panic at the boundaries, exactly as integer overflow does in debug builds. Duration::MAX + 1ns has no representable result. 160ms - 340ms would be negative, and an unsigned type cannot represent a negative value. A panic shows the bug immediately. Without the panic, an incorrect timestamp would go into later calculations.

When the operands come from untrusted input or from a calculation with no known limit, select the failure mode explicitly:

Figure: Choosing a Duration arithmetic family

11_1_instant_systemtime_duration_1

// read is 340 ms, parse is 160 ms, one_ns is Duration::from_nanos(1).
// checked_*: the boundary becomes an Option that you must examine.
assert_eq!(parse.checked_sub(read), None);              // the result would be negative
assert_eq!(Duration::MAX.checked_add(one_ns), None);    // the result would overflow
assert_eq!(read.checked_div(0), None);                  // ÷0 returns None, not a panic

// saturating_*: the result stays at the boundary (ZERO or MAX).
assert_eq!(parse.saturating_sub(read), Duration::ZERO); // a countdown stops at zero

The saturating family is useful in real designs. An exponential-backoff schedule doubles a delay for each attempt. The retry middleware for tokio and reqwest uses this type of schedule. With saturating_mul(2).min(cap), no number of attempts can cause a panic, however large the number is:

let mut delay = Duration::from_millis(100);
let cap = Duration::from_secs(5);
// The example runs the line below one time for each attempt. The delays are:
// 100ms, 200ms, 400ms, 800ms, 1.6s, 3.2s, 5s, 5s, ...
delay = delay.saturating_mul(2).min(cap);   // doubles the delay, but never above `cap`

Duration also implements Sum and Ord, so aggregation needs no unit conversions:

sum() adds an iterator of lap times directly into a total.
min and max find the fastest lap and the slowest lap.
clamp puts a timeout from the user into a permitted range.

11_02_duration_arithmetic.rs prints:

read + parse = 500ms
read - parse = 180ms
Duration::MAX + 1ns panicked as expected (caught).
parse.checked_sub(read) = None
read.checked_div(0)     = None
parse.saturating_sub(read) = 0ns

backoff schedule: [100ms, 200ms, 400ms, 800ms, 1.6s, 3.2s, 5s, 5s]

laps total = 1.216s, average = 304ms
fastest = 298ms, slowest = 312ms, clamped timeout = 50ms

All assertions passed.

Instant: The Monotonic Stopwatch

Instant::now() reads the monotonic clock of the OS:

CLOCK_MONOTONIC on Linux
CLOCK_UPTIME_RAW on macOS
QueryPerformanceCounter on Windows

Its primary guarantee is this: a later Instant::now() is never less than an earlier one. NTP corrections, leap seconds, and a manual change of the system time do not move this clock.

The cost of that guarantee is that an Instant is opaque. It has no as_secs, no calendar meaning, and no serialized form. You can only compare it with other Instant values from the same process. Do not think of this as a limitation that you must bypass. Because the type is opaque, the OS is free to use a clock that cannot jump.

Figure: Monotonic clock vs. wall clock

11_1_instant_systemtime_duration_2

The most common pattern is the stopwatch:

// data: Vec<u64> — 100_000 pseudo-random values.
let start = Instant::now();        // read the monotonic clock
data.sort_unstable();              // the workload to measure
let sort_time = start.elapsed();   // a Duration, equal to Instant::now() - start

The wall clock has no effect on this measurement, so the result is never negative and never grossly incorrect. For this reason, each benchmark harness uses Instant and never SystemTime. In tests, assert only lower bounds and very large upper limits on measured times, because CI machines pause unpredictably.

You can ask for a difference in the incorrect direction (earlier minus later). Three methods do this, and each one behaves differently:

// now: Instant — a reading of Instant::now().
let future = now + Duration::from_secs(5);   // an Instant 5 s after `now`
// Each call below asks for `now - future`, which would be negative.
assert_eq!(now.duration_since(future), Duration::ZERO);            // saturates (panic before 1.60)
assert_eq!(now.checked_duration_since(future), None);              // shows the failure as None
assert_eq!(now.saturating_duration_since(future), Duration::ZERO); // saturates, explicitly

Instant + Duration returns a later instant. This makes the deadline pattern possible:

Calculate one absolute deadline at the start.
Pass the deadline down through the call layers.
Where a layer needs the remaining time, call deadline.saturating_duration_since(Instant::now()).

A result of zero means that the budget is spent. Tutorial 11.2 uses this pattern in many places.

11_03_instant_monotonic.rs prints:

1000 successive Instant::now() calls: never decreased
sorted 100_000 u64s in 48.619375ms (varies per run)   # (varies)
earlier.duration_since(later) saturates: 0ns
earlier.checked_duration_since(later) = None
remaining budget: 345.666µs (varies per run)          # (varies)

All assertions passed.

SystemTime: The Wall Clock and UNIX_EPOCH

SystemTime::now() returns the current date and time as the OS knows them. This clock can jump: backward when NTP corrects it, and forward after a suspend. Use it when a time must have a meaning outside your process:

file modification times
cache-expiry timestamps
log timestamps
any time value that you serialize

The anchor constant UNIX_EPOCH (1970-01-01 00:00:00 UTC) gives SystemTime its external meaning. A duration relative to the epoch is a Unix timestamp:

use std::time::{Duration, SystemTime, UNIX_EPOCH};

let now = SystemTime::now();
// duration_since returns a Result. It is Err if `now` is before the epoch.
let unix_secs = now.duration_since(UNIX_EPOCH)
    .expect("system clock is set before 1970")
    .as_secs();   // u64: whole seconds, without the subsecond part

// Round trip: timestamp -> SystemTime. Whole seconds truncate, so the
// reconstructed time is less than 1 s behind the original and never ahead.
let reconstructed = UNIX_EPOCH + Duration::from_secs(unix_secs);
assert!(reconstructed <= now);

JWT exp claims and most on-disk formats use the same u64 of seconds. HTTP Expires headers have the same resolution of one second, but they contain a date text. This conversion is how you serialize time with only the standard library.

Times before the epoch are also valid: UNIX_EPOCH - Duration::from_hours(24) is 1969-12-31. Other systems represent such a time as a negative timestamp.

SystemTime implements Ord, so the comparison operators express calendar logic directly. The function below also shows a technique for tests. Pass now as a parameter, and do not read the clock in the function. Then the tests of the expiry logic are fully deterministic. The rate limiter in tutorial 11.2 uses the same technique with Instant:

// The caller supplies the current time. The function does not read the clock.
fn is_expired(expiry_unix_secs: u64, now: SystemTime) -> bool {
    now >= UNIX_EPOCH + Duration::from_secs(expiry_unix_secs)
}

// now: SystemTime — the reading from the previous snippet.
// entry_expiry: u64 — unix_secs + 300, a timestamp 5 minutes after `now`.
assert!(!is_expired(entry_expiry, now));                          // not expired at `now`
assert!(is_expired(entry_expiry, now + Duration::from_mins(10))); // expired 10 min later, no sleep

SystemTimeError: When "Later" Comes First

The two clocks have different APIs for the same operation. Thus the type system shows the difference between the clocks:

Instant::duration_since returns a Duration (it saturates). The monotonic clock makes a backward difference almost impossible. Thus std does not make each caller handle a Result.
SystemTime::duration_since returns Result<Duration, SystemTimeError>. On a wall clock, a negative result for "later minus earlier" is a legitimate runtime event, not a bug. Thus the API makes you handle it.

SystemTimeError is different from most errors: it contains data that you frequently want. Its duration() method returns how far the difference went in the incorrect direction. A backward movement of the clock thus becomes a value that you can measure and recover from, not a crash:

// Make the error deterministically: ask for t - (t + 5s).
let t = SystemTime::now();
let err = t.duration_since(t + Duration::from_secs(5))   // Err: the result would be negative
    .expect_err("t is strictly before t + 5s");          // err: SystemTimeError
assert_eq!(err.duration(), Duration::from_secs(5));      // the size of the negative difference

elapsed() is duration_since from a new SystemTime::now() reading. Thus it fails in the same way for a time that is still in the future. A safe pattern for "seconds since X" on the wall clock handles Err as "something adjusted the clock". The pattern then uses zero (or synchronizes again) and logs err.duration() as the size of the jump.

11_04_systemtime_epoch.rs prints:

seconds since UNIX_EPOCH: 1783686826 (varies)                    # (varies)
round-trip through u64 secs lost 23.318ms (varies, always < 1s)  # (varies)
t.duration_since(t + 5s) -> Err, err.duration() = 5s
1969-12-31 vs epoch -> Err, err.duration() = 86400s
cache entry expiring at 1783687126 (varies): fresh now, expired in 10 min  # (varies)

All assertions passed.

Choosing the Right Clock

The decision is simple after you identify what you measure:

Figure: Which time type do I need?

11_1_instant_systemtime_duration_3

General rules to remember:

To measure how long an operation took, use Instant. SystemTime here is a common bug: an NTP step during the measurement gives a negative or absurd result.
To record when an event occurred, use SystemTime, serialized as seconds since UNIX_EPOCH.
Never convert an Instant to a wall-clock time with a paired SystemTime::now() reading. The two clocks drift apart, and the result has the weaknesses of both.
Time zones, calendars, and formatting are out of the scope of std by design. The ecosystem crates (jiff, chrono, time) build on exactly these three primitives.

Summary

Concept	Key point
`Duration`	Unsigned, exact span: `u64` seconds + `u32` nanos. No negative spans exist.
Constructors	`from_secs`/`from_millis`/`from_micros`/`from_nanos`. `from_mins`/`from_hours` since 1.91.
Accessors	Whole-unit accessors (`as_secs`) truncate. Total-unit accessors (`as_millis`) return `u128`. `subsec_*` return the fraction.
Float conversions	`as_secs_f64`, `from_secs_f64` (panics), `try_from_secs_f64` (Result), `mul_f64`, `div_duration_f64`
Operators	`+ − × ÷` panic on overflow/underflow: an immediate, visible failure for programmer errors
`checked_*`	The boundary becomes an `Option`. For untrusted operands or operands with no known limit.
`saturating_*`	The result stays at `ZERO`/`MAX`. For countdowns and backoff schedules.
`Instant`	Monotonic, opaque, per-process. The only correct clock to measure elapsed time.
`duration_since` (Instant)	Saturates to zero on a difference in the incorrect direction. The `checked_` variant returns `None`.
Deadline pattern	Calculate `start + budget` one time. `deadline.saturating_duration_since(now)` is the remaining time.
`SystemTime`	Wall clock. It can jump in each direction. It has a meaning outside the process.
`UNIX_EPOCH`	Anchor constant. `duration_since(UNIX_EPOCH).as_secs()` is the Unix timestamp.
`SystemTimeError`	Wall-clock subtraction can legitimately fail. `err.duration()` is the size of the backward jump.

Code Examples

File	Description
`11_01_duration_construction.rs`	`Duration` constructors, normalization, accessors that truncate, float conversions, `ZERO`/`MAX`, `abs_diff`
`11_02_duration_arithmetic.rs`	Operators and their panics (caught), `checked_`, `saturating_`, a backoff that cannot overflow, `Sum`, `Ord`
`11_03_instant_monotonic.rs`	Monotonic guarantee, stopwatch pattern, saturating and checked differences, deadline pattern
`11_04_systemtime_epoch.rs`	`UNIX_EPOCH` round trips, a deterministic `SystemTimeError`, times before the epoch, expiry checks with an injected clock

11.2 · Thread Sleep, Timeouts, and Timed Operations

Domain 11 — Time and Duration Duration: ~15 minutes Library components: std::thread::sleep, std::time::Duration, std::hint::spin_loop

Introduction

Tutorial 11.1 showed the three time types: spans (Duration), the monotonic clock (Instant), and the wall clock (SystemTime). This tutorial uses those types for two tasks that each long-running program has: how to wait, and how to stop a wait.

An unbounded wait can become a hang. Examples are:

a read on a peer that never answers
a recv on a channel whose senders never send
a lock that a blocked thread holds

The standard library has a consistent answer. Almost every blocking operation has a variant that accepts a Duration and stops the wait when that time elapses.

This tutorial shows:

thread::sleep: its one guarantee (a lower bound), why simple sleep loops drift, and the solution with deadlines.
thread::sleep_until, which is still unstable as of Rust 1.99. The tutorial mentions it and does not use it.
Bounded waits on the sync primitives: Mutex::try_lock, mpsc::Receiver::recv_timeout, and Condvar::wait_timeout_while.
Bounded waits on I/O with TcpStream::set_read_timeout.
How to make a token-bucket rate limiter on Instant. The caller injects the time, so the tests are fully deterministic.
The spin-wait ladder (hint::spin_loop, thread::yield_now, and a blocking wait), and when to use each tier.

Cross-reference: Domain 6 describes the sync primitives. This tutorial shows only their timed forms.

thread::sleep: A Lower Bound, Not a Promise

thread::sleep(dur) blocks the current thread for at least dur. That is the full contract. The OS does not wake the thread earlier than the requested time. But because of scheduler latency, the thread almost always wakes somewhat later. On a loaded machine, it wakes milliseconds later. There are two consequences:

Never assert a tight upper bound on the duration of a sleep. The contract guarantees only a lower bound.
Never make a fixed-rate schedule from a sleep of a fixed length in each iteration.

The second consequence is less obvious. loop { do_work(); sleep(TICK) } has a real period of work + TICK + oversleep. Each iteration moves the schedule later, and the error accumulates. A 1-second heartbeat can be seconds late before one hour elapses.

The solution is to schedule against absolute deadlines from one anchor. Then an oversleep in one tick makes the next sleep shorter, and the error does not accumulate:

let tick = Duration::from_millis(10);
let mut next_deadline = Instant::now();   // the anchor of the schedule
loop {
    next_deadline += tick;   // anchor + n * tick: the drift cannot accumulate
    // checked_duration_since returns None if the deadline is in the past.
    if let Some(remaining) = next_deadline.checked_duration_since(Instant::now()) {
        thread::sleep(remaining);   // sleep only for the remainder of this tick
    } // None: do not sleep, do the work immediately
    do_work();   // the periodic task of your program (a placeholder)
}

On nightly, #![feature(thread_sleep_until)] gives you thread::sleep_until(next_deadline), which does this in one call. It is still unstable as of Rust 1.99. On stable, the exact equivalent is sleep(deadline.saturating_duration_since(Instant::now())). If the deadline is in the past, the saturation makes the sleep zero-length.

A zero-duration sleep returns immediately. To let a different thread run explicitly, use thread::yield_now (slide 6).

11_05_thread_sleep.rs prints:

requested 20ms, actually slept 25.010375ms (varies, always >= requested)  # (varies)
sleep(Duration::ZERO) returned immediately
3 fixed-rate ticks of 10ms took 30.906208ms (varies, always >= 30ms)      # (varies)
sleep_until: unstable on 1.99 — use the saturating_duration_since pattern

All assertions passed.

Bounded Waits on Locks and Channels

The timed variants in std::sync have one common form. You pass a Duration, and the result tells you which of three outcomes occurred: success, timeout, or permanent failure:

Figure: The bounded-wait toolbox in std

11_2_sleep_timeouts_timed_operations_1

Mutex::try_lock is the limit case: a timeout of zero duration. It takes the lock immediately or returns Err(TryLockError::WouldBlock). It never blocks. std has no lock operation with a timeout. The usual pattern is try_lock in a retry loop, or a Condvar when you need a real timed wait. Remember that the Mutex of std is not reentrant: try_lock fails even from the thread that holds the lock.

mpsc::Receiver::recv_timeout is the usual tool for a worker loop. Its two error variants tell the worker what to do next:

// job_rx: mpsc::Receiver<&str>. This match is the body of the worker loop.
// process() is the job handler of the worker (a placeholder).
match job_rx.recv_timeout(Duration::from_millis(15)) {   // waits 15 ms at most
    Ok(job) => process(job),
    Err(RecvTimeoutError::Timeout) => { /* no job yet: examine the shutdown flag, then loop */ }
    Err(RecvTimeoutError::Disconnected) => break, // all senders are gone: exit immediately
}

Timeout means "continue the loop". Disconnected arrives immediately when all the senders are gone. The receiver does not wait the full budget for messages that can never arrive. This poll loop with a timeout is how an idle worker sees a shutdown request.

Condvar::wait_timeout returns (guard, WaitTimeoutResult). Call timed_out() to learn if the deadline or a notification stopped the wait. Prefer wait_timeout_while. It examines your predicate again in a loop, and thus it handles the spurious wakeups that each OS can cause:

// job_slot: Mutex<Option<&str>>, job_ready: Condvar, patience: Duration (15 ms).
// guard: the MutexGuard from job_slot.lock().
// The wait continues while the closure returns true (the slot is empty).
let (guard, result) = job_ready
    .wait_timeout_while(guard, patience, |slot| slot.is_none())
    .expect("not poisoned");   // Err only if a thread panicked with the lock held
if result.timed_out() { /* the budget is spent and the slot is still empty */ }

11_06_sync_timeouts.rs prints:

try_lock while held -> Err(WouldBlock)
try_lock after drop -> Ok
recv_timeout with queued message -> Ok("resize-image")
recv_timeout on empty channel -> Err(Timeout) after 17.978875ms (varies)   # (varies)
recv_timeout after senders dropped -> Err(Disconnected) immediately
wait_timeout_while with no notifier -> timed_out() after 16.852542ms (varies)  # (varies)
wait_timeout_while with notifier -> woke early with job "send-report"

All assertions passed.

I/O Timeouts: TcpStream

Unbounded waits cause the most problems in network I/O. A peer can accept the connection and then send nothing. A read with no timeout then blocks forever. TcpStream::set_read_timeout sets a deadline for each read call:

// client: TcpStream — connected to a peer that sends nothing.
// buf: [u8; 16] — the read buffer.
client.set_read_timeout(Some(Duration::from_millis(25)))?;   // each read waits 25 ms at most
match client.read(&mut buf) {
    Ok(n) => { /* n bytes arrived in time */ }
    // The error kind of a timeout depends on the platform, so match the two kinds.
    Err(e) if matches!(e.kind(), ErrorKind::WouldBlock | ErrorKind::TimedOut) => {
        // the deadline passed: retry, reconnect, or fail the health check
    }
    Err(e) => return Err(e),   // a different I/O error
}

Details that are important in practice:

The error kind depends on the platform: typically WouldBlock on Unix, and possibly TimedOut on Windows. Portable code matches the two kinds.
None means "block forever" (the default).
std rejects Some(Duration::ZERO) with InvalidInput. A zero timeout would be ambiguous with "no timeout", and std does not guess. To poll and never block, use set_nonblocking (see tutorial 10.1).
A read that times out consumes no data and does not damage the connection. The connection stays healthy. A timeout applies to one call, not to the connection.
set_write_timeout is the equivalent for writes. It limits how long a write can wait for space in the kernel buffer. It expires much less frequently, because loopback and healthy links drain the buffer quickly.

11_07_tcp_read_timeout.rs prints:

connected to 127.0.0.1:58355 (port varies)                       # (varies)
default read_timeout = None (a silent peer would hang us forever)
read on silent peer -> WouldBlock after 26.013958ms (kind and time vary)  # (varies)
set_read_timeout(Some(ZERO)) -> Err(InvalidInput)
read with data available -> "pong"
write_timeout set/get round-trips; None disarms

All assertions passed.

Building a Rate Limiter with Instant

A timeout limits how long you wait. A rate limiter limits how frequently you act. The standard algorithm is the token bucket:

The bucket holds a maximum of capacity tokens.
One token comes back after each refill_interval.
Each admitted request uses one token.

The sustained throughput is one request per interval. After an idle period, a burst of up to capacity requests passes immediately.

The fixed window algorithm is simpler: "a maximum of N per second, with a reset at the boundary". But it admits up to 2N requests in a burst that crosses a boundary. The token bucket prevents exactly that burst.

Figure: Token-bucket rate limiter states

11_2_sleep_timeouts_timed_operations_2

One design decision makes the limiter testable. It is the same technique as is_expired in tutorial 11.1: the limiter never calls Instant::now(). The caller passes now as a parameter:

// Two methods of TokenBucket. The struct has these fields:
//   capacity: u32, tokens: u32, refill_interval: Duration, last_refill: Instant

// Admits one request at the time `now`. Returns true if a token is available.
fn try_acquire(&mut self, now: Instant) -> bool {
    self.refill(now);   // first add the tokens for the elapsed time
    if self.tokens > 0 { self.tokens -= 1; true } else { false }
}

fn refill(&mut self, now: Instant) {
    // If the caller passes an instant before the checkpoint, add nothing.
    let Some(elapsed) = now.checked_duration_since(self.last_refill) else { return };
    // The number of whole refill intervals in `elapsed` (integer division).
    let intervals = elapsed.as_nanos() / self.refill_interval.as_nanos();
    // Add one token for each whole interval, to a maximum of `capacity`.
    // If the bucket is not full, move the checkpoint forward by only those
    // whole intervals. The fraction that remains counts toward the next token.
    // If the bucket is full, move the checkpoint to `now`.
    ...
}

Instant + Duration makes future instants with no wait. Thus the example asserts the full behavior with zero sleeps. It gives the limiter base, base + 150ms, base + 250ms, and base + 10s, and it asserts:

the initial burst
the refill of one token per interval
the fraction that carries over to the next token
the limit at capacity
the safe result for a stale instant

At the end, one short demonstration with a real sleep tests the limiter against the real clock. It asserts only the guaranteed direction: a sleep of 25 ms or more must refill a 20 ms token.

11_08_rate_limiter.rs prints:

t=0ms:   3 admitted (burst), 4th rejected
t=150ms: 1 admitted (one interval refilled), next rejected
t=250ms: 1 admitted (fraction carried over), next rejected
t=10s:   3 admitted (refilled to cap, not beyond), 4th rejected
stale instant (t=0 again): rejected — no tokens minted
real time: admitted, refilled during a 25ms sleep, admitted again

All assertions passed.

Spin-Wait vs. Sleep: hint::spin_loop

The opposite of a sleep is a busy-wait: a tight loop that polls a flag. It has the lowest possible wake-up latency, because it does not go through the scheduler. It also has the highest cost: one core at 100% load. std::hint::spin_loop() is the one line that tells the processor about the spin loop:

// ready: AtomicBool — a different thread stores `true` with Ordering::Release.
// SPIN_CAP: u32 = 10_000_000 — the loop stops there if the flag does not change.
let mut spins = 0_u32;
while spins < SPIN_CAP && !ready.load(Ordering::Acquire) {
    hint::spin_loop();   // x86: PAUSE, AArch64: ISB. A CPU hint, NOT a syscall
    spins += 1;
}

The call tells the processor that the thread is in a busy-wait. The CPU can then decrease speculative execution, give pipeline resources to the sibling hyperthread, and save power. The thread stays runnable and keeps its core. In contrast, thread::yield_now() is a syscall, and it offers the core back to the scheduler.

Use a spin only when two conditions are true at the same time:

The expected wait is much shorter than a scheduler quantum (microseconds).
The writer runs on a different core at this moment.

In all other cases, escalate to a stronger wait. Production locks (parking_lot, and the futex-based Mutex of std) use exactly this escalation:

Figure: The wait-strategy escalation ladder

11_2_sleep_timeouts_timed_operations_3

Two rules keep a spin loop safe, and the example shows both:

Each spin has an iteration cap. If the writer dies, a spin with no cap is a hang with no end.
The last tier is a real blocking wait. Thus the code cannot livelock, whatever the scheduler does.

In the example, the slow producer takes ~10 ms, which is time for tens of millions of spins. The wait uses all of its spin budget and all of its yield budget, and then it blocks on a channel. This shows why a spin alone is a bug when the wait can be long.

11_09_spin_loop.rs prints:

fast handoff: spun 2642 times (varies) before flag/cap                        # (varies)
slow producer: finished via blocking tier (parked on the channel) (tier varies with scheduling)  # (varies)

All assertions passed.

Summary

Concept	Key point
`thread::sleep`	Guarantees only a lower bound. Oversleep is normal. Never assert tight upper bounds.
Sleep-loop drift	`work + sleep + oversleep` accumulates. Schedule against absolute deadlines from one anchor.
`thread::sleep_until`	The deadline pattern as one call. Still unstable on 1.99. Stable equivalent: `sleep(deadline.saturating_duration_since(now))`.
`Mutex::try_lock`	Zero-duration timeout: `Ok` immediately or `Err(WouldBlock)`. std has no timed lock operation.
`recv_timeout`	`Timeout` means "continue the loop". `Disconnected` is immediate when all senders are gone: the shutdown signal.
`Condvar::wait_timeout_while`	Timed condition wait. The predicate loop handles spurious wakeups. Examine `timed_out()`.
`set_read_timeout`	Deadline for each I/O call. `WouldBlock` (Unix) / `TimedOut` (Windows). std rejects `Some(ZERO)`.
Token bucket	Burst of `capacity`, then one token per interval. Inject `now: Instant` for deterministic tests.
`hint::spin_loop`	CPU hint (PAUSE/ISB), not a syscall. For busy-waits of microseconds.
`thread::yield_now`	Syscall: offers the core to the scheduler. The middle tier.
Escalation ladder	Spin (capped) → yield (capped) → block. The last tier must be a real blocking wait.

Code Examples

File	Description
`11_05_thread_sleep.rs`	The lower-bound contract of `sleep`, zero-duration sleeps, fixed-rate ticks with no drift, a note on `sleep_until`
`11_06_sync_timeouts.rs`	`Mutex::try_lock`, `recv_timeout` (all three outcomes), `Condvar::wait_timeout_while`
`11_07_tcp_read_timeout.rs`	`set_read_timeout` on loopback: the timeout on a silent peer, the rejection of `ZERO`, reads that arrive in time, write timeouts
`11_08_rate_limiter.rs`	Token-bucket limiter with injected `Instant` values: deterministic tests and one short real-time demonstration
`11_09_spin_loop.rs`	Bounded spin on an `AtomicBool`, `hint::spin_loop`, the spin → yield → block escalation ladder

12.1 · Environment Variables, Args, and Current Directory

Domain 12 — Process and Environment Interaction Duration: ~15 minutes Library components: std::env, std::env::Args, std::env::ArgsOs, std::env::Vars, std::env::VarsOs, std::env::VarError

Introduction

Each process inherits three items from its parent: an environment block, an argument list, and a working directory. The std::env module reads that inherited state, and writes it with care. The module is small, but it shows two of the most instructive design decisions of the standard library:

Each accessor is one of a pair. One version is convenient and returns UTF-8 (var, vars, args). The other version is lossless and returns OS-native strings (var_os, vars_os, args_os). No major platform guarantees that the environment and the argument list are valid Unicode. The API makes you decide what occurs when they are not.
A write to the environment is unsafe in edition 2024. env::set_var and env::remove_var look harmless. But on most Unix platforms, they can race with any other thread that reads the environment. This includes reads that are hidden in C library calls. For years, Rust treated these functions as a safe API. Then Rust concluded that they were never safe.

This tutorial shows:

How to read variables: var, var_os, and the two VarError variants.
How to change the environment soundly.
How to iterate the environment: vars and vars_os.
The argument iterators: args and args_os.
The location state of the process: current_dir, set_current_dir, current_exe, temp_dir, and home_dir (no longer deprecated).

Tutorial 12.2 continues from this tutorial. The state that a process inherits here is exactly the state that Command lets you set for a child.

Reading Variables: `var`, `var_os`, and `VarError`

env::var(key) returns Result<String, VarError>. It can fail in exactly two ways, and the two ways have very different meanings:

VarError::NotPresent: the variable is not set. Usually this is not an error. Use a default value as the alternative.
VarError::NotUnicode(OsString): the variable is set, but its value is not valid UTF-8. The error contains the raw bytes, so no data is lost. If you silently treat this case as "not set", that is a bug, because the user configured something.

env::var_os(key) returns Option<OsString> instead. It cannot fail. A variable that is not set gives None. Any value, however unusual, comes back unchanged as an OsString (tutorial 3.4 explains OsString).

Figure: Choosing between var and var_os

12_1_env_args_current_dir_1

The realistic pattern to decode a variable handles each of the three results separately:

// The example sets TUT12_01_LOG_LEVEL to "trace" before this match.
let effective = match env::var("TUT12_01_LOG_LEVEL") {
    Ok(value) => value,                                // explicit override
    Err(VarError::NotPresent) => String::from("info"), // not set: use the default
    Err(VarError::NotUnicode(raw)) => {
        // raw: OsString, the value as the OS stores it. Report the bad value.
        panic!("TUT12_01_LOG_LEVEL is not valid Unicode: {}", raw.display());
    }
};
// effective is "trace"

On Unix, an environment value is any byte sequence without a NUL byte. Thus it is easy to make a NotUnicode case for a test (the byte 0xFF never occurs in UTF-8):

#[cfg(unix)]
{
    use std::os::unix::ffi::OsStrExt;
    // invalid: &OsStr of 6 bytes. The byte 0xFF makes it invalid UTF-8.
    let invalid = std::ffi::OsStr::from_bytes(b"deb\xFFug");
    // SAFETY: the program has only one thread, so no other thread reads the environment.
    unsafe { env::set_var("TUT12_01_LOG_LEVEL", invalid) };

    // var refuses the value and returns the raw bytes in the error.
    assert!(matches!(env::var("TUT12_01_LOG_LEVEL"), Err(VarError::NotUnicode(_))));
    // var_os returns the same 6 bytes and no error.
    assert!(env::var_os("TUT12_01_LOG_LEVEL").is_some());
}

12_01_env_var_basics.rs prints:

before set: Err(NotPresent)
after set:  Ok("debug")
after overwrite: Ok("trace")
effective log level: trace
NotUnicode: raw value is 6 bytes      # (unix only)
var_os still sees all 6 bytes         # (unix only)
after remove: Err(NotPresent)

All assertions passed.

Writing the Environment: Why `set_var` Is `unsafe`

In edition 2024, env::set_var and env::remove_var are unsafe functions. Their behavior did not change. Only the signature changed, and it is now honest.

The process environment is one global, mutable table that all threads share. On most Unix platforms, setenv may reallocate or rewrite the environ block while another thread reads it. The reader is not always your code, because C library routines read the environment internally. For example, localtime reads TZ, and resolvers read LOCALDOMAIN. A getenv call that runs concurrently with setenv is a data race and a possible use-after-free. That is undefined behavior, and safe Rust must not be able to cause it.

Figure: The setenv/getenv race the unsafe contract prevents

12_1_env_args_current_dir_2

Before you call these functions, you must be able to write the soundness argument:

// SAFETY: this program has only one thread. It does not spawn a thread,
// so no other thread can read the environment during the change.
unsafe {
    env::set_var("TUT12_01_LOG_LEVEL", "debug");   // creates or overwrites the variable
    env::remove_var("TUT12_01_STALE_FLAG");        // does nothing if the variable is not set
}

The practical rule: change the environment at startup, before you spawn a thread. After that, treat the environment as read-only. If a child process needs different variables, do not change your own environment. Command::env (tutorial 12.2) limits the change to the child.

In any edition, both functions may also panic if the key is empty or contains = or a NUL byte. set_var may also panic if the value contains a NUL byte.

Iterating the Environment: `vars` and `vars_os`

env::vars() yields (String, String) pairs, and env::vars_os() yields (OsString, OsString) pairs. Each function captures a snapshot of the environment at the time of the call.

There is one hazard. The vars() iterator panics during iteration if any key or value in the full environment is not valid Unicode. This includes a variable that your program never reads. A defensive CLI iterates vars_os() and makes a decision for each entry.

Iteration is very useful in the prefix-scan configuration pattern. This is the twelve-factor style that real tools use. For example, cargo scans CARGO_*, and many services scan APP_*:

// The example sets TUT12_APP_NAME, TUT12_APP_PORT, and TUT12_APP_VERBOSE first.
let config: BTreeMap<String, String> = env::vars()
    .filter(|(key, _)| key.starts_with("TUT12_APP_"))   // keep only the prefixed variables
    // Remove the prefix and make the key lowercase: TUT12_APP_PORT becomes "port".
    .map(|(key, value)| (key["TUT12_APP_".len()..].to_lowercase(), value))
    .collect();
// config: {"name": "orders-service", "port": "8080", "verbose": "true"}

BTreeMap gives a deterministic order, because the OS returns the variables in arbitrary order. The defensive version skips keys that are not valid Unicode and does not panic:

let robust_count = env::vars_os()
    // into_string() returns Err for a key that is not valid Unicode.
    // ok() changes that Err to None, and filter_map drops the entry.
    .filter_map(|(key, _)| key.into_string().ok())
    .filter(|key| key.starts_with("TUT12_APP_"))
    .count();
// robust_count is 3: the same three variables as in the scan above

The parse-with-fallback chain converts strings into typed configuration:

let port: u16 = env::var("TUT12_APP_PORT")
    .ok()                               // Result to Option: an error becomes None
    .and_then(|raw| raw.parse().ok())   // a value that is not a u16 also becomes None
    .unwrap_or(3000);                   // None: use the default
// port is 8080, because the example set TUT12_APP_PORT to "8080"

12_02_env_iteration.rs prints:

collected 3 config entries:
  name = orders-service
  port = 8080
  verbose = true
vars_os prefix scan found 3 entries
typed config: port=8080, workers=4 (default)

All assertions passed.

Arguments: `args` and `args_os`

env::args() iterates the command-line arguments as String values. env::args_os() is the lossless OsString equivalent. The same Unicode rule applies: the args() iterator panics if any argument is not valid Unicode. This can occur on Unix (arbitrary bytes) and on Windows (unpaired UTF-16 surrogates). Both iterators implement ExactSizeIterator and DoubleEndedIterator.

By convention, the first element is the path that started the program. But that is only a convention. The parent process can set argv[0] to any value, so never trust argv[0] for a security decision. Ask the OS through env::current_exe() instead:

let argv: Vec<String> = env::args().collect();
assert!(!argv.is_empty()); // argv[0] is present by convention

// The file stem is the same on all platforms (file_stem removes `.exe` on Windows).
let program = Path::new(&argv[0]).file_stem().and_then(|s| s.to_str()).unwrap();
assert_eq!(program, "12_03_args_and_dirs");

// exe: PathBuf, the path of this binary. The OS supplies it, not the parent.
let exe = env::current_exe().expect("OS can report our path");
assert!(exe.is_file());

Args deliberately does not implement Clone and does not supply indexing. It is an iterator that you can use only one time. If you need random access, call env::args() again (the call is cheap) or use collect(). Real argument parsing is the task of crates such as clap. std gives you only the raw list.

Process Locations: `current_dir`, `current_exe`, `temp_dir`, `home_dir`

env::current_dir() returns the working directory, and env::set_current_dir(path) changes it. Two facts make set_current_dir more hazardous than it looks:

The working directory is process-wide state. Each relative path in each thread resolves against it. Thus a change in a multithreaded program silently changes the meaning of relative paths in other threads. The function is safe (it causes no memory unsafety), and it is only a correctness hazard. Compare this with set_var, which can really cause undefined behavior.
Be careful when you compare paths. Temporary directories are frequently symlinks (on macOS, /var points to /private/var). On Windows, fs::canonicalize adds a \\?\ prefix. Thus canonicalize both sides before you compare them.

let original = env::current_dir()?;   // PathBuf: the directory to restore later
// scratch: PathBuf, a unique directory in the OS temporary directory.
// The process id in the name prevents a collision between parallel runs.
let scratch = env::temp_dir().join(format!("tut12_03_{}", std::process::id()));
fs::create_dir_all(&scratch)?;

env::set_current_dir(&scratch)?;
fs::write("note.txt", "written via a relative path")?; // the file goes into scratch

env::set_current_dir(&original)?;  // restore the directory FIRST:
fs::remove_dir_all(&scratch)?;     // Windows cannot delete the cwd of a process

The other path accessors are:

env::current_exe() returns the path of the current executable, from the OS and not from argv[0]. Programs use it to find resources adjacent to the executable. Tutorial 12.2 uses it for the self-spawn pattern in all its examples.
env::temp_dir() returns the path of a temporary directory. On Unix, it returns the value of TMPDIR if that variable is set. If not, the default is OS-specific (/tmp on most Unix systems, a per-user directory on macOS). On Windows, it uses GetTempPath2, which reads TMP and then TEMP. It does not check that the directory exists.
env::home_dir() returns Option<PathBuf>. The value is None if the function cannot find the home directory. Rust 1.29 deprecated the function because its Windows behavior was wrong. It read the HOME variable, which is not a Windows convention. Rust 1.85 corrected the Windows behavior, and Rust 1.87 removed the deprecation. Thus you can use the function on 1.99.

12_03_args_and_dirs.rs prints:

argc = 1
program = 12_03_args_and_dirs
current_exe file stem = 12_03_args_and_dirs
entered scratch dir inside temp_dir: true
relative write landed in scratch dir: true
restored original working directory: true
temp_dir exists = true
home_dir found  = true    # (varies)

All assertions passed.

Summary

Concept	Key point
`env::var`	Returns `Result<String, VarError>`. Convenient, UTF-8 only.
`VarError::NotPresent`	The variable is not set. Usually, use a default value.
`VarError::NotUnicode(raw)`	The variable is set but is not UTF-8. Report it. The error keeps the raw bytes.
`env::var_os`	Returns `Option<OsString>`. Lossless, cannot fail.
`env::set_var` / `remove_var`	`unsafe` in edition 2024. They may race with a hidden `getenv` on another thread.
Sound mutation rule	Change the environment only while the process is single-threaded (startup). Prefer `Command::env` for children.
`env::vars`	`(String, String)` snapshot. The iterator panics if any key or value is not Unicode.
`env::vars_os`	`(OsString, OsString)` pairs. The base for defensive iteration.
Prefix scan	Collect `PREFIX_*` variables into a map (twelve-factor configuration).
`env::args` / `args_os`	Argument iterators. `argv[0]` is a convention, and the parent controls it.
`env::current_dir` / `set_current_dir`	Process-wide working directory. A correctness hazard with threads.
`env::current_exe`	The path of the binary, from the OS. The base of the self-spawn pattern in 12.2.
`env::temp_dir`	From `TMPDIR` on Unix, or from `GetTempPath2` on Windows. It does not check that the directory exists.
`env::home_dir`	Deprecated in 1.29, corrected in 1.85, not deprecated since 1.87. You can use it again.

Code Examples

File	Description
`12_01_env_var_basics.rs`	`var` vs `var_os`, both `VarError` variants, `set_var`/`remove_var` with the single-threaded safety argument
`12_02_env_iteration.rs`	`vars`/`vars_os`, the prefix-scan config pattern, typed parsing with defaults
`12_03_args_and_dirs.rs`	`args`/`args_os`, `current_dir`/`set_current_dir` with cleanup, `current_exe`, `temp_dir`, `home_dir`

12.2 · Spawning and Managing Child Processes

Domain 12 — Process and Environment Interaction Duration: ~15 minutes Library components: std::process::Command, std::process::Child, std::process::Output, std::process::Stdio, std::process::ExitStatus, std::process::ExitCode, std::process::Termination, std::process::exit, std::process::abort

Introduction

std::process is the portable Rust interface to the oldest form of concurrency: a program that runs another program. Real tools use it constantly. For example, cargo runs rustc, and rustc runs your system linker. rust-analyzer runs cargo check and communicates with it through pipes. This tutorial shows the full lifecycle:

How to build the launch configuration of a child with Command (arg, env, current_dir, stdin/stdout/stderr).
The three methods that run the command (spawn(), output(), status()), and the different default Stdio configuration of each one.
How to connect pipes: a round trip between parent and child, a shell-style connection of two children, and the usual pipe deadlock that wait_with_output prevents.
How to supervise a child through the Child handle: wait, try_wait, kill, wait_with_output.
How a process ends: ExitStatus (what you observe about a child), ExitCode (what your own process produces), process::exit, and process::abort.
The Termination trait, which is the mechanism that lets main return Result<(), E>.

A note about the examples: a portable example cannot assume that echo or cmd exists. Thus each example in this tutorial spawns itself. It calls Command::new(env::current_exe()?) with a marker argument such as "child:upper". main examines the marker first and, if the marker is there, runs a small child role. The invocation without arguments is the parent. This technique is also useful for integration tests.

`Command`: Building the Launch Configuration

Command::new(program) names the executable. If the name is a bare name, the OS resolves it through PATH. All other settings are builder methods, and nothing occurs until you run the command. By default, a child inherits the environment and the working directory of the parent. The builder methods specify differences from that inherited state:

// scratch: PathBuf, a directory that the example created in the OS temporary directory.
let output = Command::new(env::current_exe()?)
    .arg("child:report")                        // one argument, no shell splitting
    .env("TUT12_GREETING", "hello from parent") // ADD to the inherited environment
    .env_remove("TUT12_SECRET")                 // REMOVE from the inherited environment
    .current_dir(&scratch)                      // the first working directory of the child
    .output()?;                                 // run the command (one of three methods)
// output: Output, with the exit status and the captured stdout and stderr bytes

Important details for practical use:

arg adds one argument, and args adds each argument from an iterator. Each argument goes to the child unchanged. There is no shell between parent and child, so there is no quoting, no $VAR expansion, and no * globbing. This is an advantage, because it is the reason that Command is not prone to shell injection.
env and envs add or override variables for the child only. Because of these methods, it is almost never necessary to change your own environment with env::set_var (tutorial 12.1). env_clear() discards the full inherited environment and leaves an empty environment.
A Command is reusable. You can call .status() on it repeatedly to run the same configuration several times.

The child sees the result through the same std::env APIs that tutorial 12.1 shows, for example env::var("TUT12_GREETING") and env::current_dir(). This is the reason that this domain pairs the two tutorials. Tutorial 12.1 is the read side of the inherited state, and Command is the write side.

Three Ways to Run: `spawn`, `output`, `status`

Figure: Choosing how to run a Command

12_2_child_processes_1

output() runs the child to completion and captures all its output. It returns Output { status, stdout: Vec<u8>, stderr: Vec<u8> }. Note that the payloads are bytes, not String values. The output of a child is not necessarily UTF-8 (the same lesson as var compared with var_os).
status() runs the child to completion while the child shares your terminal. It returns only the ExitStatus. It is the correct selection when you run a tool interactively for the user.
spawn() creates the child and returns a Child handle immediately. It is the primitive on which the standard library builds the other two methods. It is the only method that lets you communicate with a child while the child runs.

// exe: PathBuf, the path of this example binary from env::current_exe().
let mut child = Command::new(&exe).arg("child:announce").spawn()?;   // child: Child
// ... the parent can do other work here while the child runs concurrently ...
let status = child.wait()?;   // ALWAYS reap: a child without wait() stays a zombie on Unix
assert!(status.success());

12_04_command_output_status.rs prints the output below. The two [child] lines come from the children of status() and spawn(), which write directly to the stdout of the parent:

captured from child:report:
  greeting = hello from parent
  secret   = absent
  child cwd matches Command::current_dir: true
[child] announcing directly on the parent's stdout
child:fail exited with code Some(3)
[child] announcing directly on the parent's stdout
spawned child finished: success = true

All assertions passed.

`Stdio`: `piped`, `null`, `inherit`

You configure each of the three standard streams of the child independently with a Stdio value. There are only three configurations, and each one specifies a different destination for the bytes:

Figure: The three Stdio configurations and where each fits

12_2_child_processes_2

Stdio::inherit(): the child shares the stream of the parent. Use it when the child must write to the user or read from the user directly: progress bars, prompts, interactive tools.
Stdio::piped(): a new kernel pipe connects parent and child. The parent end of the pipe is available on the Child as the stdin, stdout, or stderr handle. Use it when the parent supplies the input of the child or reads its output programmatically.
Stdio::null(): the stream connects to the null device. The OS discards each write, and each read returns EOF immediately. Use it to discard the output of a noisy worker, or to make sure that a child cannot wait for input.

The defaults depend on the method that runs the command. spawn() and status() inherit all three streams. output() pipes stdout and stderr (it must, to capture them) and connects stdin to the null device.

A fourth source of Stdio values is important for pipelines: the From conversions. A ChildStdout converts directly into a Stdio, and this is how you chain children together (next slide). File and owned OS handles convert too. Thus you can redirect the output of a child directly into a log file without an intermediary.

Pipes: Round Trips and Pipelines

The full write-then-read exchange with a child has three necessary steps:

Take the stdin handle from the Child, and write the input.
Drop the handle. The drop closes the pipe, and that close is the EOF of the child. If the handle stays open, the child never exits (the usual cause of this hang).
Collect the output with wait_with_output.

Figure: Parent–child pipe round trip

12_2_child_processes_3

// exe: PathBuf from env::current_exe(). The child:upper role makes each stdin line uppercase.
let mut child = Command::new(&exe).arg("child:upper")
    .stdin(Stdio::piped())                    // the parent writes here
    .stdout(Stdio::piped())                   // the parent reads here
    .spawn()?;

// take() moves the ChildStdin out of the Child, so that the parent can drop it.
let mut stdin = child.stdin.take().expect("stdin was piped");
stdin.write_all(b"hello\npipeline world\n")?;
drop(stdin);                                  // closes the pipe: the child sees EOF

let output = child.wait_with_output()?;       // reads to EOF, then reaps the child
assert_eq!(String::from_utf8(output.stdout)?, "HELLO\nPIPELINE WORLD\n");

Use wait_with_output. Do not call wait() first and read the output afterward. The reason is a deadlock. Pipes have finite kernel buffers (approximately 64 KiB on Linux). A child that blocks on a write into a full stdout pipe never exits. A parent that blocks in wait() never drains the pipe. Each process waits for the other, forever.

wait_with_output reads both streams to EOF before it reaps the child, so the full buffer can never block the two processes. To write the stdin and read the stdout of a long-running child concurrently, you need a thread or an async runtime.

To connect two children in shell style (upper | exclaim), use the From<ChildStdout> for Stdio conversion. The data flows from child to child through the kernel and does not go through the parent:

// upper: Child that runs the child:upper role, spawned with piped stdin and stdout
// (the same configuration as `child` in the previous snippet).
let upper_stdout = upper.stdout.take().expect("stdout was piped");   // ChildStdout
let exclaim = Command::new(&exe).arg("child:exclaim")
    .stdin(Stdio::from(upper_stdout))   // exclaim reads the output of upper directly
    .stdout(Stdio::piped())             // the parent reads the result of the pipeline
    .spawn()?;
// The example then writes "one\ntwo\n" to the stdin of upper.
// The parent reads "ONE!\nTWO!\n" from the stdout of exclaim.

12_05_pipes_roundtrip.rs prints:

upper produced:
HELLO
PIPELINE WORLD
pipeline produced:
ONE!
TWO!

All assertions passed.

Supervising a Child: `wait`, `try_wait`, `kill`

The Child handle supplies the tools of a supervisor. A test runner with a timeout, a service manager, and an editor that terminates a language server all have this structure:

wait() blocks until the child exits, returns its ExitStatus, and reaps the child. To reap a child is to release the OS records for it. On Unix, a child that exited but that the parent did not reap is a zombie.
try_wait() is the non-blocking probe. It returns Ok(Some(status)) if the child exited, and Ok(None) if the child still runs. Thus a watchdog can poll the child between other tasks.
kill() is forceful termination: SIGKILL on Unix, TerminateProcess on Windows. The child cannot do its cleanup, so treat kill() as the escalation, not as the shutdown protocol. Real supervisors first send a request to exit (a message through a pipe, or a platform signal). They call kill() only after a timeout.

// exe: PathBuf from env::current_exe(). The child:patient role sleeps for up to 30 seconds.
let mut child = Command::new(&exe).arg("child:patient")
    .stdin(Stdio::null())
    .stdout(Stdio::null())      // discard the output of the worker
    .spawn()?;

assert!(child.try_wait()?.is_none());   // Ok(None): the child still runs

child.kill()?;                          // sends the termination request (SIGKILL on Unix)
let status = loop {                     // polls until the OS reports the exit status
    if let Some(status) = child.try_wait()? { break status; }
    thread::sleep(Duration::from_millis(10));
};
assert!(!status.success());             // a killed child did not exit normally

The example checks two details:

kill() only sends the termination request. The exit status arrives a moment later, and that is the reason for the poll loop.
After the parent reaps the child, the Child keeps the status in a cache. Thus later wait() and try_wait() calls return the same status and not an error.

12_06_kill_try_wait.rs prints:

supervising child pid = 8805            # (varies)
try_wait while running: Ok(None)
killed child success = false
unix: code() = None, signal() = Some(9) [SIGKILL]
wait() after try_wait() returns the same cached status

All assertions passed.

How Processes End: `ExitStatus`, `ExitCode`, `exit`, `abort`

Two types with similar names are on opposite sides of the process boundary:

ExitStatus is what you observe about a finished child: success(), and code() -> Option<i32>. The Option is necessary. On Unix, a child that a signal killed has no exit code. code() returns None, and the Unix-only ExitStatusExt::signal() gives the signal number. Portable code must handle None.
ExitCode is what you produce for your own process when you return it from main. It is deliberately opaque (it has no getter), and you construct it from a u8. 0..=255 is the only range that has a portable meaning. ExitCode::SUCCESS and ExitCode::FAILURE are sufficient for the common cases. ExitCode::from(42) selects a documented custom code. Such codes are the contract that scripts and CI systems use (grep documents 0/1/2 in exactly this way).

Two functions end a process immediately, without a return through main:

process::exit(code) terminates the process immediately with the given code. Destructors do not run, so the data in buffers that you did not flush is lost. It is legitimate as the last line of the error path of a CLI. Deep in library code, it is a sign of a design problem.
process::abort() is abnormal termination: no destructors run, Termination does not apply, and catch_unwind cannot catch it. It exists for the cases where the state may be corrupt and the program must not touch memory again. On Unix, it raises SIGABRT, so a parent observes code() == None and signal() == Some(6). Windows reports a specific failure code instead. Do not assert on the value of that code in portable code.

// exe: PathBuf from env::current_exe(). The child:abort role calls process::abort().
let aborted = Command::new(&exe).arg("child:abort").output()?;
assert!(!aborted.status.success());
#[cfg(unix)]
{
    use std::os::unix::process::ExitStatusExt;     // supplies signal() on Unix
    assert_eq!(aborted.status.code(), None);       // a signal ended the child: no exit code
    assert_eq!(aborted.status.signal(), Some(6));  // signal 6 is SIGABRT
}

`Termination`: How `main() -> Result` Works

fn main() -> Result<(), io::Error> compiles because main may return any type that implements std::process::Termination. After main returns, the runtime calls Termination::report(value). That call converts the value into the ExitCode that the OS sees. The standard implementations are:

`main` returns	`report()` produces
`()`	`ExitCode::SUCCESS`
`ExitCode`	The same value (full manual control)
`Result<(), E: Debug>`	`Ok`: `SUCCESS`. `Err(e)`: prints `Error: {e:?}` to stderr, then `FAILURE`
`!` / `Infallible`	(never returns)

You can implement the trait on stable Rust. Thus a type can document the exit codes that a tool uses, because each variant maps to one documented code. A return through main (unlike process::exit) still runs destructors:

// Each variant is one possible result of a run of the tool.
#[derive(Debug)]
enum Outcome { Success, BadConfig }

impl Termination for Outcome {
    // The runtime calls report() after main returns.
    fn report(self) -> ExitCode {
        match self {
            Outcome::Success => ExitCode::SUCCESS,       // exit code 0
            Outcome::BadConfig => {
                eprintln!("error: bad configuration");   // last diagnostic
                ExitCode::from(42)                       // custom exit code 42
            }
        }
    }
}

// main returns an Outcome, and report() converts it into the process exit code.
fn main() -> Outcome { /* … */ }

Because ExitCode is opaque, you can test this only from outside the process. 12_07_exitcode_termination.rs spawns itself and asserts on the ExitStatus that it observes. It prints:

child:ok          -> code Some(0)
child:bad-config  -> code Some(42)
child:abort       -> success = false
unix: child:abort -> signal() = Some(6) [SIGABRT]

All assertions passed.

Summary

Concept	Key point
`Command::new` + `arg`/`args`	Arguments go to the child unchanged: no shell, no injection, no globbing
`env` / `env_remove` / `env_clear`	Differences from the inherited environment, for the child only
`current_dir`	The first working directory of the child
`output()`	Blocks and captures `Output { status, stdout, stderr }` (bytes, not `String`)
`status()`	Blocks with inherited stdio and returns only `ExitStatus`
`spawn()`	Returns a `Child` immediately: the concurrent primitive that you can supervise
`Stdio::inherit / piped / null`	Share the terminal / connect to the parent / discard
Pipe round trip	take stdin → write → drop (EOF) → `wait_with_output`
`wait_with_output`	Reads to EOF before it reaps the child. Prevents the full-pipe deadlock.
`Stdio::from(ChildStdout)`	Chains children in shell style. The data does not go through the parent.
`wait` / `try_wait` / `kill`	Reap (blocking) / poll (non-blocking) / terminate forcefully
`ExitStatus::code()`	`Option<i32>`. `None` on Unix means that a signal killed the child.
`ExitCode`	Opaque, in the `u8` range. Return it (or `SUCCESS`/`FAILURE`) from `main`.
`process::exit` / `abort`	Immediate exit (no destructors) / abnormal abort (`SIGABRT` on Unix)
`Termination`	`report()` converts the return value of `main` into the OS exit code

Code Examples

File	Description
`12_04_command_output_status.rs`	`Command` builder (`arg`, `env`, `env_remove`, `current_dir`), `output()` compared with `status()` and `spawn()`, the self-spawn pattern
`12_05_pipes_roundtrip.rs`	`Stdio::piped` round trip, EOF via dropped stdin, `wait_with_output`, two-child pipeline via `Stdio::from(ChildStdout)`
`12_06_kill_try_wait.rs`	Watchdog pattern: `try_wait` polling, `kill`, `Stdio::null`, signal vs code on Unix/Windows
`12_07_exitcode_termination.rs`	Custom `Termination` impl, `ExitCode::from(42)` observed from the parent, `process::exit` vs `process::abort`

13.1 · PartialEq, Eq, PartialOrd, Ord: The Comparison Hierarchy

Domain 13 — Comparing, Ordering, and Hashing Duration: ~15 minutes Library components: std::cmp::PartialEq, std::cmp::Eq, std::cmp::PartialOrd, std::cmp::Ord, std::cmp::Ordering, std::cmp::Reverse, std::cmp::min, std::cmp::max, std::cmp::min_by, std::cmp::max_by, std::cmp::min_by_key, std::cmp::max_by_key

Introduction

Most languages have one equality operator and one ordering operator. Rust has four traits, and each one has a purpose. Together, the traits put two mathematical facts into the type system. In other languages, these facts can cause errors at run time:

Some types have values that are not equal to themselves (NaN).
Some types have pairs of values that have no order.

This tutorial shows:

Why the hierarchy has PartialEq/Eq and PartialOrd/Ord, and what each level guarantees.
The float problem: NaN breaks reflexivity and totality, and f64::total_cmp gives floats a total order that you can sort by.
Manual Ord implementations with Ordering combinators (then_with), and the consistency contract that the four traits must obey together.
The derive-order problem: #[derive(Ord)] compares fields in declaration order. That order is correct only if the fields of your struct are in that sequence.
Cross-type comparisons with PartialEq<Rhs>, the mechanism behind String == &str.
Reverse for descending sorts and for the min-heap pattern, and the standalone min/max/clamp family.

Why Four Traits?

Each trait adds one guarantee to the guarantees of the previous trait:

PartialEq gives you == with symmetry and transitivity, but not reflexivity.
Eq adds the reflexivity guarantee (a == a, always) and nothing else. It is a marker trait with no methods.
PartialOrd gives you <, >, <=, and >= through partial_cmp, which returns Option<Ordering>. None means that the two values have no order.
Ord adds the last guarantee. cmp returns a plain Ordering, so each pair of values is comparable.

Figure: The four comparison traits and what each level adds

13_1_comparison_hierarchy_1

These guarantees make APIs available. HashMap keys require Eq, because the map could never find a key that is not equal to itself. slice::sort, BTreeMap, Iterator::max, and binary_search require Ord, because a sort has a meaning only with a total order. If your type implements only the Partial traits, a call to those APIs is a compile error. That error is intentional.

let nan = f64::NAN;
assert_ne!(nan, nan);                          // == is not reflexive for f64
assert_eq!(nan.partial_cmp(&nan), None);       // NaN has no order, not even with itself
// The next two lines do not compile, because `sort` requires `Ord`:
// let mut xs = vec![3.0_f64, 1.0];
// xs.sort();
// error[E0277]: the trait bound `f64: Ord` is not satisfied

f64 cannot give the two guarantees. Thus it implements PartialEq and PartialOrd, but not Eq and not Ord. For this reason, you cannot call plain .sort() on a Vec<f64>, and f64 cannot be a HashMap key.

Floats, NaN, and total_cmp

A common alternative is sort_by(|a, b| a.partial_cmp(b).unwrap()). It works until the first NaN gets to the comparator. Then it panics during the sort. The solution of the standard library is f64::total_cmp (and f32::total_cmp). It implements the IEEE 754 totalOrder predicate, which is a total order of all float bit patterns:

-NaN < -∞ < negative numbers < -0.0 < +0.0 < positive numbers < +∞ < +NaN

// Sensor readings. A failed measurement is NaN.
let mut readings = vec![22.4, f64::NAN, 19.8, 25.1, f64::NAN, 21.0];
// f64::total_cmp is the comparator: it returns an Ordering for each pair.
readings.sort_by(f64::total_cmp);              // no unwrap, no panic
// readings is now [19.8, 21.0, 22.4, 25.1, NaN, NaN]

The sort always puts the NaN values at one end of the slice. There you can remove them before you calculate a percentile or a median. Two details are important:

total_cmp is finer than ==. It puts -0.0 before +0.0 (IEEE == says that they are equal), and it orders NaN values by their sign bit.
total_cmp has a different policy from the inherent f64::max, which ignores NaN (f64::max(NAN, 3.0) == 3.0). Floats have at least three comparison semantics, and you select one at each call site.

13_01_nan_partialord_total_cmp.rs prints:

NaN == NaN            → false
1.0.partial_cmp(&NaN) → None
sort_by(partial_cmp().unwrap()) with NaN → panicked (as expected)
sorted readings       → [19.8, 21.0, 22.4, 25.1, NaN, NaN]
median of valid part  → 22.4
total order           → [NaN, -inf, -2.5, -0.0, 0.0, 1.5, inf, NaN]
f64::max(NaN, 3.0)    → 3.0 (NaN ignored)

All assertions passed.

Manual Ord and the Consistency Contract

When the derive cannot give the order that you need, implement Ord manually. Chain the field comparisons with Ordering::then_with. A typical case is semver. Semver says that 1.0.0-alpha < 1.0.0, but the Ord of Option<String> says that None < Some(_), which is the opposite. Cargo implements the same semver rule:

// Version has the fields major, minor, patch (each u32) and pre (Option<String>).
// `pre` is the pre-release tag, for example Some("alpha"). A release has None.
impl Ord for Version {
    fn cmp(&self, other: &Self) -> Ordering {
        self.major.cmp(&other.major)                      // most significant field first
            .then_with(|| self.minor.cmp(&other.minor))   // runs only if major is Equal
            .then_with(|| self.patch.cmp(&other.patch))   // runs only if minor is also Equal
            .then_with(|| match (&self.pre, &other.pre) {
                (None, None) => Ordering::Equal,
                (None, Some(_)) => Ordering::Greater,  // release > pre-release
                (Some(_), None) => Ordering::Less,     // pre-release < release
                (Some(a), Some(b)) => a.cmp(b),        // two tags: String order
            })
    }
}

then_with is lazy. The closure runs only when all the comparisons before it returned Equal. Thus a fast integer comparison can make the slower comparisons unnecessary. then is the eager variant. Use it when you already have the second Ordering.

Two rules keep the four traits consistent:

Define PartialOrd through Ord when you implement Ord manually. The body of partial_cmp is Some(self.cmp(other)) and nothing else. If the two implementations have independent logic, they can become different.
Obey the consistency contract. a == b must be true exactly when a.cmp(&b) == Ordering::Equal, and partial_cmp must always return Some(cmp(..)). A violation is a logic error. When the traits disagree, sort, BTreeMap, and binary_search operate incorrectly and report no error.

13_02_manual_ord_semver.rs prints:

string order: "1.10.0" < "1.2.0"  (wrong for versions)
Version ord:  1.2.0 < 1.10.0        (numeric, correct)
semver rule:  1.0.0-alpha < 1.0.0
sorted:       0.9.9 < 1.0.0-alpha < 1.0.0-beta.2 < 1.0.0 < 1.2.0 < 1.10.0
newest:       1.10.0
consistency:  ==, partial_cmp, cmp all agree on 36 pairs
Ordering:     Less.is_lt()=true, Less.reverse()=Greater

All assertions passed.

The Derive-Order Problem

#[derive(PartialOrd, Ord)] compares fields in declaration order, from top to bottom. It orders enum variants by their discriminants, which are in declaration order unless you set explicit discriminants. The derive is correct only when your declaration order is the same as your semantic order:

#[derive(PartialEq, Eq, PartialOrd, Ord)]
struct Release {
    codename: String,  // ← compared FIRST: releases sort alphabetically
    year: u16,         // the derive uses the date fields only for equal codenames
    month: u8,
    day: u8,
}

releases.iter().max() now returns the release with the last codename in alphabetical order. Thus it reports the 2023 release zephyr as more recent than the 2025 release aurora. Nothing reports the failure: the code compiles, the sorts run, and the answer is wrong.

There are two solutions:

Change the field order so that the most significant field is first (year, month, day, codename). This solution has the lowest cost when no code depends on the struct layout.
Implement Ord manually with the tuple-of-keys pattern. Tuples compare lexicographically, so put the fields into a tuple in semantic order:

// Release keeps its field order (codename first) and does not derive PartialOrd or Ord.
impl Ord for Release {
    fn cmp(&self, other: &Self) -> Ordering {
        // The tuples compare element by element: year, then month, day, and codename.
        // &self.codename borrows the String, so there is no clone.
        (self.year, self.month, self.day, &self.codename)
            .cmp(&(other.year, other.month, other.day, &other.codename))
    }
}
// The manual PartialOrd returns Some(self.cmp(other)), as in the previous section.

Enums have the same problem. If you put the variants in alphabetical order, the result is Critical < Error < Info < Warning. Then log.iter().max() reports a Warning as the worst event. Declare the variants in semantic order (Info, Warning, Error, Critical), and the derive is correct.

Figure: Which comparison trait do I derive or implement?

13_1_comparison_hierarchy_2

Cross-Type Comparisons: `PartialEq<Rhs>`

PartialEq is generic over its right-hand side: PartialEq<Rhs = Self>. Because of the default parameter, == usually compares two values of the same type. But you can add a second implementation with a different Rhs. The standard library uses this mechanism to make String == &str, Vec<T> == [T; N], and PathBuf == String compile.

You can use the same mechanism for your own newtypes. For example, a service receives raw u64 ids from the network and keeps them internally in a UserId wrapper. With cross-type implementations, you do not need to write .0 at each comparison:

// A newtype for a raw u64 id. `assert_eq!` needs Debug to print a failure.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct UserId(u64);

// UserId == u64
impl PartialEq<u64> for UserId {
    fn eq(&self, other: &u64) -> bool { self.0 == *other }
}
// u64 == UserId: ALWAYS supply the mirror implementation
impl PartialEq<UserId> for u64 {
    fn eq(&self, other: &UserId) -> bool { *self == other.0 }
}

assert_eq!(UserId(1001), 1001_u64);    // uses the first implementation
assert_eq!(1001_u64, UserId(1001));    // uses the mirror implementation

PartialOrd<Rhs> has the same pattern, and its supertrait bound is PartialEq<Rhs>. With a PartialOrd<u64> implementation, id > RESERVED_RANGE_END compares an id with a raw u64 threshold. Cross-type comparison has two limits:

There is no cross-type Eq. The declaration is Eq: PartialEq<Self>, because reflexivity has a meaning only in one type.
Cross-type == does not let get on a HashMap<UserId, V> accept a u64. Map lookups use the Borrow trait (tutorial 1.2).

The contract also applies across types. If a == b compiles in the two directions, the two implementations must agree (symmetry). A chain of comparisons through a third type must be transitive.

Reverse: Descending Sorts and Min-Heaps

std::cmp::Reverse<T> is a zero-cost wrapper. Its Ord is the opposite of the Ord of its contents. With Reverse, you state the opposite order in the type, and you do not write a comparator. It has two common uses:

Descending sorts. sort_by_key(|x| Reverse(key)) is easier to read than a comparator with swapped arguments. The sort stays stable: elements with equal keys keep their initial relative order:

// leaderboard: Vec<(&str, i32)> of (player, score) pairs, initially
// [("nova", 1450), ("rex", 2210), ("ash", 1450), ("kai", 3100)]
leaderboard.sort_by_key(|&(_, score)| Reverse(score));   // highest score first
// [("kai", 3100), ("rex", 2210), ("nova", 1450), ("ash", 1450)]
// "nova" stays before "ash": the two have equal scores

Min-heaps. BinaryHeap is always a max-heap (tutorial 4.4), and there is no MinHeap type. If you put each element in a Reverse wrapper, the largest wrapper contains the smallest value:

// Each element is Reverse((deadline, job name)).
let mut jobs: BinaryHeap<Reverse<(u32, &str)>> = BinaryHeap::new();
jobs.push(Reverse((30, "compress logs")));
jobs.push(Reverse((10, "rotate keys")));
// pop returns the largest Reverse, which contains the smallest tuple.
let Reverse((deadline, name)) = jobs.pop().unwrap();  // (10, "rotate keys")

The two together give the usual top-k pattern. Keep the k largest items of a stream in a BinaryHeap<Reverse<T>> that has a maximum of k elements. The root of the heap is the smallest of these items. That item is the one to remove when a larger item arrives. The cost is O(n log k), and no full sort is necessary.

13_04_reverse_minmax_clamp.rs prints the lines below. The last three lines are for the next section:

leaderboard: [("kai", 3100), ("rex", 2210), ("nova", 1450), ("ash", 1450)]
run order  : [(10, "rotate keys"), (20, "send digest"), (30, "compress logs")]
top-3      : [705, 830, 990]
tie rules  : min_by_key → ("alice", 30), max_by_key → ("bob", 30)
min_by     : min_by(0.5, NaN, total_cmp) = 0.5
clamp      : volume 1.3 → 1

All assertions passed.

min, max, and clamp: The Standalone Family

std::cmp has free functions that compare two values. You do not need to make a collection to select one of two values:

Function	Signature sketch	Use when
`min(a, b)` / `max(a, b)`	Requires `Ord`	The values have a plain total order
`min_by(a, b, cmp)` / `max_by(a, b, cmp)`	Takes an explicit comparator	The values are floats (with `total_cmp`), or you need a custom order
`min_by_key(a, b, f)` / `max_by_key(a, b, f)`	Takes a function that gets a key	You compare structs by one field

The documentation specifies the result for equal values: min returns the first argument, and max returns the second. Because of this asymmetry, a fold with repeated max calls returns the last of the equal values. The asymmetry also keeps min/max consistent with stable sorting.

min_by and max_by are a good match for total_cmp. The comparator parameter is the reason that floats work here without Ord:

// +NaN is the maximum of the total order, so 0.5 is the minimum.
assert_eq!(min_by(0.5_f64, f64::NAN, f64::total_cmp), 0.5);

clamp limits a value to a closed range: 15.clamp(0, 10) == 10. For integers, it is a method of Ord. For f32 and f64, it is an inherent method, which also rejects NaN bounds. clamp panics if min > max: it checks the bounds and does not swap them. Thus it is the correct tool for input that a user controls, such as a volume slider.

Summary

Concept	Key point
`PartialEq`	`==` with symmetry and transitivity. The trait does not guarantee reflexivity.
`Eq`	Marker trait that adds reflexivity. `HashMap` keys require it.
`PartialOrd`	`partial_cmp` returns `Option<Ordering>`. `None` means that the values have no order.
`Ord`	Total order. `sort`, `BTreeMap`, `max`, and `binary_search` require it.
`NaN`	Breaks reflexivity and totality. For this reason, `f64` implements only the `Partial` traits.
`f64::total_cmp`	IEEE 754 totalOrder: sortable floats, `NaN` at the ends, `-0.0 < +0.0`
Consistency contract	`==`, `partial_cmp`, and `cmp` must agree. A violation corrupts sorted structures and reports no error.
`then_with`	Lazy chain of field comparisons for a manual `Ord`
Derive order	Fields compare in declaration order, and enum variants too. Change the order or implement `Ord` manually.
Tuple-of-keys	`(a, b, c).cmp(&(x, y, z))`: a manual `Ord` in one line
`PartialEq<Rhs>`	Cross-type `==` (newtype and inner type, `String` and `&str`). Always add the mirror implementation.
`Reverse`	Zero-cost opposite order: descending `sort_by_key`, min-heap with `BinaryHeap<Reverse<T>>`
`min`/`max` family	Documented result for equal values (`min` returns the first, `max` returns the second). The `_by` variants accept `total_cmp`.
`clamp`	Limits a value to a range. Panics if `min > max`.

Code Examples

File	Description
`13_01_nan_partialord_total_cmp.rs`	`NaN` breaks reflexivity and totality, the `partial_cmp().unwrap()` panic, a float sort with `total_cmp`
`13_02_manual_ord_semver.rs`	A manual `Ord` for semver versions with `then_with`, the consistency contract, `Ordering` combinators
`13_03_derive_order_gotcha.rs`	The derive problems of field order and variant order, with the two solutions (a new field order, a manual tuple-of-keys implementation)
`13_04_reverse_minmax_clamp.rs`	`Reverse` for a descending sort, a min-heap, and the top-k pattern. Also `min`/`max`/`min_by`/`max_by_key`/`clamp`.
`13_05_cross_type_eq.rs`	`PartialEq<u64>`/`PartialOrd<u64>` for a `UserId` newtype, and the `String`/`&str` and `Vec`/array comparisons of the standard library

13.2 · Custom Hashing: Implementing Hash and Building Hashers

Domain 13 — Comparing, Ordering, and Hashing Duration: ~15 minutes Library components: std::hash::Hash, std::hash::Hasher, std::hash::BuildHasher, std::hash::BuildHasherDefault, std::hash::RandomState, std::hash::DefaultHasher

Introduction

Tutorial 4.2 introduced the Hash, Hasher, and BuildHasher traits briefly, as the mechanism behind HashMap. This tutorial examines each of the three traits in detail. You will:

See the byte stream that a Hash implementation gives to a hasher.
Implement a real hashing algorithm (FNV-1a) as a Hasher.
Connect that hasher to HashMap in two different ways.
Break the Hash/Eq contract on purpose, and see the deterministic result.

This tutorial shows:

The pipeline: the Hash::hash method of a value sends write_* calls to a Hasher state machine, and finish() returns the u64.
What you may assert about DefaultHasher (properties) and what you must never assert (exact values).
How to write FNV-1a as a Hasher and connect it to HashMap with BuildHasherDefault.
Why BuildHasher is a separate factory trait. A seeded hasher factory, which has the same design as RandomState, shows the reason.
RandomState, HashDoS resistance, and when a deterministic hasher is the correct selection.
The Hash/Eq contract, and the two symptoms of a violation: lookups that miss equal keys, and sets that contain "duplicates".

The Pipeline: Hash → Hasher → finish

The design divides the work into two parts, and neither part needs the details of the other. The Hash implementation of a type selects the bytes that represent the identity of a value. A Hasher converts bytes into a u64. Each Hash type works with each Hasher. That is the purpose of the generic signature fn hash<H: Hasher>(&self, state: &mut H).

Figure: The hashing pipeline from value to u64

13_2_custom_hashing_1

Hasher requires only two methods: write(&mut self, &[u8]) and finish(&self) -> u64. It supplies the typed methods (write_u8, write_u32, write_usize, …) as default methods that call write. finish takes &self: it reads the accumulated state and does not consume the hasher.

#[derive(Hash)] hashes each field in declaration order. That structure is the documented contract of the derive. The exact byte encoding of each field is an implementation detail.

Watching the Pipeline: A Tracing Hasher

Hasher is only a trait, so a short decorator can make the pipeline visible. The decorator records each write_* call in a log and then delegates to a real DefaultHasher. When you hash a struct with a derived Hash through the decorator, the log shows what the derive sends:

// log gets one entry for each call that the hasher receives.
struct TraceHasher { inner: DefaultHasher, log: Vec<String> }

impl Hasher for TraceHasher {
    // The result comes from the inner DefaultHasher.
    fn finish(&self) -> u64 { self.inner.finish() }
    fn write(&mut self, bytes: &[u8]) {
        // Record the call, with the bytes in hexadecimal.
        self.log.push(format!("write({} bytes: {bytes:02x?})", bytes.len()));
        // Then give the same bytes to the real hasher.
        self.inner.write(bytes);
    }
    // write_u8, write_u32, and write_usize record and delegate in the same way
}

// The example hashes this value through a TraceHasher:
// #[derive(Hash)] struct Packet { seq: u32, source: String, urgent: bool }
// packet = Packet { seq: 7, source: "gateway-1".to_string(), urgent: true }

13_06_hash_pipeline.rs prints:

Hash::hash(&packet) produced 4 hasher calls:
  write_u32(7)
  write(9 bytes: [67, 61, 74, 65, 77, 61, 79, 2d, 31])
  write_u8(0xff)
  write_u8(0x01)
finish() → 0x228a20ddaab16c60  (value not asserted — see below)  # (may change across Rust versions)

The trace shows an important design decision: after the bytes of a str, the pipeline sends a 0xff terminator. Without the terminator, the tuples ("ab", "c") and ("a", "bc") would give identical byte streams to the hasher. Then each composite key with adjacent strings would have serious collisions. This is a prefix collision. The terminator keeps the streams different. The same example prints:

("ab", "c") stream:
  write(2 bytes: [61, 62])
  write_u8(0xff)
  write(1 bytes: [63])
  write_u8(0xff)
("a", "bc") stream:
  write(1 bytes: [61])
  write_u8(0xff)
  write(2 bytes: [62, 63])
  write_u8(0xff)
streams differ → hashes differ

Examine this protocol, but do not depend on it. The byte encoding is not specified and may change between Rust releases.

What You May (and May Not) Assert About DefaultHasher

DefaultHasher is currently SipHash-1-3, but the documentation explicitly does not guarantee the algorithm. The algorithm changed before and may change again. The practical rules are:

Never assert, store, or send exact DefaultHasher output. A hash that one Rust version writes to disk may not match the hash that the next version calculates.
Do assert properties. The whole ecosystem depends on one property: equal values have equal hashes. HashMap<String, V>::get(&str) works because of it. The Borrow contract (tutorial 1.2) requires hash(owned) == hash(borrowed). For this reason, String deliberately has the same hash as the str that it derefs to.

// One RandomState makes hashers that all have the same keys.
let state = RandomState::new();
// The same value gives the same hash. The exact number varies between runs.
assert_eq!(state.hash_one("commit-3f9a"), state.hash_one("commit-3f9a"));

let owned = String::from("release");
// &String on the left, &str on the right: the two hashes are equal.
assert_eq!(state.hash_one(&owned), state.hash_one("release"));

One detail: DefaultHasher::new() uses constant keys, so it is deterministic in one toolchain. The same input gives the same output on each run. That determinism applies to one toolchain and is not permanent. You can reproduce the numeric value, but do not use it as a stable artifact. When you need values that you can assert exactly, implement the algorithm yourself. The next slide shows how.

Writing a Real Hasher: FNV-1a

FNV-1a is a well-known hash for small keys. The fnv and rustc-hash crates have the same function for cargo, rustc, and Firefox. The full algorithm has one u64 of state. For each byte, it does one XOR and one multiplication:

// The published 64-bit FNV-1a parameters.
const FNV_OFFSET_BASIS: u64 = 0xcbf2_9ce4_8422_2325;
const FNV_PRIME: u64 = 0x100_0000_01b3;

struct Fnv1a { state: u64 }   // the full state of the hasher

// A new hasher starts at the offset basis. BuildHasherDefault requires this Default.
impl Default for Fnv1a {
    fn default() -> Self { Self { state: FNV_OFFSET_BASIS } }
}

impl Hasher for Fnv1a {
    fn finish(&self) -> u64 { self.state }
    fn write(&mut self, bytes: &[u8]) {
        for &byte in bytes {
            self.state ^= u64::from(byte);                     // XOR the byte into the state
            self.state = self.state.wrapping_mul(FNV_PRIME);   // multiply, overflow wraps
        }
    }
}

The algorithm is yours, so its outputs are part of its specification. Exact assertions against the published FNV-1a test vectors are legitimate. They pass on every Rust version, with no time limit:

// fnv1a(bytes) is a helper: Fnv1a::default(), then write(bytes), then finish().
assert_eq!(fnv1a(b""), 0xcbf2_9ce4_8422_2325);       // empty input = offset basis
assert_eq!(fnv1a(b"a"), 0xaf63_dc4c_8601_ec8c);
assert_eq!(fnv1a(b"foobar"), 0x8594_4171_f739_67e8);

BuildHasherDefault<H> is a zero-sized adapter. It makes the factory that HashMap needs from any Hasher that implements Default. One type alias is the full integration:

type FnvBuildHasher = BuildHasherDefault<Fnv1a>;
// The third type parameter of HashMap is the hasher factory.
let mut interned: HashMap<&str, u32, FnvBuildHasher> = HashMap::default();

HashMap::new() exists only for the RandomState default. For a different hasher, use HashMap::default() or HashMap::with_hasher.

13_07_fnv_hasher.rs prints:

fnv1a(b"")       = 0xcbf29ce484222325
fnv1a(b"a")      = 0xaf63dc4c8601ec8c
fnv1a(b"foobar") = 0x85944171f73967e8

interned paths   = [("Cargo.toml", 2), ("src/lib.rs", 1), ("src/main.rs", 0)]
two FNV builders agree: 0xad4b47697711d239

RandomState A    : 0xd9bf5732a8b70e3e  (varies per run)
RandomState B    : 0xda91be1ab29a3d15  (varies per run)

All assertions passed.

BuildHasher: Why the Factory Exists

A Hasher is one hash calculation. But a HashMap needs a new hasher for each key operation, and all the hashers must have the same configuration. If they did not, the same key would have different hashes on insert and on lookup. BuildHasher is that long-lived configuration:

pub trait BuildHasher {
    type Hasher: Hasher;                             // the hasher type that the factory makes
    fn build_hasher(&self) -> Self::Hasher;          // makes one new hasher
    fn hash_one<T: Hash>(&self, x: T) -> u64 { … }  // convenience: build + hash + finish
}

The separation becomes necessary when the configuration has state. A seeded FNV factory has the same structure as RandomState, but you control the randomness. A Hasher alone cannot express it, because the seed must live longer than one calculation:

// The factory keeps the seed. Fnv1a is the hasher from the previous section.
struct SeededFnv { seed: u64 }

impl BuildHasher for SeededFnv {
    type Hasher = Fnv1a;
    // Each new hasher starts from the seed, not from the FNV offset basis.
    fn build_hasher(&self) -> Fnv1a { Fnv1a { state: self.seed } }
}

Figure: One factory per map, one fresh hasher per key operation

13_2_custom_hashing_2

RandomState is this same pattern, with a seed from the operating system. Each HashMap::new() gets its own random SipHash keys. Thus an attacker cannot calculate in advance a set of keys that all go into one bucket (HashDoS). The visible cost is that the iteration order is different between runs. For this reason, tests sort the entries before they compare them. The table shows how to select a factory:

Factory	Determinism	Use for
`RandomState` (default)	Random for each map	All keys that come from untrusted input
`BuildHasherDefault<H>`	Fully deterministic, no configuration	Tests, snapshots, internal hot paths
Manual `BuildHasher`	Deterministic, with parameters	Seeds that processes or nodes share

13_08_randomstate_buildhasher.rs prints:

same seed        : nodes agree on 0x30456221a034866a
different seed   : staging disagrees, 0x765d0095564d32db
routing table    : [("user:1001:profile", 3), ("user:2002:cart", 1)]

RandomState #1   : 0x6b8a8444bfce1207  (varies per run)
RandomState #2   : 0x46438868a596b51a  (varies per run)

All assertions passed.

The Hash/Eq Contract and Its Violation

The Hash documentation states the contract: if k1 == k2, then hash(k1) == hash(k2). To obey it safely, hash exactly the fields that eq compares. Hash no more fields and no fewer fields, and use the same normalized form.

The violation below can look correct in a code review. It is a case-insensitive asset key. The author made eq ignore case and did not change hash:

// BuggyKey is a newtype: struct BuggyKey(String). It also implements Eq.
impl PartialEq for BuggyKey {
    // Equality ignores ASCII case: "Logo.PNG" == "logo.png".
    fn eq(&self, other: &Self) -> bool { self.0.eq_ignore_ascii_case(&other.0) }
}
impl Hash for BuggyKey {
    // BUG: hashes the raw bytes, so "Logo.PNG" and "logo.png" get different hashes.
    fn hash<H: Hasher>(&self, state: &mut H) { self.0.hash(state) }
}

With a deterministic FNV hasher, the failure occurs on each run. With RandomState, the key has the same defect, but the failure is more difficult to show. There are two symptoms:

Lookups miss equal keys. "Logo.PNG" and "logo.png" compare equal, but their hashes go to different buckets. HashMap uses the hash first, and it never calls eq on keys in other buckets. Thus get returns None for a key that is == to a key in the map.
Sets hold duplicates. insert("BANNER.JPG") after insert("banner.jpg") returns true, because the set treats the value as new. The set then has two elements that compare equal.

13_09_hash_eq_contract.rs prints the lines below. CiKey is the key type with the corrected hash:

BuggyKey: probe == stored        → true
BuggyKey: cache.get(probe)       → None      ← equal key, lookup missed!
BuggyKey: cache.get(exact case)  → Some("89 50 4e 47 …")
BuggyKey: set.len()              → 2 (holding two equal elements)

CiKey   : cache.get("logo.png")  → hit (hash folds case, like eq)
CiKey   : set.len()              → 1 (duplicate rejected)

All assertions passed.

None of this is undefined behavior. The documentation calls it a "logic error". In this example, the program does not crash and has no memory unsafety. The only damage is to the invariants of the map, and nothing reports it. The solution is one line: hash the case-folded form, so that hash and eq use the same identity.

A related failure is more difficult to find. If you mutate a key after insertion (through interior mutability or unsafe), its hash changes while the key stays in the old bucket. Keep keys immutable for as long as they are in the map.

Summary

Concept	Key point
`Hash`	Selects which bytes are the identity of a value. The derive hashes fields in declaration order.
`Hasher`	State machine that converts bytes into a `u64`. The trait requires only `write` and `finish`. The typed `write_*` methods call `write` by default.
`finish`	Takes `&self`: reads the state and does not consume the hasher
Prefix collisions	`str` hashing appends a terminator, so `("ab","c")` ≠ `("a","bc")`. This detail is not specified: examine it, but do not depend on it.
`DefaultHasher`	SipHash-1-3 today. The algorithm and the keys are not stable across releases. Assert properties, never exact values.
Equal → equal	The one property that you can rely on. `Borrow`-based lookups depend on it (`String` hashes as its `str`).
FNV-1a	A real hasher in ~10 lines. It is your algorithm, so exact test-vector assertions are legitimate.
`BuildHasherDefault<H>`	Zero-sized factory adapter for any `Hasher: Default`. Makes `HashMap::default()` available.
`BuildHasher`	The factory trait: one factory for each map, and a new hasher with the same configuration for each operation. `hash_one` is a convenience method.
`RandomState`	The seeded factory pattern with OS randomness. It is the HashDoS defense. The iteration order varies between runs.
Deterministic hashers	For tests, snapshots, shared seeds, and hot paths with trusted keys. Never for untrusted input.
Hash/Eq contract	`k1 == k2 ⇒ hash(k1) == hash(k2)`. A violation is a logic error: missed lookups, duplicate set entries.
Key mutation	A change to the hashed state of a stored key leaves the key in its old bucket. Keep keys immutable.

Code Examples

File	Description
`13_06_hash_pipeline.rs`	A `Hasher` decorator that records the `write_*` calls of `Hash::hash`, the prefix-collision terminator, property assertions for `DefaultHasher`
`13_07_fnv_hasher.rs`	FNV-1a as a `Hasher`, exact test vectors, `BuildHasherDefault` integration, a comparison with `RandomState`
`13_08_randomstate_buildhasher.rs`	A manual seeded `BuildHasher` factory (the `RandomState` pattern), `hash_one`, how to select a factory
`13_09_hash_eq_contract.rs`	A deterministic violation of the Hash/Eq contract: missed `HashMap` lookups and `HashSet` "duplicates", and the solution

14.1 · Arithmetic and Bitwise Operators

Domain 14 — Operator Overloading Duration: ~15 minutes Library components: std::ops::Add through std::ops::ShrAssign, std::ops::Neg, std::ops::Not

Introduction

Each arithmetic operator and each bitwise operator in Rust is a call to a trait method. a + b is exactly Add::add(a, b). x <<= 4 is exactly ShlAssign::shl_assign(&mut x, 4). The compiler has no special rule for the built-in numbers: i32 implements the same std::ops traits that your types can implement. Because of that one design decision, you can give your domain types a real algebra:

distances that you divide by durations to get speeds
permission sets that you join with |
bitboards that move all the chess pieces of one side with one <<

This tutorial shows:

The std::ops trait family: Add, Sub, Mul, Div, Rem, the unary Neg, the six bitwise traits (BitAnd, BitOr, BitXor, Not, Shl, Shr), and each *Assign variant.
The structure of an operator trait: why Output is an associated type, why Rhs is a generic parameter, and what each decision gives you.
The primary pattern: operators with a different RHS and a different Output (Meters / Seconds = MetersPerSecond). This pattern changes unit errors into compile errors.
How to implement the assign family and write the arithmetic only one time.
Bitflags and bitboards: the two usual applications of the bitwise operators on newtypes.

The std::ops Trait Map

Each operator token maps to exactly one trait, and each binary trait has a related assign trait. An implementation of one trait never gives you the other trait. You implement each row of the figure separately.

Figure: The operator trait family in std::ops

14_1_arithmetic_bitwise_operators_1

All the binary operator traits have the same shape. Memorize it:

// `Rhs` is the type of the right operand. Its default is `Self`.
pub trait Add<Rhs = Self> {
    type Output;                              // the type of the result of `a + b`
    fn add(self, rhs: Rhs) -> Self::Output;   // takes the two operands by value
}

These four lines contain three design decisions:

Rhs is a generic parameter, and its default is Self. Thus one type can have an operator impl for several right-hand types. For example, Meters * f64 and f64 * Meters are two different impls.
Output is an associated type, not a parameter. For one (Self, Rhs) pair there is exactly one result type. Thus the compiler can infer the type of a + b without annotations.
The method takes self and rhs by value. The operation consumes the operands. For Copy newtypes this has no cost. For larger types, you can also implement the trait for references (impl Add for &Matrix). The standard library does exactly this for the primitives: &i32 + i32, i32 + &i32, and &i32 + &i32 all exist.

Type-Safe Unit Arithmetic: the Primary Pattern

The most valuable use of operator overloading is not to make the math look good. It is to make incorrect math impossible. Put each raw f64 in a newtype, and implement only the combinations that have a physical meaning:

Figure: Cross-type division — each meaningful combination is one impl

14_1_arithmetic_bitwise_operators_2

use std::ops::Div;

// Three newtypes. Each one contains one f64, but they are three different types.
struct Meters(f64);
struct Seconds(f64);
struct MetersPerSecond(f64);

// Div where Rhs ≠ Self AND Output ≠ Self: distance / time = speed.
impl Div<Seconds> for Meters {
    type Output = MetersPerSecond;
    fn div(self, rhs: Seconds) -> MetersPerSecond {
        MetersPerSecond(self.0 / rhs.0)      // divides the two inner f64 values
    }
}

let speed = Meters(100.0) / Seconds(12.5);   // speed: MetersPerSecond, value 8.0

At runtime, each of the three types is only an f64: the compiler removes the newtypes completely. But at compile time, only the operations that you implemented exist:

Meters + Meters compiles.
Meters / Seconds gives a MetersPerSecond.
The compiler rejects Meters + Seconds with error[E0308]: mismatched types (expected Meters, found Seconds).

The failure mode of the Mars Climate Orbiter (a silent unit error) becomes a compile error.

Multiplication by a scalar uses the Rhs parameter in the opposite direction, and it shows a detail of the orphan rule. Meters * 2.0 needs impl Mul<f64> for Meters. 2.0 * Meters needs impl Mul<Meters> for f64. The second impl is on a foreign type (f64), but the compiler permits it. The orphan rule examines the full impl header, and Meters is local to your crate.

Rem also has a use in this family. Meters(1000.0) % Meters(400.0) is the position in the current lap of a 400 m track. Thus the remainder is real distance algebra, and not only an operation for integers.

The Assign Family: Write the Math Once

The compiler does not derive AddAssign and the other assign traits from the related binary traits. If you implement Add but not AddAssign, then a += b does not compile. For Copy types, the idiomatic pattern is to implement each assign variant with the binary operator. Then the arithmetic is in exactly one place:

// Vec2 is a Copy struct with two f64 fields, x and y.
impl Add for Vec2 {
    type Output = Vec2;
    fn add(self, rhs: Vec2) -> Vec2 {
        Vec2::new(self.x + rhs.x, self.y + rhs.y)   // adds each component
    }
}

impl AddAssign for Vec2 {
    fn add_assign(&mut self, rhs: Vec2) {
        *self = *self + rhs;      // calls `+`, so `+=` always agrees with `+`
    }
}

Note the different shape of the signature: add_assign(&mut self, rhs) mutates in place and returns nothing. It has no Output. If the two impls disagree, a += b and a = a + b give different results. That bug is very confusing for the users of the type.

With the full family, numeric code looks like the textbook equation that it implements:

// velocity, position, gravity: Vec2. dt: f64, the seconds in one tick.
velocity += gravity * dt;      // AddAssign + Mul<f64>: v += g·dt
position += velocity * dt;     // p += v·dt

For non-Copy types (matrices, polynomials, bignums), the delegation usually goes in the opposite direction. You implement the in-place *Assign as the primitive, and the by-value operator calls it. The reason is that mutation in place prevents an allocation.

Bitflags on a Newtype

Flag sets are the usual application of BitOr, BitAnd, BitXor, and Not. A u8 in a newtype, some const flags, and four operator impls give you the core of the popular bitflags crate:

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct Permissions(u8);                              // one bit for each flag

impl Permissions {
    const EXEC:  Permissions = Permissions(0b001);
    const WRITE: Permissions = Permissions(0b010);
    const READ:  Permissions = Permissions(0b100);
    const ALL:   Permissions = Permissions(0b111);   // all the valid bits

    // Returns true if `self` has each flag in `required`.
    fn contains(self, required: Permissions) -> bool {
        self & required == required          // intersect, then compare
    }
}

// The BitOr and BitAnd impls are not shown. They apply `|` and `&` to the inner u8.
let owner = Permissions::READ | Permissions::WRITE;   // union: 0b110
assert!(owner.contains(Permissions::READ));

The operators map to set algebra:

| is union.
& is intersection (the membership test).
^ is toggle.
! is complement.

One detail makes a flags type correct: Not must mask its result to the valid bits.

impl Not for Permissions {
    type Output = Permissions;
    fn not(self) -> Permissions {
        // `!self.0` inverts all 8 bits. The mask keeps only the 3 valid bits.
        Permissions(!self.0 & Permissions::ALL.0)
    }
}

A plain !self.0 sets the five unused bits of the u8. The result is a Permissions value that no combination of flags can give. The inner u8 is private, so the mask in Not keeps each reachable value a legal flag combination. Invalid states stay unrepresentable. Then you write the usual operations with operators:

grant with mode |= flag
revoke with mode &= !flag
toggle with mode ^= flag

Shifts on a Bitboard

Shl and Shr are different from the other binary operators in one way. Their usual RHS is a shift amount, not a second value of Self. (For the primitives, the standard library generates a full matrix of impls: you can shift a u64 by u8, u32, usize, and more.)

The usual application of shifts on a newtype is the bitboard. One u64 holds an 8×8 game board, with one bit for each square. A shift of the integer moves all the pieces in one operation. Chess engines generate moves in this way:

struct Bitboard(u64);   // bit 0 = a1, bit 63 = h8

impl Shl<u32> for Bitboard {
    type Output = Bitboard;
    fn shl(self, n: u32) -> Bitboard {
        Bitboard(self.0 << n)      // a shift by 8 moves each piece one rank north
    }
}

// RANK_2 is an associated constant: Bitboard(0x0000_0000_0000_FF00).
let pawns = Bitboard::RANK_2;      // eight pawns, bits 8–15
let advanced = pawns << 8;         // ALL of them move one rank north, to bits 16–23

BitOr merges two boards, and BitAnd detects collisions. With these and the shifts, a small number of operator impls make board logic look like geometry. The example binary 14_04_bitboard_shifts.rs moves a rook north with rook <<= 8 until it collides with a blocker. That is ShlAssign in a movement loop.

One caution from the primitives also applies here. A shift of a u64 by 64 or more panics in debug builds. In release builds, the shift amount wraps. A production bitboard type would clamp or mask its shift amounts at the API boundary.

Design Guidelines

It is easy to misuse operator overloading. These guidelines keep its use correct.

Guideline	Why
Overload only when the operation is the math	`path1 + path2` for concatenation is a surprise. `duration1 + duration2` is not. The standard library has no `String + String`. But `String + &str` exists, and many programmers think that it is a design defect.
Keep `a + b` and `a += b` consistent	Make one operator call the other. Never write the arithmetic two times.
Make impossible combinations impossible	Implement only the `(Self, Rhs)` pairs that have a meaning. A missing impl is a feature.
Make `Output` the true result type	A division of units changes the unit. Put the new unit in the type.
Mask `Not` to valid bits on flag types	`!` must not make values that the type cannot represent.
For non-`Copy` types, add reference impls	With `impl Add<&T> for &T`, callers do not have to clone. The standard library does this for each primitive.
Do not overload to be clever	`<<` for "send to channel" (the C++ iostream style) is legal. Almost all programmers regret it.

The trait bounds are also important for generic code. A function that is generic over T: Add<Output = T> + Copy accepts i32, f64, Meters, and Vec2. Your newtypes go into the same generic numeric code as the primitives, because they implement the same traits.

Summary

Concept	Key point
`a + b` desugar	It is exactly `Add::add(a, b)`. Operators are ordinary trait calls.
`Rhs` generic parameter	One type can combine with many right-hand types. The default is `Self`.
`Output` associated type	Each `(Self, Rhs)` pair has exactly one result type, which makes inference possible.
Operands by value	This is cheap for `Copy` newtypes. Add `&T` impls for large types.
Unit newtypes	`Meters / Seconds = MetersPerSecond`. Unit errors become compile errors.
Orphan rule detail	`impl Mul<Meters> for f64` is legal, because the impl header contains a local type.
`*Assign` traits	You implement each one separately. Make it call the binary operator (or the opposite).
`Neg` / `Not`	These are the unary `-` and `!`. Mask `Not` to the valid bits on flag newtypes.
Bitflags	`\|` union, `&` intersection and test, `^` toggle, `&= !flag` revoke
`Shl` / `Shr`	The RHS is a shift amount. A bitboard moves all its pieces in one shift.
Debug-mode shifts	A shift ≥ the bit width panics in debug builds. Guard the shift amounts in production code.

Code Examples

File	Description
`14_01_unit_arithmetic.rs`	`Meters`/`Seconds`/`MetersPerSecond` newtypes: `Add`, `Sub`, `Neg`, `Rem`, `AddAssign`, scalar `Mul`, and `Div` with a different RHS type and Output type
`14_02_vec2_ops_family.rs`	The full arithmetic and assign family on a 2D vector, and a physics integrator that uses `position += velocity * dt`
`14_03_bitflags_newtype.rs`	Bitflags-style `Permissions(u8)`: `BitOr`, `BitAnd`, `BitXor`, masked `Not`, and the assign variants
`14_04_bitboard_shifts.rs`	`Shl`/`Shr`/`ShlAssign`/`ShrAssign` on a chess-style `Bitboard(u64)`

14_01_unit_arithmetic.rs prints:

--- same-type arithmetic: Add, Sub, Neg, Rem, AddAssign ---
total = Meters(750.0), fade = Meters(-50.0)
1000 m into 400 m laps → lap position Meters(200.0)
odometer after two splits: Meters(750.0)

--- scalar arithmetic: Mul<f64> for Meters, Mul<Meters> for f64 ---
stride Meters(1.25) × 4.0 = Meters(5.0), 2.0 × stride = Meters(2.5)

--- cross-type arithmetic: Div<Seconds> for Meters = MetersPerSecond ---
Meters(100.0) / Seconds(12.5) = MetersPerSecond(8.0)
Meters(300.0) at MetersPerSecond(8.0) → ETA Seconds(37.5)

All assertions passed.

14.2 · Index, Deref, and the Range Operator Family

Domain 14 — Operator Overloading Duration: ~15 minutes Library components: std::ops::Index, std::ops::IndexMut, std::ops::Deref, std::ops::DerefMut, std::ops::Range, std::ops::RangeInclusive, std::ops::RangeFrom, std::ops::RangeTo, std::ops::RangeToInclusive, std::ops::RangeFull, std::ops::RangeBounds, std::ops::Bound, std::range, std::range::legacy

Introduction

Tutorial 14.1 showed the operators that compute. This tutorial shows the operators that access. It has three families, and you use them on almost every line of Rust:

Index/IndexMut: the [] syntax, and why it panics and does not return an Option.
Deref/DerefMut: the mechanism of smart pointers, of method auto-deref, and of the coercion that silently changes &String into &str. (Tutorial 4.1 showed its best-known application: Vec<T> gets every slice method.)
The range operators .. and ..=: syntactic sugar for six ordinary structs. The RangeBounds trait lets one function accept each of them. The new std::range generation, stabilized in Rust 1.96 (RFC 3550), makes ranges Copy.

Index and IndexMut: `[]` on Your Own Types

grid[i] is syntactic sugar for *Index::index(&grid, i). Note the dereference: index returns a reference, and the [] expression is the place that the reference points to. IndexMut is the &mut equivalent. The compiler uses it when the expression is in a write position.

As with the Rhs of the arithmetic traits, the index is a generic parameter. Thus one type can accept several index types at the same time. A row-major grid can accept a (row, col) tuple and also a plain row number:

// A row-major grid: `cells` holds `rows * cols` elements in one flat Vec.
struct Grid<T> { rows: usize, cols: usize, cells: Vec<T> }

impl<T> Index<(usize, usize)> for Grid<T> {
    type Output = T;
    fn index(&self, (row, col): (usize, usize)) -> &T {
        // Index cannot return an error, so an invalid index must panic.
        assert!(row < self.rows && col < self.cols,
                "grid index ({row}, {col}) out of bounds");
        &self.cells[row * self.cols + col]       // the row-major offset
    }
}

impl<T> Index<usize> for Grid<T> {   // second impl, same type!
    type Output = [T];               // a whole row, as a slice
    fn index(&self, row: usize) -> &[T] {
        &self.cells[row * self.cols..(row + 1) * self.cols]
    }
}

// terrain: Grid<i32> with 3 rows and 4 columns, and each cell is 0.
// The IndexMut impl for the tuple is not shown.
terrain[(1, 2)] = 8;                 // IndexMut with a tuple
let row: &[i32] = &terrain[1];       // Index with a usize: [0, 0, 8, 0]

The contract has two important points:

Index cannot report a failure. Its signature returns &Output, so the only possible result for an index out of bounds is panic!. This is intentional: [] is for indices that must be valid, and a violation is a bug. For an index that can be out of range, add a get method that returns Option<&T>. Slices have the same pair: slice[i] and slice.get(i).
IndexMut: Index, and it uses Index::Output again. Thus the mutable view and the shared view always have the same type.

Deref and the Smart Pointer Pattern

*x on a non-pointer type is syntactic sugar for *Deref::deref(&x). An implementation of Deref<Target = T> declares: this type is a handle that "is really" a T. That sentence is also the design rule. Box, Rc, Arc, String (→ str), Vec<T> (→ [T]), and MutexGuard all obey it. A Deref impl that simulates struct inheritance does not obey it. That use is the anti-pattern that the Rust ecosystem agrees on most.

The practical wrapper below uses the secrecy crate as its model. It is a secret that you can use fully, but that never appears in logs:

struct Secret<T> { inner: T }               // Secret::new(value) is not shown

impl<T> Deref for Secret<T> {
    type Target = T;
    fn deref(&self) -> &T { &self.inner }   // gives a reference to the payload
}

impl<T> fmt::Debug for Secret<T> {          // `{:?}` uses this impl of the OUTER type
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("Secret([REDACTED])")   // never prints `inner`
    }
}

// token: Secret<String>
let token = Secret::new(String::from("eyJhbGci.payload.sig"));
assert_eq!(token.len(), 20);                 // str::len, two deref steps down
println!("{token:?}");                       // prints Secret([REDACTED])

DerefMut is the mutable equivalent. It is a subtrait (DerefMut: Deref) and has no Target of its own, so the shared view and the mutable view always agree. With DerefMut, token.push_str(".extra") also operates through the wrapper.

Note why the redaction is reliable. {:?} uses the Debug impl of the outer type, Secret<T>, and that impl exists. Thus {:?} never gets to the inner String. len is not on the wrapper, so the method lookup continues through the deref chain.

Deref Coercion Rules

Method calls and arguments of a reference type both cause deref coercion. The compiler inserts as many .deref() steps as necessary to get a type that fits.

Figure: The coercion chain — two hops, fully automatic

14_2_index_deref_range_family_1

// Takes &str, the most general borrowed form of a string.
fn count_segments(token: &str) -> usize { token.split('.').count() }

// `token` is the Secret<String> from the previous snippet.
count_segments(&token);                      // &Secret<String> → &String → &str, gives 3
count_segments(&String::from("a.b"));        // &String → &str, gives 2
count_segments("no dots here");              // already &str, gives 1

The exact rules:

Coercion applies at coercion sites: an argument position, a let with a type annotation, and a function return. It converts only &U to &Target, or &mut U to &mut Target. A mutable reference coerces to a shared reference, but never the opposite.
Coercion follows chains transitively. Secret<String> to str is two steps.
Owned values never coerce. A Secret<String> does not become a String. Only references convert.
Generic parameters are not coercion sites. fn f<S: AsRef<str>>(s: S) does no deref coercion. AsRef (tutorial 1.3) is for that case.
Method resolution tries the receiver type, and then each deref step in sequence. It applies autoref at each level. That is why token.len() finds str::len two steps down.

The Legacy Range Family and Its Sugar

Each .. expression makes one of six plain structs from std::ops:

Syntax	Type	Bounds	Iterator?
`..`	`RangeFull`	unbounded / unbounded	no
`a..`	`RangeFrom<T>`	included / unbounded	yes
`..b`	`RangeTo<T>`	unbounded / excluded	no
`a..b`	`Range<T>`	included / excluded	yes
`a..=b`	`RangeInclusive<T>`	included / included	yes
`..=b`	`RangeToInclusive<T>`	unbounded / included	no

The ranges are ordinary values. (1..5).map(|n| n * n) is valid because Range is itself an iterator. Slide 7 shows the effects of this design decision. All the fields are public (r.start, r.end), with one exception: RangeInclusive hides its endpoints behind the start() and end() getters. The reason is that it also has a private exhausted flag, which makes inclusive iteration possible.

Do not read start() or end() after you iterate a legacy RangeInclusive to its end. The values in that state are not specified. Rust 1.99 changed them when it made inclusive iteration faster. After a full iteration, 1..=3 now reports start() == 4 and end() == 3. The standard library specifies only that is_empty() returns true in that state, so use is_empty() for the test.

Ranges are also patterns, which have a grammar that is separate from the grammar of range values. Inclusive ..= patterns are an old stable feature. Exclusive .. patterns became stable in Rust 1.80. With them, the boundaries of adjacent buckets align, and you do not need off-by-one corrections:

// ms: u32, a latency in milliseconds. The match gives a &'static str.
match ms {
    0..100 => "fast",          // exclusive end: 0 to 99, and you do not write 0..=99
    100..500 => "degraded",    // 100 to 499
    500..=1000 => "slow",      // inclusive end: 500 to 1000
    _ => "timeout",
}

RangeBounds: One Function, Any Range

Vec::drain, String::replace_range, and BTreeMap::range accept each of the six range forms. They use the RangeBounds<T> trait, which has two methods: start_bound() and end_bound(). Each method returns a Bound<&T>: Included, Excluded, or Unbounded.

A short function with two match expressions converts arbitrary bounds to concrete half-open indices. Conceptually, each standard library API that takes a range does the same operation internally:

// Converts each range form to (start, end) indices: start included, end excluded.
fn normalize(range: impl RangeBounds<usize>, len: usize) -> (usize, usize) {
    let start = match range.start_bound() {
        Bound::Included(&s) => s,         // 3.. or 3..7
        Bound::Excluded(&s) => s + 1,     // no range syntax gives an excluded start
        Bound::Unbounded => 0,            // ..7 or ..
    };
    let end = match range.end_bound() {
        Bound::Included(&e) => e + 1,     // ..=e  →  half-open e + 1
        Bound::Excluded(&e) => e,         // ..7
        Bound::Unbounded => len,          // 3.. or ..
    };
    // Clamp the two indices to `len`. The end is never before the start.
    (start.min(len), end.min(len).max(start.min(len)))
}

With that helper, a lines(log, range) utility accepts .., 3.., ..2, 1..3, 2..=3, and ..=1. It also accepts a manually built (Bound::Excluded(2), Bound::Unbounded) tuple, because (Bound<T>, Bound<T>) implements RangeBounds too. Write your slicing APIs with an impl RangeBounds<usize> parameter. Then callers get the same flexibility that the standard library gives.

The New `std::range` Generation (Rust 1.96, RFC 3550)

In 2015, Rust made Range be an Iterator, and that decision had a cost. An iterator must mutate as it advances, so ranges could not be Copy. A loop counter that the program copies silently is a source of bugs. RangeInclusive also needed the hidden exhausted: bool. For eleven years, users asked why they could not put a Range in a Copy struct. Then the redesign of RFC 3550 arrived in std::range:

Figure: Legacy vs. new range generations

14_2_index_deref_range_family_2

The new types implement IntoIterator and not Iterator. The for loop does not change, but the range itself holds no iteration state. All the other properties are a result of that:

They are Copy. A selection rectangle can be plain data. A struct with a range::Range<usize> field and a range::RangeInclusive<usize> field can derive Clone and Copy. The legacy types make this impossible (error[E0204]).
You can iterate two times. The loop consumes a copy. Iterator adapters need an explicit .into_iter() or .iter() first (r.map(...) is E0599).
RangeInclusive stores start and last. The two fields are public, and there is no exhausted flag. For usize endpoints on a 64-bit target, the size is 16 bytes and not 24. The name of the field is last, not end, on purpose. Thus you cannot silently confuse an inclusive endpoint with an exclusive endpoint.
From/Into conversions exist in the two directions for all four redesigned types. The new types also operate directly in slice indexing and in RangeBounds APIs. For example, after v.drain(r), r is still usable, because it is Copy.

Migration: the a..b syntax still makes the legacy types today. A future edition is expected to change it to the new generation. Code that uses impl RangeBounds (slide 6) will not see the change.

Since 1.98, the iterator-style types are also available at std::range::legacy. They are the same types as std::ops::{Range, RangeFrom, RangeInclusive, RangeToInclusive}. With this name, a signature can say that it takes the generation that .. still makes. RangeTo and RangeFull were already Copy, and they are in std::range itself.

Summary

Concept	Key point
`Index`/`IndexMut`	They are the `[]` syntax. The index type is generic: a tuple, a range, or any other type.
`Index` contract	It returns `&Output`, so it must panic on failure. Add a `get` method that returns `Option`.
`Deref<Target>`	"This type is a handle to a `Target`". Use it for smart pointers only, not for inheritance.
`DerefMut: Deref`	It uses the same `Target`. The mutable view always agrees with the shared view.
Deref coercion	`&U` → `&Target`, transitively, at coercion sites. Owned values never coerce.
Method auto-deref	The receiver tries each deref step. Impls on the outer type have priority (redacted `Debug`).
Range syntax	Six structs are behind `..`/`..=`. `Range`, `RangeFrom`, and `RangeInclusive` iterate.
Range patterns	`match` arms: `..=` is an old stable feature, exclusive `..` since 1.80
`RangeBounds` + `Bound`	One `impl RangeBounds<usize>` parameter accepts every range form
`std::range` (1.96)	`Copy` ranges, `IntoIterator` not `Iterator`, `last` not `end`, no hidden flag
`std::range::legacy` (1.98)	Alias of `ops::Range` and the related types: the generation that `..` still makes
Conversions	`From`/`Into` between the generations. The new types index slices directly.

Code Examples

File	Description
`14_05_grid_index.rs`	`Index`/`IndexMut` on a row-major grid: the `(row, col)` tuple index, the whole-row index, and `get` compared with `[]`
`14_06_deref_smart_pointer.rs`	`Secret<T>` wrapper: `Deref`/`DerefMut`, method auto-deref, the coercion chain, and the redacted `Debug`
`14_07_rangebounds_normalize.rs`	The six legacy range types, `RangeBounds`/`Bound` normalization, and exclusive range patterns in `match`
`14_08_new_range_types.rs`	The new `std::range` generation: `Copy` structs, double iteration, `From` conversions, layout, and std integration
`14_13_range_legacy.rs`	`std::range::legacy` (1.98): the same types as `ops::Range*`, with a name for the generation that `..` still makes

14_08_new_range_types.rs prints:

--- Copy ranges in a struct + iterating twice ---
selection Selection { rows: 1..4, cols: 0..=2 } copied, original still usable
iterated rows twice: 9 cells then 3 rows
rows doubled via .iter().map(): [2, 4, 6]

--- From conversions and the layout win ---
round-tripped: 10..14, 97..=101, 100.., ..=9
size: range::RangeInclusive<usize> = 16, ops::RangeInclusive<usize> = 24

--- integration: indexing, drain, legacy boundaries ---
data[2..5] = [30, 40, 50], drained = [3, 4, 5]
legacy_api(r.into()) = 3

All assertions passed.

14.3 · Fn, FnMut, FnOnce: Closures as Trait Objects

Domain 14 — Operator Overloading Duration: ~15 minutes Library components: std::ops::Fn, std::ops::FnMut, std::ops::FnOnce

Introduction

You can also overload the call operator (). The Fn traits are that overload. When you write f(x) on a value that is not a function, the compiler calls Fn::call, FnMut::call_mut, or FnOnce::call_once. In the same way, a + b calls Add::add. Closures are the types that implement these traits automatically. Which traits a closure gets depends only on what its body does with the captured environment.

This tutorial shows:

What a closure is at runtime: an anonymous struct whose fields are its captures.
The three call traits and their hierarchy (FnOnce ⊇ FnMut ⊇ Fn). The use of the captures in the body selects the implemented set.
What move changes, and what it does not change.
The three ways to accept or store callables: impl Fn generics, Box<dyn FnMut> trait objects, and plain fn pointers.
The sizes of closures, and the exact moment of a heap allocation.

Closures Are Structs, and Calling Is an Operator

Each closure literal defines a new struct type that you cannot name. The captures are its fields. The body becomes the implementation of a call trait. Conceptually:

let factor = 3;
let scale = move |x: i64| x * factor;    // `move` copies `factor` into the closure

// The compiler generates approximately this struct and these impls:
struct __Scale { factor: i64 }                   // one field for each capture
impl FnOnce<(i64,)> for __Scale { /* body */ }   // and FnMut, and Fn

Figure: Closure Memory Layout

Closure memory layout

This has three immediate effects:

The size of a closure is the sum of its captures. It is zero for a closure that captures nothing. Each capture by reference adds one pointer, and each capture by value adds its full payload. The closure is where you put it: on the stack, in a struct field, or in a Box.
No two closures have the same type, even if their text is identical. Thus collections of closures need trait objects (slide 5).
The three traits are different only in how they take self. That one detail of the signature controls how many times you can call the closure:

// Simplified signatures. `Args` is a tuple of the argument types.
pub trait FnOnce<Args> { fn call_once(self, args: Args) -> Self::Output; }   // consumes
pub trait FnMut<Args>: FnOnce<Args> { fn call_mut(&mut self, ...); }         // mutates
pub trait Fn<Args>: FnMut<Args> { fn call(&self, ...); }                     // shares

The Hierarchy: FnOnce ⊇ FnMut ⊇ Fn

The supertrait chain Fn: FnMut: FnOnce means that you can also use each Fn closure as FnMut and as FnOnce. Read the names as capabilities that the caller demands, not as properties of the closure. An API that requires FnOnce demands the least: one call, which can consume the closure. An API that requires Fn demands the most: many calls through a shared reference.

Figure: Capture behavior decides the implemented traits

14_3_fn_fnmut_fnonce_closures_1

The example binary 14_09_closure_capture_modes.rs shows the three tiers in one pipeline:

// samples: Vec<i32>. total: usize, declared with `let mut`. report: String.
let count_high = || samples.iter().filter(|&&s| s > 10).count();  // reads → Fn
let mut accumulate = |amount| { total += amount; total };          // writes → FnMut
let deliver = move || report;                                      // moves out → FnOnce

An API rejects a closure of the wrong tier with a precise diagnostic. For example, bind a FnMut closure to a variable. Then pass the variable to a function that requires Fn. The compiler reports:

error[E0525]: expected a closure that implements the `Fn` trait, but this closure only implements `FnMut`

The compiler reports a second call of a FnOnce closure as a move error. call_once takes self, so the first call consumes the closure value (error[E0382]: use of moved value).

The example shows one more detail. A closure whose captures are all Copy (for example, one shared reference) is itself Copy. You can pass it by value many times, and you do not have to borrow it.

What `move` Does

move changes how the closure captures the environment: by value and not by reference. It does not change which traits the closure implements. The body still decides that:

let label = String::from("sensor-A");
let tag = move || format!("[{label}]");   // owns label, but only READS it
tag(); tag();                             // still Fn: each call gives "[sensor-A]"
// println!("{label}");                   // error[E0382]: borrow of moved value: `label`

Thus the two axes are independent:

	body reads	body mutates	body consumes
without `move`	`Fn`, borrows `&T`	`FnMut`, borrows `&mut T`	`FnOnce`, takes ownership anyway
with `move`	`Fn`, owns `T`	`FnMut`, owns `T`	`FnOnce`, owns `T`

You need move when the closure must outlive the scope that made it:

a factory function returns the closure
a struct with a long life stores the closure
you send the closure to a different thread

Without move, the closure holds references into a stack frame that ends soon. The borrow checker rejects that (error[E0373]: closure may outlive the current function).

The FnMut tier has a related borrow-checker effect. A closure that mutably borrows a local variable holds that &mut until the last use of the closure. Until then, you cannot read the original variable. This is the usual behavior of the borrow checker. But it is easy not to see, because the borrow is not visible in the closure.

Storing Callbacks: `Box<dyn FnMut>` in a Struct

Each closure has its own type. Thus a Vec of the closures that a user registers is impossible without type erasure. Box<dyn FnMut(...)> is the standard solution: one fat pointer type for any closure. The cost is a heap allocation and dynamic dispatch:

struct Downloader {
    on_chunk: Vec<Box<dyn FnMut(u64)>>,                    // observers, called many times
    on_complete: Option<Box<dyn FnOnce(u64) -> String>>,   // hook, called one time
}

// A method of `impl Downloader`. It boxes the closure and stores it.
fn subscribe(&mut self, callback: impl FnMut(u64) + 'static) {
    self.on_chunk.push(Box::new(callback));   // the Box erases the closure type
}

The example binary 14_10_boxed_callbacks.rs shows three patterns:

A call of a stored FnMut needs &mut access at each level: for callback in &mut self.on_chunk { callback(chunk_bytes); }.
A call of a stored FnOnce needs a move out of the struct first. self.on_complete.take() puts None in the field, and then hook(total) consumes the box. Option with take is the standard pattern. With it, the struct can detect a second completion, and the second completion does not panic.
The 'static bound on subscribe exists because Box<dyn Trait> means Box<dyn Trait + 'static> by default. Stored callbacks cannot borrow from stack frames with a short life. A callback that needs outside state must own it (move a counter into the closure) or share it through Rc<Cell<_>>. The example uses the two methods, and main reads the shared accumulator after dispatch.

Producing Closures: Factories and `impl Fn`

To return a closure, you must name a type that has no name. impl Fn does this and keeps static dispatch:

fn tax(rate: f64) -> impl Fn(f64) -> f64 {
    move |price| price * (1.0 + rate)     // move is mandatory: `rate` ends at the return
}

// flat_discount(amount) is a second factory. It returns |price| (price - amount).max(0.0).
fn compose(first: impl Fn(f64) -> f64, second: impl Fn(f64) -> f64) -> impl Fn(f64) -> f64 {
    move |x| second(first(x))             // the returned closure owns the two parts
}

let checkout = compose(flat_discount(10.0), tax(0.25));   // discount first, then tax
checkout(100.0);                          // 112.5. Built one time, called many times.

Sometimes the program selects the closure at runtime, and different match arms give different closure types. Then put them behind one Box<dyn Fn> type, which is the dynamic equivalent of impl Fn. In the example, rounding_mode(mode) returns f64::ceil, f64::floor, or an identity closure in this way.

Guideline:

Accept impl Fn… generics in function parameters (zero cost, monomorphized).
Return impl Fn from factories.
Use Box<dyn …> only when you store heterogeneous closures, or when an API boundary must hide the type.

`fn` Pointers and Coercion

A closure that captures nothing coerces to a plain function pointer. Its struct has no fields and is zero-sized, so a code address alone describes it completely:

let double: fn(i64) -> i64 = |x| x * 2;      // coercion: closure to fn pointer
let negate: fn(i64) -> i64 = |x| -x;
fn increment(x: i64) -> i64 { x + 1 }         // ordinary function

// One array type holds the two closures and the function.
let pipeline: [fn(i64) -> i64; 3] = [double, increment, negate];
// Applied in sequence to 7, the steps give 14, then 15, then -15.

Closures and named functions can be together in such a table. There is no boxing and no dyn, and each entry is one machine word. A capturing closure cannot coerce. The compiler reports error[E0308] with the note "closures can only be coerced to fn types if they do not capture any variables". The state of the closure must be somewhere, and a code pointer alone has no space for it.

The relation also applies in the opposite direction. fn pointers implement all three call traits, so you can use them in each API that accepts a closure. iter.map(double) is equivalent to iter.map(|x| x * 2).

The sizes complete the model. These are the values that 14_12_closure_size.rs prints on a 64-bit target:

A closure that captures nothing is 0 bytes. That is smaller than the 8-byte fn pointer to the same code.
Each capture by reference adds one word.
A payload that move captures adds its full size.
A Box<dyn Fn> handle is a 16-byte fat pointer: a data pointer and a vtable pointer.

Closures never allocate by themselves. Boxing is the first operation that uses the heap.

Summary

Concept	Key point
Closures are structs	Captures become fields. The body implements the call traits.
`FnOnce`	`call_once(self)` consumes the closure. Each closure implements it.
`FnMut: FnOnce`	`call_mut(&mut self)`: repeatable with exclusive access
`Fn: FnMut`	`call(&self)`: repeatable through a shared reference
Trait selection	The body moves a capture → `FnOnce` only. It mutates → `FnMut`. It reads → `Fn`.
`move` keyword	It changes the capture to by-value. It never changes the implemented traits.
When you need `move`	The closure outlives the scope that made it: factories, stored callbacks, threads
`impl Fn` parameter/return	Static dispatch. It is the correct default to accept and to return closures.
`Box<dyn FnMut>`	Type erasure for stored or heterogeneous callbacks. Call it through `&mut`.
Stored `FnOnce`	`Option<Box<dyn FnOnce…>>` with `take()`: move out, then call
`fn` pointer coercion	Only closures that capture nothing coerce. `fn` pointers implement all three traits.
Size and allocation	Size = sum of captures (0 if none). Only `Box::new` allocates.

Code Examples

File	Description
`14_09_closure_capture_modes.rs`	A body that borrows, mutates, or consumes its captures gives `Fn`, `FnMut`, or `FnOnce`. The example also shows the hierarchy in use and what `move` changes.
`14_10_boxed_callbacks.rs`	A struct with `Vec<Box<dyn FnMut(u64)>>` observers and an `Option<Box<dyn FnOnce…>>` hook that runs one time
`14_11_closure_factories.rs`	Factories that return `impl Fn`, composition, `Box<dyn Fn>` for a selection at runtime, and `fn`-pointer coercion
`14_12_closure_size.rs`	`size_of_val` on closures: zero-sized closures, one word for each reference, by-value payloads, and fat-pointer boxes

14_09_closure_capture_modes.rs prints:

--- shared borrow  => Fn (usable as all three) ---
Fn closure ran 5 times across both harnesses; samples intact: [12, 45, 7, 30]

--- mutable borrow => FnMut ---
FnMut closure accumulated total = 16

--- capture by value + move out => FnOnce ---
FnOnce closure delivered: "Q3 numbers: all green"

--- `move` changes capture, not the trait ---
move-closure is still Fn: [sensor-A] [sensor-A]

All assertions passed.

15.1 · Future, Poll, and the Async Machinery

Domain 15 — Asynchronous Programming Primitives Duration: ~15 minutes Library components: std::future::Future, std::future::poll_fn, std::future::pending, std::future::ready, std::task::Poll, std::task::Context, std::task::Waker, std::task::RawWaker, std::task::RawWakerVTable, std::task::Wake

Introduction

The machinery behind async/await is small: one trait (Future), one enum (Poll), and one callback handle (Waker). The standard library intentionally supplies only these interfaces and no runtime. The interfaces are the traits and types that let futures, executors, and I/O sources from different crates operate together. An executor has real design trade-offs (work stealing, I/O reactors, priorities). Thus std defines the contract, and crates such as tokio and smol supply the scheduling policy. The 30-line executor that this tutorial shows is one more alternative.

This tutorial shows:

The Future trait and its one method, poll, which returns Poll::Ready or Poll::Pending.
Context and Waker: how a future requests a new poll, and the rule that it arranges a wake before it returns Pending.
Three ways to get a Waker: Waker::noop(), the safe Wake trait, and the raw RawWaker/RawWakerVTable layer.
How to write a minimal, correct block_on executor with std only: Wake, Arc, and thread::park/unpark.
The futures that std supplies: future::ready, future::pending, and future::poll_fn for ad-hoc futures.
What async fn desugars to: a lazy state machine. Its size is the set of locals that are live across an .await.

The Future Trait and Poll

A future is an inert value that describes work that may not be complete yet. All the async support in std depends on one trait:

pub trait Future {
    // The type of the value that the future gives when it completes.
    type Output;
    // The only method. The caller supplies a pinned reference and a Context.
    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output>;
}

Poll<T> is an enum with two variants:

Poll::Ready(T): the work is complete, and this is the value.
Poll::Pending: the work is not complete. The future arranges a notification for the time when a new poll is useful.

There are no callbacks and no heap-allocated continuations. The owner of the future calls poll, and the future completes or returns Pending. Tutorials 15.2 and 15.3 explain the Pin<&mut Self> receiver. For the Unpin types in this tutorial, it behaves as a plain &mut self.

A manual implementation of the trait makes the model concrete. With remaining: 3, this future returns Pending three times before it completes:

struct Countdown {
    remaining: u32,   // the number of `Pending` results that are still to come
    polls: u32,       // the number of `poll` calls (the output of the future)
}

impl Future for Countdown {
    type Output = u32;

    // Countdown has only integer fields, so it is `Unpin`.
    // Thus `self.polls += 1` operates through the `Pin` as through `&mut self`.
    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<u32> {
        self.polls += 1;
        if self.remaining == 0 {
            Poll::Ready(self.polls)   // complete: do not poll this future again
        } else {
            self.remaining -= 1;
            cx.waker().wake_by_ref(); // arrange the subsequent poll (next slide)
            Poll::Pending
        }
    }
}

Remember two rules of the contract:

Do not poll a future again after it returned Ready. Many futures panic if you do.
A future that returns Pending must arrange a wake first. If it does not, the executor may never poll it again. This is the async equivalent of a deadlock.

Context and Waker

The second argument of poll is a Context. Currently, a Context only carries a &Waker. The Waker is a handle to the executor that requests a new poll of the task. It is cheap to clone, and it is thread-safe. The three parties use this protocol:

Figure: Executor, future, and waker interaction

15_1_future_poll_machinery_1

These rules make the protocol correct:

Arrange a wake for each Pending. A future must make sure that a wake call comes after each Pending that it returns. Countdown wakes immediately (it calls wake_by_ref before it returns), because a new poll can make progress at once. This also guarantees that the park call of the executor in this tutorial never blocks forever.
Register the waker again on each poll. An executor may pass a different waker on each call. Thus a future stores a new cx.waker().clone() each time it returns Pending.
The contract permits spurious polls. A future must accept a poll that no wake came before. A poll call is an idempotent examination of the state, not an event.

wake() consumes the waker, and wake_by_ref() borrows it. Use wake_by_ref() when you keep the waker.

Getting a Waker: noop, Wake, and RawWaker

To call poll yourself, you need a Context, and a Context needs a Waker. There are three sources, in order from the simplest to the most general:

Waker::noop() (stable since 1.85) is a waker that does nothing. It is sufficient for manual polling in a busy loop, where you poll again unconditionally:

// A Context needs a Waker. The no-op waker is sufficient for a busy loop.
let mut cx = Context::from_waker(Waker::noop());
// `pin!` pins the future on the stack. fut: Pin<&mut Countdown>
let mut fut = pin!(Countdown { remaining: 3, polls: 0 });
loop {
    // `as_mut()` reborrows the pinned future, so each iteration can poll it.
    match fut.as_mut().poll(&mut cx) {
        Poll::Pending => println!("  poll -> Pending"),                    // 3 times
        Poll::Ready(polls) => break println!("  poll -> Ready({polls})"),  // polls is 4
    }
}

The Wake trait (stable since 1.51) is the safe source. Implement wake(self: Arc<Self>) on your own type. Then convert the type with Waker::from(Arc<...>). It is the correct selection for executors (see the next slide).
RawWaker + RawWakerVTable is the raw layer below the two other sources. It is a data pointer and a manually written vtable of four function pointers: (clone, wake, wake_by_ref, drop). The construction of a Waker from these parts is unsafe, because the compiler cannot make sure that your vtable obeys the contract (thread safety, no use-after-free). This layer is for executors with special allocation strategies. All other code should use Wake or Waker::noop():

// The vtable has four function pointers. Each one receives the data pointer.
const NOOP_VTABLE: RawWakerVTable = RawWakerVTable::new(
    |_| RawWaker::new(ptr::null(), &NOOP_VTABLE), // clone: a new no-op RawWaker
    |_| {},                                       // wake: no task to notify
    |_| {},                                       // wake_by_ref: no task to notify
    |_| {},                                       // drop: no resource to release
);
// `Waker::new` takes the data pointer and the vtable. There is no data here.
// SAFETY: each function in the vtable ignores `data` and touches no state.
let handmade = unsafe { Waker::new(ptr::null(), &NOOP_VTABLE) };

15_01_future_poll_by_hand.rs prints:

Polling Countdown { remaining: 3 } by hand:
  poll -> Pending
  poll -> Pending
  poll -> Pending
  poll -> Ready(4)

future::ready(..)   -> Ready on poll #1
future::pending::<()>() -> Pending forever (polled 3 times)
hand-rolled RawWaker no-op waker polled future::ready(99) -> Ready(99)

All assertions passed.

A Minimal block_on Executor

An executor is the poll loop and a strategy to sleep efficiently between polls. With std threads, the usual strategy is thread::park. To wake a task, the waker unparks the thread that polls the task.

// The waker of the executor. It holds a handle to the thread that polls.
struct ThreadWaker {
    thread: Thread,
}

impl Wake for ThreadWaker {
    fn wake(self: Arc<Self>) {
        self.thread.unpark(); // gives the park token to that thread
    }
    // The default `wake_by_ref` clones the Arc and calls `wake`.
    fn wake_by_ref(self: &Arc<Self>) {
        self.thread.unpark(); // this override does not clone the Arc
    }
}

fn block_on<F: Future>(fut: F) -> F::Output {
    let mut fut = pin!(fut); // pin the future before the first poll (tutorial 15.2)
    // One waker for the full run. It unparks the current thread.
    let waker = Waker::from(Arc::new(ThreadWaker { thread: thread::current() }));
    let mut cx = Context::from_waker(&waker);
    loop {
        match fut.as_mut().poll(&mut cx) {
            Poll::Ready(value) => return value,
            // Block until a wake. If the wake came first, `park` returns immediately.
            Poll::Pending => thread::park(),
        }
    }
}

This executor has no lost-wakeup race. An unpark call for a thread that is not parked yet stores a token. If wake runs between the poll call and the park call, park returns immediately. Tutorial 6.1 explains the semantics of park and unpark.

The example binary runs three futures on this executor, and each one is more realistic than the one before:

the manually written Countdown
an async block that awaits two futures
a OneShot future that a different thread completes and wakes while the executor sleeps

15_02_block_on_executor.rs prints:

Countdown finished after 4 polls (3 Pending + 1 Ready)
async block awaited two Countdowns: 5 total polls
OneShot delivered: "payload from worker"

All assertions passed.

ready, pending, and poll_fn

std::future supplies three small constructors that are sufficient for many practical cases:

future::ready(value) completes on the first poll. It is the usual fast path for a cache hit: the API must return a future, but the value is already known.
future::pending::<T>() never completes and never wakes a task. It is a typed placeholder for a branch that never resolves.
future::poll_fn(closure) is the most general of the three. It makes a future from any FnMut(&mut Context) -> Poll<T> closure. The captures of the closure are the fields of the future. Thus a future for one use needs no struct and no impl block. Runtimes supply the operation below as yield_now. With poll_fn, it is a short function:

// Returns a future that is Pending on the first poll and Ready on the second.
fn yield_now() -> impl Future<Output = ()> {
    let mut yielded = false; // the `move` closure owns this flag between polls
    future::poll_fn(move |cx| {
        if yielded {
            Poll::Ready(())           // second poll: complete
        } else {
            yielded = true;
            cx.waker().wake_by_ref(); // request a new poll on the next loop of the executor
            Poll::Pending             // first poll: other tasks can run now
        }
    })
}

You can use all three in async blocks without adapters. .await operates on any Future, manually written or compiler-generated.

What async fn Desugars To

async fn and async {} do not run code. They construct a state machine. The compiler rewrites the body into a structure that is similar to an enum, with one state for each .await. The fields of each state are exactly the locals that must stay alive across that suspension.

Figure: States of a one-await async fn

15_1_future_poll_machinery_2

The example binary contains this state machine, manually written, for an async fn with one .await. It makes sure that the two forms give the same result. Two effects of the desugaring are important in daily work:

Futures are lazy. The construction of a future runs no part of the body. (A JavaScript Promise is different: it runs eagerly.) The first poll runs the body until the first .await, and no code runs before that. .await itself desugars to a loop: poll the inner future, and on Pending, return Pending to the caller.
A future has the size of its live state. The machine must store a 1 KiB buffer that the body uses across an .await. The same buffer adds no bytes if its last use is before the .await. When futures nest, the sizes add together. For this reason, deep async call trees sometimes box the inner futures.

15_04_async_desugaring.rs prints:

async fn      -> 42
hand machine  -> 42

poll #1: Start -> Awaiting -> Pending
poll #2: Awaiting -> Done -> Ready(10)

lazy: body ran only once polled (value = 7)
buffer held across await:  1026 bytes    # (varies by compiler version)
buffer dropped before await: 8 bytes     # (varies by compiler version)

All assertions passed.

Summary

Concept	Key point
`Future`	One method: `poll(Pin<&mut Self>, &mut Context) -> Poll<Output>`
`Poll::Ready(T)`	The work is complete. Do not poll the future again.
`Poll::Pending`	The work is not complete. The future must arrange a wake first, or the task never continues.
`Context` / `Waker`	The callback handle of the executor. Clone it and store it on each `Pending`.
`wake` vs `wake_by_ref`	`wake` consumes the waker. `wake_by_ref` borrows it.
`Waker::noop()`	A waker that does nothing, for manual polling in a busy loop (1.85)
`Wake` trait	Safe construction of a waker: `Waker::from(Arc<impl Wake>)`
`RawWaker`/`RawWakerVTable`	The unsafe raw layer below each waker. You rarely need it directly.
`block_on`	A poll loop and `thread::park()`. `unpark` stores a token, so no wake is lost.
Why no runtime in `std`	`std` standardizes the interface. Crates supply the scheduling policy.
`future::ready` / `pending`	A future that is complete immediately / a future that never completes
`future::poll_fn`	Ad-hoc futures from closures. The captures are the fields.
`async fn` desugaring	A lazy state machine with one state for each `.await`. Its size is the locals that are live across an `.await`.

Code Examples

File	Description
`15_01_future_poll_by_hand.rs`	A manual `Future` implementation, manual polling with `Waker::noop()`, `future::ready`/`pending`, and a manually written `RawWakerVTable`
`15_02_block_on_executor.rs`	A minimal `block_on` from `Wake`, `Arc`, and park/unpark. It runs a manually written future, an `async` block, and a one-shot future that a second thread completes
`15_03_poll_fn_adhoc_futures.rs`	`future::poll_fn`: `yield_now`, ad-hoc futures with state, composition with `.await`, and `future::ready` as a fast path
`15_04_async_desugaring.rs`	The state machine of an `async fn`, manually written. Lazy futures. The size of a future compared with the data that is live across an `.await`

15.2 · Pin: Why Futures Need Pinning

Domain 15 — Asynchronous Programming Primitives Duration: ~15 minutes Library components: std::pin::Pin, std::pin::pin!, std::boxed::Box::pin, std::marker::Unpin, std::marker::PhantomPinned

Introduction

Tutorial 15.1 used the Pin<&mut Self> receiver of Future::poll without an explanation. This tutorial explains it. Pin exists because of one concrete problem: a suspended async future may contain pointers into itself. A Rust move is a shallow memcpy that never updates pointers. Pin is a promise at the type level: the value behind this pointer will never move again. This promise is exactly what makes a poll of such a future sound.

This tutorial shows:

The self-referential problem, in a safe demonstration: a move changes addresses, and interior pointers do not follow.
Why an async block that borrows a local across an .await is a self-referential struct, and why it is !Unpin.
The Pin contract: what it prevents, what it promises, and which code relies on it.
The two pinning tools: the pin! macro (stack, no cost, limited to the scope) and Box::pin (heap, movable handle).
Unpin, the auto trait that makes Pin a no-op for almost every type, and Pin::new for those types.
PhantomPinned, the marker that makes the full pin contract apply to a type again.

The Self-Referential Problem

In Rust, a move copies the bytes of a value to a new location, and the program does not use the old bytes again. No code examines the value for pointers to the old location. That makes moves cheap, and it makes self-reference dangerous:

Figure: A move dangles any pointer into the moved value

Moving a self-referential struct leaves its internal pointer aiming at the old address

The example shows the address change. It never dereferences a stale pointer, because that is undefined behavior:

// Request has one field: `line: String`.
let request = Request { line: String::from("GET /index.html") };
// The address of the `line` field while `request` is on the stack.
let stale: *const String = &raw const request.line;

let moved = Box::new(request);                 // the move: stack -> heap
// The address of the same field after the move.
let fresh: *const String = &raw const moved.line;

// The field is now at a different address. The code only compares the
// two addresses. It never reads through `stale`.
assert!(!ptr::eq(stale, fresh));

The heap buffer of the String did not move. But the String struct itself (pointer, length, capacity) is now at a different address. If Request kept stale in one of its own fields, the move would silently invalidate that field. The borrow checker of safe Rust cannot express a reference into the same value, because no lifetime means "as long as this value". Thus this structure occurs only with raw pointers or with compiler-generated futures.

Async Blocks Are Self-Referential

Tutorial 15.1 showed that an async block compiles into a state machine. The fields of the state machine are the locals that are live across each .await. This example shows the result when one of those locals borrows a different one:

// `yield_once()` returns a future that is Pending on the first poll, then Ready.
let fut = async {
    let city = String::from("Lisbon");
    let name = &city;        // a borrow of a local...
    yield_once().await;      // ...that stays live across a suspension point
    name.len()               // 6
};

While the state machine is suspended at the .await, it stores city and name. name points at city, which is a field of the same struct. Thus the future is a self-referential struct. A move between polls would make name a dangling pointer. For that reason the compiler makes the future !Unpin, and the type system rejects each operation that could move it:

// `fut` is the async block from the previous snippet. `cx` is a `Context`.
fn assert_unpin<T: Unpin>(_: &T) {}   // compiles only if T is Unpin
assert_unpin(&fut);
// error[E0277]: `{async block@...}` cannot be unpinned
//     = note: consider using the `pin!` macro

fut.poll(&mut cx);   // `poll` needs Pin<&mut Self>, and `fut` is not pinned
// error[E0599]: no method named `poll` found for `async` block `{async block@...}`
//     help: consider pinning the expression

One detail is important. Futures are lazy (15.1), and the body creates the self-references while it runs, not at construction. Thus you can still move a future freely before its first poll. This is why Box::pin(fut) is valid, and why you can pass a future by value. The promise must apply only from the moment when polling starts. The Pin<&mut Self> receiver of poll marks exactly that moment.

The Pin Contract

Pin wraps a pointer P (such as &mut T or Box<T>) and protects the pointee:

What it prevents: safe code cannot get &mut T from a Pin<P<T>> (unless T: Unpin). Without &mut T, there is no mem::swap, no mem::replace, and no move out of the value. Each safe way to move the pointee needs a mutable reference.
What it promises: the pointee stays at its address from the moment of pinning until its destructor runs. Code that receives Pin<&mut T>, such as Future::poll, may rely on this, also in unsafe blocks. That reliance is the purpose of Pin.
What it does not do: it does not lock the value at run time. Pin is a compile-time rule with no run-time cost. Pin<&mut T> has the size of one pointer, and the guarantee exists only in the type system.

Note the direction of the contract. Pin restricts the users of the value. As a result, the implementation (the poll method of the future) can trust its own interior pointers.

pin! and Box::pin

Two safe constructors are sufficient for pinning in practice. The selection between them obeys a fixed rule:

Figure: Choosing a pinning tool

15_2_pin_fundamentals_1

pin! (stable since 1.68) pins a value in the current stack frame and returns Pin<&mut F>. The macro moves the value into a hidden local variable that no code can name. Thus no code can access the unpinned value again, and that makes the macro safe. There is no cost. The limit is the lifetime: the pin cannot leave the current scope, so a function cannot return it.

// `cx` is a `Context` from `Waker::noop()`. `yield_once()` is Pending once, then Ready.
let mut fut = pin!(async { yield_once().await; 7 });     // fut: Pin<&mut {async block}>
// `as_mut()` reborrows the pin for each poll. A direct `fut.poll(..)` consumes `fut`.
assert_eq!(fut.as_mut().poll(&mut cx), Poll::Pending);   // first poll: suspended at the .await
assert_eq!(fut.as_mut().poll(&mut cx), Poll::Ready(7));  // second poll: complete

Box::pin puts the future on the heap and returns Pin<Box<F>>. This handle is an ordinary value, and you can do these things with it:

move it into a Vec
return it from a function
erase its type to Pin<Box<dyn Future<Output = T>>> (the usual "boxed task" type of executors)

A move of the handle moves a pointer. The pinned future stays at its address. Box::into_pin converts an existing Box<T> at no cost. This is safe because a box is the only owner of its allocation.

// The return type erases the concrete type of the future. `pin!` is not
// possible here: the future must stay alive after `make_task` returns.
fn make_task(id: u32) -> Pin<Box<dyn Future<Output = u32>>> {
    Box::pin(async move { yield_once().await; id * 10 })   // make_task(3) gives 30
}

15_06_pin_stack_heap.rs prints:

pin!  : polled incrementally on the stack -> Ready(7)
Box::pin: drove 3 boxed tasks from a queue, total = 60
Box::into_pin: existing Box pinned in place
Pin::new: Unpin future pinned from a plain &mut

All assertions passed.

Unpin: The Exemption

If Pin restricted every type, async Rust would be very hard to use. Unpin is an auto trait. It means that a move of the type is safe also after pinning. This is true for any type without interior pointers: integers, String, Vec, references, boxes, and many futures (for example future::Ready). For an Unpin type, the pin contract has no effect, and Pin gives back all the usual access:

let mut name = String::from("alpha");
let mut pinned = Pin::new(&mut name);            // safe: String is Unpin
pinned.as_mut().get_mut().push_str("-beta");     // get_mut gives &mut String
// `mem::replace` moves the "pinned" String out. An Unpin type permits this.
let old = mem::replace(pinned.get_mut(), String::from("gamma"));
// old is "alpha-beta", name is "gamma"

Remember two results:

Pointers are always Unpin, whatever the pointee is. Box<T>, &mut T, and Pin<Box<T>> all move freely, because a move of a pointer never moves its pointee. Unpin describes only the behavior of a type as the pinned pointee.
Box::pin gives you an Unpin handle. An async block is !Unpin, but Pin<Box<F>> is Unpin and still implements Future. This is the standard method to pass a !Unpin future to an API with an F: Future + Unpin bound. Such bounds are common in combinator libraries, which need Pin::new(&mut f) internally.

Tutorial 17.2 explains the mechanics of marker traits from the side of the type system: why auto traits propagate through fields.

PhantomPinned: Opting Out

Unpin is an auto trait, so your own types are Unpin by default. This default is dangerous for a type that you intend to make self-referential. PhantomPinned is a zero-sized marker field that makes the type that contains it !Unpin. Then the pin contract applies again:

struct Cursor {
    pos: usize,
    _pin: PhantomPinned,      // zero bytes, but the struct is now !Unpin
}

let mut c = Cursor { pos: 0, _pin: PhantomPinned };
let p = Pin::new(&mut c);     // `Pin::new` needs a pointee that is Unpin
// error[E0277]: `PhantomPinned` cannot be unpinned
//     = note: consider using the `pin!` macro

let mut on_heap = Box::pin(Cursor { pos: 3, _pin: PhantomPinned });   // Pin<Box<Cursor>>
assert_eq!(on_heap.pos, 3);   // a read through Pin is always possible (Deref)
on_heap.as_mut().get_mut();   // `get_mut` also needs a pointee that is Unpin
// error[E0277]: `PhantomPinned` cannot be unpinned

You can read a pinned !Unpin value freely, because Pin always implements Deref. But no safe API gives &mut to the value or moves it. To mutate its fields through the pin, you need the unsafe projection methods of tutorial 15.3. In that tutorial, PhantomPinned is the marker that makes manually built self-referential types sound. Those types use NonNull (tutorial 16.3).

15_07_unpin_escape_hatch.rs prints:

i32, String, Vec, &str, future::Ready: all Unpin
Pin<&mut String> allowed mutation AND replacement (String: Unpin)
Cursor (PhantomPinned): pinned, readable, but no safe &mut
Box<Cursor>, &mut Cursor, Pin<Box<Cursor>>: Unpin (pointers move freely)
poll_once: rejected bare async block, accepted Box::pin'd one

All assertions passed.

Summary

Concept	Key point
The problem	A move is a shallow memcpy. Interior self-pointers dangle silently.
Async futures	A borrow of a local across an `.await` makes the state machine self-referential and `!Unpin`
Lazy futures	A future may move before its first poll. The pin promise starts at the first poll.
`Pin<P>` prevents	Safe `&mut T` access to a `!Unpin` pointee, and thus `mem::swap`, `mem::replace`, and a move out
`Pin<P>` promises	The address of the pointee is stable from pinning until the destructor runs
Cost	Zero. `Pin` is a compile-time rule, and one pointer at run time.
`pin!`	Stack pinning at no cost, but the pin cannot leave the scope (1.68)
`Box::pin` / `Box::into_pin`	Heap pinning. You can move, return, and type-erase the handle.
`Unpin`	Auto trait: a move is safe also after pinning. Almost every type implements it.
`Pin::new` / `get_mut`	The safe path without restrictions for `Unpin` types
Pointers	`Box<T>`, `&mut T`, and `Pin<Box<T>>` are always `Unpin`. With `Pin<Box<F>>`, you can use APIs that need `Unpin`.
`PhantomPinned`	A zero-sized field that makes a type `!Unpin`, so the pin contract applies again

Code Examples

File	Description
`15_05_self_referential_problem.rs`	A move changes addresses (a safe demonstration). An async block that borrows across an `.await` is `!Unpin`. Pinning lets you poll it
`15_06_pin_stack_heap.rs`	Incremental polling with `pin!`, a `Box::pin` task queue with `dyn Future`, `Box::into_pin`, and `Pin::new` for `Unpin` futures
`15_07_unpin_escape_hatch.rs`	`Unpin` in almost every type: safe `Pin::new`/`get_mut`/`mem::replace`, the `PhantomPinned` opt-out, pointers that are `Unpin`, and combinators with an `Unpin` bound

15.3 · Advanced Pinning: Unsafe Construction and Structural Projection

Domain 15 — Asynchronous Programming Primitives Duration: ~15 minutes Library components: std::pin::Pin, std::ptr::NonNull, std::marker::PhantomPinned

Introduction

Tutorial 15.2 used Pin as a consumer. On that side, pin!, Box::pin, and Unpin keep all the code safe. This tutorial shows the producer side:

the code that pin! and Box::pin contain
the code in every future combinator
the code behind the widely used pin-project crate

On this side you need unsafe, always in small blocks with a full justification. The reason is that you now keep the pin contract, where before you relied on it.

This tutorial shows:

Pin::new_unchecked and the pledge that you make: the value stays pinned until its destructor runs, also after the Pin itself is gone.
The accessor matrix: as_ref/as_mut/get_ref (safe), get_mut/into_inner (safe, Unpin only), get_unchecked_mut/into_inner_unchecked (unsafe), and Pin::set (safe for every type).
Structural and non-structural pinning: the decision, for each field, whether the pin of your wrapper also pins the field.
A manually written pin projection for a wrapper future: the pin-project pattern, with documented unsafe blocks.
A correct self-referential type with PhantomPinned and NonNull (16.3), which you construct behind Box::pin.

Pin::new_unchecked and the Pledge

Pin::new_unchecked(pointer) wraps a pointer in a Pin and does not check the pointer. It is unsafe because the full pin system depends on the pledge that you make here. The pledge is: the pointee will never move again, from this moment until its destructor runs. The pledge continues to apply after the Pin value itself is gone.

The pledge also has a part about the pointer type itself. Its Deref, DerefMut, and Drop implementations must not move the pointee, and each dereference must give the same object. Since Rust 1.99, the documentation states these requirements in the PinSafePointer trait, together with requirements for Clone and the formatting traits. The trait is unstable, but the pointer type that you pass to Pin::new_unchecked must obey its rules. &mut T and Box<T> obey them, and these are the only pointer types in this tutorial.

// `cx` is a `Context` from `Waker::noop()`.
let mut fut = async { 40 + 2 };   // a !Unpin future in a local variable
// SAFETY: the second `fut` binding shadows this one, so no code can name the
// future again. Nothing can move it before it drops at the end of the scope.
// The pointer is a `&mut T`, which obeys the pointer rules of `Pin`.
let mut fut = unsafe { Pin::new_unchecked(&mut fut) };   // fut: Pin<&mut {async block}>
assert_eq!(fut.as_mut().poll(&mut cx), Poll::Ready(42));

The shadowing is not a style decision. It is the proof that no code can move the value: the original binding has no name after that line. The pin! macro uses the same principle. It moves the value into a hidden local variable, and then it calls Pin::new_unchecked on a reference to that value. The misuse below shows why the pledge does not end with the Pin value:

// MISUSE: do not write this. `poll` represents any code that polls the pinned future.
let mut a = async { /* borrows a local across .await */ };
let pinned = unsafe { Pin::new_unchecked(&mut a) };
poll(pinned);      // the future now points into itself
// The Pin<&mut A> is gone here, but the pledge REMAINS:
let b = a;         // breaks the pledge: a move after the pin and the poll (possible UB later)

The drop of a Pin<&mut T> does not release the pledge. You must treat the value as pinned until its destructor runs. pin! enforces this with a hidden local variable. Box::pin enforces it because the box owns the allocation. With new_unchecked, you enforce it yourself.

The Accessor Matrix

The table shows each way to get access through a Pin or to unwrap it, and the condition for each one:

API	Returns	Safety	Why
`as_ref()`	`Pin<&T>`	safe	Shared reborrow
`as_mut()`	`Pin<&mut T>`	safe	Pinned reborrow, which you use for each poll
`get_ref()`	`&T`	safe	A `&T` cannot move a value
`get_mut()`	`&mut T`	safe, `T: Unpin`	A move of an `Unpin` value does no harm
`into_inner()`	`P`	safe, `T: Unpin`	The same reason. It unwraps the pointer.
`set(value)`	—	safe, any `T`	It drops the old value in place. The new value is pinned from that moment.
`get_unchecked_mut()`	`&mut T`	unsafe	You promise not to move `T` through it
`into_inner_unchecked()`	`P`	unsafe	You promise to continue to treat `T` as pinned

Two rows need more explanation. Pin::set is safe also for !Unpin types. The pin contract always permits the destruction of a value in place, and the new value was never pinned before. Thus the call breaks no promise. The sections on pin projection and on the self-referential type depend on get_unchecked_mut. It is sound when you use the &mut only to mutate the value in place, and not to move it:

// slot: Pin<Box<Tracked>>. Tracked is !Unpin and has an `id: u32` field.
// SAFETY: the code writes one field in place. It does not move, swap,
// or replace the Tracked value.
unsafe { slot.as_mut().get_unchecked_mut().id = 11; }

15_08_pin_unchecked_api.rs prints the output below. The drop: Tracked #11 line comes from Pin::set, which drops the old value in place.

new_unchecked: hand-pinned async block -> Ready(42)
as_ref/get_ref: shared access to a pinned value is unrestricted
get_unchecked_mut: in-place field write on a pinned value
Pin::set replaces the pinned value:
  drop: Tracked #11
into_inner_unchecked + Box::into_pin: unwrap/re-pin, value never moved
  drop: Tracked #2

All assertions passed.

Structural vs Non-Structural Pinning

Think of a wrapper future (a timeout, a tracing span, a metrics counter) that holds an inner future and some bookkeeping data. When the wrapper is pinned, each field can be pinned too, or not. You make this design decision for each field. The compiler does not infer it:

Figure: Deciding whether a field's pinning is structural

15_3_advanced_pinning_projection_1

A field is structurally pinned when "self is pinned" must imply "the field is pinned". This is true for an inner future that you poll through your own Pin, and for any field that a self-pointer points at. Then you may give Pin<&mut Field> to callers, but never &mut Field.
A field is non-structural when no code depends on its address (counters, flags, configuration). A pinned wrapper may freely give &mut Field to callers.

Pin projection is the division of one Pin<&mut Self> into a projection for each field. std gives you only the unsafe primitives to write it. You must keep the four obligations in the figure, which come from the structural-pinning requirements in the std::pin documentation. Those requirements also include the Drop guarantee: no code may use the storage of a pinned struct again before the destructors of its structural fields run. A struct that does not manage memory manually keeps this guarantee automatically.

Hand-Written Pin Projection

The example wraps any future and counts its polls. It has one structural field, one non-structural field, and one projection that needs unsafe:

struct Instrumented<F> {
    inner: F,     // STRUCTURAL: polled through the pin of the wrapper
    polls: u32,   // non-structural: plain bookkeeping data
}

impl<F: Future> Future for Instrumented<F> {
    // The output is the result of the inner future and the number of polls.
    type Output = (F::Output, u32);

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // SAFETY: the code uses `this` only to get access to the two fields.
        // It moves nothing out of `this`. It pins `inner` again immediately, and
        // the full module treats `inner` as structurally pinned (no Drop, no
        // &mut accessor, no repr(packed), Unpin only from the auto trait).
        let this = unsafe { self.get_unchecked_mut() };   // this: &mut Instrumented<F>
        this.polls += 1;                                   // non-structural: plain &mut access
        // SAFETY: `inner` is a structurally pinned field of a pinned value, so it
        // never moves again. The pointer is a `&mut F`, which obeys the pointer rules.
        let inner = unsafe { Pin::new_unchecked(&mut this.inner) };   // inner: Pin<&mut F>
        match inner.poll(cx) {
            Poll::Ready(value) => Poll::Ready((value, this.polls)),
            Poll::Pending => Poll::Pending,
        }
    }
}

Instrumented keeps the four obligations:

Unpin: the automatic Unpin implementation makes Instrumented<F> Unpin if and only if F is Unpin. A blanket impl<F> Unpin for Instrumented<F> would let safe code call Pin::new and mem::replace on a pinned async block. Then the unsafe code in poll would be unsound, although that code did not change.
Drop: there is no Drop implementation, so no destructor code can move the fields.
Accessors: no method returns &mut F or moves inner out of a pinned wrapper. You may add an accessor for the non-structural counter: fn polls_mut(self: Pin<&mut Self>) -> &mut u32.
Layout: the struct is not #[repr(packed)].

The pin-project crate generates and checks exactly this code for you. Use the crate in production code, but write the projection manually one time to understand it.

15_09_pin_projection.rs prints:

inner result = 42, polled 3 times (2 Pending + 1 Ready)
polls counter reset through Pin<&mut Instrumented<..>>
Unpin propagation verified: Instrumented<Ready<u32>> is Unpin

All assertions passed.

A Correct Self-Referential Type

The last example uses all the parts: a struct that really points into itself. Three components make it sound: PhantomPinned (15.2), NonNull (16.3), and Box::pin:

struct Record {
    line: String,             // the owned data
    current: NonNull<String>, // INVARIANT: points at self.line after initialization
    _pin: PhantomPinned,      // !Unpin: safe code can never move the Record out of the Pin
}

The field is a NonNull<String> and not a &'a String, because no lifetime can express "as long as this value". A raw pointer is not subject to the borrow checker, and Pin supplies the safety that the borrow checker cannot. Construction has two phases, and the order of the phases is the important point:

Figure: Pin<Box<Record>> — movable handle, immovable pointee

Heap layout of a pinned self-referential Record: the stack handle moves freely while NonNull points at the Record's own field

// An associated function of `Record`.
fn new(text: &str) -> Pin<Box<Self>> {
    // Phase 1: make the Record with a placeholder pointer and pin it on the heap.
    let mut boxed = Box::pin(Record {
        line: String::from(text),
        current: NonNull::dangling(),   // placeholder: never read
        _pin: PhantomPinned,
    });
    // The address is now FINAL: Pin + !Unpin prevent every later move.
    // Phase 2: take the address of the `line` field at its final location.
    let self_ptr = NonNull::from(&boxed.line);
    // SAFETY: the code writes one field in place. It does not move the Record,
    // no &mut Record leaves this block, Record has no Drop that moves fields,
    // and Record is !Unpin, so safe code can never break the pin afterwards.
    unsafe { Pin::as_mut(&mut boxed).get_unchecked_mut().current = self_ptr; }
    boxed
}

It is wrong to set the pointer before pinning: the value still moves (into the box) after the code takes the pointer. When the code sets the pointer after Box::pin, the address in the pointer can never change again. The next slide gives the safety argument for a read through the pointer. It also shows that safe code cannot break the invariant.

The Safety Argument

A read through the self-pointer is one unsafe expression. Its SAFETY comment answers each obligation:

// A method of `Record`. It reads `line` through the self-pointer.
fn via_pointer(&self) -> &str {
    // SAFETY: `new` set `current` to &self.line after pinning. The value did not
    // move since then (Pin + PhantomPinned), so the pointer is valid and aligned,
    // and it points at a live String. The function returns a shared reference
    // with the lifetime of &self, and no code mutates `line` in that lifetime.
    unsafe { self.current.as_ref() }   // &String, which coerces to &str
}

The invariant holds because each attempt of safe code to move a Record out of its pin is a compile error. The unsafe code stays behind an API that is fully safe:

// archive: Vec<Pin<Box<Record>>>
let by_value: Record = *archive.remove(0);           // tries to move the Record out of its box
// error[E0507]: cannot move out of dereference of `Pin<Box<Record>>`

let unpinned = Pin::into_inner(archive.remove(0));   // `into_inner` needs Record: Unpin
// error[E0277]: `PhantomPinned` cannot be unpinned

The handle stays easy to use. The example does these steps with Pin<Box<Record>> values:

It puts them into a Vec.
It rotates the Vec.
It passes the Vec into a function and receives it back.

The Record values on the heap never move, so ptr::eq(current, &line) stays true in all the steps.

If you remove PhantomPinned, the design is no longer sound. The type becomes Unpin, Pin::into_inner compiles, and safe code can move the struct while current still points at the old address. One zero-sized field is the difference between a sound API and a soundness hole.

15_10_self_referential_nonnull.rs prints:

self-pointer reads own field: "GET /health HTTP/1.1"
2 records moved via Vec + function boundary: pointers intact

All assertions passed.

Summary

Concept	Key point
`Pin::new_unchecked`	Your pledge: no move from now until the destructor runs. The pledge continues to apply after the `Pin` value is gone. The pointer type must obey the `PinSafePointer` rules.
`pin!` / `Box::pin` internally	They keep that pledge with a hidden local variable (`pin!`) or because the box owns the allocation (`Box::pin`)
`as_ref` / `as_mut` / `get_ref`	Safe reborrows. Use `as_mut()` for each poll.
`get_mut` / `into_inner`	Safe only for an `Unpin` pointee
`get_unchecked_mut`	Unsafe `&mut`: sound if and only if no code moves the value through it
`into_inner_unchecked`	Unsafe unwrap: continue to treat the pointee as pinned
`Pin::set`	Safe for any `T`: the old value drops in place, and the new value is pinned from that moment
Structural field	The address is important (the code polls the field, or a self-pointer points at it). Project it as `Pin<&mut Field>`. The four obligations apply.
Non-structural field	Plain data. Project it as `&mut Field`. There are no obligations.
Projection obligations	`Unpin` only from the auto trait. No `Drop` that moves fields. No accessor that gives `&mut` or moves the field out. No `repr(packed)`.
Self-referential type	`PhantomPinned` + `NonNull` + `Box::pin`. Set the pointer after pinning.
Containment	All the `unsafe` code is behind safe APIs. Each attempt of safe code to move the value out is a compile error.

Code Examples

File	Description
`15_08_pin_unchecked_api.rs`	Correct use of `Pin::new_unchecked` (and the misuse that its contract forbids), the accessor matrix, the drop in place of `Pin::set`, and `into_inner_unchecked`
`15_09_pin_projection.rs`	A manually written pin projection: a wrapper future with a structural field and a non-structural field, the four obligations, and `Unpin` propagation
`15_10_self_referential_nonnull.rs`	A self-referential `Record` with `PhantomPinned` and `NonNull`, construction in two phases with `Box::pin`, and moves of the handle that keep the pointers valid

16.1 · std::mem: Swaps, Sizes, and Transmutes

Domain 16 — Memory Management and Unsafe Primitives Duration: ~15 minutes Library components: std::mem::*

Introduction

std::mem is the standard library module that operates on values as memory. Its functions do these tasks:

They measure values.
They move values without a clone.
They control when destructors run.
The most dangerous functions reinterpret the bits of a value.

Most of the module is safe. A small part (transmute, zeroed) is among the most dangerous code that you can write in Rust.

This tutorial shows:

How to measure types and values with size_of, size_of_val, align_of, and align_of_val. This includes padding, niches, and fat pointers.
The raw-pointer versions size_of_val_raw and align_of_val_raw (stabilized in 1.99).
The three functions that move a value out from behind a &mut: mem::swap, mem::replace, and mem::take.
Destructor control: mem::drop (run the destructor now), mem::forget (never run it), and ManuallyDrop (run it yourself, in the order that you select).
mem::transmute: why the compiler checks only the size, which few uses are sound, and the safe APIs that replace it.
mem::zeroed and the deprecated mem::uninitialized. Tutorial 16.2 is about its replacement, MaybeUninit.
mem::discriminant, which compares enum variants without PartialEq.

Sizes, Alignment, and Niches

Each type has a size and an alignment. The size is the number of bytes that a value occupies, padding included. The alignment is a power of two, and the address of a value must be a multiple of it. Since Rust 1.80, size_of and align_of are in the prelude of every edition, so the mem:: prefix is optional.

Padding comes from the two properties together. The alignment of a struct is the alignment of its most-aligned field. The size of a struct must be a multiple of its alignment. Thus a struct with a u8 field, a u32 field, and a u16 field is 8 bytes, not 7:

struct Reading {
    sensor_id: u8,    // 1 byte
    micro_volts: u32, // 4 bytes, its offset must be a multiple of 4
    sequence: u16,    // 2 bytes
}
// The default representation lets the compiler reorder the fields. Here the result is 8 bytes.
// With #[repr(C)] (declaration order), the same fields occupy 12 bytes.
assert_eq!(size_of::<Reading>(), 8);  // 7 bytes of data + 1 byte of padding
assert_eq!(align_of::<Reading>(), 4); // the alignment of u32, the most-aligned field

A niche is a bit pattern that a type can never hold. For example, &T is never null, and NonZeroU32 is never zero. The compiler stores the None of an Option in the niche, at no cost in size:

use std::num::NonZeroU32;

assert_eq!(size_of::<Option<&u8>>(), size_of::<&u8>());          // niche: null
assert_eq!(size_of::<Option<NonZeroU32>>(), size_of::<u32>());   // niche: zero
assert!(size_of::<Option<u32>>() > size_of::<u32>());            // no niche: 8 bytes, not 4

size_of_val and align_of_val measure values. Thus they also operate on dynamically sized types, for which size_of::<T>() does not compile:

For a &[u32] of 4 elements, size_of_val returns 16 bytes.
For a &dyn Display, size_of_val reads the vtable at run time.

References are not always one word in size. &[T] and &dyn Trait are two words. The first word is a pointer to the data. The second word is the length of the slice or a pointer to the vtable.

size_of_val needs a reference, and a reference must point to a live, initialized value. Rust 1.99 stabilized mem::size_of_val_raw and mem::align_of_val_raw, which take a raw pointer. They read only the pointer metadata (the slice length or the vtable) and do not read the value. Thus they also operate on a value that is not initialized or is already dropped.

The two functions are unsafe. For a slice, a str, or a trait object, the size that the metadata gives must fit in isize. Layout::for_value_raw (tutorial 16.5) returns the two numbers together:

use std::alloc::Layout;
use std::fmt::Debug;
use std::ptr::{self, NonNull};

// `Packet` is a struct with a u64 field and a u16 field: 16 bytes, alignment 8. It derives Debug.

// A dangling slice pointer: no u32 values exist behind it. Only the length (5) is real.
let dangling: *const [u32] = ptr::slice_from_raw_parts(NonNull::<u32>::dangling().as_ptr(), 5);
// SAFETY: 5 is a valid slice length, and 5 * 4 bytes fits in isize.
assert_eq!(unsafe { std::mem::size_of_val_raw(dangling) }, 20);

// A trait object: the size comes from the vtable.
let raw: *mut dyn Debug = Box::into_raw(Box::new(Packet { id: 9, flags: 0 }));
// SAFETY: `raw` comes from Box::into_raw, so it is valid and aligned. This is the only drop.
unsafe { ptr::drop_in_place(raw) };                   // the value is dead: `&*raw` is not permitted
// SAFETY: the vtable part of `raw` comes from an unsizing coercion of a `Packet`.
// The vtable is static data, and the size that it gives (16 bytes) fits in isize.
let layout = unsafe { Layout::for_value_raw(raw) };   // no reference is necessary
assert_eq!(layout, Layout::new::<Packet>());
// SAFETY: Box::new allocated this block with the layout of `Packet`. No code uses `raw` again.
unsafe { std::alloc::dealloc(raw.cast::<u8>(), layout) };

16_19_size_of_val_raw.rs prints:

Packet: size = 16, align = 8
dangling *const [u32] of length 5: size = 20
dyn Debug after drop: size = 16, align = 8
memory freed with Layout::for_value_raw
size_of_val(&[u16; 3]) = 6

All assertions passed.

swap, take, and replace

Rust never lets a &mut T point at a location that holds no valid value. You cannot move a field out of a borrowed struct and fill the empty location later. swap, take, and replace solve this problem: each one always leaves a valid value in the location. All three are safe. All three move only the header of the value (for a String: pointer, length, and capacity). The heap contents do not move, and no reallocation occurs.

Figure: mem::swap exchanges stack headers, not heap buffers

mem::swap and mem::take memory behavior

The usual take pattern moves a buffer out of &mut self:

// `self.partial` is a String: the current incomplete line.
// `self.lines` is a Vec<String>: the complete lines.
fn feed(&mut self, chunk: &str) {
    for ch in chunk.chars() {
        if ch == '\n' {
            // Move `partial` out. A new empty String (no allocation) replaces it.
            let full = mem::take(&mut self.partial);
            self.lines.push(full);
        } else {
            self.partial.push(ch);
        }
    }
}

replace does the work of take for types that have no useful Default. You supply the new value, and replace returns the old value. Then you can match on the old value by value and move fields out of it. This is the standard method to write state-machine transitions behind &mut self:

// `self` is a `&mut Connection`. `Connection` is an enum: Idle, Connected, or Draining.
// `replace` puts Idle into `*self` and returns the previous state by value.
match mem::replace(self, Connection::Idle) {
    Connection::Connected { session_id } => {
        // Use `session_id` from the old state to make the new state.
        *self = Connection::Draining { session_id, pending: 3 };
        "connected -> draining"
    }
    // ... the arms for Idle and Draining
}

swap exchanges two values through &mut in O(1). A common use is double buffering. A consumer processes front while a producer fills back, and then the two buffers exchange roles.

16_02_mem_swap_take_replace.rs prints:

before swap: front=[110, 220, 330] back=[]
after swap:  front=[] back=[110, 220, 330]

complete lines: ["GET /index.html", "Host: example.com"]
still partial:  "Accept: */*"

shutdown #1: connected -> draining
shutdown #2: already draining

new high score 1450 (previous 1200)

All assertions passed.

How to Select a mem:: Function

Figure: Decision tree for the mem:: value-movers

16_1_mem_swaps_sizes_transmutes_1

A note on drop: its full definition is fn drop<T>(_x: T) {}. The function takes ownership of the value, and that is the destruction mechanism. drop is in the prelude, so you rarely write mem::drop.

forget, Drop Order, and ManuallyDrop

Two rules control automatic destruction:

Struct fields drop in declaration order.
Local variables drop in reverse declaration order.

Sometimes the declaration order and the necessary destruction order are different. For example, a client handle must drop before the runtime that it belongs to. For these cases, you have two tools:

mem::forget(v) does not run the destructor at all. It is safe (a leak is not UB), but the resources stay unavailable until the process exits. There are two legitimate uses. The first is after you extract a raw pointer that foreign code now owns. The second is to prevent a second drop of a value that ptr::read moved out (see 16.3).
ManuallyDrop<T> suppresses the automatic drop of a field. Your Drop impl then runs the drop explicitly, in the order that the code gives and not in the order of the field list:

// `Tracked` is a type whose destructor writes its name to a log.
struct ServiceExplicit {
    runtime: ManuallyDrop<Tracked>,  // declared first on purpose: the wrong order for the drop
    client: ManuallyDrop<Tracked>,
}

impl Drop for ServiceExplicit {
    fn drop(&mut self) {
        // SAFETY: this code drops each field exactly once and does not use a field after its drop.
        unsafe {
            ManuallyDrop::drop(&mut self.client);  // client first
            ManuallyDrop::drop(&mut self.runtime); // runtime second, the field order has no effect
        }
    }
}

GPU crates and async crates use this pattern (device handles drop before instances). A refactor that reorders the fields can no longer change the destruction order silently. The order is now code, and code gets a review.

One documented special case (fixed in 1.96, guaranteed in the 1.98 docs): after ManuallyDrop::drop on a ManuallyDrop<Box<T>>, a move of the wrapper is not UB. Before 1.96, the compiler treated the inner Box as a Box that you must not move after deallocation. You still must not dereference the dropped box.

transmute: Capability and Risk

mem::transmute::<A, B>(a) reinterprets the bits of an A as a B. The compiler checks only one condition: the two types must have the same size (different sizes give error E0512). All other conditions are your responsibility. A transmute is sound only when every bit pattern of A is a valid B:

// `checksum` is a u32 with the value 0xDEAD_BEEF.
// SAFETY: u32 and [u8; 4] have the same size, and each bit pattern is valid for both types.
// Thus the conversion cannot break an invariant in one direction or the other.
let bytes: [u8; 4] = unsafe { mem::transmute::<u32, [u8; 4]>(checksum) };
// On a little-endian target, bytes is [0xEF, 0xBE, 0xAD, 0xDE].

The standard library already has this conversion as a safe function with explicit endianness:

assert_eq!(bytes, checksum.to_ne_bytes());       // the safe form ("ne" = native endian)
let round_trip = u32::from_ne_bytes(bytes);      // round_trip == checksum

The same is true for floats: f32::to_bits and f32::from_bits do the work of a transmute in reviewed library code. Current versions of rustc also have an unnecessary_transmutes lint that recommends these APIs. These well-known misuses are all UB:

transmute::<&T, &mut T> makes an aliased &mut, which is immediate UB.
transmute::<u8, bool>(2) makes an invalid value.
transmute::<u32, char>(0xD800) makes a surrogate, which is not a char.

Because of these misuses, the rule is: transmute is the last alternative, never the first choice.

mem::zeroed::<T>() returns a value in which all bits are zero. It is sound only when zero is valid for T:

Zero is valid for integers, floats, raw pointers, and arrays of those types.
Zero is not valid for references, Box, NonZero*, or fn().

mem::uninitialized is deprecated and effectively always UB. It tells the compiler that a T exists before the program writes its bytes. Tutorial 16.2 is about its replacement, MaybeUninit.

discriminant: Comparing Variants Without PartialEq

mem::discriminant(&e) returns an opaque Discriminant<E> that identifies which variant e is. It ignores all payload data. It does not need PartialEq on the enum. This is important, because payloads that contain closures or sockets never implement PartialEq:

// `Job` is an enum: Fetch { url: String }, Compress { level: u8 }, or Shutdown.
// It does not implement PartialEq.
fn same_kind(a: &Job, b: &Job) -> bool {
    mem::discriminant(a) == mem::discriminant(b)
}

// `a_url` and `b_url` are two different Strings.
assert!(same_kind(
    &Job::Fetch { url: a_url },   // same variant,
    &Job::Fetch { url: b_url },   // different payload: the discriminants are equal
));

Discriminant<E> implements Eq and Hash, so you can use it as a HashMap key. The example binary 16_03_mem_transmute_discriminant.rs uses it to group consecutive queue entries of the same kind into batches. Compare discriminant with matches!. matches! tests a value against one pattern that you name at compile time. discriminant compares two runtime values.

Summary

Function	What it does	Safety
`size_of` / `align_of`	Compile-time size and alignment of a type (padding included)	Safe
`size_of_val` / `align_of_val`	Run-time size and alignment of a value, also for slices, `str`, and `dyn Trait`	Safe
`size_of_val_raw` / `align_of_val_raw` (1.99)	The same from a raw pointer. They read only the metadata, so the value can be dead	`unsafe`
`mem::swap`	Exchanges two values through `&mut`: headers only, O(1)	Safe
`mem::take`	Moves the value out and leaves `Default::default()`	Safe
`mem::replace`	Moves the value out, leaves a supplied value, and returns the old value	Safe
`mem::drop`	Runs the destructor now (the prelude `drop`)	Safe
`mem::forget`	Never runs the destructor: a leak, but no UB	Safe
`ManuallyDrop<T>`	Suppresses the automatic drop. You run the drop explicitly with `ManuallyDrop::drop`. A move of `ManuallyDrop<Box<T>>` after the drop is not UB (1.98 docs)	`unsafe` to drop
`mem::transmute`	Reinterprets bits. The compiler checks only the size	`unsafe`, the last alternative. Prefer `to_ne_bytes`, `to_bits`, casts
`mem::zeroed`	All-zero value, sound only if zero is valid for `T`	`unsafe`
`mem::uninitialized`	Deprecated, effectively always UB. Use `MaybeUninit` (16.2)	Never
`mem::discriminant`	Opaque variant identifier. It compares variants without `PartialEq`	Safe

Code Examples

File	Description
`16_01_mem_sizes_alignment.rs`	`size_of`, `align_of`, `size_of_val` on DSTs, padding, niches, fat pointers
`16_02_mem_swap_take_replace.rs`	Double buffering with `swap`, a buffer moved out with `take`, a state machine with `replace`
`16_03_mem_transmute_discriminant.rs`	A sound transmute compared with `to_ne_bytes`/`to_bits`, `zeroed`, variant batches with `discriminant`
`16_04_manually_drop_forget.rs`	Observable drop order, `ManuallyDrop` for an explicit order, `drop` and `forget`, and a move of `ManuallyDrop<Box<T>>` after its drop (1.98 docs)
`16_19_size_of_val_raw.rs`	`size_of_val_raw`, `align_of_val_raw`, `Layout::for_value_raw` (1.99): size and alignment from a raw pointer, also after `drop_in_place`

16.2 · MaybeUninit: Safe Patterns for Uninitialized Memory

Domain 16 — Memory Management and Unsafe Primitives Duration: ~15 minutes Library components: std::mem::MaybeUninit

Introduction

The Rust type system has one rule without exceptions: each value of type T is always valid. A String that is not yet initialized cannot exist. From the moment a String exists, its pointer, length, and capacity must be consistent. But real programs need storage that exists before its contents exist:

buffers that wait for a read,
array slots that the program fills one at a time,
pool slots between uses.

MaybeUninit<T> is the approved type for such storage. It is a container with the same size and alignment as T, and it makes no validity claim about its bytes. You make the claim later, at one point that a reviewer can audit: the unsafe call to assume_init. With that call, you guarantee that the initialization is complete.

This tutorial shows:

Why MaybeUninit replaced the deprecated mem::uninitialized (16.1).
The lifecycle: uninit() → write() → assume_init(), and also assume_init_ref, assume_init_mut, and assume_init_drop.
The drop trap: MaybeUninit never runs the destructor of T.
MaybeUninit::zeroed as the version of mem::zeroed that you can control.
How to initialize an array one element at a time. The last step uses the MaybeUninit<[T; N]> ⇄ [MaybeUninit<T>; N] conversions (stabilized in Rust 1.95).
A use case: a vector with a fixed capacity that is only on the stack. This is the core of arrayvec and SmallVec.

Why MaybeUninit Exists

The old mem::uninitialized::<T>() returned a T directly. Thus, for one moment, a "value" of T existed with uninitialized bytes. For types with validity invariants (bool, references, enums), that moment is undefined behavior even if you never read the value. The function is deprecated, and you effectively cannot use it.

MaybeUninit<T> solves the problem in the model, not in the symptom. The uninitialized bytes are in a type whose contract is "contents unknown", so they break no invariant. MaybeUninit<T> is also zero-cost:

// `Session` is a struct with one field, `id: u32`, and a Drop impl.
assert_eq!(size_of::<MaybeUninit<Session>>(), size_of::<Session>());   // no extra bytes
assert_eq!(align_of::<MaybeUninit<Session>>(), align_of::<Session>());

It is fully safe to create a MaybeUninit. The danger starts only when you claim the contents:

let mut slot: MaybeUninit<Session> = MaybeUninit::uninit(); // safe: no Session exists yet
let session_ref = slot.write(Session { id: 41 });           // safe: returns &mut Session
// SAFETY: the `write` above fully initialized `slot`.
let session: Session = unsafe { slot.assume_init() };       // the one unsafe step

write stores a value and does not drop the previous content. This behavior is necessary, because a drop of uninitialized bytes would be UB.

The Lifecycle

Figure: MaybeUninit's three states — only the validity claim changes

MaybeUninit lifecycle

The middle state (initialized, but not yet claimed) has its own accessors. Each accessor has the same proof obligation as assume_init: the storage must really hold an initialized value.

Method	Gives you	Typical use
`assume_init()`	`T` by value	Extraction: the usual end of the lifecycle
`assume_init_ref()`	`&T`	Read the value while it stays in the storage
`assume_init_mut()`	`&mut T`	Change the value in place before extraction
`assume_init_read()`	`T` (a bitwise copy, the storage keeps the bytes)	Move a value out of one slot of a buffer
`assume_init_drop()`	Nothing (it runs `Drop`)	Destroy the value without extraction

MaybeUninit::zeroed() fills the storage with zero bytes but still makes no claim. mem::zeroed::<T>() from 16.1 asserts validity immediately. With MaybeUninit::zeroed(), the danger moves to the assume_init line, where a reviewer can audit it:

let zeros: MaybeUninit<[u32; 4]> = MaybeUninit::zeroed();   // 16 zero bytes, no claim yet
// SAFETY: the all-zero bit pattern is a valid [u32; 4].
let zeros = unsafe { zeros.assume_init() };                 // [0, 0, 0, 0]

The Drop Problem

MaybeUninit does not know whether it holds a value, so it never runs the destructor. If you initialize a slot and then do nothing more, the payload leaks silently:

let mut slot: MaybeUninit<Session> = MaybeUninit::uninit();
slot.write(Session { id: 7 });
drop(slot);            // the MaybeUninit is gone, but the Session destructor did NOT run

The example binary 16_05_maybeuninit_lifecycle.rs proves this with a drop counter: the count does not change. This is a leak, not UB. But a leak of a connection pool or a file handle is still a bug. When you must destroy a value and not extract it, use assume_init_drop:

// `slot` is a new `MaybeUninit::<Session>::uninit()`, as in the previous snippet.
slot.write(Session { id: 8 });
// SAFETY: the `write` above initialized `slot`, and no code uses `slot` after this line.
unsafe { slot.assume_init_drop() };   // the destructor runs exactly once

Remember this rule: with MaybeUninit, initialization tracking and destruction are fully your responsibility. Each sound wrapper type that uses MaybeUninit (slide 6) has two parts:

a len or a flag that records what is initialized,
a Drop impl that drops exactly that region.

Element-by-Element Array Initialization

The primary use case is to build a [T; N] when T has no Default, no Clone, and no cheap placeholder value. Then [T::default(); N] and [value; N] do not compile. The pattern has three steps:

Figure: The array initialization pipeline (stable since 1.95)

16_2_maybeuninit_uninitialized_memory_1

// `CalibrationCurve` is a struct with no Default and no Clone.
// `calibrate(i)` returns the CalibrationCurve for channel `i`.

// 1. An array of 8 INDEPENDENT uninitialized slots. This step is fully safe.
//    The inline const is necessary: MaybeUninit<T> is not Copy when T is not Copy.
let mut slots: [MaybeUninit<CalibrationCurve>; 8] =
    [const { MaybeUninit::uninit() }; 8];

// 2. Initialize each slot.
for (i, slot) in slots.iter_mut().enumerate() {
    slot.write(calibrate(i));
}

// 3. Convert the array of initialized MaybeUninit<T> into a MaybeUninit of an array.
//    The two types have the same layout, so the From impl (stable 1.95) has no cost.
//    Then claim the full array in one call.
let whole: MaybeUninit<[CalibrationCurve; 8]> = MaybeUninit::from(slots);
// SAFETY: the loop above wrote each of the 8 elements.
let table: [CalibrationCurve; 8] = unsafe { whole.assume_init() };

The conversion also operates in the opposite direction. <[MaybeUninit<u16>; 4]>::from(block) changes one large uninitialized block into one slot for each element. You can then fill the slots one at a time. APIs that fill storage from the caller use this form. After a partial fill, you can claim only the initialized prefix, one element at a time.

On nightly, the two conversions are also available as .transpose(). On stable, From is the supported method.

One hazard: if step 2 panics before the loop is complete, the elements that the loop already wrote leak (slide 4: MaybeUninit never drops). That is memory-safe but wrong. Production implementations protect the loop with a drop guard, which destroys the initialized prefix on unwind.

Use Case: A Fixed-Capacity Stack Buffer

FixedVec<T, N> uses all the techniques from the previous slides. It stores a maximum of N elements inline, with no heap and no trait bounds on T. This is the core of the arrayvec crate and of the inline half of SmallVec:

struct FixedVec<T, const N: usize> {
    /// The slots at index >= len are uninitialized.
    data: [MaybeUninit<T>; N],
    /// The invariant of the type: data[..len] is always fully initialized.
    len: usize,
}

Each unsafe block in the type relies on that one invariant. Each mutation has an order that keeps the invariant true at each step: write the slot first, increment len after. This order gives panic safety by construction:

fn push(&mut self, value: T) -> Result<(), T> {
    if self.len == N { return Err(value); }   // full: return the value, no panic, no drop
    self.data[self.len].write(value);         // first initialize the slot,
    self.len += 1;                            // then count it
    Ok(())
}

fn as_slice(&self) -> &[T] {
    // SAFETY: MaybeUninit<T> has the same layout as T, and data[..len] is
    // initialized (the invariant). Thus this region is a valid &[T].
    unsafe { std::slice::from_raw_parts(self.data.as_ptr().cast::<T>(), self.len) }
}

The Drop impl drops exactly the initialized prefix. The example binary uses a drop counter to assert this: the drop of a buffer with 3 live elements runs exactly 3 destructors:

impl<T, const N: usize> Drop for FixedVec<T, N> {
    fn drop(&mut self) {
        // Only the slots below `len` hold values. MaybeUninit never drops them automatically.
        for slot in &mut self.data[..self.len] {
            // SAFETY: the slot is in the initialized prefix, and this loop drops it exactly once.
            unsafe { slot.assume_init_drop() };
        }
    }
}

16_07_fixed_capacity_stack_buffer.rs prints:

len after 4 pushes: 4
rejected at capacity: Packet("overflow")
contents: ["syn", "syn-ack", "ack", "data"]
popped: Packet("data")
buffer drop destroyed 3 live elements

All assertions passed.

Summary

Concept	Key point
`MaybeUninit<T>`	Storage with the size and alignment of `T` and no validity claim. It is zero-cost
`uninit()` / `new(v)` / `zeroed()`	All are safe to create. The danger starts only at `assume_init`
`write(v)`	Initializes without a drop of the previous content, and returns `&mut T`
`assume_init()`	The one `unsafe` step: you guarantee that the initialization is complete
`assume_init_ref` / `_mut` / `_read` / `_drop`	Borrow, copy out, or destroy while the value stays in the storage
Never drops	Payloads that are initialized but not claimed leak. Track them and drop them yourself
Array init (1.95)	`[const { MaybeUninit::uninit() }; N]` → write each slot → `MaybeUninit::from` → `assume_init`
Panic safety	Order the mutations so that the "initialized prefix" invariant is true at each step
vs `mem::uninitialized`	Deprecated and effectively always UB. `MaybeUninit` is its replacement

Code Examples

File	Description
`16_05_maybeuninit_lifecycle.rs`	`uninit`/`write`/`assume_init`, the drop trap (`MaybeUninit` never drops), `assume_init_ref`/`_mut`/`_drop`, `zeroed`
`16_06_maybeuninit_array_init.rs`	Array initialization one element at a time with the 1.95 `From` conversions, in both directions
`16_07_fixed_capacity_stack_buffer.rs`	`FixedVec<T, N>`: `push`, `pop`, `as_slice`, and `Drop`, which all rely on the initialized-prefix invariant

16.3 · Raw Pointers in Practice: NonNull, Copying, and Raw References

Domain 16 — Memory Management and Unsafe Primitives Duration: ~15 minutes Library components: std::ptr::NonNull, std::ptr::null, std::ptr::read, std::ptr::write, std::ptr::copy, std::ptr::copy_nonoverlapping, std::ptr::drop_in_place, &raw operators

Introduction

A raw pointer (*const T or *mut T) is an address without the guarantees of a reference:

It may be null, dangling, or unaligned.
It has no lifetime.
The borrow checker ignores it.

The rule about safety is exact. It is always safe to create a raw pointer. Only a dereference (and a few operations that imply access) is unsafe.

This tutorial is about the raw-pointer patterns that occur in real safe wrapper code, not about pointer trivia. Examples of such code are the containers of the standard library, arrayvec, and arena allocators. The tutorial shows:

The &raw const and &raw mut operators (stable 1.82), and why packed structs make them necessary.
ptr::null/null_mut compared with Option<&T>, and ptr::from_ref/from_mut.
NonNull<T>: the non-null guarantee, the niche optimization that it permits, and the owning-handle pattern.
How to move values through memory: ptr::read, ptr::write, and copy compared with copy_nonoverlapping (memmove compared with memcpy).
ptr::drop_in_place, which runs destructors through a pointer: in pools, on slices, and through vtables.

Figure: Where raw pointers come from and what consumes them

16_3_raw_pointers_in_practice_1

&raw: Addresses Without References

&raw const place and &raw mut place take the address of a place and do not create a reference first. This is important, because a reference asserts alignment and validity from the moment it exists. A misaligned &u32 is UB even if no code uses it. Packed structs are a concrete case:

#[repr(C, packed)]   // no padding: 1 + 4 + 2 = 7 bytes
struct WireHeader {
    version: u8,
    count: u32, // offset 1: misaligned for u32
    crc: u16,   // offset 5: misaligned for u16
}
// `header` is a WireHeader in which `count` is 123_456.

// let r = &header.count;
// error[E0793]: reference to field of packed struct is unaligned

let count_ptr: *const u32 = &raw const header.count;   // only an address, no reference
// SAFETY: `count_ptr` points into the live `header`. read_unaligned has no alignment requirement.
let count = unsafe { count_ptr.read_unaligned() };     // 123_456

Three related notes:

Plain by-value access to a packed field (let c = header.crc;) is safe. The compiler emits an unaligned copy.
Before 1.82, the addr_of! and addr_of_mut! macros did this operation.
When you already hold a reference, ptr::from_ref(r) converts it without an as cast. Unlike as, it cannot silently change the pointee type.

For null pointers, ptr::null() and ptr::null_mut() exist for FFI and for sentinels. is_null() is the check that you do before each dereference. In pure Rust APIs, prefer Option<&T> or Option<NonNull<T>>. They have the same size as a pointer (next slide), and the compiler forces the check.

NonNull and the Niche

NonNull<T> is a *mut T with a static guarantee that it is not null. This one guarantee has two benefits:

The niche optimization: the compiler stores None as the null pattern, which a NonNull can never hold:

assert_eq!(size_of::<Option<NonNull<u8>>>(), size_of::<*mut u8>()); // Option adds no bytes
assert!(size_of::<Option<*mut u8>>() > size_of::<*mut u8>());      // nullable: one more word

Covariance: NonNull<T> is covariant in T, as &T is, while *mut T is invariant. Owning containers need this property. Internally, Box, Vec, and Rc all use NonNull. For this reason, Option<Box<T>> has no additional cost.

The constructors, in order of preference:

NonNull::new(ptr) does a check at run time and returns an Option.
NonNull::from(&mut x) converts a reference and cannot fail.
NonNull::dangling() returns a pointer that is well-aligned and non-null but points at no value. An empty Vec<T> stores this sentinel and does not allocate.
Use new_unchecked only where a checked form is not possible.

The curriculum calls the next pattern "safe wrapper code". The example is an owning handle:

struct OwnedAudio { samples: NonNull<Vec<i16>> }

impl OwnedAudio {
    fn new(samples: Vec<i16>) -> Self {
        // Move the Vec to the heap. `into_raw` gives the ownership of the allocation to this code.
        let raw = Box::into_raw(Box::new(samples));   // raw: *mut Vec<i16>
        Self { samples: NonNull::new(raw).expect("Box::into_raw never returns null") }
    }
    fn peak(&self) -> i16 {
        // SAFETY: the pointer comes from Box::into_raw, and only Drop frees it.
        // `&self` proves that no `&mut` exists, so a shared borrow of the pointee is valid.
        let samples = unsafe { self.samples.as_ref() };   // samples: &Vec<i16>
        samples.iter().copied().max().unwrap_or(0)        // the largest sample, or 0 if empty
    }
}

impl Drop for OwnedAudio {
    fn drop(&mut self) {
        // SAFETY: the pointer comes from Box::into_raw, and this is the only Box::from_raw.
        // The drop of the new Box drops the Vec and frees the memory, exactly once.
        drop(unsafe { Box::from_raw(self.samples.as_ptr()) });
    }
}

Each safety comment derives the borrow rules again from &self or &mut self. The wrapper converts facts that the compiler checks into arguments for pointer validity.

read and write: Bitwise Moves

ptr::read(src) does a bitwise move out of a location. The source bytes do not change, but you must treat the value as if it now lives in the new location. If the two copies both drop, the result is a double free.

ptr::write(dst, v) moves a value in and does not drop the old contents of the destination. This is necessary when the destination holds uninitialized bytes (16.2) or the stale bytes of a moved-out value.

Together, these two functions are the core of mem::swap. This is the full sequence, written without library helpers:

/// # Safety
/// x and y must be valid for reads and writes, aligned, and NOT overlap.
unsafe fn swap_raw<T>(x: *mut T, y: *mut T) {
    // SAFETY: the caller guarantees that x and y are valid, aligned, and do not overlap.
    unsafe {
        let tmp: T = ptr::read(x);         // move *x out (x now holds stale bits)
        ptr::copy_nonoverlapping(y, x, 1); // memcpy *y over *x, no drop
        ptr::write(y, tmp);                // move tmp in, no drop of the stale *y
    }
}

Each value exists exactly once at each step. The example binary proves it with a payload that counts drops. The drop count at the end must be exactly the number of values (the output calls this a balanced drop ledger).

Example 16_10 shows one more pattern of this family. After ptr::read, the source still holds the stale bytes (the output calls them a hull). If the source would drop them, call mem::forget (16.1) on the source to prevent the second drop.

copy vs copy_nonoverlapping

The two functions copy count * size_of::<T>() bytes. The difference is one guarantee from the caller:

Figure: memmove vs memcpy — the overlap rule

ptr::copy vs ptr::copy_nonoverlapping

ptr::copy(src, dst, n) is memmove: the source and the destination may overlap. This is the core operation of Vec::remove. A shift of the tail to the left by one position means that the ranges share cells, and only memmove is correct for that:

// `lanes` is [10_u32, 99, 20, 30, 40], and `p` is `lanes.as_mut_ptr()`.
// Remove the 99: move the 3 elements at indexes 2..5 to indexes 1..4.
// SAFETY: the two ranges are in `lanes`, and ptr::copy permits overlap.
unsafe { ptr::copy(p.add(2), p.add(1), 3) };
// `lanes` is now [10, 20, 30, 40, 40]. The last slot is stale.

ptr::copy_nonoverlapping(src, dst, n) is memcpy: the ranges MUST be disjoint. An overlap is UB, not only wrong data. Two separate buffers are disjoint by their structure, so you get the full speed of memcpy at no cost.

General rule: for one buffer, use copy. For two different buffers, use copy_nonoverlapping.

Neither function runs destructors. For Copy types, that has no effect. For owning types, the copy duplicates ownership, and you must make sure that the program treats only one side as live. swap_raw above does exactly that.

16_10_ptr_read_write_copy.rs runs swap_raw, the overlapping shift, a copy between two separate buffers, and the ptr::read pattern with mem::forget. It prints:

after swap_raw: a=Token("beta") b=Token("alpha")

after overlap-shift: [10, 20, 30, 40] (slot 4 is stale)

frame after splice: [00, 00, DE, AD, BE, EF, 00, 00]

ptr::read moved the value out; hull neutralized with forget

drop ledger balanced: 3 Tokens dropped exactly once

All assertions passed.

drop_in_place: Destructors by Pointer

ptr::drop_in_place(p) runs the destructor of the value at p and does not move the value. Three situations make it necessary:

Managed storage (pools, arenas, MaybeUninit slots): the destructor must run in place, and then the slot is available for a new value. assume_init_drop from 16.2 is a wrapper for exactly this call.
Whole slices: drop_in_place accepts unsized pointees. Thus one call on a *mut [T] from ptr::slice_from_raw_parts_mut destroys N values. Vec drops its elements in this way. A provenance detail from the example: get the base pointer from the whole array (pool.as_mut_ptr()), not from pool[0]. The provenance of a pointer from pool[0] includes only element 0.
Trait objects: you cannot move the value behind a *mut dyn Trait, because the value is unsized. drop_in_place finds the concrete destructor in the vtable. The drop of a Box<dyn Trait> uses this mechanism:

// `raw` is a `*mut dyn Debug` from Box::into_raw. The value behind it is live.
// SAFETY: `raw` comes from Box::into_raw, so `&*raw` is a valid reference.
let layout = Layout::for_value(unsafe { &*raw });  // get the layout BEFORE the drop
// SAFETY: `raw` is valid and aligned, and this is the only drop of the value.
unsafe { ptr::drop_in_place(raw) };                // the vtable gives the destructor
// SAFETY: Box::new allocated this memory with this layout. No code uses `raw` again.
unsafe { std::alloc::dealloc(raw.cast::<u8>(), layout) }; // free the memory: a SEPARATE step (16.5)

The last line shows the important point: the destructor and the deallocation are two independent steps, and Box::drop does both. After drop_in_place, the memory is "logically uninitialized". The bytes stay, but it is UB to use them as a value again.

Since Rust 1.99, Layout::for_value_raw(raw) gets the same layout directly from the raw pointer, with no &*raw reference. It is also correct after drop_in_place, because it reads only the vtable (tutorial 16.1 shows it).

Summary

Tool	What it does	Key obligation
`&raw const` / `&raw mut`	Address of a place, with no reference	None: the creation is safe (a dereference is not)
`ptr::from_ref` / `from_mut`	Reference → raw pointer, with no `as` cast	None
`ptr::null` / `null_mut` / `is_null`	Null sentinels for FFI	Check before each dereference, or use `Option<&T>`
`NonNull<T>`	Non-null `*mut T` with a niche and covariance	The pointer must be non-null at construction
`NonNull::dangling`	Aligned, non-null, no allocation	Never dereference it
`ptr::read`	Bitwise move out	Treat the source as logically moved-from
`ptr::write`	Move in, with no drop of the old contents	The old value of the destination must not need a drop
`ptr::copy`	memmove: the ranges may overlap	Ranges valid for reads/writes
`ptr::copy_nonoverlapping`	memcpy: faster	The ranges MUST be disjoint (UB if they are not)
`ptr::drop_in_place`	Runs the destructor and does not move the value. Also for slices and `dyn`	The value must be valid. Drop it exactly once. Free the memory separately

Code Examples

File	Description
`16_08_raw_refs_and_null.rs`	`&raw` on packed fields, `read_unaligned`, `from_ref`, `.cast()`, null sentinels
`16_09_nonnull_niche.rs`	Assertions for the niche optimization, `NonNull` constructors, an owning-handle wrapper
`16_10_ptr_read_write_copy.rs`	`swap_raw` made from `read`/`copy_nonoverlapping`/`write`, an overlapping shift, a drop count check
`16_11_drop_in_place.rs`	In-place destruction in pool slots, on whole slices, and through vtables

16.4 · Pointer Arithmetic, Volatile, and Provenance

Domain 16 — Memory Management and Unsafe Primitives Duration: ~15 minutes Library components: std::ptr, pointer primitive methods, std::ptr::dangling

Introduction

This tutorial shows the three topics of raw pointers that are the most difficult to use correctly:

Arithmetic: add, sub, and offset. The surprising rule: to compute an out-of-bounds pointer is undefined behavior (UB), even with no dereference.
Provenance: the record of the allocation that a pointer can access. Each pointer has this record, but you cannot see it. The strict provenance APIs (stabilized in Rust 1.84) make it explicit: addr, with_addr, map_addr, expose_provenance, with_exposed_provenance.
Volatile: the guarantees of read_volatile and write_volatile. They guarantee that each access is a side effect that the compiler must do. They do not guarantee atomicity or synchronization.

The common idea is that a pointer is more than an address. The optimizer uses the origin of each pointer in its analysis. The APIs in this tutorial keep your model of a pointer the same as the model of the optimizer.

Pointer Arithmetic and Its UB Rules

p.add(n), p.sub(n), and p.offset(i) count in elements, not in bytes. (byte_add is for formats that give offsets in bytes.) The most important rule is:

The result must stay inside the same allocation, or point exactly one past its last element. If you break this rule, the result is UB even if you never dereference the pointer, because the optimizer assumes that this cannot occur.

The one-past-the-end pointer is the legal exclusive bound. The standard library uses it for its own iteration: the core of slice::Iter is a loop with two pointers. The function below sums a slice in the same way:

fn sum_by_pointer(data: &[u32]) -> u64 {
    let mut cursor: *const u32 = data.as_ptr();   // points to element 0
    // SAFETY: data.len() elements from the base is one past the last element,
    // which is in bounds of the allocation of the slice.
    let end: *const u32 = unsafe { cursor.add(data.len()) };

    let mut total = 0_u64;
    while cursor != end {   // you can compare with `end`, but never dereference it
        // SAFETY: cursor != end, and cursor moves one element at a time from
        // the base. Thus it points to an initialized element in bounds.
        total += u64::from(unsafe { *cursor });
        // SAFETY: cursor is before `end`, so +1 is in bounds or one past the end.
        cursor = unsafe { cursor.add(1) };
    }
    total
}
// sum_by_pointer(&[3, 1, 4, 1, 5, 9, 2, 6]) returns 31. An empty slice gives 0.

Distances, Wrapping, and Byte Offsets

The inverse operations give distances. p2.offset_from(p1) returns the signed distance in elements. offset_from_unsigned returns the unsigned distance. Use it when you know that the receiver is not before the argument (a debug build checks this). The two methods require that the two pointers are in the same allocation:

// samples = [5_u32, 10, 15, 20, 25] and base = samples.as_ptr().
// third = base.add(2) points to 15. fifth = base.add(4) points to 25.
// SAFETY: the two pointers are derived from the same array.
assert_eq!(unsafe { fifth.offset_from(third) }, 2);
assert_eq!(unsafe { third.offset_from(fifth) }, -2);
// SAFETY: same array, and fifth is not before third.
assert_eq!(unsafe { fifth.offset_from_unsigned(third) }, 2);

Some code really needs addresses that are out of range. Examples are a scan with a stride that checks the bounds separately, and a hash of an address. For this code, wrapping_add and wrapping_sub remove the in-bounds rule. Wrapping arithmetic is not unsafe, because the calculation of an address is not dangerous. A dereference of an out-of-bounds address is still UB. The wrapping methods make the arithmetic legal, but never the access:

// samples = [1_u32, 2, 3] and base = samples.as_ptr().
let way_out = base.wrapping_add(1_000_000); // far out of bounds: legal to compute and keep
let back = way_out.wrapping_sub(1_000_000); // == base, and it keeps the provenance of base
// SAFETY: back is equal to base in address and in provenance. Element 0 is initialized.
assert_eq!(unsafe { *back }, 1);

Some file formats and device datasheets give offsets in bytes. For these, byte_add does not multiply the offset by the element size. base.byte_add(4).cast::<u32>().read() reads the field that a header puts at byte 4. byte_add is the pointer-arithmetic equivalent of mem::offset_of!.

Provenance: A Pointer Is More Than an Address

Two pointers can hold the same address and not be interchangeable. Each pointer has provenance: a compile-time record of the allocation that the pointer is derived from and can access. Alias analysis uses provenance. The conclusion "these two stores cannot alias, because they have different provenance" lets the optimizer change the order of memory operations. Those changes would be unsound if any integer could become any pointer.

The strict provenance APIs (stable since 1.84) make the concept concrete:

Figure: The two roads from integer back to pointer

16_4_pointer_arithmetic_volatile_provenance_1

p.addr() returns the numeric address, for logs, alignment checks, and hashing. The result has no provenance: no strict provenance API can make a usable pointer from that usize alone.
base.with_addr(a) and base.map_addr(f) return a new pointer that has the provenance of base at a new address. You do integer arithmetic and you keep the provenance. If you can name the pointer that your address comes from, use these methods.
p.expose_provenance() with ptr::with_exposed_provenance(a) is the alternative for an address that really must exist as an integer for some time. Examples are the user-data slot of a C callback (which has the size of a void*) and XOR linked lists. An exposed provenance removes some alias-analysis capability from the optimizer. Thus use this pair only when the provenance chain really goes through an integer.
ptr::without_provenance(a) makes a pointer from an address that you will never dereference, and the call states that intent. ptr::dangling() is the well-aligned variant: it is the sentinel that Vec::<T>::new() stores before its first allocation.

Use Case: A Tagged Pointer

Because of alignment, you can prove that the low bits of most pointers are zero. For example, align_of::<u32>() == 4, so bits 0 and 1 of a pointer to u32 are free. Compilers, interpreters, and garbage collectors put flags into those bits to save one word for each object. map_addr lets you do this soundly. The address bits change and the provenance stays. Thus the untagged pointer is still fully valid.

Figure: One word holds both the pointer and the mark flag

Tagged pointer bit layout

use std::num::NonZeroUsize;
use std::ptr::NonNull;

/// An owning pointer to a heap u32. Bit 0 of the address holds the mark flag.
struct TaggedNode {
    /// INVARIANT: tagged.addr() & !1 is a live Box<u32> allocation.
    tagged: NonNull<u32>,
}

impl TaggedNode {
    fn set_marked(&mut self, marked: bool) {
        // map_addr changes the address bits and keeps the provenance.
        // `a` is the current address as a NonZeroUsize.
        self.tagged = self.tagged.map_addr(|a| {
            // Set bit 0 to mark the node. Clear bit 0 to remove the mark.
            let bits = if marked { a.get() | 0b1 } else { a.get() & !0b1 };
            // The untagged address is a non-zero multiple of 4, so `bits` is never 0.
            NonZeroUsize::new(bits).expect("untagged address is aligned, hence nonzero")
        });
    }
}

The invariant of the type enforces two rules:

Always untag before you dereference. On a misaligned access, the hardware would fault or do worse.
Always untag before you free. The allocator must receive the exact address that it gave.

The example binary asserts that size_of::<TaggedNode>() is equal to size_of::<usize>(). A (pointer, bool) pair uses two words. The example also does a pass in the style of mark-and-sweep: it sets the flag on some nodes and does not change their data. 16_14_tagged_pointer.rs prints:

TaggedNode: 8 bytes vs (ptr, bool): 16 bytes   # (on a 64-bit target)
values:  [0, 111, 222, 333]
marked:  [true, false, true, false]
values still intact: [0, 111, 222, 333]

All assertions passed.

Volatile: Every Access Is a Side Effect

read_volatile and write_volatile mark a memory access as an observable side effect. The compiler must do exactly the volatile accesses that you wrote. It does not remove an access, it does not merge two accesses, and it does not change the order of volatile accesses. That is the full guarantee. Its purpose is memory-mapped I/O (MMIO), where an "address" is really a device register and each access has an effect on the hardware:

// tx: *mut u32 and status: *const u32 point to the two registers of a UART.
// The example binary simulates the registers with a local struct.
for byte in b"OK" {
    // The two stores can have equal values. The compiler could merge two
    // normal stores into one. The device must receive each byte.
    // SAFETY: tx points into a live, aligned, initialized local struct.
    unsafe { tx.write_volatile(u32::from(*byte)) };

    // The compiler could move a normal read out of the loop ("memory never
    // changes"), and then the loop would never stop. The compiler must do a
    // volatile read on each iteration, because the DEVICE may change the value.
    // SAFETY: status points into the same live struct.
    while unsafe { status.read_volatile() } & 0b1 != 0 {}
}

The guarantees that volatile does not give are as important:

No atomicity: a volatile u64 store may tear.
No ordering for the non-volatile accesses around it.
No visibility guarantees across threads.

A flag that threads share through volatile accesses "usually works", but it is a data race, and a data race is UB. The correct tool for signals between threads is AtomicU32 or AtomicU64 with acquire/release ordering (Domain 6). Volatile is for communication with hardware, not with other threads.

Summary

Tool	What it does	The rule
`add` / `sub` / `offset`	Arithmetic in elements	Stay in bounds or one past the end. To compute a pointer outside is UB
`offset_from` / `offset_from_unsigned`	Distance in elements between two pointers	The two pointers must be in the same allocation
`byte_add`	Arithmetic in bytes	For formats that give byte offsets
`wrapping_add` / `wrapping_sub`	Arithmetic with no bounds rule, not `unsafe`	Makes the arithmetic legal, never the access
`addr()`	Address as `usize`, with no provenance	Only for logs, alignment checks, and hashing
`with_addr` / `map_addr`	New address, same provenance	The preferred, strict method for integer arithmetic
`expose_provenance` / `with_exposed_provenance`	A real round trip from pointer to integer to pointer	Use only when necessary. It decreases alias analysis. Document the reason
`ptr::without_provenance` / `ptr::dangling`	Pointers that have only an address, aligned sentinels	Never dereference
`read_volatile` / `write_volatile`	Accesses that the compiler must do exactly as written	MMIO only. No atomicity. Not for thread synchronization

Code Examples

File	Description
`16_12_pointer_arithmetic.rs`	Slice sum with two pointers, `add`/`offset`/`offset_from`, wrapping arithmetic, `byte_add`
`16_13_provenance.rs`	`addr`, `with_addr`/`map_addr`, round trip through an exposed provenance, `without_provenance`, `dangling`
`16_14_tagged_pointer.rs`	Mark bit in a `Box<u32>` pointer with `map_addr`, and the sound rules to tag, untag, and free
`16_15_volatile.rs`	UART transmit loop in the form of MMIO code, and the guarantees that volatile gives and does not give

16.5 · Global Allocator and Allocation APIs

Domain 16 — Memory Management and Unsafe Primitives Duration: ~15 minutes Library components: std::alloc::GlobalAlloc, std::alloc::Layout, std::alloc::alloc, std::alloc::dealloc, std::alloc::realloc, std::alloc::handle_alloc_error, std::alloc::System

Introduction

Each heap operation in a Rust program goes through one global allocator: each Box::new, and each growth of a Vec or a String. This tutorial goes down that stack in three steps:

Layout: the language of size and alignment in which you request memory.
The free functions alloc, dealloc, realloc, and alloc_zeroed, which take a Layout.
The GlobalAlloc trait and the #[global_allocator] attribute, which let you replace the allocator for the full program. Allocator replacements such as jemalloc and mimalloc use this mechanism, and so do memory profilers.

Figure: The allocation call chain

16_5_global_allocator_allocation_apis_1

The tutorial also shows the layout composition methods. extend is stable since Rust 1.44. repeat, repeat_packed, extend_packed, and dangling_ptr are stable since Rust 1.95. With them, you calculate the layout of an aggregate at run time in the same way as the compiler does at compile time. The last topic is the parameterized Allocator trait: why it is still unstable, and why you should monitor it.

Layout: The Request Language

A Layout is two numbers: size (in bytes) and align (a power of two). Each request for memory uses a Layout. The constructors are:

use std::alloc::Layout;
use std::fmt::Debug;

// A trait object and a raw pointer to the same value, for the run-time constructors.
let boxed_dyn: Box<dyn Debug> = Box::new(7_u64);
let raw_dyn: *const dyn Debug = &raw const *boxed_dyn;

// `?` returns a LayoutError to the caller of the function that contains these lines.
let one  = Layout::new::<u32>();             // size 4, align 4
let many = Layout::array::<u32>(4)?;         // size 16, align 4 (Err if the size overflows)
let vals = Layout::for_value(&*boxed_dyn);   // size 8, align 8: from the vtable at run time (16.3)
// SAFETY: raw_dyn points to a live value, so it is safe to reborrow as a reference.
let same = unsafe { Layout::for_value_raw(raw_dyn) };   // 1.99: the same, from a raw pointer (16.1)
let raw  = Layout::from_size_align(5, 4)?;   // manual: size does NOT have to be a multiple of align

The last line shows an important point. For each real Rust type, size is a multiple of align, so each element of an array is aligned. A Layout that you make manually does not have this rule. For that reason, the composition methods on the next slides have padded variants and packed variants.

pad_to_align() rounds the size up to a multiple of the alignment. It is the last step of a struct layout calculation.

Composing Layouts: extend and repeat

The composition methods do, at run time, the layout algorithms that the compiler does at compile time. extend adds a field in the style of repr(C). It does three steps:

It adds padding until the offset agrees with the alignment of the field.
It increases the alignment of the aggregate, if necessary.
It returns the offset of the field.

/// Calculates the layout of a #[repr(C)] struct from the layouts of its fields.
/// Returns the struct layout and the offset of each field.
fn repr_c(fields: &[Layout]) -> (Layout, Vec<usize>) {
    // Start with an empty layout: size 0, align 1.
    let mut layout = Layout::from_size_align(0, 1).unwrap();
    let mut offsets = Vec::with_capacity(fields.len());
    for &field in fields {
        // extend returns the larger layout and the offset of the new field.
        // It returns Err only if the new size is too large for a Layout.
        let (grown, offset) = layout.extend(field).unwrap();
        layout = grown;
        offsets.push(offset);
    }
    (layout.pad_to_align(), offsets)   // last rule: round the size up to align
}

The example binary gives this function the layouts of u8, u32, and u16. It asserts that the result is exactly equal to the compiler result for a real #[repr(C)] struct: size_of, align_of, and each mem::offset_of!.

Figure: What the calculator reproduces — repr(C) padding vs packed

repr(C) layout with padding vs packed

Arrays, Packed Layouts, and the Zero-Size Sentinel

repeat(n) gives the layout of an array of n elements. It also returns the stride: the distance between the start of one element and the start of the next element. Layout::array::<T>(n) is the short form for a Rust type:

// Telemetry is the #[repr(C)] struct of the figure: u8, u32, u16 (size 12, align 4).
// `?` returns a LayoutError if the total size is too large for a Layout.
let (array, stride) = Layout::new::<Telemetry>().repeat(16)?;
assert_eq!(stride, size_of::<Telemetry>());              // 12 (real types: stride == size)
assert_eq!(array.size(), size_of::<[Telemetry; 16]>());  // 192 = 16 × 12

The _packed variants add no padding:

extend_packed adds a field at the current size and does not increase the alignment. This is the #[repr(C, packed)] algorithm. For the struct on the previous slide, it gives 7 bytes and align 1.
repeat_packed puts each element directly after the previous one and does not round the element size.

For real Rust types (where the size is a multiple of the alignment), repeat and repeat_packed give the same layout. For a manual layout with size 5 and align 4, the layouts are different. repeat(3) gives stride 8. repeat_packed(3) gives size 15, and elements 1 and 2 are misaligned. Packed layouts agree with wire formats byte for byte. The cost, as in tutorial 16.3, is that you cannot make references to misaligned fields.

The last method is dangling_ptr(). The global allocator does not accept zero-size requests: a call to alloc with a zero-size layout is UB. But Vec::<u8>::new() needs some pointer value. dangling_ptr() returns the standard sentinel for this purpose: a non-null pointer, aligned to the layout, with no allocation behind it. It is the Layout equivalent of ptr::dangling (tutorial 16.4).

Never dereference this pointer. Its address can be equal to the address of a valid pointer, so do not use it as a test for "not allocated".

16_16_layout_composition.rs prints:

calculated: size=12 align=4 offsets=[0, 4, 8]
compiler:   size=12 align=4 offsets=[0, 4, 8]

packed: size=7 align=1

[Telemetry; 16]: size=192 stride=12
(5,4) repeat(3): size=21 stride=8; repeat_packed(3): size=15

zero-size sentinel: non-null, aligned to 8 — no allocator call needed

All assertions passed.

The Raw Allocation API

Each collection uses the four free functions in std::alloc as its base. Their contract is in the style of C, and the caller must obey all of it:

alloc(layout) returns uninitialized memory, or null on failure. A read before a write is UB (the subject of tutorial 16.2). On failure, call handle_alloc_error(layout). It reports the failure and aborts the process, because by default an allocation failure must not unwind.
realloc(ptr, layout, new_size) makes a block larger or smaller. It keeps the data up to the smaller of the two sizes, but the block may move. Never keep pointers into a reallocated block: use only the returned pointer. An access through the old pointer is UB, even if the block did not move. On failure, realloc does not change the original block.
dealloc(ptr, layout) has the most dangerous rule: it must receive the same layout that you used to allocate the block. The allocator does not keep the layout for you. Vec keeps its capacity for exactly this purpose: to calculate this layout again.
alloc_zeroed(layout) is the equivalent of calloc. It returns memory that is all zeros, and the allocator often takes that memory directly from new OS pages. Bytes that are all zero are initialized data for integer types.

use std::alloc::{self, Layout};

let four = Layout::array::<u32>(4)?;         // size 16, align 4
// SAFETY: `four` has non-zero size.
let raw: *mut u8 = unsafe { alloc::alloc(four) };
// Null means that the allocation failed. handle_alloc_error does not return.
if raw.is_null() { alloc::handle_alloc_error(four); }
let buf = raw.cast::<u32>();                 // *mut u32, aligned: `four` has the align of u32
for i in 0..4 {
    // SAFETY: i < 4 is in bounds of the block. `write` does not read or drop the old value.
    unsafe { buf.add(i).write(1 << i) };     // the block becomes [1, 2, 4, 8]
}
// ... later:
// SAFETY: this allocator allocated `raw` with layout `four`. This is the only free.
unsafe { alloc::dealloc(raw, four) };

The example binary does the full cycle:

Allocate a block with alloc.
Initialize the elements.
Double the capacity with realloc.
Free the block with dealloc and the larger layout.

This is the internal work of Vec, written manually one time. It shows the work that the safe type hides.

#[global_allocator]: Replacing the Allocator

To replace the allocator for the full program, do two steps:

Write unsafe impl GlobalAlloc for YourType.
Put the #[global_allocator] attribute on one static of that type.

The usual first allocator is a wrapper around System that counts calls. It is the minimum version of the work that memory profilers do:

use std::alloc::{GlobalAlloc, Layout, System};
use std::sync::atomic::{AtomicUsize, Ordering};

struct CountingAllocator;   // no fields: the counter is a static

static ALLOC_CALLS: AtomicUsize = AtomicUsize::new(0);

// SAFETY: each method passes its arguments unchanged to System, which obeys the
// GlobalAlloc contract. The atomic counter has no effect on the returned memory.
unsafe impl GlobalAlloc for CountingAllocator {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        ALLOC_CALLS.fetch_add(1, Ordering::Relaxed);   // an atomic add does not allocate
        // SAFETY: the caller guarantees that `layout` has a non-zero size.
        unsafe { System.alloc(layout) }
    }
    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        // (The example binary also counts the dealloc calls here.)
        // SAFETY: the caller guarantees that this allocator allocated `ptr` with `layout`.
        unsafe { System.dealloc(ptr, layout) }
    }
}

// This one static sends each heap operation of the program to CountingAllocator.
#[global_allocator]
static GLOBAL: CountingAllocator = CountingAllocator;

Obey three rules:

The allocator must not allocate, because that call would go into the allocator again. Use only atomics and System calls.
realloc and alloc_zeroed have default implementations. If you pass them to System, you keep the optimized paths of System, and your counts stay correct.
Assert only monotonic facts. This rule controls the assertions of the example.

The runtime, println!, and the OS may allocate at any moment, so exact counts have no meaning. The example asserts this fact: the collection of 1000 elements into a Vec<u64> increased the alloc count and requested at least 8000 bytes. That fact is true in each run of the example. It is not a guarantee: the GlobalAlloc documentation lets the optimizer remove or merge allocations. Thus unsafe code must not rely on a count.

A last note, which the curriculum also makes: the parameterized Allocator trait (as in Vec<T, A>) is still unstable. It will permit a custom allocator for each collection, with no replacement of the global allocator. #[global_allocator] is the stable tool today. Monitor the progress of Allocator.

Summary

Tool	What it does	Key obligation
`Layout`	Size and alignment: the language of each memory request	The size, rounded up to `align`, is not more than `isize::MAX`. `align` is a power of two
`Layout::extend` (1.44)	Adds a field in the style of `repr(C)` and returns its offset	—
`Layout::repeat` (1.95)	Array layout, and it returns the stride	—
`extend_packed` / `repeat_packed` (1.95)	Composition with no padding, for wire formats	Elements may be misaligned
`Layout::dangling_ptr` (1.95)	Aligned non-null sentinel for zero-size "allocations"	Never dereference
`Layout::for_value_raw` (1.99)	Layout of the value behind a raw pointer. It reads only the pointer metadata	`unsafe`. The call is safe if the pointer is safe to reborrow as `&T`. If not, a `Sized` type is always safe. For a slice, a `str`, or a trait object, the size of the entire value must fit in `isize`
`alloc`	Uninitialized memory or null	Non-zero size. Check for null, then call `handle_alloc_error`
`alloc_zeroed`	Zeroed memory (`calloc`)	Same as `alloc`
`realloc`	Larger or smaller block. It keeps the data, but the block may move	Same layout that you used to allocate the block. Use only the returned pointer
`dealloc`	Returns a block to the allocator	Exactly the same layout: the allocator does not keep it
`GlobalAlloc` + `#[global_allocator]`	Replace the allocator for the full program	Obey the contract. Never allocate in the allocator
`System`	The OS allocator, always available for a wrapper	—
`Allocator` trait	Allocators for each collection (`Vec<T, A>`)	Still unstable: monitor it, do not use it

Code Examples

File	Description
`16_16_layout_composition.rs`	`repr(C)` calculator with `extend` and `pad_to_align`, checked against `offset_of!`. Also packed variants, `repeat`, `dangling_ptr`
`16_17_raw_alloc_realloc.rs`	Full cycle of `alloc`, initialization, `realloc`, and `dealloc`. Also `alloc_zeroed`, and `handle_alloc_error` on null
`16_18_counting_global_allocator.rs`	`#[global_allocator]` wrapper around `System` that counts calls, with monotonic assertions

17.1 · Any and Dynamic Typing

Domain 17 — Type System Utilities Duration: ~15 minutes Library components: std::any::Any, std::any::TypeId, std::any::type_name, std::any::type_name_of_val

Introduction

Rust answers almost every type question at compile time. This is a basic design decision of the language. But a small set of problems must find the type of a value at runtime:

A plugin host receives opaque state from plugins that it does not know.
A middleware stack attaches arbitrary data to each request.
A panic handler gets a message from the value that a panic! call carried.

For these cases, std::any gives a minimal, safe form of dynamic typing.

This tutorial shows:

TypeId::of::<T>(): an opaque, globally unique runtime identifier for each 'static type.
The Any trait and its three downcasting tools: downcast_ref, downcast_mut, and Box::downcast.
The heterogeneous registry (HashMap<TypeId, Box<dyn Any>>). This is the "type map" pattern that http::Extensions and the resources of Bevy use.
Panic payloads as Box<dyn Any + Send>, and why Any requires 'static.
type_name and type_name_of_val for diagnostics, and why you must not use their output as a stable value.

Keep this model in mind for the next 15 minutes. TypeId is an identity, type_name is a label, and Any gives a value back only as its exact original type.

TypeId: Runtime Type Identity

TypeId::of::<T>() returns an opaque value. The value is unique for each type during one run of a program. TypeId implements Eq, Ord, and Hash, so you can use it with each comparison and collection in the standard library. To find at runtime if two types are the same, compare their TypeId values. This comparison is the reliable method:

use std::any::TypeId;

assert_eq!(TypeId::of::<u32>(), TypeId::of::<u32>());   // the same type, the same TypeId
assert_ne!(TypeId::of::<u32>(), TypeId::of::<i32>());   // same size, different types
// A generic argument is part of the identity: Vec<u8> is not Vec<u16>.
assert_ne!(TypeId::of::<Vec<u8>>(), TypeId::of::<Vec<u16>>());

Two details are important in practice:

Aliases share identity, newtypes do not. type Meters = u64; is the same type as u64 and has the same TypeId. struct MetersNewtype(u64); is a distinct type with its own TypeId. For this reason, the newtype pattern (tutorial 26.1) gives real type safety, and an alias does not.
The 'static bound. TypeId::of::<T> requires T: 'static. A TypeId cannot encode lifetimes. A type such as &'a str, with a borrowed lifetime 'a, has no well-defined runtime identity. Thus the compiler does not make a TypeId for it. The same rule applies to the Any trait on the subsequent slides.

type_name: a label, not an identity

type_name::<T>() and type_name_of_val(&v) return a best-effort, readable name of a type. Use this name in log lines and generic error messages:

use std::any::type_name;

// No parameter has the type T, so the caller gives T explicitly.
fn decode_error<T>(offset: usize) -> String {
    format!("failed to decode `{}` at byte offset {offset}", type_name::<T>())
}

// `MetersNewtype` is the newtype from the list above.
let err = decode_error::<MetersNewtype>(11);
// In the example binary, `err` is:
// "failed to decode `17_01_typeid_type_name::MetersNewtype` at byte offset 11"

The documentation gives no guarantee for two properties. The output format may change between compiler versions. Two different types may have the same name (for example, the same type from two versions of one crate).

Do not use the output of type_name for serialization, as a lookup key, or in assertions that compare the exact text. The example binaries assert only loose properties, such as ends_with("MetersNewtype") and contains("Vec"). The exact text (alloc::vec::Vec<u8> today) can change, and then an exact assert_eq! fails.

The Any Trait and Downcasting

A blanket implementation in the standard library implements Any for each 'static type. Any is useful when you erase a concrete type into &dyn Any or Box<dyn Any> and get it back later. The operation that gets the concrete type back is downcasting. There is one tool for each ownership mode:

Tool	Signature (essence)	Use when
`is::<T>()`	`-> bool`	You need only a yes or no answer
`downcast_ref::<T>()`	`-> Option<&T>`	You borrow the value and examine it
`downcast_mut::<T>()`	`-> Option<&mut T>`	You change the value in place
`Box::downcast::<T>()`	`-> Result<Box<T>, Box<dyn Any>>`	You take ownership

Figure: Downcasting a type-erased value — each attempt either recovers the concrete type or passes the value through intact

17_1_any_dynamic_typing_1

Internally, a downcast is an exact TypeId comparison. It does not use subtyping or coercions. For a String erased as dyn Any, downcast_ref::<&str>() returns None. The Err variant of Box::downcast returns the original box. Thus you can try one type and then a different type, and you do not lose the value:

// `Click` is a struct with two i32 fields, x and y.
// Ok: the event is a Click, and `*boxed` moves it out of the box.
// Err: the event has a different type, and the caller gets the same box back.
fn take_click(event: Box<dyn Any>) -> Result<Click, Box<dyn Any>> {
    event.downcast::<Click>().map(|boxed| *boxed)
}

Example 17_02 shows one possible error. Box<dyn Any> is also a 'static type, so it implements Any too. A .type_id() call on the box gives the TypeId of the box. To get the TypeId of the contents, call (*boxed).type_id().

The is, downcast_ref, and downcast_mut methods do not have this problem. They are inherent methods on dyn Any, so auto-deref gets to the contents first. Clippy has a lint (type_id_on_box) for this error.

The Heterogeneous Registry

The best-known std::any pattern uses the two parts together. TypeId is hashable, and Box<dyn Any> is type-erased. Thus HashMap<TypeId, Box<dyn Any>> is a container that holds a maximum of one value of each type. You get a value by its type only. There are no string keys that you can misspell, and no central enum that you must maintain. A new middleware can attach new state, and the definition of the registry does not change.

Figure: A type map — each entry is keyed by the TypeId of the value it stores, so the downcast on retrieval cannot fail

17_1_any_dynamic_typing_2

// Registry has one field: `entries: HashMap<TypeId, Box<dyn Any>>`.
impl Registry {
    // Stores `value` and returns the previous value of type T, if there was one.
    fn insert<T: Any>(&mut self, value: T) -> Option<T> {
        self.entries
            .insert(TypeId::of::<T>(), Box::new(value))   // Option<Box<dyn Any>>: the old entry
            // The old box was stored under the TypeId of T, so this downcast cannot fail.
            .and_then(|old| old.downcast::<T>().ok())
            .map(|boxed| *boxed)                          // Box<T> -> T
    }

    // Borrows the stored value of type T, if there is one.
    fn get<T: Any>(&self) -> Option<&T> {
        self.entries
            .get(&TypeId::of::<T>())                      // Option<&Box<dyn Any>>
            .and_then(|boxed| boxed.downcast_ref::<T>())
    }
}

One invariant makes this pattern safe: the registry stores each value only under the TypeId of that value. Thus the downcast always succeeds when you get the value. http::Extensions uses this pattern to attach middleware state to requests. The app_data of actix-web and the ECS resource storage of Bevy use it too. Example 17_03 contains the full registry with insert, get, get_mut, and remove.

Panic Payloads and the 'static Requirement

You do not need an unusual architecture to see Box<dyn Any>. Each Rust panic has a payload of this type. When its closure panics, std::panic::catch_unwind returns Err(Box<dyn Any + Send>). JoinHandle::join (Domain 6) returns the same box from a thread that panicked. The payload type depends on the call that started the panic:

panic!("literal") → &'static str
panic!("format {x}") → String
std::panic::panic_any(value) → the type of value (any Send + 'static type)

To get a message from the payload, use a chain of downcasts. The chain is possible because Err returns the box:

// Tries the known payload types in sequence. Each Err gives the box back for the next try.
fn payload_message(payload: Box<dyn Any + Send>) -> String {
    match payload.downcast::<&'static str>() {
        Ok(message) => (*message).to_string(),                // from panic!("literal")
        Err(payload) => match payload.downcast::<String>() {
            Ok(message) => *message,                          // from panic!("format {x}")
            Err(_) => String::from("<opaque panic payload>"), // a different payload type
        },
    }
}

Why `'static`?

dyn Any is always dyn Any + 'static. Suppose that a Box<dyn Any> could contain borrowed data. Then a later downcast could give a reference to memory that the program already freed. The TypeId cannot hold the lifetime, so the borrow checker could not reject that downcast. Thus the compiler rejects the erasure where it occurs:

// This function does not compile: `&'a str` is not `'static`.
fn erase<'a>(s: &'a str) -> Box<dyn Any> {
    Box::new(s)
}
// error: lifetime may not live long enough
//     coercion requires that `'a` must outlive `'static`

Owned data (String, Vec, your own structs) and &'static str all satisfy 'static. Thus a panic payload can contain them. Tutorial 2.3 covers the panic mechanics. In this tutorial, the payload is an example of type erasure.

When to Use Any (and When Not To)

std::any has a deliberately narrow scope. Use it when the set of types is open: you cannot know the types at the point where you define the container or channel.

Extension maps and type maps: middleware state and the data that plugins attach (example 17_03).
Plugin architectures: hosts that store state for separately compiled plugins.
Error downcasting: Box<dyn Error> gives you downcast_ref, which uses the same TypeId comparison (tutorial 2.2).
Panic payloads: you have no alternative, because the API gives you Box<dyn Any + Send> (example 17_04).
Diagnostics: type_name in log lines and generic error messages (example 17_01).

Prefer the static alternatives when the set of types is closed:

Situation	Better tool
Fixed, known set of variants	`enum` and `match`: exhaustive, faster, and with no `Option` to handle
One behavior for many types	Generics, or `dyn Trait` with a trait that has the behavior. Callers use methods, not type tests
Registry keys that are strings	Newtypes and the type map: use `TypeId` as the key

A long if-else chain of downcast_ref calls on a set of types that you fully control does the work of an enum. But the compiler does not check that the chain is exhaustive. The chain also has a real dispatch cost, although it uses no HashMap. Use Any only where you cannot write an enum.

Summary

Concept	Key point
`TypeId::of::<T>()`	Opaque, unique runtime identity that is `Eq + Ord + Hash`. It requires `T: 'static`
Aliases vs. newtypes	`type X = Y` has the same `TypeId` as `Y`. `struct X(Y)` has its own
`type_name::<T>()`	Readable label for diagnostics. The format is not stable and not unique. Do not serialize it or compare its exact text
`Any`	A blanket implementation covers each `'static` type. It makes safe downcasting possible
`downcast_ref` / `downcast_mut`	Borrow the concrete value: `Option<&T>` / `Option<&mut T>`
`Box::downcast`	Takes ownership: `Result<Box<T>, Box<dyn Any>>`. `Err` returns the original box
Type map	`HashMap<TypeId, Box<dyn Any>>`: one value for each type, and you get it by type. `http::Extensions` and Bevy resources use it
Panic payloads	`Box<dyn Any + Send>` from `catch_unwind` and `join`. It contains `&'static str`, `String`, or a `panic_any` value
`'static` requirement	`TypeId` cannot encode lifetimes, so you cannot erase borrowed data into `dyn Any`
Design guidance	Closed set of types → enum or trait. Open set → `Any`

Code Examples

File	Description
`17_01_typeid_type_name.rs`	`TypeId` identity (aliases, newtypes, generics). `type_name` and `type_name_of_val` for diagnostics. Why the assertions on names are loose
`17_02_downcasting.rs`	`downcast_ref`, `downcast_mut`, and `Box::downcast` on a UI event bus. The `.type_id()` error on a `Box<dyn Any>`
`17_03_heterogeneous_registry.rs`	The type map: `HashMap<TypeId, Box<dyn Any>>` with `insert`, `get`, `get_mut`, and `remove`, as middleware extensions use it
`17_04_panic_payloads.rs`	How to get `&'static str`, `String`, and custom payloads from `catch_unwind`. `panic_any`. The `'static` rule

17_03_heterogeneous_registry.rs prints:

registry holds 3 entries (one per type)
re-insert replaced Some(RequestId(42)), len still 3
user = ada (admin: true)
get::<Vec<u8>>() -> None (never attached)
metrics after three queries = Metrics { db_queries: 3 }
removed RequestId(43); len now 2

All assertions passed.

17.2 · Marker Traits: Sized, Copy, Send, Sync, Unpin

Domain 17 — Type System Utilities Duration: ~15 minutes Library components: std::marker::Sized, std::marker::Copy, std::marker::Send, std::marker::Sync, std::marker::Unpin, std::marker::PhantomData, std::marker::PhantomPinned

Introduction

std::marker is a small module of the standard library, and arguably the module with the largest effect. Its traits have no methods. Each trait is a compile-time property that a type has or does not have:

Sized: the compiler knows the size of the type.
Copy: a copy of the bits is a valid copy of the value.
Send: a value can move to a different thread.
Sync: threads can share a value.
Unpin: the address of a value is not important to the value.

The compiler uses these markers constantly. You usually see them only when a bound fails.

This tutorial shows:

Sized, the implicit default bound on each generic type parameter, and ?Sized, which relaxes it.
Thin pointers and fat pointers: the location of the size of an unsized value.
Copy and Clone: the meaning of assignment, and the types that can be Copy.
Send and Sync as auto traits: the fields of a type give the markers to the type, or remove them. This is the type-system view of the traits that Domain 6 uses in practice. Tutorial 6.8 covers concurrency in depth.
Unpin, and the zero-size markers PhantomPinned and PhantomData<T>. You add these markers to a type to change the properties that the compiler assumes for it.

Figure: The marker traits at a glance — automatic properties above, opt-in/opt-out markers below

17_2_marker_traits_1

Sized: The Bound You Never Wrote

Each generic type parameter has an implicit bound. fn process<T>(value: T) means fn process<T: Sized>(value: T). The compiler must know the size of a type to pass a value of that type or to keep it in a local variable. The same is true for a field of a struct. Thus the compiler requires Sized by default, and you do not write the bound.

But Rust has dynamically sized types (DSTs). The size of a DST exists only at runtime. For each DST, the compiler does not know one item:

str: the number of bytes
[T]: the number of elements
dyn Trait: the concrete type behind the trait object

A DST value cannot be directly on the stack. It exists only behind a pointer. ?Sized ("maybe sized") relaxes the implicit bound, so one generic signature accepts sized types and DSTs:

use std::fmt::Debug;

// Accepts &u32, &str, &[u8], and &dyn Debug: sized and unsized types.
// `value` is a reference, so the parameter has a known size for each T.
fn describe<T: Debug + ?Sized>(label: &str, value: &T) {
    println!(
        "  {label:<14} value = {value:?}, size_of_val = {} bytes",
        size_of_val(value)   // reads the size from the value at runtime
    );
}

describe("u32 (sized)", &42_u32);            // T = u32:  4 bytes
describe("str", "héllo");                    // T = str:  6 bytes (é is 2 bytes of UTF-8)
describe("[u8] slice", &[1_u8, 2, 3][..]);   // T = [u8]: 3 bytes

The relaxed bound helps only behind a pointer. A parameter that takes an unsized value directly is still not possible:

// This function does not compile: it takes `value` directly, and T can be unsized.
fn broken<T: ?Sized>(value: T) {}
// error[E0277]: the size for values of type `T` cannot be known at
//               compilation time
// help: function arguments must have a statically known size, borrowed types
//       always have a known size

For this reason, std uses the bounds Box<T: ?Sized>, Rc<T: ?Sized>, and &T where T: ?Sized. One impl<T: ?Sized> Deref for Box<T> applies to Box<u32>, Box<str>, and Box<dyn Trait>. Zero-sized types such as () are still Sized, because a size of zero is a known size.

Thin and Fat Pointers

If the type does not give the size, the pointer must give it. A reference to a Sized type is one machine word: only an address. A reference to a DST is two words: the address and the metadata that gives the size at runtime.

Figure: Where the size information lives — thin pointers for Sized types, pointer + metadata for DSTs

Thin vs. fat references

// `Debug` is std::fmt::Debug. `word` is 8 on a 64-bit platform.
let word = size_of::<usize>();
assert_eq!(size_of::<&u32>(), word);           // thin: address only
assert_eq!(size_of::<&str>(), 2 * word);       // address + byte length
assert_eq!(size_of::<&[u8]>(), 2 * word);      // address + element count
assert_eq!(size_of::<&dyn Debug>(), 2 * word); // address + vtable pointer

For slices and str, the metadata is a length. For trait objects, the metadata is a pointer to the vtable. The vtable contains the size and alignment of the concrete type, together with its method table (Domain 24 examines the vtable). size_of_val is the runtime counterpart of size_of. It reads the metadata from the value, so it works where size_of::<T>() does not compile.

Copy vs. Clone

Clone is an ordinary trait. It makes a duplicate when you call it, and the call may allocate and run arbitrary code. Copy is a marker that changes the meaning of assignment. For a Copy type, let b = a; copies the bits, and a stays usable. For all other types, the same line is a move (the central topic of Domain 1).

#[derive(Debug, Clone, Copy, PartialEq)]
struct Point { x: i32, y: i32 }

let a = Point { x: 1, y: 2 };
let b = a;              // bitwise copy, NOT a move
assert_eq!(a, b);       // `a` is still usable after the assignment

The compiler enforces two rules for Copy:

All fields must be Copy. A struct with a String field cannot be Copy (error[E0204]). A copy of the pointer bits would make two owners of one heap buffer.
No Drop implementation. A type cannot implement both Copy and Drop (error[E0184]). A bitwise copy of a value that has cleanup code would run that cleanup twice.

Clone is a supertrait of Copy (Copy: Clone). Thus a type that derives Copy must also implement Clone, usually with derive. These rules have useful results:

Copy composes structurally: tuples, arrays, and Options of Copy types are Copy.
&T is always Copy, and &mut T is never Copy. Two live unique borrows would break the aliasing rules.
Iterators have .copied() and .cloned(), so the code shows the cost of the conversion from &T to T.

Use Copy for small, plain-old-data types (a few machine words at most). Let all other types move.

Send and Sync: Auto Traits

Send means that a value is safe to move to a different thread. Sync means that a value is safe to share: &T can go to a different thread. The two are auto traits: the compiler implements them for each type in which all fields have them. In safe code, you do not opt in manually. A type opts out when it contains a field that is not thread-safe. One Rc field removes the markers from the full struct.

Figure: Auto-trait composition — one non-Send field removes the marker for the whole type

17_2_marker_traits_2

Common types, their markers, and the reasons:

Type	`Send`	`Sync`	Reason
`Rc<T>`	✗	✗	The reference count is not atomic
`RefCell<T>`	✓	✗	The borrow flag is not atomic. A move of the full cell is safe (if `T: Send`), but shared access is not
`const T` / `mut T`	✗	✗	The compiler cannot reason about raw aliasing
`Arc<Mutex<T>>`	✓	✓	Atomic count and synchronized access (if `T: Send`)

A pattern with no runtime cost makes compile-time assertions from these properties. The tokio and crossbeam crates and the tests of std use it:

// Each function compiles only for a T that has the marker. The body is empty.
fn assert_send<T: Send>() {}
fn assert_sync<T: Sync>() {}

// `Job` is a struct with three fields: u64, Vec<u8>, and Arc<Mutex<u64>>.
assert_send::<Job>();      // compiles, because each field of Job is Send
// assert_send::<Rc<u8>>();
// error[E0277]: `Rc<u8>` cannot be sent between threads safely

To opt out deliberately on stable Rust, add a PhantomData<*const ()> field (negative impls are unstable). This field has zero size, and its raw pointer type removes the two markers. It is the standard technique for a wrapper of an FFI handle that must stay on one thread.

Tutorial 6.8 covers the meaning of Send and Sync in concurrent code. The important point here is the mechanism: structural composition that the compiler checks fully at compile time.

Unpin, PhantomPinned, and PhantomData

Unpin and PhantomPinned

Unpin is a third auto trait. It means that a move of the value is always safe, also after you pin the value. Almost all types are Unpin. For these types, Pin<&mut T> is a wrapper with no effect (Pin::get_mut requires T: Unpin). The exception is a type that holds pointers into itself. The primary examples are the async futures that the compiler generates.

PhantomPinned is the marker that opts out. It is a zero-size field that removes Unpin. When such a value is behind Pin, safe code cannot move it again. Tutorials 15.2 and 15.3 cover Pin in depth. In this tutorial, it is one more example of the marker mechanism.

use std::marker::PhantomPinned;

struct Parser {
    buffer: String,
    _pinned: PhantomPinned,   // zero bytes, and Parser is now !Unpin
}
// The marker field adds no bytes to the struct.
assert_eq!(size_of::<Parser>(), size_of::<String>());

PhantomData: type-level bookkeeping in zero bytes

PhantomData<T> tells the compiler to treat the type as if it contained a T, but it stores no T. size_of::<PhantomData<T>>() is 0 for each T. It has three uses:

To use a type parameter that no field uses. Typed IDs, quantities with a unit tag, and typestate are examples. Id<User> and Id<Order> have the same u64 bits, but a mix of the two is error[E0308]. The sum of a Quantity<Meters> and a Quantity<Seconds> does not compile.
To connect a raw pointer to a lifetime. A struct that holds *const T and PhantomData<&'a [T]> borrows the same as &'a [T]. std::slice::Iter<'a, T> uses this technique: it uses raw pointers internally and stays safe.
To control variance and drop-checking. Select the form that agrees with the real use of T in your type:
- PhantomData<T>: the type behaves as an owner of a T.
- PhantomData<fn() -> T>: the type only produces T. With this form, the type stays Send, Sync, and Copy for each T.
- PhantomData<fn(T)>: the type consumes T.

Example 17_08 shows one more detail. derive(Clone, Copy) adds a bound (T: Copy) to each type parameter, also to a phantom parameter. Thus the typed-ID wrapper implements Clone, Copy, and PartialEq manually. Then it stays usable with tag types that implement no traits.

Summary

Marker	What it asserts	How a type gets it
`Sized`	The compiler knows the size	Automatic. It is an implicit bound on each `T`, and `?Sized` relaxes it
`Copy`	Assignment makes a valid copy of the bits	`derive`, if all fields are `Copy` and there is no `Drop`
`Clone`	Explicit duplication	`derive` or a manual `impl` (supertrait of `Copy`)
`Send`	A value can move to a different thread	Auto: all fields are `Send`
`Sync`	`&T` can go to a different thread	Auto: all fields are `Sync`
`Unpin`	A move is safe, also after you pin the value	Auto: all fields are `Unpin`
`PhantomPinned`	— (removes `Unpin`)	Add it as a zero-size field
`PhantomData<T>`	The type behaves as if it contained a `T`	Add it as a zero-size field for unused parameters, lifetimes, and variance

Key points:

Marker traits are properties, not interfaces. They have no methods, only a meaning.
Auto traits compose structurally: one field without the marker removes it from the composite type. A zero-size phantom field can remove a marker deliberately, or connect the type to a lifetime.
References to DSTs are fat: pointer and length, or pointer and vtable. This metadata holds the size information that a type without Sized does not have.
fn assert_send<T: Send>() {} gives a compile error when a type is not thread-safe as you expect. It is protection at no cost for public types.

Code Examples

File	Description
`17_05_sized_unsized.rs`	The implicit `Sized` bound, `?Sized` generics, `size_of_val`, and the sizes of thin and fat pointers
`17_06_copy_clone.rs`	`Copy` semantics and `Clone`, the rules for `Copy` (E0204 and E0184 as commented errors), structural composition, `copied()` and `cloned()`
`17_07_send_sync_unpin.rs`	Auto-trait composition, compile-time probes such as `assert_send`, the `Rc` and `RefCell` opt-outs, `PhantomData<*const ()>`, and `PhantomPinned`
`17_08_phantomdata.rs`	`PhantomData`: typed IDs, quantities with a unit tag, a lifetime for a raw pointer, variance forms, and zero-size assertions

17_05_sized_unsized.rs prints (on a 64-bit platform):

by_value(42u32)      = 42
by_value([1, 2, 3])  = [1, 2, 3]

describe over sized AND unsized types:
  u32 (sized)    value = 42, size_of_val = 4 bytes
  str            value = "héllo", size_of_val = 6 bytes
  [u8] slice     value = [1, 2, 3], size_of_val = 3 bytes
  dyn Debug      value = [1, 2, 3], size_of_val = 24 bytes

pointer sizes (machine word = 8 bytes):
  &u32           = 8 bytes (thin)
  &str           = 16 bytes (ptr + len)
  &dyn Debug     = 16 bytes (ptr + vtable)
  Box<dyn Debug> = 16 bytes (ptr + vtable)

All assertions passed.

17.3 · The Default Trait and Default Values

Domain 17 — Type System Utilities Duration: ~15 minutes Library components: std::default::Default

Introduction

Default is a small trait with one associated function, fn default() -> Self, and many parts of std use it. The trait gives a reasonable value of a type for the case where the caller does not specify one. Numbers give 0, bool gives false, String and each collection give an empty value, and Option gives None.

Much everyday Rust code uses this one function:

struct initialization with ..Default::default()
the #[derive(Default)] chain
Option::unwrap_or_default
the HashMap entry pattern entry(k).or_default()
mem::take, the standard technique to move a value out from behind &mut

This tutorial goes through these items in sequence, from the definition of defaults to the patterns that use them.

Figure: One trait, many consumers — the APIs that use T: Default

17_3_default_trait_1

Deriving Default

#[derive(Default)] implements default() as a call of Default::default() on each field. Use it when the default of each field is the correct default for the type. Examples are counters at zero, empty buffers, and options with no value.

#[derive(Debug, Default, PartialEq)]
struct RequestStats {
    requests: u64,                  // default: 0
    errors: u64,                    // default: 0
    last_path: String,              // default: ""
    peak_latency_ms: Option<u32>,   // default: None
}

let stats = RequestStats::default();
// stats is RequestStats { requests: 0, errors: 0, last_path: "", peak_latency_ms: None }

The derive requires that the type of each field implements Default. One field without it gives error[E0277]: the trait bound ... Default is not satisfied. Types such as TcpListener deliberately have no default, because there is no reasonable "default socket".

For an enum, you must tell the derive which variant is the default. Put #[default] on exactly one unit variant:

#[derive(Debug, Default, PartialEq)]
enum LogLevel {
    Debug,
    #[default]   // LogLevel::default() returns this variant
    Info,
    Warn,
    Error,
}

assert_eq!(LogLevel::default(), LogLevel::Info);

Manual Default and `..Default::default()`

When zero or empty is not a sensible initial state, implement Default manually. A server configuration with all fields at zero (port 0, empty host, zero workers) does not work. The manual implementation contains the values that are useful:

// ServerConfig has five fields: host: String, port: u16, workers: usize,
// verbosity: LogLevel (the enum above), and tls: bool.
impl Default for ServerConfig {
    fn default() -> Self {
        ServerConfig {
            host: String::from("127.0.0.1"),
            port: 8080,
            workers: 4,
            verbosity: LogLevel::default(), // uses the Default of the field type: Info
            tls: false,
        }
    }
}

The benefit is struct update syntax, which Rust uses in the place of keyword arguments with defaults. Callers set the fields that are important to them and take the other fields from the default:

let production = ServerConfig {
    host: String::from("0.0.0.0"),
    tls: true,
    ..Default::default()   // the other fields: port 8080, workers 4, verbosity Info
};

This pattern is a simple alternative to the builder pattern. If each field of a configuration type has an independent default that callers can override, Default with struct update often replaces a builder fully. When construction needs validation or ordering constraints, use a real builder.

Three forms call the same function:

Default::default() (the compiler infers the type from the context)
ServerConfig::default()
<ServerConfig as Default>::default()

Prefer the form with the type name when the type is not clear at the call site. The default_trait_access lint of Clippy recommends the same.

Where Default Is Most Useful: `unwrap_or_default` and `or_default`

Code rarely calls Default directly. Its value is in the std methods that have a T: Default bound. These methods use the default value when there is no other value.

Option::unwrap_or_default() changes an optional value into a definite value, and you do not write an alternative value at each call site. The method name says that an absent value means zero or empty. Result has the same method, and it combines well with parsing:

// line.bytes_sent: Option<&str>, the text of a log field that can be absent.
let bytes: u64 = line.bytes_sent
    .unwrap_or_default()              // Option<&str> -> &str:  None gives ""
    .parse()                          // &str -> Result<u64, ParseIntError>
    .unwrap_or_default();             // Result -> u64:  Err gives 0
// Some("512") gives 512. None gives 0, because "" does not parse.

The unwrap_or_* family has three members:

unwrap_or(v) is eager: the program makes v even when it does not use it.
unwrap_or_else(f) is lazy, and the closure can return an arbitrary value.
unwrap_or_default() is lazy and self-documenting: the alternative value is the canonical default of the type.

Entry::or_default() is the usual pattern for aggregation (tutorial 4.2 introduced the Entry API). It gets the value for a key and returns &mut to it. If the key is new, it inserts the default first:

// statuses: an array of u16 HTTP status codes: [200, 200, 404, 200, 500, 404].
let mut counts: HashMap<u16, u32> = HashMap::new();
for status in statuses {
    *counts.entry(status).or_default() += 1;   // a new counter starts at 0
}
// counts: 200 -> 3, 404 -> 2, 500 -> 1

// requests: an array of (user, path) pairs, each of type (&str, &str).
let mut by_user: HashMap<&str, Vec<&str>> = HashMap::new();
for (user, path) in requests {
    by_user.entry(user).or_default().push(path); // a new group starts as an empty Vec
}

or_default() is for the case where the initial value is empty or zero. Most real aggregation code is this case. Use or_insert(v) or or_insert_with(f) for an initial value that is not the default.

Default + `mem::take`: Moving Out of `&mut`

You cannot move a field out through &mut self. The move would make the struct invalid, and the borrow checker rejects it (error[E0507]: cannot move out of ... which is behind a mutable reference). mem::take(&mut place) is the solution. It returns the value and puts T::default() in its place, so the place is never invalid.

Figure: mem::take swaps a fresh default into the slot and gives you the original — no clone, no allocation

17_3_default_trait_2

Example 17_11 shows the emit-and-refill pattern. Real tokenizers and codecs use this pattern:

// A method of LineAccumulator. `self.current` is a String that holds the line so far.
fn push(&mut self, ch: char) -> Option<String> {
    if ch == '\n' {
        // Moves the String out and puts an empty String in its place.
        // The line is complete: the caller gets it, and the next line starts empty.
        Some(mem::take(&mut self.current))
    } else {
        self.current.push(ch);
        None
    }
}

The simple alternative is self.current.clone() and then clear(). It costs one allocation for each line, with no benefit.

Two related functions complete the group. mem::replace(&mut place, new) lets you specify the replacement, and it has no Default bound. take(p) is exactly replace(p, Default::default()). Option::take() puts None in the place of the value.

The same trait makes the reset pattern possible. In a reset(&mut self) method, *self = Self::default() resets each field, also the fields that you add later. Thus the method always agrees with the struct.

Default and `#[non_exhaustive]`

#[non_exhaustive] on a struct tells downstream crates that the struct can get more fields later. To enforce this, the compiler forbids two operations outside the crate that defines the struct: construction with a struct literal, and exhaustive destructuring. The first rule also applies to functional update:

// In crate `config_lib`:
#[non_exhaustive]
#[derive(Default)]
pub struct Options { pub retries: u32, pub verbose: bool }

// In YOUR crate. The two statements below do not compile:
let opts = Options { retries: 3, verbose: true };
// error[E0639]: cannot create non-exhaustive struct using struct expression

let opts = Options { retries: 3, ..Default::default() };
// error[E0639] also: an expression with `..` is still a struct expression

A downstream crate can create the struct only with the functions that the crate of the struct supplies. Default::default() is the most common one. Thus the pattern to configure such a struct is mutate-after-default:

let mut opts = config_lib::Options::default();   // all fields have their defaults
opts.retries = 3;                                // you can still write to a public field

The two features work together deliberately. With #[non_exhaustive], a new field is not a breaking change. Default makes sure that each field, present and future, has a value that your code does not name. std uses the pattern too: std::io::Empty and std::io::Sink are #[non_exhaustive] structs that implement Default.

The compile error occurs only between two crates. Thus the example binaries describe it in comments and do not show it.

Summary

Concept	Key point
`Default::default()`	The canonical value of the type when the caller does not specify one
`#[derive(Default)]`	Each field gets its default. Each field type must implement `Default`
`#[default]` on enums	Marks the default unit variant for the derive
Manual `impl Default`	For defaults with specific values (port 8080, 4 workers) where zero or empty is wrong
`..Default::default()`	Struct update syntax: set some fields and take the others from the default. A simple alternative to a builder
`unwrap_or_default()`	On `Option` and `Result`: an absent value or an error means zero or empty. Self-documenting
`entry(k).or_default()`	Gets the value or inserts the default, and returns `&mut`. The pattern for counters and groups
`mem::take`	Moves a value out of `&mut` and puts `T::default()` in its place. Equal to `replace(p, Default::default())`
Reset pattern	`*self = Self::default()` resets all fields, also the fields that you add later
`#[non_exhaustive]`	Blocks downstream struct literals (also with `..`). `Default` is the approved method to create the value

Code Examples

File	Description
`17_09_derive_manual_default.rs`	How to derive `Default` (structs, enums with `#[default]`), manual implementations, `..Default::default()`, and `T: Default` in generic bounds
`17_10_default_in_apis.rs`	`unwrap_or_default` on `Option` and `Result`, parse chains with a default value, and `Entry::or_default` for counters and groups
`17_11_mem_take_reset.rs`	A parser that emits and refills with `mem::take`, `mem::replace`, `Option::take`, and the reset pattern `*self = Self::default()`

17_11_mem_take_reset.rs prints:

emitted ["GET /a", "POST /b"] (2 lines), accumulator ready for more
mem::take moved [80, 95, 71] out; original now [100]
mem::replace swapped in [0, 0, 0], returned [1, 2, 3]
Option::take() -> Some("db-7"), slot now None
session.reset() -> Session { user: None, permissions: [], request_count: 0 }

All assertions passed.

18.1 · Integer Methods: Checked, Wrapping, Saturating, Overflowing

Domain 18 — Numeric Types and Math Duration: ~15 minutes Library components: Primitive integer types, std::num::NonZero*, std::num::Wrapping, std::num::Saturating

Introduction

The primitive integer types of Rust have an unusually large set of methods. The standard library treats the result of arithmetic that does not fit as an API design decision, not as an accident. This tutorial shows:

The four overflow strategies (checked_*, wrapping_*, saturating_*, overflowing_*) on the same operation. The Wrapping<T> and Saturating<T> wrapper types make a strategy a property of the type.
pow and isqrt for integer powers and integer square roots.
How to examine bits: count_ones, leading_zeros, trailing_zeros, rotate_left/rotate_right, reverse_bits, swap_bytes.
The bit-query family that Rust 1.97 stabilized: bit_width, highest_one, lowest_one, isolate_highest_one, isolate_lowest_one.
Endian conversion (to_le_bytes, from_be_bytes, and the related methods) as the primary tool to encode binary protocols. [u8]::escape_ascii shows the results.
The NonZero integer types, and the niche optimization that makes Option<NonZeroU32> the same size as u32.

The common idea: Rust makes you select a strategy where C silently selects one for you.

The Four Overflow Strategies

The plain + operator on integers has two behaviors when the result does not fit. Assume that a u8 variable holds 250 and the program adds 10 at run time. In a debug build, the addition panics ("attempt to add with overflow"). In a release build, the addition wraps to 4. If the compiler can calculate the result at compile time, it rejects the addition: the arithmetic_overflow lint is deny-by-default.

This difference is deliberate. The overflow check is a tool to find bugs, and the wrap in a release build is a compromise, not a feature. When overflow is possible, use the explicit methods. They say what must occur, and they do the same in every build profile.

Figure: Choosing an overflow strategy

18_1_integer_methods_1

The four strategies exist for add, sub, mul, div, and pow on every integer type. Some other operations have only three of them: for example, there is no saturating_shl.

let x: u8 = 250;                              // u8::MAX is 255
assert_eq!(x.checked_add(10), None);          // None: 260 does not fit in a u8
assert_eq!(x.wrapping_add(10), 4);            // 260 mod 256
assert_eq!(x.saturating_add(10), 255);        // stops at u8::MAX
assert_eq!(x.overflowing_add(10), (4, true)); // (wrapped value, overflow flag)

Remember one special case for signed integers. The mathematical result of i32::MIN / -1 is one more than i32::MAX, so the plain / operator panics in all build profiles. The explicit methods apply their strategy:

checked_div returns None.
saturating_div returns i32::MAX.
wrapping_div returns i32::MIN.

Write the limits as associated constants: i32::MAX, u8::MIN. Rust 1.99 deprecates the legacy forms, so the compiler gives a warning for each use of them:

the module constants (std::i32::MAX)
the functions i32::max_value() and i32::min_value()

The same rule applies to the float module constants such as std::f64::EPSILON. Write f64::EPSILON.

pow takes a u32 exponent. On overflow, pow behaves as * does (it panics in a debug build), and checked_pow applies a strategy. isqrt (1.84) returns the floor of the square root as an integer, and it does not convert through a float. This is important for large u64 values that f64 cannot represent exactly.

`Wrapping<T>` and `Saturating<T>`: The Strategy as a Type

When every operation on a value must use the same strategy, a wrapping_mul call at each location adds repetition. std::num::Wrapping and std::num::Saturating are wrapper types: you put the integer in the wrapper one time. Then the ordinary operators apply the strategy. The FNV-1a hash is the typical example, because its multiplication must overflow:

use std::num::Wrapping;

let mut hash = Wrapping(0xcbf2_9ce4_8422_2325_u64); // FNV offset basis
let prime = Wrapping(0x0000_0100_0000_01b3_u64);    // FNV prime
for &byte in b"rust" {                              // byte: u8
    hash ^= Wrapping(u64::from(byte));              // widen the u8 to u64, then XOR
    hash *= prime; // ordinary `*`: it wraps on overflow and does not panic
}
assert_eq!(hash.0, 0xbffe_df1f_6f66_c727);          // .0 is the plain u64

Saturating<T> (1.74) does the same for the saturating strategy. The example is a health meter that cannot go past the limits of its type:

use std::num::Saturating;

let mut health = Saturating(200u8);
health += Saturating(100); // 200 + 100 stops at u8::MAX: 255
health -= Saturating(200); // 255 - 200 = 55
health -= Saturating(200); // 55 - 200 stops at 0, it does not wrap
assert_eq!(health.0, 0);   // .0 is the plain u8

18_01_overflow_strategies.rs prints:

checked_add:     250 + 10 = None
wrapping_add:    250 + 10 = 4
saturating_add:  250 + 10 = 255
overflowing_add: 250 + 10 = (4, overflowed: true)

i32::MIN / -1: checked=None, saturating=2147483647
2^10 = 1024, isqrt(17) = 4

FNV-1a("rust") = 0xbffedf1f6f66c727
Saturating health after +100, -200, -200: 0

All assertions passed.

Bit Counting and the 1.97 Bit-Query Family

The classic count methods are count_ones (population count), count_zeros, leading_zeros, and trailing_zeros. Each one typically compiles to one CPU instruction or a small number of instructions. Before Rust 1.97, you derived the answer to a common question such as "which is the highest set bit?" from these methods. It is easy to make an error in that arithmetic: 31 - x.leading_zeros() is wrong for zero. The bit-query family gives each question a name:

Method	Result	For `0b1010_1000u32`	For `0`
`bit_width()`	The number of bits that the value needs	`8`	`0`
`highest_one()`	The index of the highest set bit	`Some(7)`	`None`
`lowest_one()`	The index of the lowest set bit	`Some(3)`	`None`
`isolate_highest_one()`	A mask that keeps only the highest set bit	`0b1000_0000`	`0`
`isolate_lowest_one()`	A mask that keeps only the lowest set bit	`0b0000_1000`	`0`

A job scheduler keeps a u32 bitmask of ready queues. If bit i is set, queue i has work. A short loop serves the queues in order of priority, highest first:

let mut pending: u32 = 0b1010_1000;               // queues 3, 5, and 7 have work
let mut served = Vec::new();                      // the order of service
// highest_one() returns None when the mask is 0, and the loop stops.
while let Some(queue) = pending.highest_one() {
    served.push(queue);                           // 7, then 5, then 3
    pending &= !pending.isolate_highest_one();    // clear the served bit
}
assert_eq!(served, [7, 5, 3]);

On the NonZero integer types, the same queries return plain values, not Option. The type system already excludes the empty mask. isolate_* returns a NonZero value, because a mask with one set bit is never zero.

The first lines of the output of 18_02_bit_queries.rs:

ready mask       = 0b10101000
count_ones=3, leading_zeros=24, trailing_zeros=3
bit_width=8, highest_one=Some(7), isolate_highest_one=0b10000000
service order    = [7, 5, 3]

Rotations, `reverse_bits`, and `swap_bytes`

The << operator discards the bits that it moves past the end. rotate_left and rotate_right put those bits back at the opposite end. Ciphers and hash functions use rotations very frequently. For example, SipHash (the default HashMap hasher) uses rotations:

let pattern: u8 = 0b1100_0001;
assert_eq!(pattern.rotate_left(2), 0b0000_0111); // the top two bits go to the bottom
assert_eq!(pattern.rotate_left(8), pattern);     // 8 positions on a u8: no change

It is easy to confuse two operations that reverse an order:

reverse_bits reverses the order of all the bits. CRC algorithms and some DSP formats need it.
swap_bytes reverses the order of the bytes. This is endianness conversion, the subject of the next slide.

let x: u16 = 0b1000_0000_0000_0010;                   // bits 15 and 1 are set
assert_eq!(x.reverse_bits(), 0b0100_0000_0000_0001);  // bit 15 goes to bit 0, bit 1 to bit 14
assert_eq!(x.swap_bytes(),   0b0000_0010_1000_0000);  // the two bytes change places

The rest of the output of 18_02_bit_queries.rs follows. The line about 3000 comes from a part of the example that uses bit_width to round a buffer size up to a power of two:

rotate_left(2):  0b11000001 -> 0b00000111
reverse_bits:    0b1000000000000010 -> 0b0100000000000001
swap_bytes:      0b1000000000000010 -> 0b0000001010000000

next power of two >= 3000: 4096

All assertions passed.

Every binary file format and every network protocol must specify a byte order. Big-endian ("network order") stores the most significant byte first. Little-endian (x86, Apple silicon, most ARM configurations) stores it last. The *_bytes methods are the primary tool of the standard library for this conversion:

to_be_bytes and to_le_bytes return a fixed-size array ([u8; 4] for u32).
from_be_bytes and from_le_bytes decode such an array.

The result is the same on every platform.

Figure: The same u32 in two byte orders

Endianness — the same u32 in two byte orders

To encode a telemetry record with a fixed layout, call extend_from_slice for each field. To decode, take a sub-slice of the buffer for each field. Then convert that &[u8] to a fixed-size array with try_into():

// In Record::encode: buf is a Vec<u8>, and self.timestamp is a u32.
buf.extend_from_slice(&self.timestamp.to_le_bytes());      // appends 4 bytes

// In Record::decode: buf is a &[u8], and the function returns Option<Record>.
// try_into() converts the 4-byte slice to [u8; 4].
// `.ok()?` returns None from the function if the length is wrong.
let ts = u32::from_le_bytes(buf[4..8].try_into().ok()?);   // ts: u32

to_ne_bytes and from_ne_bytes use the native endianness. Use them only for memory that stays on the same machine. The methods to_be, to_le, from_be, and from_le operate on the full value: they arrange the bytes but do not make an array. The byte-array forms are usually clearer.

[u8]::escape_ascii (1.60) shows a raw byte buffer in a form that a person can read. Printable ASCII stays as it is. The other bytes become escapes, most of them in the \xNN form. Thus you can log binary payloads and you do not need a hex-dump helper.

18_03_endian_encoding.rs prints:

0x1F90 BE bytes = [1f, 90]
0x1F90 LE bytes = [90, 1f]

wire bytes      = [ef, be, 01, 07, 80, 51, 01, 00, 00, 00, bc, 41]
escape_ascii    = "\xef\xbe\x01\x07\x80Q\x01\x00\x00\x00\xbcA"

decoded         = Record { magic: 48879, flags: 1, channel: 7, timestamp: 86400, reading: 23.5 }
BE bytes misread as LE: 2152792320 (expected 86400)

All assertions passed.

`NonZero` Types and the Niche Optimization

std::num::NonZeroU32 puts the rule "this value is never zero" in the type system. Every integer type has an equivalent NonZero type. The constructor does the check one time, at the boundary: NonZeroU32::new(0) is None. All the code after the boundary can assume the invariant.

The benefit is the niche optimization. Option<u32> needs 8 bytes: every u32 bit pattern is a valid value, so None needs a separate tag. NonZeroU32 does not use one bit pattern (zero), and the compiler stores None in that pattern:

Figure: Option<u32> vs Option<NonZeroU32> memory layout

NonZero niche optimization

use std::num::NonZeroU32;

// `size_of` is in the prelude, so it needs no `use` line.
assert_eq!(size_of::<Option<u32>>(), 8);        // 4 bytes of value + the tag and its padding
assert_eq!(size_of::<Option<NonZeroU32>>(), 4); // None is the all-zero pattern

A wrapper struct keeps the niche: with struct RowId(NonZeroU64), Option<RowId> is 8 bytes. The same mechanism makes Option<&T> and Option<Box<T>> the size of a pointer. The NonZero types have more properties that help:

NonZeroU32::MIN is 1 (not 0). The signed variants have the full range of the primitive type, without zero.
new, trailing_zeros, ilog2, and is_power_of_two are const fn. Thus a constant that carries the invariant needs no lazy initialization.
ilog2 returns a plain u32, not an Option, because a nonzero value always has a logarithm.
The division of a primitive by a NonZero divisor cannot panic. Thus u32 / NonZeroU32 exists as a Div implementation that cannot fail, and you do not need checked_div.

18_04_nonzero_niche.rs prints:

NonZeroU32::new(42) = 42, new(0) = None

size_of: u32=4, Option<u32>=8, Option<NonZeroU32>=4
NonZeroU32::MIN = 1, NonZeroI32::MIN = -2147483648

find_row(9001) = Some(RowId(9001))
find_row(0)    = None

All assertions passed.

Summary

Concept	Key point
`checked_*`	Overflow becomes `None`, and the caller must decide what to do
`wrapping_*`	Modular arithmetic, the same in debug builds and release builds
`saturating_*`	Stops at `MIN`/`MAX`. For gauges, meters, and counters
`overflowing_*`	Returns `(wrapped, bool)`. It is the primitive operation behind the other three
`Wrapping<T>` / `Saturating<T>`	The type applies one strategy to every operator
`pow` / `isqrt`	Integer power and integer square root (no conversion through a float)
`bit_width`, `highest_one`, `lowest_one`, `isolate_*` (1.97)	Named bit queries. `highest_one`/`lowest_one` return `Option` on primitives and plain values on `NonZero`
`rotate_left` / `reverse_bits` / `swap_bytes`	Rotate the bits, reverse the bits, reverse the bytes
`to_le_bytes` / `from_be_bytes` etc.	Binary encoding that is the same on every platform. The primary tools for protocols
`[u8]::escape_ascii`	Shows a raw byte buffer in a form that a person can read
`NonZero*`	One zero check at the boundary. `Option<NonZero*>` has no size cost
Niche optimization	The unused zero pattern stores `None`. Wrapper types keep the niche

Code Examples

File	Description
`18_01_overflow_strategies.rs`	The four strategies on one operation, `pow`/`isqrt`, `Wrapping<T>` (FNV-1a), `Saturating<T>`
`18_02_bit_queries.rs`	Classic count methods, the 1.97 bit-query family, `NonZero` variants, rotations, `reverse_bits`/`swap_bytes`
`18_03_endian_encoding.rs`	Round trip of a little-endian record (encode, decode), `to_ne_bytes`, `escape_ascii`, a read with the wrong byte order
`18_04_nonzero_niche.rs`	`NonZero` construction, size assertions for the niche, `MIN`/`MAX`, const methods, division that cannot fail

18.2 · Floating-Point: IEEE 754, Special Values, and Math Constants

Domain 18 — Numeric Types and Math Duration: ~15 minutes Library components: f32, f64, std::f32::consts, std::f64::consts

Introduction

f32 and f64 are IEEE 754 floating-point numbers. Their unusual behaviors (0.1 + 0.2 != 0.3, NaN != NaN, a negative zero) all come from one design. The design divides a fixed number of bits into sign, exponent, and mantissa. This tutorial shows:

The method families:
- rounding (floor, ceil, round, trunc, fract, round_ties_even)
- sign manipulation (abs, signum, copysign)
- roots and powers (sqrt, cbrt, powi, powf, hypot)
- mul_add (fused multiply-add, const-evaluable since 1.94)
- the 1.98 algebraic operators that permit reassociation
The transcendental methods: ln, log2, log10, exp, exp2, and the trigonometric family, which includes atan2.
The constants in std::f64::consts, with EULER_GAMMA and GOLDEN_RATIO (stabilized in 1.94).
The special values (NAN, INFINITY, NEG_INFINITY, signed zero, subnormals) and the predicates that detect them (is_nan, is_finite, is_subnormal, classify).
total_cmp: the total order that makes floats sortable, NaN included.

The Anatomy of an `f64`

Every f64 has 64 bits: 1 sign bit, 11 exponent bits (biased by 1023), and 52 mantissa bits. The exponent field selects a mode: IEEE 754 reserves its extreme values for zeros, subnormals, infinities, and NaN. If you remember this layout, the unusual float behaviors become direct results of it.

Figure: IEEE 754 f64 bit layout and what the exponent selects

f64 bit layout

Two facts about the structure cause all the other behaviors:

Floats have logarithmic spacing. Half of all f64 values are between −1 and 1, and the gap between adjacent floats increases with magnitude (~2 × 10⁻⁶ near 10¹⁰). Thus a comparison with a tolerance must include the scale of the values.
== is not the same relation as bit equality. -0.0 == 0.0 is true, but the bits are different. NaN == NaN is false, but the bits are possibly identical.

Rounding and Sign Manipulation

floor, ceil, trunc, and round are four rounding modes, and each one rounds in a different direction. The differences show only on negative values and on ties:

Method	Direction	`2.7`	`-2.7`	`2.5`
`floor`	toward −∞	`2.0`	`-3.0`	`2.0`
`ceil`	toward +∞	`3.0`	`-2.0`	`3.0`
`trunc`	toward zero	`2.0`	`-2.0`	`2.0`
`round`	nearest, ties away from zero	`3.0`	`-3.0`	`3.0`
`round_ties_even` (1.77)	nearest, ties to even	`3.0`	`-3.0`	`2.0`

round_ties_even is banker's rounding. IEEE 754 and most finance rules expect it, because it does not bias sums upward. fract returns the fractional part that trunc removes (x == x.trunc() + x.fract()).

The sign methods are a small family of their own:

abs
signum (0.0.signum() is 1.0, because positive zero counts as positive)
copysign, which gives a the sign of b and needs no if

let step = 0.25_f64;
// The result has the magnitude of `step` and the sign of the argument.
assert_eq!(step.copysign(-8.0), -0.25);

Roots, Powers, `hypot`, and `mul_add`

IEEE 754 guarantees that the result of sqrt is correctly rounded (64.0.sqrt() == 8.0 exactly). powf calculates through exp and ln and has no such guarantee, so compare its results with a tolerance. powi takes an integer exponent. It is cheaper, and it is exact for small powers of exact values. cbrt accepts negative inputs ((-27.0).cbrt() == -3.0), where sqrt returns NaN.

Two methods exist because the textbook formula fails in floating-point arithmetic:

hypot(a, b) calculates sqrt(a² + b²) without intermediate overflow. The square of 3e200 overflows to infinity. hypot rescales internally and returns approximately 5e200 for the 3-4-5 triangle at that scale.
a.mul_add(b, c) calculates a*b + c as one fused operation with one rounding step. The plain expression 0.1 * 10.0 - 1.0 first rounds 0.1 * 10.0 to exactly 1.0, which destroys the residue. 0.1f64.mul_add(10.0, -1.0) keeps the residue: 5.55e-17, the true error of "0.1 × 10". Since 1.94, mul_add is const-evaluable, so tables that the compiler calculates get full FMA precision.

Since 1.98, five methods calculate the same IEEE operations as + - * / %: algebraic_add, algebraic_sub, algebraic_mul, algebraic_div, and algebraic_rem. But they permit the compiler to apply algebraic identities that floats do not obey: reassociation, distribution, vectorization. a + b + c + d is strictly ((a+b)+c)+d. The compiler may group a chain of algebraic_add calls as (a+b)+(c+d). The results are not guaranteed to be bit-identical across compilers, but the methods never introduce UB. Use them in hot reductions where the freedom of the optimizer is more important than a stable sum.

18_05_float_arithmetic.rs prints:

-2.7: floor=-3, ceil=-2, trunc=-2, round=-3
2.5: round=3, round_ties_even=2
0.25.copysign(-8.0) = -0.25
2^10 = 1024, sqrt(2) = 1.4142135623730951
naive sqrt(a²+b²) = inf, hypot = 5.000e200

0.1*10 - 1  plain = 0e0, mul_add = 5.551115123125783e-17
marathon = 42.194988 km (computed at compile time)

All assertions passed.

18_14_algebraic_ops.rs uses values that are exact, so each algebraic result is equal to the result of the operator. It prints:

f64: 1+2=3, 6-2=4, 3*4=12, 8/2=4, 7%3=1
f32: 1.5+2.5=4, 9/3=3
strict sum = 10, algebraic sum = 10

All assertions passed.

Logarithms, Trigonometry, and the Constants

The transcendental methods are ln, log2, log10, exp, exp2, and the trigonometric family. They are typically accurate to one unit in the last place, but they are rarely bit-exact. Compare their results with a tolerance, never with ==. Three practical uses:

Decibels are 10.0 * ratio.log10().
Half-life decay is (-t * LN_2 / t_half).exp().
atan2(y, x) is the arctangent that keeps the quadrant, and every heading calculation needs it. atan(y/x) loses the quadrant and divides by zero when x == 0.

The trigonometric methods use radians. to_radians and to_degrees convert. PI.sin() is not 0.0 but 1.2e-16, because consts::PI is the closest f64 to π, not π itself.

Rust 1.94 added two constants to std::f64::consts:

EULER_GAMMA (γ ≈ 0.5772) is the Euler–Mascheroni constant, the limit of H_n - ln(n). It appears in the bounds on harmonic numbers in the analysis of algorithms.
GOLDEN_RATIO (φ ≈ 1.6180) is (1 + √5)/2, and it satisfies φ² = φ + 1. The ratios of consecutive Fibonacci numbers converge to it.

use std::f64::consts;

let n = 100_000_u32;
// harmonic is H_n = 1 + 1/2 + ... + 1/n.
let harmonic: f64 = (1..=n).map(|k| 1.0 / f64::from(k)).sum();
let estimate = harmonic - f64::from(n).ln();   // H_n - ln(n), approximately 0.57722
// The error decreases as 1/(2n): approximately 5e-6 for n = 100_000.
assert!((estimate - consts::EULER_GAMMA).abs() < 1e-5);

18_06_float_transcendental.rs prints:

e = 2.718281828459045, ln(e) = 1
doubling power  = +3.01 dB
C-14 after 11460y = 25.0% remains
sin(PI)         = 1.2246467991473532e-16  (PI is not exactly pi!)
heading         = 45.0° (atan2 keeps the quadrant)

EULER_GAMMA     = 0.5772156649
H_n - ln(n)     = 0.5772206649 (n = 100000)
GOLDEN_RATIO    = 1.6180339887, phi^2 - phi - 1 = 0e0

All assertions passed.

Special Values

NaN is the result of undefined operations (0.0/0.0, inf - inf, (-1.0).sqrt()). It compares unequal to everything, itself included: the implementation of is_nan is x != x. NaN also propagates: one NaN contaminates every later calculation that uses it. Thus check the values at the boundary where NaN can first appear.

Infinities come from overflow (f64::MAX * 2.0) and from division by zero. A float division by 0.0 does not panic, but an integer division by zero does. Infinity behaves as a valid extreme value until you combine infinities, which gives NaN.

Signed zero: -0.0 == 0.0 is true, but to_bits() gives different values, and 1.0 / -0.0 is NEG_INFINITY. The sign stays through underflow and shows again in division.

Subnormals fill the gap between zero and MIN_POSITIVE (~2.2 × 10⁻³⁰⁸). They exchange precision for range, so very small differences do not become zero immediately (gradual underflow). is_subnormal detects them. classify returns one of the five FpCategory variants (Nan, Infinite, Zero, Subnormal, Normal). is_finite is the usual check, and it accepts the last three.

EPSILON is the gap between 1.0 and the next larger float. A plain < EPSILON tolerance is applicable only to values near 1.0. At 10¹⁰, even adjacent floats are ~2 × 10⁻⁶ apart. Thus multiply the tolerance by the magnitude of the operands (relative comparison).

18_07_float_special_values.rs prints:

NAN == NAN      -> false
1.0/0.0         -> inf, MAX*2 -> inf
-0.0 bits       = 0x8000000000000000
0.1 + 0.2       = 0.30000000000000004 (!= 0.3 exactly)
MIN_POSITIVE/8  = 2.781342323134e-309 (subnormal)
    2.35e1  finite=true   Normal
       NaN  finite=false  Nan
       inf  finite=false  Infinite
      -0e0  finite=true   Zero
2.781342323134e-309  finite=true   Subnormal

All assertions passed.

`total_cmp`: Making Floats Sortable

NaN has no order relation to the other values, so f64 implements only PartialOrd and not Ord. This has two effects:

vec.sort() does not compile (the trait bound `f64: Ord` is not satisfied).
The common workaround sort_by(|a, b| a.partial_cmp(b).unwrap()) panics on the first NaN in real data.

total_cmp (1.62) implements the totalOrder predicate of IEEE 754, in which every value has a position:

Figure: The total_cmp ordering of all f64 values

18_2_floating_point_1

// Latency samples. One measurement failed and recorded NaN.
let mut samples = vec![12.5_f64, 3.1, f64::NAN, 8.4, 0.9, 42.0];
samples.sort_by(f64::total_cmp);   // does not panic, and the NaN goes to the end
// samples is now [0.9, 3.1, 8.4, 12.5, 42.0, NaN]

total_cmp separates values that == cannot separate: -0.0 comes before +0.0. Thus it is a consistent key for binary_search, dedup, and map wrappers. For the maximum and the minimum, select the NaN behavior deliberately:

A fold with f64::max ignores NaN (it returns the other operand).
Iterator::max_by(f64::total_cmp) returns NaN (NaN sorts above +inf).

Tutorial 13.1 explains the PartialOrd/Ord hierarchy that total_cmp relates to.

18_08_total_cmp_sorting.rs prints:

partial_cmp().unwrap() sort with NaN: panicked = true
total_cmp sort: [0.9, 3.1, 8.4, 12.5, 42.0, NaN]
p50 latency   = 8.4 ms (5 valid samples)
fold(f64::max) = 42, max_by(total_cmp) = NaN

All assertions passed.

Summary

Concept	Key point
Bit layout	Sign, exponent, mantissa. The extreme exponent values encode the special values
`floor` / `ceil` / `trunc` / `round`	Four directions. The differences show on negative values and ties
`round_ties_even` (1.77)	Banker's rounding, which does not bias sums
`copysign`	Magnitude of `self`, sign of the argument
`sqrt` vs `powf`	`sqrt` is correctly rounded. `powf` is not, so compare with a tolerance
`hypot`	`sqrt(a² + b²)` without intermediate overflow
`mul_add`	Fused multiply-add with one rounding step. `const`-evaluable since 1.94
`algebraic_*` (1.98)	Same operations as `+ - * / %`, but the compiler may reassociate and vectorize
`EULER_GAMMA`, `GOLDEN_RATIO` (1.94)	New constants in `std::f64::consts`
NaN	Unequal to everything, itself included. It propagates
Signed zero	`==`-equal to `+0.0`, but the bits are different. The sign shows again in `1/x`
Subnormals	Gradual underflow below `MIN_POSITIVE`. Detect them with `is_subnormal` or `classify`
`EPSILON`	Meaningful only near 1.0. Use a relative tolerance at other scales
`total_cmp` (1.62)	IEEE totalOrder: sortable floats, with NaN at a deterministic position

Code Examples

File	Description
`18_05_float_arithmetic.rs`	Rounding modes, `fract`, sign manipulation, roots/powers, `hypot`, const `mul_add`
`18_14_algebraic_ops.rs`	`algebraic_{add,sub,mul,div,rem}` (1.98) on `f32`/`f64`
`18_06_float_transcendental.rs`	`ln`/`exp`/`log2`/`log10`, trigonometry and `atan2`, `EULER_GAMMA` and `GOLDEN_RATIO`
`18_07_float_special_values.rs`	NaN, infinities, signed zero, the limits of an `EPSILON` tolerance, subnormals, `classify`
`18_08_total_cmp_sorting.rs`	Why `sort()` does not compile, the `partial_cmp().unwrap()` panic, a sort with `total_cmp`, the NaN behavior of each maximum method

18.3 · Parsing Numbers: FromStr, Radix, and Formatting

Domain 18 — Numeric Types and Math Duration: ~15 minutes Library components: std::str::FromStr, integer and float from_str_radix, std::num::ParseIntError, std::num::ParseFloatError, std::num::IntErrorKind

Introduction

Numbers go into and out of a program as text: config files, CLI arguments, wire formats, log lines. This tutorial shows the full round trip and the checks that keep it safe:

str::parse and the FromStr trait behind it, and how to implement FromStr for your own types.
Exactly what the integer parser and the float parser accept, and what they reject (no whitespace, no underscores, no 0x prefixes).
from_str_radix for binary, octal, hex, and every base up to 36, which includes NonZero*::from_str_radix (1.98).
ParseIntError::kind() and IntErrorKind: how to convert parse failures to precise messages for the user.
How to format numbers as text: width, alignment, zero padding, signs, precision, and the radix specifiers.
How to convert between integer sizes safely with TryFrom, the checked alternative to as. Tutorial 1.3 covers the conversion traits in general, and this tutorial covers the numeric cases.

`str::parse` and `FromStr`

parse is a thin generic wrapper: s.parse::<T>() only calls T::from_str(s). There is no other mechanism. Thus each type that implements FromStr gets:

the "...".parse() call
errors that work with ?
compatibility with every generic API that takes T: FromStr

Figure: The parse → FromStr plumbing

18_3_parsing_numbers_1

parse is generic over its return type, so you must supply the type with a turbofish or with an annotated binding:

let port = "8080".parse::<u16>().unwrap();   // turbofish: port is a u16
let retries: i32 = "3".parse().unwrap();     // the annotation selects i32

The integer parser is strict. It accepts an optional + or - sign and then ASCII digits, and nothing else:

The parser does not trim whitespace, so " 42" fails. The usual pattern is to call trim() first.
Underscores and 0x prefixes are conveniences of source code. They do not exist in data.

The float parser accepts more. Scientific notation, inf, and NaN (case-insensitive) all parse. Overflow becomes ±inf and is not an error. Thus malformed text is almost the only cause of a ParseFloatError.

Implementing `FromStr` for Your Own Types

A config loader reads WIDTHxHEIGHT strings. It implements FromStr one time and gets structured errors that the caller can distinguish:

// Resolution is a struct with two u32 fields: width and height.
// ParseResolutionError is an enum: MissingSeparator or BadNumber(ParseIntError).
impl FromStr for Resolution {
    type Err = ParseResolutionError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        // split_once returns None if there is no 'x'. ok_or changes None to the error.
        let (w, h) = s.split_once('x').ok_or(ParseResolutionError::MissingSeparator)?;
        // Each `?` converts a ParseIntError to BadNumber through the From impl.
        Ok(Self { width: w.trim().parse()?, height: h.trim().parse()? })
    }
}

The error type implements From<ParseIntError>, so ? converts the inner failures automatically. The benefit is composition: one collect parses a full config line.

// str::parse calls Resolution::from_str for each item.
// The first Err stops the collect and becomes the result. Here the result is Ok.
let modes: Result<Vec<Resolution>, _> =
    "640x480,1280x720,1920x1080".split(',').map(str::parse).collect();

18_09_parse_fromstr.rs prints:

parsed port=8080, retries=3
" 42" as i32   -> Err("invalid")
"6.02e23"      -> Ok(6.02e23)
"1920x1080"    -> Resolution { width: 1920, height: 1080 }
parsed 3 display modes

All assertions passed.

`from_str_radix`: Any Base from 2 to 36

i32::from_str_radix(src, radix) parses text in bases 2 to 36, and every integer type has the same function. A radix outside that range panics, because it is an error of the programmer, not of the input. Radix 10 behaves exactly as parse does. Above base 10, letters are digits (a=10 … z=35, case-insensitive). For this reason, base 36 is popular for compact IDs.

assert_eq!(u32::from_str_radix("ff", 16), Ok(255));    // hex
assert_eq!(u32::from_str_radix("1010", 2), Ok(10));    // binary
assert_eq!(u32::from_str_radix("755", 8), Ok(493));    // octal: 493 is 0o755
assert_eq!(i32::from_str_radix("-7f", 16), Ok(-127));  // a signed type accepts '-'

One mistake is common: "0x1F" fails with InvalidDigit. The radix comes from the argument, so the prefix is only an invalid character. Remove it with strip_prefix("0x"). Two common applications are short:

Parse each two-digit slice of a #RRGGBB hex color with u8::from_str_radix(.., 16).
Parse a "755" file mode with radix 8. The bits of the result then align with the Unix permission mask.

For the opposite direction, use the {:x}/{:o}/{:b} format specifiers (slide 6).

Since 1.98, the same function exists on the NonZero* types. NonZeroU32::from_str_radix("FF", 16) is Ok(255), and "0" is a parse error (IntErrorKind::Zero). The parser checks the nonzero invariant at the boundary, so a successful result is already nonzero.

18_10_from_str_radix.rs prints:

ff(16)=Ok(255) 1010(2)=Ok(10) zz(36)=Ok(1295)
"0x1F" raw     -> Err; after strip_prefix -> 31
#1E90FF        -> r=30, g=144, b=255
"755" (oct)    -> owner=rwx group=r-x other=r-x

All assertions passed.

18_15_nonzero_from_str_radix.rs prints:

FF(16)=Ok(255) 1010(2)=Ok(10) 755(8)=Ok(493)
from_str_radix("0", 10) = Err(ParseIntError { kind: Zero })
worker_threads = 16

All assertions passed.

`IntErrorKind`: Precise Parse Errors

"Port missing", "port is not a number", and "port too large" need three different messages. ParseIntError::kind() (1.55) distinguishes them, and you do not have to scan the string again:

Figure: How integer parse failures map to IntErrorKind

18_3_parsing_numbers_2

// err: &ParseIntError, from a failed "...".parse::<NonZeroU16>().
// The match gives the message for the user, a &'static str.
match err.kind() {
    IntErrorKind::Empty        => "no number given",
    IntErrorKind::InvalidDigit => "not a valid number",
    IntErrorKind::PosOverflow  => "number too large for this field",
    IntErrorKind::NegOverflow  => "number too small for this field",
    IntErrorKind::Zero         => "zero is not allowed here",
    _                          => "invalid number", // mandatory: the enum is non_exhaustive
}

Three details are important:

IntErrorKind is #[non_exhaustive], so the wildcard arm is mandatory.
"-1".parse::<u8>() gives InvalidDigit, not NegOverflow. The sign is not a valid character for an unsigned type.
Zero occurs only when you parse into a NonZero* type ("0".parse::<NonZeroU16>()).

from_str_radix returns the same error type and the same kinds. Match on kind(), never on the Display text, because the text of the message is not a stable API.

18_11_int_error_kind.rs prints:

"8080" -> port 8080
"" -> error: --port "": no number given
"eighty" -> error: --port "eighty": not a valid number
"70000" -> error: --port "70000": number too large for this field
"0" -> error: --port "0": zero is not allowed here
" 443 " -> port 443

Display: "invalid digit found in string" / kind: InvalidDigit

All assertions passed.

Formatting Numbers: Width, Precision, Sign, Radix

The opposite direction uses the format-string syntax, and you do not need external crates:

Specifier	Effect	Example
`{:8}` / `{:<8}` / `{:^8}`	Minimum width. Numbers align to the right by default	`" 42"`
`{:08}`	Zero padding that keeps the sign first	`-42` → `"-0000042"`
`{:+}`	A sign on positive numbers too	`"+42"`
`{:.2}`	Float precision (it rounds, and it pads with zeros)	`12.3456` → `"12.35"`
`{:10.3}` / `{:+010.3}`	Width and precision together	`" 12.346"`
`{:e}` / `{:.2e}`	Scientific notation	`123456.789` → `"1.23e5"`
`{:x}` / `{:X}` / `{:o}` / `{:b}`	Hex, octal, binary	`2561` → `"a01"`
`{:#06x}`	`0x` prefix, which counts toward the width	`"0x0a01"`
`{:>w$.p$}`	Width and precision from arguments	Table columns with a size from run time

Be careful with three behaviors:

The width never truncates. A number that is too wide extends past its column.
Radix formatting of a negative number shows the two's-complement bits, not a minus sign: format!("{:#06x}", -1_i16) is "0xffff".
Float Display without a precision prints the shortest string that parses back to the same float. For this reason, 0.1 + 0.2 prints as 0.30000000000000004.

18_12_number_formatting.rs prints:

[      42] [42      ] [   42   ]
delta = +42, padded = -0000042
price = 12.35, aligned = [    12.346]
avogadro = 6.022e23
flags = 0x0a01 = 0b101000000001

-- invoice --
subtotal     99.99
tax           8.25
total       108.24

All assertions passed.

Safe Narrowing with `TryFrom`

A parsed number rarely has the width that you need, and a narrowing as cast always produces a value:

300_u16 as u8 is 44 (the cast keeps the low bits).
-1_i32 as u32 is 4294967295 (the cast reinterprets the bits).

The language defines these behaviors, and they are useful in bit manipulation. As conversions, they are a source of silent errors.

TryFrom (stabilized 1.34) exists for every pair of integer types where a value might not fit. It returns Err(TryFromIntError), not corrupted data:

assert_eq!(u8::try_from(200_u16), Ok(200));   // 200 fits in a u8
assert!(u8::try_from(300_u16).is_err());      // u8::MAX is 255
assert!(u32::try_from(-1_i32).is_err());      // a negative value never fits an unsigned type

// TryFromIntError implements Display. Since Rust 1.99, the text says which limit applies.
let too_large = u8::try_from(300_u16).unwrap_err().to_string();
assert_eq!(too_large, "number too large to fit in target type");
let too_small = u8::try_from(-1_i16).unwrap_err().to_string();
assert_eq!(too_small, "number too small to fit in target type");

Before Rust 1.99, the two cases gave one generic message ("out of range integral type conversion attempted"). Do not compare error messages in production code. Match on the Result. Print the message only for a person to read.

Combinators make the failure policy explicit and easy to review:

u8::try_from(n).unwrap_or(u8::MAX) saturates.
.unwrap_or_default() uses zero as the alternative.
? propagates the error.

A widening conversion needs no check, and u32::from(byte) documents that it cannot fail. But there is deliberately no From<u64> for usize. The size of usize depends on the platform, so the conversion can fail in general. usize::try_from makes the failure on a 32-bit platform explicit.

The same pattern protects indexing with offsets from an external source. usize::try_from(offset).ok()? rejects negative values before they can wrap to very large indices.

18_13_tryfrom_narrowing.rs prints:

300u16 as u8   = 44   (silent truncation)
-1i32 as u32   = 4294967295   (sign reinterpreted)
u8::try_from(300u16) = Err(TryFromIntError(PosOverflow))
1000 -> clamp: 255, default: 0
payload(len 2^40) on this platform: ok=false
lookup(2)=Some(30) lookup(-1)=None

All assertions passed.

Summary

Concept	Key point
`str::parse`	Thin wrapper over `FromStr::from_str`. The type comes from a turbofish or an annotation
Integer parse rules	Sign and ASCII digits only: no whitespace, underscores, or prefixes
Float parse rules	Accepts scientific notation, `inf`, `NaN`. Overflow gives `±inf`, not an error
Custom `FromStr`	One implementation gives `parse()`, `?` integration, and generic composition
`from_str_radix`	Bases 2–36. Remove the `0x`/`0o`/`0b` prefixes in your code
`NonZero*::from_str_radix` (1.98)	Same bases. `"0"` is `Err`, so a success is already nonzero
`ParseIntError::kind()`	`Empty`, `InvalidDigit`, `PosOverflow`, `NegOverflow`, `Zero`, and a mandatory wildcard arm (`#[non_exhaustive]`)
`"-1"` as unsigned	`InvalidDigit`, not `NegOverflow`: the sign is an invalid character
Width/precision/sign	`{:08}`, `{:+}`, `{:.2}`, `{:>w$.p$}`. Zero padding keeps the sign first
Radix formatting	`{:x}`, `{:#06x}` (the prefix counts in the width). Negative numbers show raw bits
`as` narrowing	Always gives a value: truncation and sign reinterpretation
`TryFrom` narrowing	An out-of-range value becomes `Err`. Combinators select the failure policy

Code Examples

File	Description
`18_09_parse_fromstr.rs`	`parse`/turbofish, integer vs float parse rules, custom `FromStr` with structured errors
`18_10_from_str_radix.rs`	Bases 2–36, prefix removal, hex colors, octal permissions
`18_15_nonzero_from_str_radix.rs`	`NonZeroU32::from_str_radix` (1.98), which rejects zero at the boundary
`18_11_int_error_kind.rs`	The five stable `IntErrorKind` variants, a CLI port validator, `kind()` vs `Display`
`18_12_number_formatting.rs`	Width/alignment, zero padding, signs, precision, scientific notation, radix, width from arguments
`18_13_tryfrom_narrowing.rs`	Hazards of `as`, `TryFrom`/`try_into`, failure policies, checks of a wire length and of an index

19.1 · Essential Macros: assert, dbg, todo, unimplemented, unreachable

Domain 19 — Macros from the Standard Library Duration: ~15 minutes Library components: assert!, assert_eq!, assert_ne!, assert_matches!, debug_assert!, debug_assert_eq!, debug_assert_ne!, debug_assert_matches!, dbg!, todo!, unimplemented!, unreachable!, panic!

Introduction

The standard library has a small set of macros that you will use almost every day. They are macros, not functions, because a macro can do these things:

It can record the source location of the call site.
It can copy the tokens of an expression into an error message.
It can accept the format! syntax.
It can expand to an expression of the never type !, which is valid in any position.

This tutorial explains the most common macros in three groups:

Check: assert!, assert_eq!, assert_ne!, and assert_matches! (stabilized in 1.96), which uses a pattern. Each one has a debug_* variant that the compiler removes in release builds.
Inspect: dbg! prints an expression and its value to stderr, then returns the value.
Signal intent: todo!, unimplemented!, unreachable!, and panic! with a formatted message. These are four ways to panic, and each one tells the reader something different.

Figure: The essential macros, grouped by purpose

19_1_essential_macros_1

The assert! Family

assert!(cond) panics if the condition is false. All the arguments after the condition are format! syntax. The program evaluates those message arguments only when the assertion fails. Thus a detailed message has no cost when the assertion passes:

// order: &Order. An Order has a numeric `id` and a list of `items` (name, price in cents).
// Plain form. A failure prints: assertion failed: !order.items.is_empty()
assert!(!order.items.is_empty());
// Form with a message. The program formats the message only on failure.
assert!(
    order.total() > 0,   // total() returns the sum of the item prices
    "order {} has a zero total: {:?}",
    order.id,
    order.items
);

Use assert_eq! and assert_ne! instead of assert!(a == b). On failure, they print both operands with Debug formatting. assert!(a == b) can only tell you that the comparison was false. The failure payload has a stable shape that is easy to recognize. Learn this shape, because you will read it in test output for years:

let computed = 1499_u32;
let expected = 1550_u32;
// The two values are different, so this assertion fails.
assert_eq!(computed, expected, "rounding bug in order 42");
// panics with:
//   assertion `left == right` failed: rounding bug in order 42
//     left: 1499
//    right: 1550

The example binary captures this payload with panic::catch_unwind (Tutorial 2.3) and makes assertions about its shape. The payload of a formatted assertion failure is a String. 19_01_assert_family.rs prints:

assert!: order 42 passed basic invariants
assert_eq!/assert_ne!: totals verified

captured assert_eq! failure payload:
assertion `left == right` failed: rounding bug in order 42
  left: 1499
 right: 1550

debug assertions are: enabled

All assertions passed.

debug_assert_*: Checks That Do Not Run in Release

debug_assert!, debug_assert_eq!, and debug_assert_ne! are the same as the macros without the debug_ prefix, with one difference. When debug_assertions is off, the compiler removes the full check, which includes the condition. debug_assertions is off by default in --release builds. Thus these macros are the correct place for checks that are too expensive for production but useful during development:

// order: the same &Order as in the first snippet.
// An O(n²) scan for duplicate names: acceptable in a debug build, wasteful in a release build.
debug_assert!(
    order
        .items
        .iter()
        .enumerate()
        // For the item at index i: no item before it has the same name.
        .all(|(i, (name, _))| !order.items[..i].iter().any(|(n, _)| n == name)),
    "order {} contains duplicate line items",
    order.id
);

Obey two rules to use debug_assert! safely:

Do not put side effects in the condition. They disappear from release builds without a warning.
Do not use it for a check that safety depends on, or for a check of user input. Release builds must be correct without the check. debug_assert! checks your own logic, not external data.

To find which mode the build uses, read cfg!(debug_assertions). It is a compile-time boolean that Tutorial 19.2 explains in detail. The example prints debug assertions are: enabled because cargo run uses the dev profile.

assert_matches!: Pattern-Shaped Assertions (1.96)

assert_eq! needs a complete expected value and a PartialEq implementation. Frequently, a test is only about the shape of a value. For example: this transition must give a timeout error, and the exact delay is not important. assert_matches!, stabilized in Rust 1.96, does this check. It accepts the full pattern grammar of a match arm:

| alternatives
.. rest patterns
bindings
if guards

use std::assert_matches;

// drive(n) is a function of the example. It returns the Connection state after attempt n.
// Only the variant is important here. `..` ignores the fields.
assert_matches!(drive(1), Connection::Connecting { .. });
// The pattern binds `after_ms`, and the `if` guard tests it.
assert_matches!(
    drive(2),   // Connection::Failed(ConnError::Timeout { after_ms: 3000 })
    Connection::Failed(ConnError::Timeout { after_ms }) if after_ms >= 1000
);
// `|` accepts one of two error variants.
assert_matches!(
    drive(3),   // Connection::Failed(ConnError::Refused)
    Connection::Failed(ConnError::Refused | ConnError::Timeout { .. })
);

Two points frequently cause confusion:

Its location: the macro is at the std crate root, adjacent to assert_eq!. Call it as std::assert_matches!(…), or import it with use std::assert_matches;. The nightly path from before the stabilization, std::assert_matches::assert_matches, does not exist now.
Why it is not in the prelude: a new macro name in the prelude can break current code. Many crates already define or import their own assert_matches! (for example, the widely used assert_matches crate). Thus std makes the adoption optional: you add one use line.

On failure, the macro prints the actual value and the pattern that did not match. assert!(matches!(…)) can only report that the result was false. 19_02_assert_matches.rs asserts that drive(0), which is Idle, matches Connection::Connected { .. }. It captures the failure and prints the payload:

shape assertions passed for all transitions

captured assert_matches! failure payload:
assertion `left matches right` failed
  left: Idle
 right: Connection::Connected { .. }

All assertions passed.

debug_assert_matches! obeys the debug_assert! rule: it is active only when debug assertions are on.

dbg!: Inspect Without Restructuring

dbg!(expr) prints [file:line:column] expr = value to stderr with pretty Debug formatting ({:#?}). Then it returns the value. The return value is the important property. You can put dbg! around any subexpression in the middle of a chain. You do not need temporary variables, and you do not need to divide the expression:

// item: LineItem { unit_cents: 1250, quantity: 3 }, shipping: 499_u32
// Each dbg! prints its operand and returns it, so `total` is 3750 + 499 = 4249.
let total = dbg!(item.unit_cents * item.quantity) + dbg!(shipping);

let discounted: Vec<u32> = [1000_u32, 2500, 400]
    .into_iter()
    .map(|price| dbg!(price * 9 / 10)) // print each discounted price
    .filter(|&price| price >= 500)     // 360 does not pass the filter
    .collect();                        // [900, 2250]

These two statements write the lines below to stderr. With cargo, the path is relative to the workspace root. In a different project, the path and the line numbers are different:

[domain-19-std-macros/examples/src/bin/19_03_dbg_macro.rs:30:17] item.unit_cents * item.quantity = 3750   # (stderr)
[domain-19-std-macros/examples/src/bin/19_03_dbg_macro.rs:30:57] shipping = 499   # (stderr)
[domain-19-std-macros/examples/src/bin/19_03_dbg_macro.rs:38:22] price * 9 / 10 = 900   # (stderr)
[domain-19-std-macros/examples/src/bin/19_03_dbg_macro.rs:38:22] price * 9 / 10 = 2250   # (stderr)
[domain-19-std-macros/examples/src/bin/19_03_dbg_macro.rs:38:22] price * 9 / 10 = 360   # (stderr)

The details that are important in practice:

dbg! takes ownership of its argument and returns it. To look at a non-Copy value that you use again later, pass a reference: dbg!(&cart).
With two or more arguments, dbg! returns a tuple: let (a, b) = dbg!(x, y);
With no arguments, dbg! prints only the location. Use this form to mark that execution got to that line.
The compiler does NOT remove dbg! in release builds, unlike debug_assert!. dbg! is a temporary development tool. The optional Clippy restriction lint dbg_macro can prevent accidental dbg! calls in committed code.

todo!, unimplemented!, unreachable!: Panics That Signal Intent

All three macros expand to a panic, and all three have the type ! (never). Thus they satisfy any expected type. For this reason, you can use them as placeholders in code that must compile now.

Figure: Choosing the right panic macro

19_1_essential_macros_2

// Codec is an enum of the example. Its variants are Flac, Opus, Aac, and Midi.
fn transcode(codec: Codec) -> &'static str {
    match codec {
        Codec::Flac => "flac: transcoded",
        Codec::Opus => "opus: transcoded",
        // todo! says "I intend to write this". It tells the team about the plan.
        Codec::Aac => todo!("AAC support is planned for the next sprint"),
        // unimplemented! says "I do not intend to support this". It states the scope.
        Codec::Midi => unimplemented!(),
    }
}

todo!() has the type !, so each arm still has the type &'static str, and the function passes the type check. You can write the implementation later. The arms that are complete operate correctly now.

unreachable! is different. It documents a branch that cannot occur if your logic is correct. You can prove that n % 3 is 0, 1, or 2. But the compiler still requires an exhaustive match, and the _ arm states your proof. If a later code change breaks the invariant, the program panics with a clear message. It does not continue with a wrong result:

// n: u32. For an unsigned integer, n % 3 is always 0, 1, or 2.
match n % 3 {
    0 => "red",
    1 => "yellow",
    2 => "green",
    // The compiler does not know the proof, so the match needs this arm.
    _ => unreachable!("n % 3 is always 0, 1, or 2"),
}

panic! Message Formatting and Payloads

panic! accepts the same syntax as format!. The form of the message sets the payload type that catch_unwind gives you:

panic!("literal") keeps the literal as a &'static str payload. It does not allocate.
panic!("no codec at index {i}") formats the message at panic time, and its payload is a String.

todo! and unimplemented! obey the same rule. todo! puts not yet implemented: before your message, and unimplemented! puts not implemented: before it.

19_04_todo_unreachable_panic.rs calls each macro on purpose inside panic::catch_unwind. It replaces the default panic hook with an empty hook, so the panics print nothing. It makes assertions about the payloads, and thus it still exits with code 0:

implemented codecs transcode fine
traffic_light never hit unreachable! for n in 0..9

todo!            -> not yet implemented: AAC support is planned for the next sprint (payload type: &'static str)
unimplemented!   -> not implemented (payload type: &'static str)
panic! literal   -> codec table corrupted (payload type: &'static str)
panic! formatted -> no codec at index 7

All assertions passed.

For the full mechanics of a panic (hooks, unwind or abort, catch_unwind), see Tutorial 2.3. The important point here is the vocabulary. The macro that you select tells the subsequent reader what the panic is:

unreachable!: a bug
todo!: code that is not written yet
unimplemented!: a limit of the scope
panic!: a fatal condition

Summary

Macro	Key point
`assert!`	Panics when the condition is false. The message arguments are `format!` syntax, and the program evaluates them only on failure.
`assert_eq!` / `assert_ne!`	Print both operands (`Debug`) on failure. Use them instead of `assert!(a == b)`.
`assert_matches!`	An assertion with a pattern (1.96). It is at the std crate root, not in the prelude.
`debug_assert_*`	The compiler removes them when `debug_assertions` is off. Do not put side effects in the condition.
`debug_assert_matches!`	`assert_matches!` with the removal rule of `debug_assert!`.
`dbg!`	Prints `[file:line:column] expr = value` to stderr and returns the value. Release builds keep it.
`todo!`	"Not written yet". It has the type `!`, so a placeholder passes the type check.
`unimplemented!`	"Not supported, by design". It states scope, not schedule.
`unreachable!`	"Provably impossible". A broken invariant becomes a panic with a clear message.
`panic!`	A fatal error with a formatted message. A literal gives a `&'static str` payload, and a formatted message gives a `String`.

Code Examples

File	Description
`19_01_assert_family.rs`	`assert!`/`assert_eq!`/`assert_ne!` with custom messages, failure payload shape, `debug_assert_*`
`19_02_assert_matches.rs`	`assert_matches!`/`debug_assert_matches!` (1.96): patterns, guards, crate-root import, failure message
`19_03_dbg_macro.rs`	`dbg!` in expression chains: returns the value, stderr output, ownership, tuple form
`19_04_todo_unreachable_panic.rs`	The intent of `todo!`, `unimplemented!`, and `unreachable!`, and the `panic!` payload types with `catch_unwind`

19.2 · Compile-Time Macros: cfg, env, include, concat, stringify

Domain 19 — Macros from the Standard Library Duration: ~15 minutes Library components: cfg!, cfg_select!, env!, option_env!, include_str!, include_bytes!, include!, concat!, stringify!, file!, line!, column!, module_path!, compile_error!

Introduction

Each macro in this tutorial runs while rustc runs. These macros do their work before your program exists:

They read environment variables.
They embed files.
They join string literals.
They change tokens into text.
They can stop the build.

Remember one question during the next 15 minutes. For each macro, ask "when does this macro read its input?" The answer is always "at compile time". That is the advantage: there is no runtime cost, and the data is always in the binary. It is also the constraint: to change the input, you must compile again.

Figure: Compile time vs. run time — who reads what, when

19_2_compile_time_macros_1

Three Ways to Branch on Configuration

Rust has three mechanisms that change the code for a target configuration. Their rules are very different:

Mechanism	Position	Code that does not match
`cfg!(pred)`	expression that gives a `bool`	The compiler still type-checks it on each target.
`#[cfg(pred)]`	attribute on items or expressions	The compiler removes it before type checking.
`cfg_select!` (1.95)	expression and item position	The compiler removes it. Only the first arm that matches stays.

cfg! makes the smallest change. At compile time, it becomes a plain true or false, and the code around it stays ordinary Rust. Both arms of an if cfg!(…) must compile on each target. That is its limit: it cannot protect APIs that exist on one platform only. It is also its convenience: you do not write an item definition two times.

// Each cfg! call becomes `true` or `false` at compile time.
// All three branches must compile on each target.
let family = if cfg!(unix) {
    "unix"
} else if cfg!(windows) {
    "windows"
} else {
    "other"
};
let is_64_bit = cfg!(target_pointer_width = "64");               // true on a 64-bit target
let unixish_desktop = cfg!(all(unix, not(target_os = "ios")));   // true on unix, but not on iOS

#[cfg(...)] removes the annotated item from the token stream before type checking. Thus the versions that the compiler removes can use platform-only APIs that do not compile on other targets:

// The compiler keeps exactly one of these three functions.
#[cfg(unix)]
fn family_via_cfg_attr() -> &'static str { "unix" }

#[cfg(windows)]
fn family_via_cfg_attr() -> &'static str { "windows" }

// The target is not unix and not windows.
#[cfg(not(any(unix, windows)))]
fn family_via_cfg_attr() -> &'static str { "other" }

cfg! and #[cfg] accept the same predicate syntax: any(), all(), not(), and key = "value" pairs such as target_os and target_pointer_width.

cfg_select!: A Match on Configurations (1.95)

cfg_select!, stabilized in Rust 1.95, is a compile-time "match on cfg predicates". The compiler tries the arms from top to bottom and selects the first arm with a true predicate. It compiles only the tokens of that arm. cfg_select! is the standard-library replacement for the long-established cfg-if crate. Unlike an if cfg! chain, the compiler never type-checks the arms that it does not select. Thus each arm can contain code for one target only.

cfg_select! is valid in expression position:

// On a unix target, the result is "unix". The compiler discards the other arms.
let family = std::cfg_select! {
    unix => { "unix" }
    windows => { "windows" }
    _ => { "other" }   // the arm for all other targets
};

// The bare name, without the std:: path, is also valid, as for cfg! and env!.
let separator = cfg_select! {
    windows => { '\\' }
    _ => { '/' }
};

It is also valid in item position, where each arm defines full items:

// Only one of these three function definitions stays in the program.
std::cfg_select! {
    unix => {
        fn family_via_cfg_select() -> &'static str { "unix" }
    }
    windows => {
        fn family_via_cfg_select() -> &'static str { "windows" }
    }
    _ => {
        fn family_via_cfg_select() -> &'static str { "other" }
    }
}

The _ arm is the arm for all other targets. Without it, a target that matches no arm causes a compile error. That is useful when you want the build to fail on a target that you do not support. The general rule:

Use cfg! for a branch inside code that compiles on all targets.
Use #[cfg] to include or exclude one item.
Use cfg_select! for several alternatives that are mutually exclusive.

19_05_cfg_family.rs prints the lines below on a 64-bit unix target. The first four lines are different on other platforms:

cfg!(target_pointer_width = "64") = true   # (varies by platform)
cfg!(all(unix, not(ios)))          = true   # (varies by platform)
path separator: '/'   # (varies by platform)
family: unix (cfg! == #[cfg] == cfg_select!)   # (varies by platform)

All assertions passed.

env! and option_env!: Build-Time Environment

env!("VAR") reads an environment variable when rustc runs. It stores the value in the binary as a &'static str. Cargo sets the CARGO_PKG_* variables for each compilation, so they are deterministic inputs. This is the idiomatic way for a program to report its own name and version. clap uses it to print --version:

// Cargo sets these two variables from the package manifest.
let name = env!("CARGO_PKG_NAME");         // &'static str: "domain-19-std-macros"
let version = env!("CARGO_PKG_VERSION");   // &'static str: "0.1.0" for this package
println!("{name} v{version}");

A variable that is not set causes a compile error, not a runtime error. A second argument replaces the error text with your own message:

// error: environment variable `HOME_SWEET_HOME` not defined at compile time
// let home = env!("HOME_SWEET_HOME");

// With a second argument, the compile error is your text:
// error: set APP_SIGNING_KEY before building
// env!("APP_SIGNING_KEY", "set APP_SIGNING_KEY before building");

option_env! returns Option<&'static str> and does not cause an error. Use it for optional build metadata, such as a commit hash that the CI system supplies:

// The result is None when the variable is not set during the build, as in this example.
let commit: Option<&'static str> = option_env!("DOMAIN19_BUILD_COMMIT");
println!("build commit: {}", commit.unwrap_or("unknown (dev build)"));

These macros are different from std::env::var, which reads the environment of the running process at runtime (Tutorial 12.1). env! answers the question "what was true when the compiler built this binary?". env::var answers the question "what is true now?".

include_str!, include_bytes!, include!

The include_* macros embed the contents of a file into your binary at compile time. The path is relative to the source file that contains the macro call. It is not relative to the crate root or to the working directory:

// src/bin/19_06_env_include.rs embeds src/embedded/motd.txt:
let motd: &'static str = include_str!("../embedded/motd.txt");     // the file as text
let raw: &'static [u8] = include_bytes!("../embedded/motd.txt");   // the same file as bytes
assert_eq!(raw.len(), motd.len()); // same file, same bytes (65 bytes)

include_str! gives a &'static str. The compiler makes sure that the file is valid UTF-8.
include_bytes! gives a raw &'static [u8] and does not require UTF-8. Use it for icons, fonts, lookup tables, and test fixtures.
include! inserts the file as Rust tokens. The compiler parses them as an expression or as items at the call site. Its primary use in practice is to include code that a build script generates: include!(concat!(env!("OUT_DIR"), "/tables.rs")).

Remember the trade-off. All the data that include_*! embeds makes the binary larger, and you must compile again to change it. Configuration that is different for each deployment belongs in runtime files or in std::env::var. Embed an asset when it must always be available: help text, licenses, shader sources, migration SQL.

19_06_env_include.rs prints the lines below. The example also calls include! on a file that contains the expression 21 * 2:

domain-19-std-macros v0.1.0   # (varies with the package version)
build commit: unknown (dev build)

--- embedded motd ---
Welcome to domain 19!
Macros embedded this file at compile time.
--- 65 bytes embedded ---
included expression evaluated to 42

All assertions passed.

concat! and stringify!: Token Operations

concat! joins literals into one &'static str that the compiler stores in the binary. It accepts string, integer, float, bool, and char literals. It does not accept variables, because it runs before any values exist. A common use is together with env!:

// The integer literals 1 and 2 become text: "domain19-bot/1.2 (rust)".
const USER_AGENT: &str = concat!("domain19-bot/", 1, ".", 2, " (rust)");
// env! expands to a string literal first: "domain-19-std-macros says hello".
const BANNER: &str = concat!(env!("CARGO_PKG_NAME"), " says hello");

stringify! changes the tokens of an expression into a &'static str and never evaluates the expression. The macro copies the tokens as text, and the program does not run them. When the tokens come directly from your source, rustc keeps the spacing between them. It prints each run of whitespace as one space. When a different macro generates the tokens, the spacing can be different:

// stringify! does not evaluate its argument. It only copies the tokens as text.
let s = stringify!(items.iter().map(|x| x * 2).sum::<u32>());
assert_eq!(s, "items.iter().map(|x| x * 2).sum::<u32>()");
// No variable `items` exists. The code compiles because stringify! never resolves names.

Macros of the standard library use stringify! internally. dbg! uses it to print the expression, and assert_matches! uses it to print the pattern. To show the technique, the example builds a small dbg! clone, trace!, from only concat! and stringify!:

macro_rules! trace {
    ($e:expr) => {{
        let value = $e;   // evaluate the expression one time
        // concat! builds the format string at compile time.
        // For trace!(21 * 2), the format string is "[trace 21 * 2] = {:?}".
        // println! writes to stdout. The real dbg! writes to stderr.
        println!(concat!("[trace ", stringify!($e), "] = {:?}"), value);
        value             // return the value, as dbg! does
    }};
}

let doubled = trace!(21 * 2); // prints: [trace 21 * 2] = 42

concat! makes one string literal at compile time, so the format string has no cost at runtime.

Source-Location Macros

file!, line!, column!, and module_path! expand to the location of the macro call itself. dbg! uses file!, line!, and column! to label its output. panic! and assert! also print a location, but they get it through #[track_caller]:

println!("file!():        {}", file!());        // the path that rustc received
println!("line!():        {}", line!());        // the line of this call (the first line is 1)
println!("module_path!(): {}", module_path!()); // for example, my_crate::parser

// Each call reports ITS OWN position. These two calls are on adjacent lines.
let here = line!();
let next = line!();
assert_eq!(next, here + 1);
// Two calls on the same line have different columns.
let (col_a, col_b) = (column!(), column!());
assert!(col_a < col_b);

With cargo, file! is relative to the workspace root. In the root of a bin crate, module_path! is only the crate name. There is one design note for library authors. If your helper panics on bad input, line!() inside the helper gives the location of your code. It does not give the location of the bug in the caller. Use #[track_caller] with std::panic::Location::caller(), which gives the location of the caller (see Tutorial 2.3).

compile_error!: Failing the Build on Purpose

compile_error!("message") always stops compilation with your message. Alone, it has no use. It becomes useful behind a #[cfg]. Then it stops the build only for a configuration that you do not support:

// The build fails only when the two features are on together.
#[cfg(all(feature = "sync", feature = "async"))]
compile_error!("features `sync` and `async` are mutually exclusive");

// The build fails on a target that is not little-endian and not big-endian.
#[cfg(not(any(target_endian = "little", target_endian = "big")))]
compile_error!("this crate does not support exotic endianness");

The second common location is the last arm of a macro_rules! definition, the arm that matches all other input. There it changes "no rule matched" from an unclear error into a clear error.

compile_error! is like a panic! that occurs earlier. The failure occurs on the machine of the developer at build time, not on the machine of a user at run time. A failure at build time always has the lower cost.

19_07_concat_stringify_location.rs prints:

concat! user agent: domain19-bot/1.2 (rust)
stringify!: items.iter().map(|x| x * 2).sum::<u32>()
[trace 21 * 2] = 42
[trace [1, 2, 3].iter().sum::<i32>()] = 6
file!():        domain-19-std-macros/examples/src/bin/19_07_concat_stringify_location.rs   # (the separator varies by platform)
line!():        57   # (changes when the file changes)
module_path!(): 19_07_concat_stringify_location

All assertions passed.

Summary

Macro	Key point
`cfg!`	A compile-time predicate that becomes a runtime `bool`. The compiler still type-checks both branches.
`#[cfg(...)]`	Removes items before type checking, so it can protect platform-only APIs.
`cfg_select!`	A "match on cfgs" that selects the first arm that matches (1.95). It is valid in expression and item position, and it replaces `cfg-if`.
`env!`	Gives a build-time environment variable as a `&'static str`. A variable that is not set causes a compile error, and you can supply the message.
`option_env!`	The same, but it gives `Option<&'static str>`. Use it for optional build metadata.
`include_str!`	Embeds a UTF-8 file as a `&'static str`. The path is relative to the source file.
`include_bytes!`	Embeds any file as a `&'static [u8]`.
`include!`	Inserts a file as Rust tokens. Use it to include code that a build script writes to `OUT_DIR`.
`concat!`	Joins literals at compile time into one `&'static str`.
`stringify!`	Changes tokens into a string and never evaluates them. `dbg!` uses it for its labels.
`file!` / `line!` / `column!`	Give the location of the call site. In library helpers, use `#[track_caller]`.
`module_path!`	Gives the full module path at the call site.
`compile_error!`	Stops the build with your message. The failure occurs on the machine of the developer, not of the user.

Code Examples

File	Description
`19_05_cfg_family.rs`	`cfg!`, `#[cfg]`, and `cfg_select!` (1.95) in expression and item position, with the same result from all three
`19_06_env_include.rs`	`env!`/`option_env!` with `CARGO_PKG_*`, `include_str!`/`include_bytes!`/`include!` of embedded assets
`19_07_concat_stringify_location.rs`	`concat!`, `stringify!`, a small `dbg!` clone named `trace!`, `file!`/`line!`/`column!`/`module_path!`, `compile_error!`

19.3 · vec!, format!, write!, and Other Constructor Macros

Domain 19 — Macros from the Standard Library Duration: ~15 minutes Library components: vec!, format!, write!, writeln!, matches!, thread_local!

Introduction

The two previous tutorials explained macros that check and macros that run at compile time. This tutorial explains the macros that build things:

vec! builds vectors.
format! builds strings.
write! and writeln! build output into any sink.
matches! builds a boolean from a pattern.
thread_local! builds per-thread statics.

There is a reason why these are macros. A function signature cannot express a variadic element list, the formatting mini-language, or item generation.

The examples are small but realistic:

a game leaderboard (vec! and format!)
a sensor report that the example writes one time into a reusable buffer (write!)
a pipeline that processes log events (matches!, thread_local!, and a todo! placeholder for a backend that does not exist yet)

vec!: List Form and Repeat Form

vec![a, b, c] expands to approximately a boxed array that moves into a Vec. The result is one allocation of the exact size. There is no reallocation, unlike the growth steps of repeated push calls (Tutorial 4.1):

let scores = vec![1250, 990, 875, 640];   // Vec<i32> with 4 elements
assert_eq!(scores.capacity(), 4); // exact-size allocation

The repeat form vec![elem; n] requires Clone. The macro evaluates the element expression one time, then clones the value n − 1 times. This form is ideal for a buffer of zeros. It also causes a well-known error with shared pointers:

use std::rc::Rc;

let buffer = vec![0_u8; 8]; // eight zeros, one allocation

// vec![rc; 3] gives three clones of the SAME Rc. They are three handles to
// one allocation, not three independent values.
let shared = Rc::new(String::from("hall of fame"));
let handles = vec![Rc::clone(&shared); 3];   // Vec<Rc<String>> with 3 handles
assert_eq!(Rc::strong_count(&shared), 4); // original + 3 clones

// Repeat forms can nest. This is a grid of zeros with 3 rows and 4 columns.
let grid = vec![vec![0_u32; 4]; 3];

format!: The Formatting Mini-Language

format! builds a String with the same mini-language that println!, write!, and panic! use. You learn the mini-language one time and use it in all these macros. The modern style is inline captured identifiers (1.58+):

// player: "ada", points: 1250_u32, width: 10, ratio: 2.0_f64 / 3.0
let line = format!("{player} scored {points}");   // "ada scored 1250"

// With positional and named arguments, one value can appear more than one time.
let echo = format!("{0}, I repeat: {0}, over {channel}", "bravo", channel = 9);
// echo is "bravo, I repeat: bravo, over 9"

// Alignment and fill. The leaderboard loop supplies name, score, and rank.
//   {:<8} left in 8 cols   {:>6} right in 6   {:*^9} centered, '*' fill
println!("{name:<8}|{score:>6}|{rank:*^9}");

// Width and precision can come from variables with `$`. Here `width$` reads `width`.
let cell = format!("[{ratio:>width$.3}]");   // "[     0.667]"

// Debug, hex with leading zeros, binary, literal braces:
assert_eq!(format!("{:?}", vec![1, 2]), "[1, 2]");
assert_eq!(format!("{:#06x}", 255), "0x00ff");
assert_eq!(format!("{:08b}", 5), "00000101");
assert_eq!(format!("{{{player}}}"), "{ada}");

19_08_vec_format.rs prints:

scores: [1250, 990, 875, 640] (cap = 4)
vec![rc; 3]: 3 handles point at one "hall of fame"

player  | score|**rank***
ada     |  1250|****1****
grace   |   990|****2****
linus   |   875|****3****

precision from variables: [     0.667]

All assertions passed.

Know the cost: each format! call allocates a new String. If you build a large text from concatenated format! results, the program copies the accumulated text many times. To append many pieces into one buffer, use write!, which is the subject of the subsequent section.

write! and writeln!: One Macro, Two Traits

write!(dst, …) expands to dst.write_fmt(format_args!(…)). Thus it accepts any destination that has a write_fmt method, and the trait that supplies the method is not important. Two traits supply it:

Figure: write! dispatches on the destination's trait

19_3_constructor_macros_1

When you append to a String with fmt::Write, the program appends all rows to one buffer. It does not allocate for each row:

use std::fmt::Write as _; // brings write_fmt for String into scope

// readings: [Reading; 3]. A Reading has `sensor: &'static str` and `celsius: f64`.
let mut report = String::with_capacity(256);
let _ = writeln!(report, "== sensor report ==");   // writeln! adds a newline
for r in &readings {
    // {:<8} aligns the name to the left. {:>7.1} aligns the number to the right, 1 decimal.
    let _ = writeln!(report, "{:<8} {:>7.1}°C", r.sensor, r.celsius);
}

A write to a String cannot fail, but write! must return fmt::Result for generic sinks. For this reason the snippet uses let _ =. In a function that returns Result, you can use ?. With io::Write, the same macro writes bytes, and the Result is important (the disk can be full, or the pipe can be closed):

use std::io::Write as _;

// Vec<u8> implements io::Write. Here it is a substitute for a file or a socket.
let mut csv: Vec<u8> = Vec::new();
writeln!(csv, "sensor,celsius").expect("writing to Vec<u8> cannot fail");

// Lock stdout one time, then write many times. ripgrep does exactly this in
// its performance-critical printing code.
let stdout = std::io::stdout();
let mut out = stdout.lock();   // the lock stays until `out` drops
writeln!(out, "wrote {} csv rows via locked stdout", readings.len())
    .expect("stdout write failed");

Display Implementations: The Primary Use of write!

In practice, the most common location of a write! call is inside impl Display. The formatter f implements fmt::Write, and ? propagates fmt::Error. Callers get to_string(), format!("{}"), and println!("{}") support at no cost:

impl fmt::Display for Reading {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        // write! returns fmt::Result, which is also the return type of this method.
        write!(f, "{} at {:.1}°C", self.sensor, self.celsius)
    }
}

// readings[1] is Reading { sensor: "core", celsius: 68.309 }.
let display = readings[1].to_string();   // to_string() uses the Display implementation
assert_eq!(display, "core at 68.3°C");

Implement Display one time with write!. Then all code in the ecosystem that formats values can print your type.

19_09_write_writeln.rs prints:

== sensor report ==
intake      21.4°C
core        68.3°C
exhaust     40.0°C
csv bytes decoded:
sensor,celsius
intake,21.42
core,68.309
exhaust,40

wrote 3 csv rows via locked stdout
Display via write!: exhaust at 40.0°C

All assertions passed.

matches!: Pattern Tests as Booleans

matches!(value, pattern) tests a value against a pattern and gives a bool. You do not need to write a full match { … => true, _ => false } block. It accepts the same grammar as a match arm: | alternatives, ranges, and if guards. It is most useful where you need a boolean in expression position:

filter closures
assert! conditions
fields of struct literals

// An Event has the fields `level: Level`, `status: u16`, and `retries: u8`.
fn keep(event: &Event) -> bool {
    // Keep warnings and errors.
    matches!(event.level, Level::Warn | Level::Error)
        // Also keep status 429 or 500 to 599, but only with fewer than 3 retries.
        // The `if` guard applies to both alternatives.
        || matches!(event.status, 429 | 500..=599 if event.retries < 3)
}

// events: a slice of Event values.
let noisy = events
    .iter()
    .filter(|e| matches!(e.level, Level::Warn | Level::Error))
    .count();   // the number of Warn and Error events

matches! is the counterpart of assert_matches! from Tutorial 19.1. It has the same pattern grammar, but it returns false where assert_matches! panics. Use matches! for control flow. Use assert_matches! for invariants, because its failure message contains the value that did not match.

thread_local! and todo! as a Typed Placeholder

thread_local! declares a static with one independent copy for each thread. It needs no locks and no atomics, and one thread cannot change the copy of a different thread. The const { … } initializer is the fast path. The compiler can put the initial value directly in thread-local storage. It does not need lazy initialization code that runs on the first access:

use std::cell::Cell;

thread_local! {
    // Each thread gets its own counter, which starts at 0.
    static DROPPED: Cell<u32> = const { Cell::new(0) };
}

// LocalKey<Cell<T>> has direct get/set helpers (1.73) and `update`
// (1.99), so simple counters do not need the with(|c| …) closure.
// `update` reads the value, applies the closure, and stores the result.
DROPPED.update(|dropped| dropped + 1);   // adds 1 to the counter of this thread
let count = DROPPED.get();               // u32: a copy of the counter of this thread

Figure: Each thread sees its own copy of a thread_local! static

19_3_constructor_macros_2

Each worker thread starts with a new counter at zero. The work of the worker threads does not change the counter of the main thread. The example asserts exactly that.

Finally, the example uses todo! from Tutorial 19.1 in its most useful role: a typed placeholder. With it, a pipeline can compile and ship while one branch is not complete. todo!() has the type !, so the incomplete arm coerces to the same type as the complete arms:

// Backend is an enum of the example. Its variants are Stdout, File, and Syslog.
fn deliver(backend: Backend) -> &'static str {
    match backend {
        Backend::Stdout => "delivered to stdout",
        Backend::File => "delivered to file",
        // This arm panics only if the program selects Syslog at runtime.
        Backend::Syslog => todo!("syslog transport lands in Q3"),
    }
}

19_10_matches_thread_local.rs prints:

matches!: 2 of 5 events are Warn/Error
main thread: kept 3, dropped 2
2 workers each dropped 2; main still at 2
implemented backends deliver fine (Syslog arm is todo!)

All assertions passed.

Summary

Macro	Key point
`vec![a, b, c]`	One allocation of the exact size from a list of elements
`vec![elem; n]`	Requires `Clone` and evaluates the element one time. `vec![rc; 3]` shares one allocation.
`format!`	Builds a `String`: inline capture, alignment and fill, `width$`/`.prec$`, hex and binary. Each call allocates.
`write!` / `writeln!`	Expand to `dst.write_fmt(…)`. They accept `fmt::Write` (text) and `io::Write` (bytes).
`write!` into `String`	One buffer for all pieces, not one `format!` allocation for each piece
`write!` in `impl Display`	The standard location for `write!`. It gives `to_string()` and `{}` support to your type.
`matches!`	Changes a pattern test into a `bool`, with `\|` and `if` guards. It is the control-flow counterpart of `assert_matches!`.
`thread_local!`	Per-thread statics. A `const { … }` initializer is the fast path. `LocalKey<Cell<T>>` has `get`/`set`, and `update` since 1.99.
`todo!`	A typed placeholder (`!` coerces to any type). Ship the pipeline now. Complete the arm later.

Code Examples

File	Description
`19_08_vec_format.rs`	`vec!` list and repeat forms, the shared-`Rc` error of the repeat form, a leaderboard with the `format!` mini-language
`19_09_write_writeln.rs`	`write!`/`writeln!` into `String`, `Vec<u8>`, and locked stdout, and `Display` with `write!`
`19_10_matches_thread_local.rs`	`matches!` filters with guards, `thread_local!` per-thread counters, `todo!` placeholder backend

20.1 · C Types and Calling Conventions

Domain 20 — FFI (Foreign Function Interface) Duration: ~15 minutes Library components: std::ffi::c_char, std::ffi::c_int, std::ffi::c_long, std::ffi::c_float, std::ffi::c_double, std::ffi::c_void

Introduction

Every FFI bug is a disagreement between Rust and C about a type, a size, or a register. The compiler cannot find the disagreement, because it never sees the C side. Thus the task in FFI is to make sure that the two sides give the same description. The standard library gives you three tools for that task:

C type aliases in std::ffi (c_int, c_long, c_char, …). They encode the C ABI of each platform, so your signatures match the C side on every target.
extern blocks and calling conventions (extern "C", extern "system"). They tell the compiler how arguments and return values move between the caller and the callee.
#[repr(C)]. It makes the layout of a struct follow the rules of C, not the layout that the compiler optimizer selects.

Since Rust 1.99 there is a fourth tool: C-variadic function definitions with the VaList type. Use them for functions that C calls with a variable number of arguments.

This tutorial shows all four tools, and it checks each claim with size_of, align_of, and offset_of!. You do not need a C compiler. std already links a C runtime (libSystem, libc, or the UCRT), and that runtime supplies real foreign symbols. C-ABI function pointers let you use the calling convention fully inside one process.

Figure: The pieces of the FFI boundary

20_1_c_types_calling_conventions_1

The C Type Aliases

When you translate a C header manually, each parameter type must map to the Rust type with the same size, alignment, and signedness. This must be true on every platform on which the binding compiles. The aliases in std::ffi are plain type aliases to primitive types. The standard library selects each alias for the target, so the aliases encode exactly those platform rules.

Most mappings are the same on every 32-bit and 64-bit platform. (On the 16-bit AVR and MSP430 targets, c_int is i16.)

use std::ffi::{c_double, c_int, c_longlong, c_schar, c_short};

// Each comment gives the C type and the Rust type behind the alias.
assert_eq!(size_of::<c_schar>(), 1);    // signed char  == i8
assert_eq!(size_of::<c_short>(), 2);    // short        == i16
assert_eq!(size_of::<c_int>(), 4);      // int          == i32
assert_eq!(size_of::<c_longlong>(), 8); // long long    == i64
assert_eq!(size_of::<c_double>(), 8);   // double       == f64

Two aliases are common causes of portability errors:

c_long: the C type long is 8 bytes on 64-bit Unix (the LP64 data model), but only 4 bytes on 64-bit Windows (LLP64). A binding that hardcodes i64 for long is wrong on Windows, and the compiler gives no warning. That is the reason for the alias.
c_char: C does not specify whether plain char is signed. It is signed on x86-64 and on all Apple targets, but unsigned on AArch64 Linux and most ARM ABIs. c_char is an alias of i8 or u8 accordingly. Never assume one or the other in portable code.

use std::ffi::{c_char, c_long};

let long_bytes = size_of::<c_long>();       // 8 on 64-bit Unix, 4 on Windows
let char_is_signed = c_char::MIN < 0;       // true or false: the target decides

// A portable conversion between a byte and c_char. It does not name i8 or u8.
let as_c: c_char = c_char::from_ne_bytes([b'A']);
assert_eq!(as_c.to_ne_bytes()[0], b'A');    // the byte value does not change

The C type size_t corresponds to usize on every supported platform. For this reason, the FFI signatures in std use usize directly (for example, the return type of strlen).

c_void and Opaque Pointers

In C, void* is a pointer to a value whose type the API erased. Rust writes that type as *mut c_void or *const c_void. You cannot construct a c_void value, and this is deliberate. The type exists only as the target of a pointer.

Every C API with a context pointer uses the same round trip. You cast a typed pointer to *mut c_void when you give it to C. When C gives the pointer back, you cast it to the original type. You must remember that type yourself:

use std::ffi::c_void;

let mut answer: i32 = 42;

// Typed to untyped, as when you register a context with a C API.
let untyped: *mut c_void = std::ptr::from_mut(&mut answer).cast::<c_void>();

// Untyped to typed. The compiler does not know the real type: you supply it.
let typed: *mut i32 = untyped.cast::<i32>();
// SAFETY: `typed` came from `&mut answer` above, so the type matches.
// The borrow is live, and no other alias exists.
unsafe { *typed += 1; }
assert_eq!(answer, 43);   // the write through the pointer changed `answer`

The type system gives no protection here. A cast back to the wrong type compiles without an error, and the use of that pointer is undefined behavior. Thus each such cast needs a SAFETY: comment that names the type from which you made the pointer.

20_01_c_type_aliases.rs prints:

fixed-width C types:
  c_int      = 4 bytes
  c_longlong = 8 bytes
  c_double   = 8 bytes

c_long = 8 bytes (8 on 64-bit Unix/LP64, 4 on Windows/LLP64)   # (varies by platform)
c_char is signed (i8) on this target                            # (varies by platform)

round-trip through *mut c_void: answer = 43
usize = pointer width = 8 bytes                                 # (varies by platform)

All assertions passed.

extern "C" Blocks: Declaring Foreign Functions

An extern block declares foreign items. It never defines them: the linker finds the symbols. Since edition 2024, you must write the block as unsafe extern. With that keyword you assert that each signature in the block matches the real C symbol, and the compiler does not check this. If one signature is wrong (for example, c_int where C has long), the result is undefined behavior, not a compile error.

The functions below need no build script and no -l flags. strlen and abs are in the C runtime that the Rust standard library already links on every major platform.

use std::ffi::{CString, c_char, c_int};

unsafe extern "C" {
    /// C: `size_t strlen(const char *s);`
    /// Unsafe to call: the argument must be a valid, nul-terminated pointer.
    fn strlen(s: *const c_char) -> usize;

    /// C: `int abs(int n);`
    /// Declared `safe` (stabilized in 1.82): it takes an integer by value
    /// and cannot violate memory safety.
    safe fn abs(n: c_int) -> c_int;
}

// `abs` needs no unsafe block, because its declaration says `safe`.
assert_eq!(abs(-42), 42);

The idiomatic binding pattern puts each unsafe call in a safe Rust function. That function makes the preconditions of the C function true by construction:

// `strlen` is the foreign function from the previous snippet.
fn c_strlen(s: &CString) -> usize {
    // SAFETY: a CString always holds a valid, nul-terminated buffer, and
    // the borrow `s` keeps it alive for the duration of the call.
    unsafe { strlen(s.as_ptr()) }
}

let greeting = CString::new("hello, ffi").expect("no interior nul");
assert_eq!(c_strlen(&greeting), 10);   // 10 bytes: strlen does not count the nul

safe fn has a precise meaning: memory safety for all inputs, not full correctness. In C, abs(INT_MIN) is undefined, because int cannot represent the result. The declaration cannot change a contract that comes from C. It only promises that no input corrupts memory.

20_02_extern_c_bindings.rs prints:

abs(-42) = 42
strlen("hello, ffi") = 10
strlen("café") = 5 bytes, but 4 chars

All assertions passed.

Calling Conventions and C-ABI Function Pointers

A calling convention is the contract for how a call occurs. It specifies which registers hold the arguments, which side cleans the stack, and how a function returns a struct. In Rust, the calling convention is part of the type of a function:

extern "C" is the C convention of the platform. It is the common convention of FFI.
extern "system" is the convention of the OS interface. It is identical to "C" on every target except 32-bit x86 Windows, where it is "stdcall" (the Win32 API convention). Thus Win32 bindings that use "system" are portable across x86 and x64. If you write extern "stdcall" directly, the compiler gives a hard error (E0570) on targets that do not support it. "system" is the portable form.
No annotation gives the Rust ABI, which is not specified. Never give a function with this ABI to C.

The ABI is also part of the type of a function pointer. A plain fn(c_int) -> c_int and an extern "C" fn(c_int) -> c_int are different types, and neither one coerces to the other. On some targets, the two ABIs pass arguments differently:

use std::ffi::c_int;

// The type of a pointer to a C-ABI function that takes an `int` and returns an `int`.
type UnaryOp = extern "C" fn(c_int) -> c_int;

// A function that Rust defines with the C calling convention.
extern "C" fn double_it(x: c_int) -> c_int { x * 2 }

let f: UnaryOp = double_it;   // the function item coerces to the pointer type
assert_eq!(f(21), 42);        // a call through the pointer

// A C-ABI fn pointer is exactly one machine word: the address of the code.
assert_eq!(size_of::<UnaryOp>(), size_of::<usize>());

C APIs use NULL for "no callback". The Rust idiom is Option<extern "C" fn(...)>. The null-pointer optimization guarantees that this type has the size of a pointer, and that None is NULL:

// `UnaryOp` is the C-ABI function pointer type from the previous snippet.
type MaybeUnaryOp = Option<UnaryOp>;
assert_eq!(size_of::<MaybeUnaryOp>(), size_of::<usize>()); // Option adds no bytes

let cb: MaybeUnaryOp = None;             // C: a NULL callback
let result = cb.map_or(0, |op| op(5));   // C: cb ? cb(5) : 0
// result is 0, because `cb` is None

C-Variadic Functions and VaList (1.99)

Rust could always call a C-variadic function such as printf. Since Rust 1.99, Rust can also define one. Write ... as the last parameter of an unsafe extern "C" fn. The name before ... binds the argument list. The type of that binding is std::ffi::VaList, which is the Rust view of the C type va_list:

use std::ffi::c_int;

/// C: `int sum_ints(int count, ...);`
// `args` has the type `VaList<'_>`. It is `mut` because each read moves it forward.
unsafe extern "C" fn sum_ints(count: c_int, mut args: ...) -> c_int {
    let mut total = 0;
    for _ in 0..count {
        // SAFETY: the caller promises `count` arguments of type `int`.
        total += unsafe { args.next_arg::<c_int>() };   // C: va_arg(args, int)
    }
    total
}

// SAFETY: the count (3) matches the three `c_int` arguments.
assert_eq!(unsafe { sum_ints(3, 10, 20, 30) }, 60);

The four C macros of <stdarg.h> map to Rust as follows:

C	Rust
`va_start(ap, count)`	Automatic. The `...` parameter is ready when the function starts.
`va_arg(ap, T)`	`args.next_arg::<T>()`. It is `unsafe`, and it reads the next argument as `T`.
`va_copy(dst, src)`	`args.clone()`. It gives a second, independent cursor on the same arguments.
`va_end(ap)`	`Drop`. It runs automatically at the end of the scope.

A C-variadic function must be unsafe, because the compiler cannot check the variadic arguments. The caller must pass the number and the types of arguments that the function reads.

C promotes the arguments of a variadic call. float becomes double, and an integer type smaller than int becomes int or unsigned int. Thus next_arg accepts only types that this promotion does not change. On every platform, these types are:

c_int, c_long, and c_longlong
c_uint, c_ulong, and c_ulonglong
c_double
the raw pointers *const T and *mut T

A VaList is an ordinary value, and you can pass it to a non-variadic function. C uses the same pattern when printf calls vprintf. 20_09_c_variadic.rs shows that pattern and a variadic function pointer. It prints:

sum_ints(3, 10, 20, 30) = 60
sum_ints(0) = 0
average(4, 1.0, 2.0, 3.0, 6.0) = 3
min_and_max(3, 7, 2, 9): min = 2, max = 9
through a function pointer: 42

All assertions passed.

Callbacks and void* Context

C has no closures. Thus a C API that takes a callback usually also takes a void *user_data pointer (for example sqlite3_exec and GLib signals). The API passes that pointer back to your callback unchanged. To see this mechanism without external linkage, define an extern "C" fn in Rust. Then call it through a C-ABI function pointer. A C library that calls your callback uses exactly the same mechanism:

Figure: The callback-with-context pattern

20_1_c_types_calling_conventions_2

use std::ffi::{c_int, c_void};

// `UnaryOp` and `double_it` are from the first snippet of the previous section.
// This function has the role of the C library. It does not know the type
// behind `user_data`.
extern "C" fn apply_and_count(op: UnaryOp, x: c_int, user_data: *mut c_void) -> c_int {
    // SAFETY: the contract of this "library" is that `user_data` points to a
    // live `c_int` counter. The only caller obeys that contract.
    unsafe { *user_data.cast::<c_int>() += 1; }
    op(x)
}

let mut calls: c_int = 0;                                    // the context value
let ctx = std::ptr::from_mut(&mut calls).cast::<c_void>();   // ctx: *mut c_void
let a = apply_and_count(double_it, 10, ctx);
assert_eq!((a, calls), (20, 1));   // double_it(10) is 20, and the counter is 1

Two rules apply to real callbacks:

Never let a panic unwind across the C boundary. Catch it inside the callback with std::panic::catch_unwind (see tutorial 2.3).
Write the type contract of user_data in the SAFETY: comment, because the compiler cannot check it.

Closures do not have a C ABI. A non-capturing closure coerces to a plain fn pointer, but only an item that you declare extern "C" gives a C-ABI pointer. To use a capturing closure, you must pass it through user_data.

#[repr(C)]: Predictable Struct Layout

By default (repr(Rust)), the compiler may reorder fields to minimize padding. This is good for memory use, but it makes the type unusable for FFI, because C does not know the position of each field. #[repr(C)] selects the layout algorithm of C:

The compiler puts the fields in declaration order and never reorders them.
Each field starts at the next offset that is a multiple of its own alignment. The compiler inserts padding where necessary.
The alignment of the struct is the largest field alignment.
The compiler rounds the total size up to a multiple of the struct alignment, so each element of an array stays aligned.

Figure: #[repr(C)] layout vs. default repr — same fields, different bytes

repr(C) struct layout with padding

core::mem::offset_of! (stable since 1.77) reads the real offsets from the compiled layout. Thus these assertions prove the rules and do not assume them:

use std::ffi::{c_double, c_int};
use std::mem::offset_of;

#[repr(C)]
#[derive(Clone, Copy, Debug, PartialEq)]
struct SensorReading {
    channel: u8,     // offset 0
    raw: c_int,      // offset 4  (3 padding bytes before)
    scale: c_double, // offset 8
    flags: u8,       // offset 16 (7 trailing padding bytes after)
}

assert_eq!(offset_of!(SensorReading, raw), 4);                    // rule 2
assert_eq!(offset_of!(SensorReading, flags), 16);
assert_eq!(align_of::<SensorReading>(), align_of::<c_double>());  // rule 3
assert_eq!(size_of::<SensorReading>(), 24);                       // rule 4: 17 rounds up to 24

The two sides agree on the layout and on the convention. Thus a #[repr(C)] struct can cross the boundary by value. Here the struct goes through a C-ABI function pointer, which is ABI-identical to a call into a real C object file:

// `SensorReading` is the #[repr(C)] struct from the previous snippet.
// This function has the role of a C function that takes and returns the struct by value.
extern "C" fn calibrate(mut r: SensorReading) -> SensorReading {
    r.scale *= 2.0;
    r.flags |= 1;   // set bit 0 to mark the reading as calibrated
    r
}

let f: extern "C" fn(SensorReading) -> SensorReading = calibrate;
let out = f(SensorReading { channel: 3, raw: 1024, scale: 0.5, flags: 0 });
// out is SensorReading { channel: 3, raw: 1024, scale: 1.0, flags: 1 }
assert_eq!(out.flags, 1);

For an FFI enum, specify an explicit integer repr. A fieldless enum with #[repr(i32)] is layout-identical to a C int32_t enum. (Plain #[repr(C)] enum exists, but C compilers do not agree about the width of an enum.)

Remember one hazard: a C int can hold values that are not valid variants. Thus it is sound to receive an enum directly from real C only if the C side promises to stay in range. If the C side does not promise that, receive a c_int and match on it.

20_04_repr_c_structs.rs prints:

#[repr(C)] SensorReading — size 24, align 8:
  channel @ 0
  raw     @ 4
  scale   @ 8
  flags   @ 16

default repr — same fields, size 16 (compiler chose the order; NOT stable across rustc versions)

after calibrate(): SensorReading { channel: 3, raw: 1024, scale: 1.0, flags: 1 }
classify: ok / saturated / fault all round-tripped as i32-backed enum

All assertions passed.

Summary

Concept	Key point
`c_int`, `c_short`, …	Aliases with one width on all 32-bit and 64-bit targets. They encode the C ABI of the target, so signatures stay portable.
`c_long`	8 bytes on 64-bit Unix (LP64), 4 bytes on Windows (LLP64). Never hardcode `i64`.
`c_char`	The platform defines the signedness (`i8` on x86/Apple, `u8` on most ARM).
`c_void`	The type behind `void*`. You cannot construct it. It exists only as the target of a pointer.
`unsafe extern "C" { }`	Declares foreign symbols. The signatures are promises that the compiler does not check (edition 2024).
`safe fn` in extern block	Callable without `unsafe`. It declares memory safety for all inputs (1.82).
`extern "C"`	The C convention of the platform. It is part of a function type and of a function pointer type.
`extern "system"`	`"C"` on every target except 32-bit x86 Windows (`"stdcall"`). Use it for Win32 bindings.
`Option<extern "C" fn>`	Nullable callback. The null-pointer optimization keeps it at the size of a pointer.
`unsafe extern "C" fn f(n: c_int, args: ...)` (1.99)	Defines a C-variadic function. `args` is a `VaList` (`next_arg` = `va_arg`, `clone` = `va_copy`).
`void *user_data`	The C substitute for closures. Cast through `*mut c_void`, and document the type contract.
`#[repr(C)]`	Declaration order, with alignment padding for each field. Check it with `offset_of!`.
FFI enums	Specify `#[repr(i32)]`. Receive a `c_int` from untrusted C and match on it.

Code Examples

File	Description
`20_01_c_type_aliases.rs`	The `std::ffi` type aliases: the mappings that are the same on all platforms, the `c_long` and `c_char` mappings that change with the platform, and the `c_void` round trip
`20_02_extern_c_bindings.rs`	`unsafe extern "C"` declarations of `strlen` and `abs`, `safe fn`, and the safe wrapper pattern
`20_03_calling_conventions_callbacks.rs`	C-ABI function pointers, `Option<extern "C" fn>`, a callback with a `void*` context, and `extern "system"`
`20_04_repr_c_structs.rs`	`#[repr(C)]` layout checked with `offset_of!`, a struct passed by value, and `#[repr(i32)]` enums
`20_09_c_variadic.rs`	C-variadic function definitions and `VaList` (1.99): `next_arg`, the `v`-function pattern, `clone` as `va_copy`, and variadic function pointers

20.2 · String Marshalling: CString/CStr and OsString/OsStr at the Boundary

Domain 20 — FFI (Foreign Function Interface) Duration: ~15 minutes Library components: std::ffi::CString, std::ffi::CStr, std::ffi::NulError, std::ffi::OsStr, std::ffi::OsString, std::os::unix::ffi::OsStrExt, std::os::windows::ffi::OsStringExt

Introduction

Rust strings, C strings, and OS strings are three different data models that only look alike:

A String stores its length, is always valid UTF-8, and may contain \0.
A C string is a raw pointer to bytes that end at the first \0. It promises nothing about the encoding.
An OS string is the data that the operating system gives you: arbitrary bytes on Unix, and arbitrary (possibly ill-formed) UTF-16 on Windows.

Tutorial 3.4 introduced these types as data structures. This tutorial is about marshalling: how to move strings across the FFI boundary correctly in the two directions. It handles ownership, lifetimes, and failure cases explicitly.

The boundary that you cross selects the type. Keep the conversions at the edges of the program, and do not spread them through your core logic.

Figure: Three string worlds and the conversions between them

20_2_string_marshalling_1

Creating CStrings: CString::new and NulError

CString::new accepts each type that is Into<Vec<u8>>, for example &str, String, and Vec<u8>. It copies (or reuses) the bytes, scans them, and appends exactly one nul terminator. Rust strings are not nul-terminated, because they hold an explicit length. Thus the appended terminator is the reason for the conversion:

use std::ffi::CString;

let msg = CString::new("temperature nominal").expect("no interior nul");
assert_eq!(msg.as_bytes().len(), 19);           // the payload, like str::as_bytes
assert_eq!(msg.as_bytes_with_nul().len(), 20);  // the payload and the nul, as C sees it

The failure case is more important. A C string ends at the first \0, so a C string cannot represent a Rust string that contains one. A silent truncation would be an injection vulnerability ("safe part\0hidden part"). CString::new rejects such input and returns a NulError. The error gives you the position of the nul and returns the bytes:

let err = CString::new("user\0name").expect_err("must be rejected");   // err: NulError
assert_eq!(err.nul_position(), 4);        // the byte index of the first nul
let recovered: Vec<u8> = err.into_vec();  // the original bytes b"user\0name": nothing is lost

With these two methods you can sanitize untrusted input. Keep the prefix before the first nul, and do not fail:

fn sanitize_to_cstring(input: &str) -> CString {
    match CString::new(input) {
        Ok(cs) => cs,
        Err(nul_err) => {
            let pos = nul_err.nul_position();     // the index of the first nul
            let mut bytes = nul_err.into_vec();   // take back the original bytes
            bytes.truncate(pos);                  // keep only the prefix before the nul
            // The prefix contains no nul, so this call cannot fail.
            CString::new(bytes).expect("prefix has no interior nul")
        }
    }
}
assert_eq!(sanitize_to_cstring("user\0name").as_bytes(), b"user");

20_05_cstring_creation.rs prints (the static tag and round-tripped text lines come from the next section):

payload bytes = 19, with nul = 20
rejected "user\0name": interior nul at byte 4
sanitized to "user"
static tag "sensor-daemon", 13 bytes
round-tripped text: végül

All assertions passed.

c"..." Literals and Getting Text Back Out

Some text is constant at compile time: format names, config keys, and the names of symbols. For such text, you do not need an allocation. A c"..." literal (stable since 1.77) is a &'static CStr. The compiler adds the nul terminator, and no scan occurs at run time. An interior nul is a compile error, so the literal is valid by construction:

use std::ffi::CStr;

let log_tag: &'static CStr = c"sensor-daemon";
assert_eq!(log_tag.count_bytes(), 13);  // the length without the nul, like C `strlen` (1.79)

// let bad = c"ab\0cd";
// error: null characters in C string literals are not supported

The opposite conversion, from C bytes to Rust text, is always fallible, because C makes no promise about the encoding. CStr::to_str borrows the bytes and validates that they are UTF-8. CString::into_string does the same validation, but it consumes the buffer:

use std::ffi::CString;

let round = CString::new("végül").expect("no interior nul");
// into_string returns an Err if the bytes are not valid UTF-8.
let back: String = round.into_string().expect("was valid UTF-8");
assert_eq!(back, "végül");

Rust → C: as_ptr and the Dangling-Temporary Problem

When you pass a string into C, C borrows it. as_ptr returns a *const c_char that is valid only while the CString is alive. The example below calls the real strlen from the C runtime that std already links (see tutorial 20.1):

use std::ffi::{CString, c_char};

unsafe extern "C" {
    /// C: `size_t strlen(const char *s);`
    fn strlen(s: *const c_char) -> usize;
}

let device = CString::new("thermo-7").expect("no interior nul");
let ptr: *const c_char = device.as_ptr();   // valid only while `device` is alive
// SAFETY: `ptr` points into the buffer of `device`. A CString guarantees
// that the buffer is nul-terminated, and `device` outlives this call.
let seen_by_c = unsafe { strlen(ptr) };
assert_eq!(seen_by_c, 8);                   // C sees the 8 bytes of "thermo-7"

The classic FFI string bug is a pointer into a temporary:

// let dangling = CString::new("oops").unwrap().as_ptr();
// The CString is a temporary. Rust frees it at the END OF THE STATEMENT.
// `dangling` then points into freed memory. If you pass it to C, the
// result is a use-after-free. rustc warns about this pattern with the
// lint `dangling_pointers_from_temporaries`.

The correct version above has the solution: bind the CString to a variable that outlives every use of the pointer.

C → Rust: CStr::from_ptr

When you receive a *const c_char from C, you wrap it in a borrowed &CStr. This makes no copy and no allocation. CStr::from_ptr reads the bytes up to the nul terminator, so your SAFETY: comment must cover the full span. The pointer must be:

non-null
nul-terminated
valid for reads up to the nul
alive for as long as you use the borrow

Figure: The full round trip across the boundary

20_2_string_marshalling_2

use std::ffi::{CStr, c_char};

// Simulate a C API that returns `const char *`. This pointer is to a 'static
// literal. A real C API documents the lifetime of the pointer that it returns.
let from_c: *const c_char = c"ready".as_ptr();

// SAFETY: `from_c` is non-null and nul-terminated. It points to 'static
// memory, which is valid for reads up to the nul and outlives the borrow.
let borrowed: &CStr = unsafe { CStr::from_ptr(from_c) };

let text: &str = borrowed.to_str().expect("valid UTF-8"); // validates: C promised nothing
let owned: String = text.to_owned(); // a copy that stays valid when the C buffer is gone
assert_eq!(owned, "ready");

Two mistakes are common in this direction:

You assume that the bytes are UTF-8. Always validate them with to_str, or use a lossy conversion to display them.
You hold the &CStr after the C library frees or reuses its buffer. If you are not sure, copy the text to a String.

Reading C Buffers: from_bytes_until_nul and from_bytes_with_nul

C structs often contain character arrays with a constant size, such as char name[16]. The string ends at the first nul, and the bytes after it have no meaning. CStr::from_bytes_until_nul (stable since 1.69) decodes exactly this layout and needs no raw pointers:

use std::ffi::CStr;

// A 16-byte field: "pump-a", then the nul, then 9 bytes without meaning.
let name_field: [u8; 16] = *b"pump-a\0\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF";
// name: &CStr, a borrow of the first 7 bytes of the field
let name = CStr::from_bytes_until_nul(&name_field).expect("field contains a nul");
assert_eq!(name.to_bytes(), b"pump-a");   // the bytes before the nul

If the slice contains no nul, the method returns FromBytesUntilNulError. Since Rust 1.97 this error type is Copy (it is also Eq). Thus a parser can keep, compare, and reuse the error value without restriction:

let junk = [0x41u8; 8]; // "AAAAAAAA", with no terminator
let err = CStr::from_bytes_until_nul(&junk).expect_err("no nul present");
let kept = err;          // a plain copy: `err` stays usable (Copy since 1.97)
assert_eq!(err, kept);   // the two copies are equal

from_bytes_with_nul is the strict variant: the nul must be the last byte and the only nul. Use it when you already know the exact span, for example for a length-delimited record that you sliced yourself.

20_06_cstr_boundary_roundtrip.rs prints (the ownership round-trip line comes from the next section):

C's strlen sees 8 bytes for "thermo-7"
received from C: "ready"
char[16] field decodes to "pump-a"
unterminated buffer rejected: data provided does not contain a nul
ownership round-trip: "session-42"

All assertions passed.

Ownership Transfer: into_raw / from_raw

Some C APIs keep the string that you pass ("takes ownership of name"), and they return it later so that you can free it. CString::into_raw releases the buffer as a *mut c_char, and Rust no longer frees it. CString::from_raw takes the buffer back. You must use the two functions as a strict pair:

use std::ffi::{CString, c_char};

let token = CString::new("session-42").expect("no interior nul");
let raw: *mut c_char = token.into_raw();   // Rust no longer owns the buffer

// ... give `raw` to C, and get it back later ...

// SAFETY: `raw` came from CString::into_raw, this is the only call that
// reclaims it, and nothing changed the length of the string.
let token_back = unsafe { CString::from_raw(raw) };
assert_eq!(token_back.as_bytes(), b"session-42");
// `token_back` goes out of scope here, and Rust frees the buffer.

Two invariants apply:

The C function free() must never receive this pointer, because Rust and C may use different allocators.
from_raw must run exactly once. Zero calls cause a leak, and two calls cause a double free.

OsString/OsStr: The Other Boundary

CString is for C libraries. OsString is for the operating system. The OS does not guarantee that command-line arguments, environment variables, and file names are UTF-8 (Unix) or even valid Unicode (Windows). OsString and OsStr hold such values without loss. For this reason, every std API that faces the OS uses these types at the boundary: env::var_os, env::args_os, and fs::read_dir. A Path is an OsStr with path methods added (see tutorial 9.3).

Conversion in from UTF-8 is infallible. Conversion out is fallible, and there are three methods with different strictness:

use std::ffi::{OsStr, OsString};

let mut cmdline = OsString::from("cargo");   // from &str: cannot fail
cmdline.push(" build");                      // push accepts a &str
cmdline.push(OsStr::new(" --release"));      // push also accepts an &OsStr

// to_str()          -> Option<&str>              None if the value is not UTF-8
// to_string_lossy() -> Cow<str>                  replaces invalid data with U+FFFD
// into_string()     -> Result<String, OsString>  consumes, and returns the value on failure
assert_eq!(cmdline.to_str(), Some("cargo build --release"));

// display() (1.87) always works when you print an OsStr for a person.
println!("{}", OsStr::new("systemd-résumé").display());   // prints: systemd-résumé

The error variant of into_string returns the original OsString, so a failed conversion loses nothing. You can always use the lossy form to display the value to a person, and keep the exact value for the OS.

Platform Extensions: OsStrExt (Unix) and Wide Chars (Windows)

The portable OsStr API deliberately hides the native representation. The extension traits reveal it, and each trait exists only on its platform. This is the design: code that depends on the representation must select a platform explicitly with #[cfg(...)].

On Unix (std::os::unix::ffi::OsStrExt/OsStringExt), an OsStr is raw bytes. The kernel treats a file name as opaque bytes, and it permits each byte value except / and nul. Bytes that are not valid UTF-8 are still a legal file name:

#[cfg(unix)]
{
    use std::ffi::OsString;
    use std::os::unix::ffi::OsStringExt;   // adds from_vec and into_vec

    // 0x80 alone is a continuation byte. No &str can contain it.
    let os_name = OsString::from_vec(vec![b'd', b'a', b't', b'a', 0x80, b'.', b'l', b'o', b'g']);
    assert!(os_name.to_str().is_none());            // not valid UTF-8, so no &str
    let bytes = os_name.into_vec();                 // the exact bytes come back
    assert_eq!(bytes[4], 0x80);                     // the invalid byte is intact
}

On Windows (std::os::windows::ffi::OsStrExt/OsStringExt), the native unit is the UTF-16 code unit. The Win32 *W APIs take nul-terminated u16 buffers:

encode_wide() gives the code units without a terminator. Append 0 yourself.
from_wide makes an OsString from any u16 sequence, which includes unpaired surrogates. No valid Unicode string can represent them. (Internally, OsString stores WTF-8 to keep them.)

#[cfg(windows)]
{
    use std::ffi::{OsStr, OsString};
    use std::os::windows::ffi::{OsStrExt, OsStringExt};

    let mut wide: Vec<u16> = OsStr::new("café").encode_wide().collect();   // 4 code units
    wide.push(0);                                    // the terminator that CreateFileW needs

    let weird = OsString::from_wide(&[0x0064, 0xD800]); // 0xD800 is a lone high surrogate
    assert!(weird.to_str().is_none());               // ill-formed UTF-16, so no &str
    let back: Vec<u16> = weird.encode_wide().collect();
    assert_eq!(back, [0x0064, 0xD800]);              // the exact code units come back
}

Domain 21 gives more detail on the Windows side (handles, raw_arg, and wide-string process arguments, in tutorial 21.2). The conclusion here: a program that must record and restore names exactly (a backup tool, an archiver) uses the extension traits. All other code uses the portable API.

On a Unix platform, 20_08_platform_os_str_ext.rs prints:

--- Unix: OsStr is raw bytes ---
"café" as bytes: [99, 97, 102, 195, 169] (5 bytes)
non-UTF-8 name displays as: data�.log
round-tripped 9 bytes losslessly
path extension still parses: log

All assertions passed.

(On Windows, the same binary runs its wide-char branch. On other platforms, it prints a skip message.)

Summary

Concept	Key point
`CString::new`	Copies the bytes and appends one nul. It fails on an interior nul (`NulError`).
`NulError`	`nul_position()` and `into_vec()` let you recover the bytes and sanitize them, as an alternative to failure.
`c"..."` literal	`&'static CStr`, nul-terminated at compile time. An interior nul is a compile error (1.77).
`as_ptr`	A borrow: the pointer is valid only while the `CString` is alive. Never call it on a temporary.
`CStr::from_ptr`	Unsafe borrow of a C string. The `SAFETY:` comment must cover the full span up to the nul.
`to_str` / `into_string`	UTF-8 validation of text that comes back from C, because C promises nothing about the encoding.
`from_bytes_until_nul`	Decodes `char name[N]` fields. Its error is `Copy` since 1.97.
`from_bytes_with_nul`	Strict: the nul must be the last byte and the only nul. Use it for a span that you know exactly.
`into_raw` / `from_raw`	Ownership transfer to C and back. Pair them exactly once, with the same allocator.
`OsString` / `OsStr`	Hold OS-native names without loss. Conversion out is fallible (`to_str`, lossy, `into_string`).
`OsStrExt`/`OsStringExt` (Unix)	`as_bytes` / `from_bytes` / `from_vec`: file names are raw bytes.
`OsStrExt`/`OsStringExt` (Windows)	`encode_wide` / `from_wide`: UTF-16 code units. Unpaired surrogates stay intact.

Code Examples

File	Description
`20_05_cstring_creation.rs`	`CString::new` from `&str`, `String`, and bytes, `NulError` recovery and sanitizing, and `c"..."` literals
`20_06_cstr_boundary_roundtrip.rs`	The two boundary directions with `SAFETY:` comments, `from_bytes_until_nul` (its error is `Copy` since 1.97), and `into_raw`/`from_raw`
`20_07_osstring_osstr.rs`	How to build an `OsString`, the three conversions out, `display()`, and use with `env` and paths
`20_08_platform_os_str_ext.rs`	The cfg-gated `OsStrExt`/`OsStringExt` traits: raw bytes on Unix, wide characters on Windows, and round trips without loss

21.1 · Unix Extensions: File Descriptors, Permissions, Signals

Domain 21 — OS-Specific Extensions Duration: ~15 minutes Library components: std::os::unix::fs, std::os::unix::io, std::os::unix::net, std::os::unix::process

Introduction

The portable API of the standard library is an intersection of the platform APIs. For example, fs::Permissions knows only "readonly or not", because that is the one permission concept that Unix and Windows share. The concepts that are not in the intersection are in std::os. This module is a tree of extension traits that add OS-specific methods to the portable types. The extension traits do not duplicate a type. When you import std::os::unix::fs::MetadataExt, the same fs::Metadata value from fs::metadata gets inode and owner accessors.

This tutorial shows:

The extension-trait pattern: how std::os::unix extends the portable types and does not make a second copy of them.
PermissionsExt and OpenOptionsExt: the full st_mode bit field, octal permissions, and how to create a file with the correct mode atomically.
MetadataExt: inode, device, hard-link count, and owner (uid/gid), directly from struct stat.
I/O safety: OwnedFd, BorrowedFd, and the AsFd/AsRawFd/FromRawFd/IntoRawFd traits, and why they replaced bare i32 descriptors.
CommandExt: pre_exec (and why it is unsafe), arg0, and uid/gid. Also ExitStatusExt, which gives exit statuses with signal data.
Unix domain sockets with UnixListener/UnixStream, and the concepts of pipes and signals.

The code in this tutorial compiles only on Unix-family targets. Thus each example binary puts its logic behind #[cfg(unix)] and prints a skip message on other platforms. Your cross-platform code should use the same pattern.

The Extension Trait Pattern

std::os::unix exists only when you compile for the Unix family. The same is true for the related modules std::os::linux, std::os::windows, and std::os::wasi, each for its OS family. The module contains traits such as PermissionsExt, which std implements for exactly one portable type. To get the methods, you import the trait. Your Permissions value does not change. It only shows more of the data that the OS already gave it.

Figure: Portable types and their Unix extension traits

21_1_unix_extensions_1

This design has two effects on the code that you write:

The compiler decides, not the program at run time. std::os::unix does not exist on Windows, so code that names it must be behind #[cfg(unix)]. Apply the gate to a function, which keeps main easy to read. main calls the gated function, and prints a skip message on other platforms.
The portable API is a view of the same data. Permissions::readonly() means "no write bits in st_mode", and Metadata::len() is st_size. The extension traits show the other fields of the struct, which the OS fills in all cases.

PermissionsExt: Mode Bits

On Unix, the permissions of a file are nine bits: read, write, and execute for the owner, the group, and others. The usual notation is octal: 0o600 is rw-------, and 0o755 is rwxr-xr-x. PermissionsExt gives access to these bits with mode(), set_mode(), and Permissions::from_mode():

use std::os::unix::fs::PermissionsExt;

// secret: PathBuf, the path of a file that exists.
// 0o600 gives read and write access to the owner only (rw-------).
fs::set_permissions(&secret, fs::Permissions::from_mode(0o600))?;
let mode = fs::metadata(&secret)?.permissions().mode();   // u32: 0o100600
assert_eq!(mode & 0o777, 0o600);   // the mask removes the file-type bits

A common error is a comparison of the mode() result with a permission constant. mode() returns the full st_mode word, and its high bits contain the file type (0o100000 is a regular file). A new 0o600 file reports 0o100600. Thus you must apply the mask 0o777 before a comparison (or 0o7777 to keep setuid, setgid, and sticky).

The portable readonly() is derived data. It tells you only whether all the write bits are clear. A 0o444 file is "readonly", but root can still write to it.

If you set the permissions of a secret after you write it, there is a time window. In this window, the file has the default mode (filtered by the umask). OpenOptionsExt::mode closes this window, because it passes the mode to open(2):

use std::os::unix::fs::OpenOptionsExt;

// key: PathBuf, a path where no file exists.
let f = fs::OpenOptions::new()
    .write(true)
    .create_new(true)     // fails if the file exists
    .mode(0o600)          // open(2) applies the mode when it creates the file
    .open(&key)?;         // f: File

Tools such as ssh-keygen use this method to guarantee that a private key is never readable by all users, not for one instant.

21_01_unix_permissions_mode.rs prints:

full st_mode      = 0o100600
mode & 0o777      = 0o600
0o400 readonly()  = true
after set_mode    = 0o644
created with mode = 0o600 (no chmod window)
script mode       = 0o755 (executable)

All assertions passed.

MetadataExt: Inodes, Links, and Owners

std::os::unix::fs::MetadataExt gives access to the raw fields of struct stat:

ino(): the inode number
dev(): the id of the device or filesystem
nlink(): the hard-link count
uid() and gid(): the owner
size(), blocks(), and more

One model explains all these fields. A file is an inode, and a path is only a name that points to an inode. fs::hard_link creates a second name for the same inode. It does not copy data, and the two names are equally "the" file:

use std::os::unix::fs::MetadataExt;

// original: PathBuf, the path of a file that exists. alias: PathBuf, a new name.
fs::hard_link(&original, &alias)?;   // adds a second directory entry for the inode
let (a, b) = (fs::metadata(&original)?, fs::metadata(&alias)?);
assert_eq!(a.ino(), b.ino());   // the two paths have the same inode
assert_eq!(a.dev(), b.dev());   // on the same filesystem (device)
assert_eq!(a.nlink(), 2);       // two directory entries point here

The (dev, ino) pair is the canonical identity of a file. Backup tools that remove duplicates (and the file-locking code of cargo) use it to recognize that two paths are one file. When you unlink a name, nlink only decreases by one. The data stays until the last link is gone. For this reason, the name of the syscall is unlink(2), not delete.

Unix filesystems also differ from each other. The traditional rule says that the nlink of a directory is 2 + number_of_subdirectories. This rule is not true on all filesystems (btrfs reports 1). Use the nlink of a directory as information only.

21_02_unix_metadata.rs prints:

ino   = 43537696 (inode number, unique per filesystem)   # (varies)
dev   = 16777233 (id of the filesystem/device)           # (varies)
nlink = 1 (number of directory entries pointing here)
uid   = 501 / gid = 20 (owner)                           # (varies)
size  = 18 bytes, blocks = 8                             # (blocks varies)

after hard_link: nlink = 2 (same ino via both paths)
write via alias, read via original: "revised numbers!"
after unlink(original): nlink = 1, content intact
directory nlink = 4 (filesystem-dependent!)              # (varies)

All assertions passed.

I/O Safety: OwnedFd and BorrowedFd

Before Rust 1.63, a file descriptor that crossed an API boundary was a bare RawFd (an i32). Code could close it twice, use it after the close, or invent descriptor 7 with no open file behind it. These are the typical descriptor bugs of C programs. The io-safety types apply the ownership model of Rust to the descriptors:

Type	Semantics	Closes on drop?
`OwnedFd`	Owns the descriptor, as `Box` owns its heap value	Yes, exactly one time
`BorrowedFd<'a>`	A borrow that cannot outlive its owner, as `&`	No
`RawFd` (`i32`)	Only a number. Use it only at the FFI/syscall boundary	No

Figure: Descriptor ownership flow

21_1_unix_extensions_2

Each type that has a descriptor implements AsFd: File, TcpStream, UnixStream, the pipe ends, and OwnedFd. Thus a function that needs any live descriptor takes a BorrowedFd<'_>. The borrow checker guarantees that the descriptor stays open during the call:

use std::os::fd::{AsFd, BorrowedFd, OwnedFd};

// Accepts a borrow of any open descriptor.
fn describe(fd: BorrowedFd<'_>) { /* fd is open here, guaranteed */ }

// path: PathBuf, the path of a file that exists.
let owned: OwnedFd = File::open(&path)?.into(); // consumes the File: owned closes the fd
describe(owned.as_fd());                        // a borrow: owned keeps the fd
let dup = owned.try_clone()?;                   // a duplicate: two independent fds
let file_again = File::from(owned);             // back to a File, with no unsafe code

Only the FFI boundary needs unsafe. into_raw_fd() releases the ownership: the i32 is then only a number, and no code closes it. from_raw_fd() takes the ownership again. Write a SAFETY comment there that explains why the descriptor is valid and has no owner.

The conversions obey one rule. A conversion that transfers ownership (File ↔ OwnedFd) is a safe From impl. Only the creation of ownership from a raw integer is unsafe.

Pipes use the same model. std::io::pipe() (stable since 1.87) returns (PipeReader, PipeWriter). On Unix, the two ends are descriptors and implement AsFd. When you drop the writer, the write end closes and the reader gets EOF. Thus the life cycle of the descriptor is the protocol.

21_03_owned_borrowed_fd.rs prints:

owned: fd is live (raw number varies: true)
borrowed: fd is live (raw number varies: true)
dup: fd is live (raw number varies: true)
read through rehydrated File: "io-safety!"
reclaimed: fd is live (raw number varies: true)
pipe reader: fd is live (raw number varies: true)
pipe writer: fd is live (raw number varies: true)
pipe round-trip: "through the pipe"

All assertions passed.

CommandExt: Hooks into fork and exec

On Unix, Command::spawn does a fork(2) and then an execvp(2). (When the command has no pre_exec closure and no uid/gid, std can use posix_spawn.) std::os::unix::process::CommandExt lets your code run between the two calls:

Figure: Where pre_exec runs in the spawn sequence

21_1_unix_extensions_3

pre_exec registers a closure that runs in the child, after the fork and before the exec. This is the traditional point to start a new session, adjust resource limits, or unblock signals.

pre_exec is an unsafe fn for a specific reason. The child is a fork of a parent that possibly has many threads. A lock or a heap state that a different thread held at the time of the fork never changes in the child. Thus the closure must do only async-signal-safe operations: no allocation, no locking, and no buffered I/O:

use std::os::unix::process::CommandExt;

let mut cmd = Command::new("true");   // `true` exits with code 0 and prints nothing
// SAFETY: the closure only returns Ok(()). It does not allocate and it takes no lock.
unsafe { cmd.pre_exec(|| Ok(())); }   // registers the closure, does not run it
assert!(cmd.status()?.success());     // status() spawns the child, which runs the closure

If the closure returns Err, the spawn fails and the exec does not occur. The parent gets the error as the result of status() or spawn(). Thus the setup code in the child has a reliable way to report a failure.

The other methods of the trait control more of the child process:

arg0 sets argv[0] independently of the executable path. Multi-call binaries such as busybox read argv[0] to decide whether they are ls or cp.
uid and gid change the user and the group of the child before the exec. This is the canonical privilege drop for daemons. In practice, only root can use it.

ExitStatusExt gives data that the portable API does not have. status.signal() tells you which signal killed a child (code() returns None in that case). ExitStatus::from_raw makes a status from a raw wait(2) value, for tests.

21_04_command_pre_exec.rs prints:

pre_exec(Ok) child exited with: Some(0)
pre_exec(Err) surfaced in the parent: kind = PermissionDenied
sh saw argv[0] = i-am-argv0
`false`: code = Some(1), signal = None
simulated SIGTERM death: signal = Some(15)

All assertions passed.

Signal Concepts: What std Gives You (and What It Does Not)

Signals are the third mechanism of traditional Unix IPC. The standard library deliberately does not have an API to handle signals. A safe handler installation must obey async-signal-safety and must manage global process state. std leaves that work to crates such as signal-hook, or to the signal streams of an async runtime.

std does give you these operations that relate to signals:

See that a signal killed a child: ExitStatusExt::signal() and stopped_signal() decode how a child terminated (previous slide).
Send one signal: Child::kill() sends SIGKILL, the termination that a process cannot block.
Prepare the signal state of a child: pre_exec is the approved point to reset signal masks before the exec.
Default behavior: a Rust process that gets a fatal signal with no handler terminates as a C program does. std changes some signal dispositions at startup. It sets SIGPIPE to ignored. Thus a write to a closed pipe returns an ErrorKind::BrokenPipe error and does not kill the process. It also installs SIGSEGV and SIGBUS handlers, which report a stack overflow.

The general rule: use std for signals that are about other processes. Use a crate to handle the signals that your process receives.

Unix Domain Sockets

std::os::unix::net has UnixListener and UnixStream. Their API has the same shape as TcpListener and TcpStream, but the address is a filesystem path, not an IP address and a port. These sockets never use the network stack, and the OS checks their permissions as it does for files. They are the standard channel for local IPC. Examples are /var/run/docker.sock of Docker, the socket activation of systemd, and .s.PGSQL.5432 of Postgres.

use std::os::unix::net::{UnixListener, UnixStream};

// sock_path: PathBuf, a short path where no file exists.
let listener = UnixListener::bind(&sock_path)?;      // creates a socket FILE
let mut client = UnixStream::connect(&sock_path)?;   // connects by path, not by port
// The server calls listener.accept() to get its own UnixStream.

In practice, three facts make them different from TCP sockets:

The socket is a file. fs::metadata(&sock_path)?.file_type().is_socket() is true. The is_socket method comes from FileTypeExt, one more Unix extension trait. local_addr()?.as_pathname() returns the path. sun_path limits the path length (~104 bytes on macOS, ~108 on Linux). For this reason, sockets are in /tmp and /run, and never in deep directory trees.
The file outlives the listener. When you drop the listener, the descriptor closes but the file stays. A new bind to the same path fails with AddrInUse until you unlink the file. For this reason, daemons unlink stale socket files at startup.
Linux adds an abstract namespace. std::os::linux::net::SocketAddrExt::from_abstract_name makes an address from a name that the kernel registers, not from a path. There is no file on disk, and the cleanup is automatic when the last user closes the socket.

Sandboxes and read-only filesystems can forbid the bind. Thus a reliable example (and a reliable daemon) handles PermissionDenied on bind as a known condition, not as a crash.

21_05_unix_domain_sockets.rs prints the output below. In a sandbox that forbids the bind, it prints one skip: line.

socket file exists: is_socket = true
listening on: /var/folders/yk/rj3_nqx94_35jbstjmxqgzl80000gn/T/tut21_05_67038/control.sock   # (varies)
client got reply: "OK: 3 jobs running"
re-bind without unlink: AddrInUse
(abstract socket namespace is Linux-only — skipped on this OS)

All assertions passed.

Summary

Concept	Key point
Extension traits	`std::os::unix::*` adds methods to portable types. It exists only under `cfg(unix)`
`PermissionsExt::mode`	Returns the full `st_mode` word: apply the mask `0o777`. Use `from_mode`/`set_mode` to make or change permissions
`OpenOptionsExt::mode`	Sets the permissions atomically in `open(2)`: no chmod window for secrets
`MetadataExt`	`ino`, `dev`, `nlink`, `uid`, `gid` from `struct stat`. `(dev, ino)` is the file identity
Hard links	Two names, one inode. `nlink` counts the names. An unlink removes only one name
`OwnedFd` / `BorrowedFd`	Ownership and borrowing for descriptors. Exactly one close, and a borrow cannot dangle
`RawFd` + `from_raw_fd`	The `unsafe` FFI boundary: the only place for raw `i32` descriptors
`std::io::pipe`	Anonymous pipes (1.87). The ends are descriptors. A drop of the writer gives the reader EOF
`CommandExt::pre_exec`	Runs in the child between fork and exec. It is `unsafe`: async-signal-safe operations only
`CommandExt::arg0` / `uid` / `gid`	Set `argv[0]`. Drop privileges before exec (root only)
`ExitStatusExt::signal`	Tells which signal killed a child (`code()` is `None` then). `from_raw` is for tests
Signals	std observes and sends signals (`Child::kill`) but never handles them. Use `signal-hook` or a similar crate
`UnixListener` / `UnixStream`	An API with the shape of TCP, with filesystem paths as addresses. The socket file stays after the drop
Abstract namespace	Linux-only `SocketAddrExt`: sockets with a name as the address, no file, automatic cleanup

Code Examples

File	Description
`21_01_unix_permissions_mode.rs`	`PermissionsExt` mode bits, `from_mode`/`set_mode`, `st_mode` masking, `OpenOptionsExt::mode` to create secret files atomically
`21_02_unix_metadata.rs`	`MetadataExt`: inode/device identity, hard links and `nlink`, `uid`/`gid`, unlink semantics
`21_03_owned_borrowed_fd.rs`	Io-safety: `OwnedFd`/`BorrowedFd`/`AsFd`, `try_clone`, the `unsafe` raw-fd boundary, `std::io::pipe`
`21_04_command_pre_exec.rs`	`CommandExt::pre_exec` (with its SAFETY comments), `arg0`, `ExitStatusExt::signal`/`from_raw`
`21_05_unix_domain_sockets.rs`	`UnixListener`/`UnixStream` request–response, socket files, `AddrInUse` on stale paths, Linux abstract namespace

21.2 · Windows Extensions: Handles, Wide Strings, and Process Creation

Domain 21 — OS-Specific Extensions Duration: ~15 minutes Library components: std::os::windows::fs, std::os::windows::io, std::os::windows::process, std::os::windows::ffi

Introduction

Windows is not a Unix with different names. Four of its basic concepts are different:

A program reaches a kernel object through a HANDLE, not through a small-integer descriptor.
The native string type is UTF-16, not bytes.
The filesystem metadata is a set of attribute flags and timestamps with a 1601 epoch, not st_mode.
CreateProcessW receives the command line of the child as one string, not as an argv array.

std::os::windows gives access to each of these concepts through the same extension-trait pattern as tutorial 21.1. The portable types stay, and the Windows-only methods become available when you import the trait.

This tutorial shows:

I/O safety for handles: OwnedHandle, BorrowedHandle, and the AsHandle/AsRawHandle/FromRawHandle/IntoRawHandle traits. They correspond one-to-one to the Unix fd types.
OsStrExt::encode_wide and OsStringExt::from_wide: the conversion between UTF-8 and UTF-16, which includes the ill-formed UTF-16 that real Windows filenames can contain.
fs::MetadataExt: file_attributes, FILETIME timestamps (creation_time, last_write_time), and file_size.
process::CommandExt: creation_flags and raw_arg, and why each program has its own quoting rules for the command line on Windows.
The differences that are important when you write cross-platform code.

std::os::windows exists only on Windows targets. Thus the example binaries put their logic behind #[cfg(windows)]. On other platforms, they print what the Windows run would demonstrate, and they exit successfully. (A compile check for the x86_64-pc-windows-msvc target verifies the code of this tutorial.)

I/O Safety: OwnedHandle and BorrowedHandle

A program reaches a Windows kernel object (a file, a pipe, a process, an event) through a HANDLE. A HANDLE is an opaque value with the size of a pointer, and you must pass it to CloseHandle exactly one time. A second close of the same handle is a typical source of heap corruption in C and C++ programs. The io-safety types of Rust 1.63 gave handles the same ownership model as Unix descriptors. The two type families correspond exactly:

Figure: Unix and Windows io-safety types correspond one-to-one

21_2_windows_extensions_1

The conversions also obey the Unix rules:

A transfer of ownership is a safe From impl.
The compiler checks the lifetime of each borrow.
Only the creation of ownership from a raw pointer needs unsafe.

use std::os::windows::io::{AsHandle, BorrowedHandle, FromRawHandle, IntoRawHandle, OwnedHandle};

// Accepts a borrow of any open handle.
fn describe(handle: BorrowedHandle<'_>) { /* handle is open here, guaranteed */ }

// path: PathBuf, the path of a file that exists.
let owned: OwnedHandle = File::open(&path)?.into(); // consumes the File: owned closes the handle
describe(owned.as_handle());                        // a borrow: it cannot outlive owned
let dup = owned.try_clone()?;                       // calls DuplicateHandle internally
let file_again = File::from(owned);                 // back to a File, with no unsafe code

let raw = dup.into_raw_handle();                    // RawHandle: no owner closes it now
// SAFETY: raw comes from into_raw_handle above. It is valid and open, and it has no owner.
let reclaimed = unsafe { OwnedHandle::from_raw_handle(raw) };   // closes the handle on drop

Windows has two differences that Unix does not have:

Sockets are separate. A Winsock SOCKET is a different kind of value from a HANDLE. Thus std::os::windows::io has a parallel set of three types: OwnedSocket, BorrowedSocket, and RawSocket (with AsSocket). Unix uses plain fds for sockets and for all other resources.
Sentinel values differ. Some Win32 APIs return INVALID_HANDLE_VALUE (-1) for "no handle", and others return null. std has one FFI type for each convention: HandleOrInvalid and HandleOrNull. Each type converts to an OwnedHandle with try_from, which does the sentinel check for you.

Wide Strings: encode_wide and from_wide

Each Win32 API with a W suffix (CreateFileW, GetFullPathNameW, and others) uses UTF-16: sequences of u16 code units. Rust strings are UTF-8. std::os::windows::ffi does the conversion for OsStr and OsString:

Figure: Crossing the UTF-8 / UTF-16 boundary

21_2_windows_extensions_2

Two details cause most of the bugs in real programs:

encode_wide does not add a NUL terminator. C APIs expect a 0u16 at the end. Add it yourself: path.encode_wide().chain(std::iter::once(0)).collect::<Vec<u16>>(). This omission is the typical bug in manual Win32 calls.
Windows strings can be ill-formed UTF-16. A filename is an arbitrary sequence of u16 values, and an unpaired surrogate is legal on disk. OsString::from_wide accepts such a name with no loss (the internal storage is WTF-8). to_str() returns None for it, and encode_wide returns the original u16 values exactly. This is the reason why OsString is a separate type from String.

The portable UTF-16 functions run on each OS, and it is useful to know them too. str::encode_utf16() gives the same code units for valid Unicode. String::from_utf16 and from_utf16_lossy are the strict decoder and the lossy decoder, as from_utf8 and from_utf8_lossy are for UTF-8.

A character above U+FFFF ('🦀' is U+1F980) uses two code units, a surrogate pair. Thus the "length" depends on the encoding: 1 char, 4 UTF-8 bytes, 2 UTF-16 units.

// '🦀' is U+1F980, which is above U+FFFF.
let units: Vec<u16> = "🦀".encode_utf16().collect();
assert_eq!(units, [0xD83E, 0xDD80]);            // a surrogate pair: high, then low

let unpaired = [0x0068, 0x0069, 0xD83E];        // "hi" + a high surrogate with no low one
assert!(String::from_utf16(&unpaired).is_err()); // strict: returns an error
assert_eq!(String::from_utf16_lossy(&unpaired), "hi\u{FFFD}"); // lossy: U+FFFD replaces it

The block below shows the output of 21_07_windows_wide_strings.rs on Windows. On other platforms, the first section is the same, and the second section is one skip: line.

--- portable UTF-16 (all platforms) ---
"café"  as UTF-16: [0063, 0061, 0066, 00E9]
"🦀"    as UTF-16: [D83E, DD80] (surrogate pair)
lossy decode of unpaired surrogate: "hi�"
round-trip ok: "Zoë — 🦀 files"

--- std::os::windows::ffi ---
encode_wide produced 16 u16 units (incl. NUL)
from_wide -> C:\Users\Zoë
ill-formed UTF-16 round-tripped losslessly through OsString

Windows-specific assertions passed.

All assertions passed.

MetadataExt: File Attributes and FILETIME

Unix metadata has st_mode at its center. Windows metadata is a flag word and timestamps. std::os::windows::fs::MetadataExt reports exactly the data that GetFileInformationByHandle returns:

file_attributes(): a u32 of flags, such as FILE_ATTRIBUTE_READONLY (0x1), HIDDEN (0x2), SYSTEM (0x4), DIRECTORY (0x10), and ARCHIVE (0x20). The constants are in the Win32 headers (in Rust, in the windows-sys crate), not in std.
creation_time() / last_access_time() / last_write_time(): u64 FILETIME values. A FILETIME counts 100-nanosecond ticks since 1601-01-01 UTC. The epoch and the unit are different from the Unix seconds since 1970.
file_size(): the same value as the portable len().

use std::os::windows::fs::MetadataExt;

// path: PathBuf, the path of a regular file that exists.
// FILE_ATTRIBUTE_DIRECTORY: u32 = 0x10. Your code declares it, because std does not.
let meta = fs::metadata(&path)?;
let attrs = meta.file_attributes();                // u32 flag word
assert_eq!(attrs & FILE_ATTRIBUTE_DIRECTORY, 0);   // the flag is clear: not a directory
assert!(meta.last_write_time() >= meta.creation_time());   // u64 FILETIME ticks
assert_eq!(meta.file_size(), meta.len());          // the two methods return the same u64

The portable API is a thin view of these flags. Permissions::readonly() reads FILE_ATTRIBUTE_READONLY, and is_dir() reads FILE_ATTRIBUTE_DIRECTORY. Some concepts exist on one platform only. HIDDEN is a real attribute on Windows, but Unix hides a file by a naming convention (a dot as the first character). In the opposite direction, Windows has no equivalent of the Unix permission sets for owner, group, and others.

The block below shows the output of 21_08_windows_metadata.rs on Windows, with the two timestamps shortened. On other platforms, the example prints a summary of this demonstration.

file attributes = 0x00000020                         # (varies; ARCHIVE typical)
archive bit set = true
directory has DIRECTORY attribute: true
after set_readonly(true): READONLY flag present
creation_time    = 133899… (100ns ticks since 1601)  # (varies)
last_write_time  = 133899…                           # (varies)
file_size = 16 bytes (== portable len())

All assertions passed.

CommandExt: creation_flags and raw_arg

Process creation is the area where Unix and Windows differ most. Unix exec receives argv as an array, and no quoting layer exists. CreateProcessW on Windows receives one command-line string, and each program parses it again with its own rules. Most programs use the rules of the MSVC C runtime, but cmd.exe does not. std::process::Command quotes each arg(), so a child that uses the MSVC rules gets exactly the argv that you built.

Figure: One string, two quoting strategies

21_2_windows_extensions_3

raw_arg (stable since 1.62) disables the quoting of std for one argument. This is essential when the child is cmd.exe. Its parser gives &, |, and ^ a special meaning, and the quotes that std adds would give it a wrong command. The example binary shows the difference: it runs itself as its own child, and the child prints the argv that it received:

use std::os::windows::process::CommandExt;

// One argument that is hard to quote: it has spaces AND an embedded quote.
let tricky = r#"say "hi" please"#;
// arg(tricky): std quotes it, and the child receives ONE argument, intact.
//   child argv = [say "hi" please]
// raw_arg(tricky): std appends it verbatim, and the parser of the child splits it.
//   child argv = [say] [hi] [please]

// CREATE_NO_WINDOW: u32 = 0x0800_0000. Your code declares it, because std does not.
// The usual raw_arg case: cmd.exe needs an unquoted & to separate two commands.
Command::new("cmd").arg("/C").raw_arg("echo one & echo two")
    .creation_flags(CREATE_NO_WINDOW)
    .output()?;                    // the captured stdout has the lines "one" and "two"

creation_flags passes dwCreationFlags directly to CreateProcessW. These two flags are the most common:

CREATE_NO_WINDOW (0x0800_0000) runs a console child and does not show a console window. This is essential when a GUI app or a service runs a command-line program.
CREATE_NEW_PROCESS_GROUP (0x0000_0200) disconnects the child from the delivery of Ctrl-C.

Combine flags with bitwise OR. The constants come from windows-sys.

The block below shows the output of 21_09_windows_command.rs on Windows. On other platforms, the example prints a summary of this demonstration.

creation_flags run: flags-ok
via arg():     child argv = ["[say \"hi\" please]"]
via raw_arg(): child argv = ["[say]", "[hi]", "[please]"]
raw_arg to cmd /C ran both echos: ["one", "two"]

All assertions passed.

Writing Cross-Platform Code

In real codebases, the two tutorials of this domain lead to a small set of rules:

Use the portable API by default. fs, process, and net have the shared 90% of the functions. Use std::os only when the OS concept has no portable equivalent (mode bits, attribute flags, pre_exec, raw_arg).
Apply the gate to a function. Write a #[cfg(unix)] fn run_unix() and a #[cfg(windows)] fn run_windows(), and call them from a short main. Then the code of each platform stays together, and the other platform compiles a skip path. Each binary in this domain uses this pattern.
Use the deliberate symmetries. Examples are OwnedFd ↔ OwnedHandle, AsFd ↔ AsHandle, and try_clone on the two owned types. Code with a structure of "an owned OS resource" moves to the other platform almost mechanically.
Accept the deliberate asymmetries. There is no pre_exec on Windows (no fork) and no raw_arg on Unix (no single command-line string). There are no mode bits on Windows and no HIDDEN attribute on Unix. When a concept of one platform does not exist on the other, that absence is information. Design the abstraction for capabilities. Do not design it as if the platforms were the same.
Use OsString for platform strings. It holds arbitrary bytes on Unix and arbitrary u16 sequences (as WTF-8) on Windows. It is the only type that can hold each real filename with no loss on the two platforms.

Concern	Unix	Windows
I/O resource	`OwnedFd` / `BorrowedFd` (`RawFd = i32`)	`OwnedHandle` / `BorrowedHandle` (`RawHandle = *mut c_void`), with separate socket types
Native strings	Arbitrary bytes (`OsStrExt::as_bytes`)	UTF-16, possibly ill-formed (`encode_wide` / `from_wide`)
Permissions	9 mode bits + setuid/setgid/sticky	`READONLY` attribute (ACLs are not in the scope of std)
Timestamps	Seconds + nanos since 1970	100 ns ticks since 1601 (FILETIME)
Spawn model	fork + exec (argv array, `pre_exec` window)	single `CreateProcessW` call (one command-line string)
Local IPC	Unix domain sockets in std	Named pipes and AF_UNIX exist, but not in stable std (`std::os::windows::net` is unstable in 1.99). Use crates

Summary

Concept	Key point
`OwnedHandle` / `BorrowedHandle`	Ownership and borrowing for HANDLEs. Exactly one `CloseHandle`, and a borrow cannot dangle
`RawHandle` + `from_raw_handle`	The `unsafe` Win32 FFI boundary, as `RawFd` is on Unix
`OwnedSocket` family	Winsock `SOCKET`s have their own three io-safety types. Unix uses fds for all resources
`encode_wide`	`OsStr` → UTF-16 code units. You must append the NUL terminator
`from_wide`	`&[u16]` → `OsString`, with no loss even for unpaired surrogates (WTF-8)
`String::from_utf16(_lossy)`	Portable strict and lossy UTF-16 decoders. A char above U+FFFF is a surrogate pair
`MetadataExt::file_attributes`	Flag word (`READONLY`, `HIDDEN`, `DIRECTORY`, `ARCHIVE`…). The constants come from `windows-sys`
FILETIME timestamps	`creation_time` and the related methods count 100 ns ticks since 1601-01-01 UTC
`CommandExt::creation_flags`	Raw `dwCreationFlags` for `CreateProcessW`, for example `CREATE_NO_WINDOW`
`CommandExt::raw_arg`	Appends one argument verbatim, with no quoting by std. Use it for `cmd.exe /C` and similar programs
Quoting model	A Windows child parses ONE command-line string again. Each program has its own quoting rules
Cross-platform shape	Apply the gate to functions, use the fd↔handle symmetry, and use `OsString` for native strings

Code Examples

File	Description
`21_06_owned_borrowed_handle.rs`	`OwnedHandle`/`BorrowedHandle`/`AsHandle`, `try_clone` (DuplicateHandle), the `unsafe` raw-handle boundary
`21_07_windows_wide_strings.rs`	`encode_wide`/`from_wide`, NUL termination, unpaired surrogates, and the portable `encode_utf16`/`from_utf16` on each OS
`21_08_windows_metadata.rs`	`file_attributes` flags, the `READONLY` attribute behind `set_readonly`, FILETIME timestamps, `file_size`
`21_09_windows_command.rs`	`creation_flags` (`CREATE_NO_WINDOW`), the quoting of `arg` and `raw_arg` (the binary runs itself as a child), `cmd /C` with `&`

22.1 · std::hint: Guiding the Optimizer

Domain 22 — Compiler Hints and Low-Level Intrinsics Duration: ~15 minutes Library components: std::hint::unreachable_unchecked, std::hint::spin_loop, std::hint::black_box, std::hint::assert_unchecked, std::hint::cold_path

Introduction

All the functions in std::hint have one property in common: a call to a hint does nothing at runtime. The call computes no value, has no side effect, and changes no observable behavior. The call changes the code that the compiler generates around it. A hint is a message to the optimizer (and, for spin_loop, to the CPU). The semantics of Rust alone cannot express that message.

The module has two groups of hints:

Safe hints (black_box, spin_loop, cold_path, select_unpredictable): incorrect use can only make your program slower. It cannot make your program incorrect.
Unsafe hints (assert_unchecked, unreachable_unchecked): each one is a promise. The optimizer deletes checks and branches because of the promise. A false promise is immediate undefined behavior (UB).

This tutorial shows:

black_box: how to prevent dead-code elimination and constant folding in micro-benchmarks.
spin_loop: the CPU-level backoff hint for bounded spin-wait loops.
cold_path: how to mark the branch that almost never runs (stabilized in 1.95).
assert_unchecked: how to state an invariant that the optimizer cannot prove without help.
unreachable_unchecked: the match arm that must never run.

The Hint Spectrum: Advice vs. Promises

Use this model. A safe hint is advice: "this branch is rare", "treat this value as opaque". The compiler may use the advice or ignore it. An unsafe hint is an axiom: "this condition is true", "this point is unreachable". The compiler trusts an axiom fully and bases proofs on it.

Incorrect advice only makes the code slower. A false axiom makes each deduction from it incorrect. Thus those two functions are unsafe fn, although they generate no code.

Figure: The std::hint family — advice on the left, promises on the right

22_1_hint_guiding_optimizer_1

Obey this discipline before you use a hint. Try a safe hint when profiling suggests it. Make an unsafe hint the last transformation that you apply to a hot path. Apply it only after you measure, and only with a written SAFETY: proof that a reviewer can check locally.

black_box: Keeping Micro-Benchmarks Valid

black_box(x) is the identity function: it returns x unchanged. But the optimizer must treat the value as maximally opaque. The optimizer cannot assume where the value came from or how the program will use it. This prevents two problems that silently make a manually written benchmark incorrect:

Dead-code elimination (DCE). If nothing reads the result of a computation, the compiler deletes the computation. Then your timing loop measures no work.
Constant folding and hoisting. If the compiler knows the input, it may compute the answer one time at compile time. It may also move the call out of the loop (hoisting).

The solution has two parts. Apply black_box to the input and to the output:

use std::hint::black_box;
use std::time::Instant;

// checksum(&[u64]) -> u64 is the function under test. It has no side effects.
// data: Vec<u64> is the input. ITERS: u32 is the number of repetitions.
let start = Instant::now();
for _ in 0..ITERS {
    let result = checksum(black_box(&data)); // input: opaque, no hoisting
    black_box(result);                       // output: "used", no DCE
}
let hardened = start.elapsed();              // Duration of the full loop

criterion and the unstable cargo bench harness use this same mechanism internally. The documentation gives two caveats. First, black_box is best-effort, not a guarantee. Second, it has no security properties: it does not give you constant-time execution for cryptographic code.

22_01_black_box_benchmark.rs measures two loops. The naive loop calls checksum(&data) and discards the result. The hardened loop is the loop above.

The example prints the timings and never asserts on them. Wall-clock durations depend on the machine, the load, and the build profile. An assertion on time makes a benchmark example an unreliable test. The example prints:

naive loop    (20 iters): 1.506042ms   # (varies)
hardened loop (20 iters): 1.351334ms   # (varies)
checksum = 0x2ad2f6ce369f7ee8 (identical with and without black_box)

All assertions passed.

In this unoptimized dev-profile run, the two loops have almost the same duration. With --release, the compiler may delete the full naive loop.

spin_loop: Cooperative Busy-Waiting

spin_loop() emits the instruction that tells the CPU "this thread spins". The instruction is PAUSE on x86, ISB on AArch64, and YIELD on 32-bit ARM. spin_loop() does not yield to the OS scheduler, and the thread continues to run. It tells the CPU core to decrease its activity, with three effects:

The core saves power.
The core gives shared pipeline resources to a sibling hyper-thread.
The core prevents a memory-order mis-speculation stall when the awaited flag changes.

A realistic spin-wait is bounded. Spin for a fixed budget while you expect the wait to be microseconds. Then use thread::yield_now() (or a real parking primitive) as the alternative. Thus a delayed producer never costs you a full scheduler quantum:

use std::hint;
use std::sync::atomic::AtomicBool;
use std::sync::atomic::Ordering;
use std::thread;

// Waits until a different thread stores `true` in `flag`.
// Returns the number of spins, for information only.
fn spin_wait_bounded(flag: &AtomicBool) -> u32 {
    const SPIN_BUDGET: u32 = 10_000;        // maximum number of spins
    let mut spins = 0u32;
    // The Acquire load pairs with the Release store of the producer thread.
    while !flag.load(Ordering::Acquire) {
        if spins < SPIN_BUDGET {
            hint::spin_loop();      // the thread stays scheduled and tells the CPU
            spins += 1;
        } else {
            thread::yield_now();    // budget used: give the time slice to the OS
        }
    }
    spins
}

spin_loop is not these three things:

It is not thread::yield_now(): it does not go into the kernel.
It is not a synchronization primitive. The Acquire/Release atomics do the synchronization (see Tutorial 6.4).
It is not for long waits. An unbounded spin on a single-core machine can delay the producer for which you wait.

Tutorial 11.2 shows sleep and park, which are the tools for long waits.

In 22_02_spin_loop_wait.rs, a producer thread publishes the sum of 1 to 1,000 and then sets the flag. The main thread calls spin_wait_bounded. The example prints:

published value: 500500
spins before flag observed: 2505   # (varies)

All assertions passed.

cold_path: Marking the Rare Branch

cold_path() (stabilized in 1.95) is a safe, expression-level hint. Its message is "control flow rarely gets here". The optimizer then puts the enclosing branch away from the hot instruction stream, so the common path stays dense in the instruction cache. The optimizer also biases the branch-weight metadata. An incorrect cold_path costs some cycles on the rare branch. It can never cause UB.

The typical use is an error branch in a hot parsing loop. The error is possible, so the code must handle it. The error is rare, so its code should not be in the layout of the hot path:

use std::hint;

// BadReading { line_no: usize } is the error type of the example.
// Parses one `name=value` line, for example "t7=21", and returns the value.
fn parse_line(line_no: usize, line: &str) -> Result<i32, BadReading> {
    // value: Option<&str> is the text after '=' (None if the line has no '=').
    let value = line.split_once('=').map(|(_, v)| v.trim());
    if let Some(v) = value.and_then(|v| v.parse::<i32>().ok()) {
        Ok(v)                       // hot path: the text is a valid i32
    } else {
        hint::cold_path();          // rare: move this code out of the hot path
        Err(BadReading { line_no })
    }
}

cold_path is the expression-level form of the #[cold] function attribute. The standard library puts #[cold] on its internal panic functions. Two more hints complete the layout-hint family:

hint::likely/unlikely are still unstable as of 1.99.
hint::select_unpredictable(cond, a, b) (stable since 1.88) is for the opposite case. There, the branch predictor cannot predict the condition. The hint asks the compiler for a branchless conditional move.

22_05_cold_path.rs parses 1,000 lines, of which 2 are malformed. Then it selects one of two strings with select_unpredictable. The example prints:

parsed 998 readings, 2 malformed
sum of readings = 1495500
select_unpredictable picked: even

All assertions passed.

assert_unchecked: Promising an Invariant

unsafe fn assert_unchecked(cond: bool) tells the optimizer that cond is true. The compiler emits no runtime check. The optimizer may delete each branch, bounds check, and panic path that the condition makes impossible. If the promise is false, the behavior of the full program is undefined.

The optimizer does not always need this help. In one function, LLVM usually proves simple facts without help (x % n < n, "the code compared this index a moment ago"). The hint is useful when the proof and the use are far apart: a function boundary, a loop, or a struct field separates them. Then the optimizer no longer knows an invariant that your code does keep:

use std::hint;

struct Histogram {
    counts: [u32; 16],
    slot: usize, // invariant: always < 16 (`select` makes sure of this)
}

impl Histogram {
    // The only function that writes `slot`.
    fn select(&mut self, raw: usize) {
        self.slot = raw % 16; // the result is in 0..=15
    }

    // Adds 1 to the selected counter and returns the new count.
    fn bump(&mut self) -> u32 {
        // SAFETY: only `select` writes `slot`, and `select` reduces it
        // modulo 16. `counts` has exactly 16 elements, so `slot < 16` is true.
        unsafe { hint::assert_unchecked(self.slot < self.counts.len()) };
        self.counts[self.slot] += 1; // an optimized build has no bounds check here
        self.counts[self.slot]
    }
}

This design has an advantage over get_unchecked. The access keeps the safe indexing syntax, and only the one clearly stated proposition is unsafe. An incorrect hint is still UB, but the unsafe surface is one sentence that a reviewer can check locally.

The recommended default is still plain assert!. It keeps the runtime check, and after the check the optimizer knows the condition too. Use assert_unchecked only on measured hot paths. As an aid during development, the current standard library traps (aborts) if the condition is false while debug assertions are on. Do not rely on this behavior.

22_03_assert_unchecked.rs uses the histogram above. It also has a rounded division, div_rounded, which gives the same hint after an early return for a zero divisor. The example prints:

div_rounded(10, 4) = Some(3)
div_rounded(7, 0)  = None
bucket 3 count = 3

All assertions passed.

unreachable_unchecked: The Impossible Arm

unsafe fn unreachable_unchecked() -> ! promises that control flow can never reach this point. The optimizer deletes the branch fully: no panic code and no jump-table slot remain. If control flow reaches the point, the result is immediate UB. There is a worse effect: the compiler may delete the checks that go to that point. If your proof is incorrect, the compiler then works against you.

The typical use is a match arm that the exhaustiveness rules make you write, but that arithmetic makes impossible:

use std::hint;

// Direction is an enum with four variants: North, East, South, West.
// Decodes a 2-bit field into a Direction.
fn decode(two_bits: u8) -> Direction {
    match two_bits % 4 {
        0 => Direction::North,
        1 => Direction::East,
        2 => Direction::South,
        3 => Direction::West,
        // The matched type is u8, so the compiler demands an arm for 4..=255.
        // SAFETY: `two_bits % 4` is always in 0..=3, so the arms above are
        // exhaustive over every value that can actually occur.
        _ => unsafe { hint::unreachable_unchecked() },
    }
}

With the hint, the match has no panic branch. This example is small, so the 1.99 optimizer also finds the proof without the hint. In an optimized build, decode is one bit mask (AND) with no branch, with the hint or with unreachable!(). A reviewer can see immediately that the precondition is true. Apply that standard to each use of this hint.

Figure: The escalation ladder — justify each step with a measurement

22_1_hint_guiding_optimizer_2

As with assert_unchecked, a build with debug assertions currently traps and does not show silent UB. This is an aid for development only.

22_04_unreachable_unchecked.rs unpacks four 2-bit fields from one byte with decode. Then it decodes all 256 byte values. The example prints:

packed 0b11100100 -> [North, East, South, West]
all 256 byte values decode without reaching the impossible arm

All assertions passed.

Summary

Hint	Safety	Tells the compiler	Cost of an incorrect hint
`black_box(x)`	safe	treat `x` as opaque: no DCE, no constant folding	slower code (missed optimizations)
`spin_loop()`	safe	this thread is in a spin-wait (CPU backoff instruction)	wasted power only
`cold_path()`	safe	this branch is rare: put it away from the hot path (1.95)	slightly slower rare branch
`select_unpredictable(c, a, b)`	safe	the predictor cannot predict this branch: emit a branchless select (1.88)	slower select
`assert_unchecked(cond)`	unsafe	`cond` is true: delete the checks that it makes redundant	undefined behavior
`unreachable_unchecked()`	unsafe	control flow never reaches this point: delete the branch	undefined behavior

Obey these rules:

A hint generates no code of its own, except the one CPU instruction of spin_loop. It changes the code around it.
Use the safe hints freely where profiling suggests them.
Use the two unsafe hints as measured optimizations, and only when no other option remains. Give each one a SAFETY: comment with a proof that a reviewer can check inside the function.
Prefer the safe alternatives (assert! and unreachable!()) until a profile shows a reason to change.

Code Examples

File	Description
`22_01_black_box_benchmark.rs`	`black_box` on the input and on the output of a checksum timing loop, to prevent hoisting and dead-code elimination
`22_02_spin_loop_wait.rs`	Bounded spin-wait on an `AtomicBool` with `spin_loop`, which changes to `yield_now` after the spin budget
`22_03_assert_unchecked.rs`	`assert_unchecked` in two cases: a zero-divisor fact after an early return, and a struct-field invariant for an indexing hot path
`22_04_unreachable_unchecked.rs`	`unreachable_unchecked` on the impossible arm of a 2-bit field decoder, with a test of all 256 input bytes
`22_05_cold_path.rs`	`cold_path` on the rare branch for malformed lines in a telemetry parser, and `select_unpredictable`

22.2 · SIMD with std::arch: Platform Intrinsics

Domain 22 — Compiler Hints and Low-Level Intrinsics Duration: ~15 minutes Library components: std::arch::x86_64, std::arch::aarch64, std::arch::wasm32, is_x86_feature_detected!, is_aarch64_feature_detected!

Introduction

SIMD (Single Instruction, Multiple Data) lets a modern CPU add four floats, or compare sixteen bytes, in one instruction. std::arch gives you the vendor intrinsics for each architecture. An intrinsic is a thin wrapper for one instruction. Its name comes from the reference manuals of Intel and Arm (_mm_add_ps, vaddq_f32).

std::arch is deliberately the low-level interface to SIMD. The portable abstraction (std::simd) is still unstable as of 1.99, so std::arch is the SIMD interface of stable Rust. memchr, ripgrep, hash implementations, codecs, and simdjson-style parsers use it.

SIMD is important because hot loops in signal processing, compression, and cryptography do the same small operation on long buffers. A lane is one element of a SIMD register. An instruction that processes 16 lanes, not 1, gives a larger speed increase than almost all algorithmic micro-tuning. Auto-vectorization depends on the decisions of the optimizer. Intrinsics make the speed increase guaranteed.

This tutorial shows:

The modules of std::arch for each architecture, and the difference between compile-time detection and runtime detection.
How to detect CPU features at runtime with is_x86_feature_detected! and is_aarch64_feature_detected!.
#[target_feature(enable = "...")] and its safety contract on Rust 1.99.
Two kernels with runtime dispatch and a scalar fallback: a 4-lane f32 addition, and a byte counter that processes 16 lanes in each step.
The FP16 intrinsics (stabilized in 1.94), and why this tutorial describes them but has no example for them.

The std::arch Landscape: Two Questions, Two Tools

Portable SIMD code must answer two different questions with two different mechanisms:

For which architecture does the compiler build this binary? #[cfg(target_arch = "...")] gives the answer at compile time. The compiler does not compile code for other architectures. For example, std::arch::aarch64 does not exist in an x86-64 build.
Does the CPU that runs this binary support feature X? is_x86_feature_detected! / is_aarch64_feature_detected! give the answer at runtime. The macro queries the OS or the CPU one time and caches the result, so a probe in a loop is cheap.

You need the two mechanisms because instruction-set extensions are different between CPUs of one architecture. Each x86-64 CPU has SSE2 (it is part of the baseline), but AVX2 and AVX-512 came in later generations. With -C target-feature=+avx2, the compiler can use AVX2 in all the code. Then the binary stops with an illegal-instruction fault on older CPUs. Runtime dispatch keeps one portable binary that adapts to the machine on which it runs.

Figure: The portable SIMD dispatch pattern

22_2_simd_std_arch_1

std::arch currently has modules for x86, x86_64, arm, aarch64, wasm32, and some others. For wasm32, you detect the v128 SIMD support at compile time, not at runtime. All the items in these modules use the names of the vendor. Thus the intrinsics references of Intel and Arm are the applicable documentation.

Runtime Feature Detection

The detection macros take the feature name as a string literal. If the name has a spelling error, you get a compile error, not a silent false. Baseline features always give true (sse2 on x86-64, neon on AArch64). The results that are different between CPUs are those for the newer extensions:

// The compiler builds this block only for AArch64.
#[cfg(target_arch = "aarch64")]
{
    // Each macro call returns a bool.
    let neon = std::arch::is_aarch64_feature_detected!("neon");       // always true
    let aes = std::arch::is_aarch64_feature_detected!("aes");         // varies by CPU
    let sve = std::arch::is_aarch64_feature_detected!("sve");         // varies by CPU
    let dotprod = std::arch::is_aarch64_feature_detected!("dotprod"); // varies by CPU
    assert!(neon, "NEON is mandatory in the AArch64 baseline");
}

// The compiler builds this block only for x86-64.
#[cfg(target_arch = "x86_64")]
{
    let sse2 = std::arch::is_x86_feature_detected!("sse2");       // always true
    let avx = std::arch::is_x86_feature_detected!("avx");         // varies by CPU
    let avx2 = std::arch::is_x86_feature_detected!("avx2");       // varies by CPU
    let avx512f = std::arch::is_x86_feature_detected!("avx512f"); // varies by CPU
    assert!(sse2, "SSE2 is mandatory in the x86-64 baseline");
}

22_06_feature_detection.rs prints each result. This is its output on an Apple Silicon machine. That machine has an AArch64 CPU with the AES and dot-product extensions, but without SVE:

compiled for target_arch = aarch64   # (varies by platform)
neon    = true
aes     = true     # (varies by CPU)
sve     = false    # (varies by CPU)
dotprod = true     # (varies by CPU)

All assertions passed.

Real libraries move the detection out of the hot loop. Typically, they select a function pointer one time (memchr caches the dispatch decision in an atomic). The macros cache the CPUID/OS query, but a predictable branch for each call is still a branch.

#[target_feature] and the Safety Contract on 1.99

#[target_feature(enable = "neon")] compiles one function as if the feature were on for the full program. In that function, the compiler may use NEON registers and instructions freely, and it may auto-vectorize your ordinary code. There is one danger: if such a function runs on a CPU without the feature, the behavior is undefined.

That one fact is the reason for the safety rules. Rust 1.86 (the target_feature_11 work) and Rust 1.87 (safe intrinsics) changed the rules to the form that stable 1.99 uses:

You may declare the function safe (fn, not unsafe fn).
In the function, you can call register-only intrinsics for that feature without unsafe, because the attribute guarantees the feature. Intrinsics that access memory (vld1q_f32, _mm_loadu_ps) still need unsafe for the pointer dereference, as each raw-pointer operation does.
A call from a function that does not have the same #[target_feature] attribute still needs an unsafe block. A feature that the build configuration enables does not remove this rule. Your runtime detection check is the justification that you write in the SAFETY: comment.

#[cfg(target_arch = "aarch64")]
#[target_feature(enable = "neon")]
fn add4_neon(lhs: &[f32; 4], rhs: &[f32; 4]) -> [f32; 4] {    // a safe fn
    use std::arch::aarch64::{vaddq_f32, vld1q_f32, vst1q_f32};
    let mut out = [0.0f32; 4];
    // vld1q_f32 loads 4 f32 values through a pointer into a float32x4_t register.
    // SAFETY: the pointers come from &[f32; 4], so each 16-byte load is in bounds.
    let a = unsafe { vld1q_f32(lhs.as_ptr()) };
    let b = unsafe { vld1q_f32(rhs.as_ptr()) };
    let sum = vaddq_f32(a, b); // register-only: no unsafe needed here
    // vst1q_f32 stores the 4 lanes of `sum` through a pointer.
    // SAFETY: `out` has 4 f32 values, so the 16-byte store is in bounds.
    unsafe { vst1q_f32(out.as_mut_ptr(), sum) };
    out
}

// At the call site, in the dispatcher `add4`.
// `add4` returns ([f32; 4], &'static str): the sum and the name of the path.
if std::arch::is_aarch64_feature_detected!("neon") {
    // SAFETY: the line above checked that this CPU supports NEON. That is
    // the only precondition of `add4_neon`.
    return (unsafe { add4_neon(lhs, rhs) }, "neon");
}

This division is clear. The body is safe because the attribute guarantees the feature. The call is unsafe because only the caller knows which CPU runs the code.

A First Kernel: 4-Lane f32 Addition

The standard pattern in the ecosystem has four parts:

A plain scalar version, which is the reference result and the fallback.
One #[target_feature] function for each architecture.
A runtime dispatcher.
An assertion that the SIMD result equals the scalar result.

The assertion is exact, not approximate. SIMD lanes do ordinary IEEE-754 arithmetic. Thus, for values that are exactly representable (small integers, sums of small floats), the results are bit-identical and assert_eq! is exact.

Figure: One 128-bit instruction, four independent f32 additions

SIMD lane-wise addition

The NEON version loads the two arrays into 128-bit float32x4_t registers, adds all four lanes with one FADD, and stores the result. The SSE2 version (_mm_loadu_ps / _mm_add_ps / _mm_storeu_ps on __m128) has the same structure, line for line. This fact makes the maintenance of kernels for many architectures much easier than it seems.

22_07_simd_add_fallback.rs adds two arrays through the dispatcher and compares the result with the scalar version. The example prints:

dispatched to: neon   # (varies by platform)
[1.0, 2.0, 3.0, 4.0] + [10.0, 20.0, 30.0, 40.0]
  = [11.0, 22.0, 33.0, 44.0]
horizontal sum of lanes = 110

All assertions passed.

There are two types of operation:

The addition above is a vertical operation. Lane i of the result depends only on lane i of the inputs. This is the natural, fast case.
A sum across the lanes of one register is a horizontal reduction. NEON has vaddvq_f32 for it. On x86, it needs a sequence of shuffles.

SIMD is most effective for vertical operations.

A Realistic Kernel: Counting Bytes 16 at a Time

Byte search is the typical workload for SIMD. It is the kernel in memchr, ripgrep, and simdjson. Compare each byte of a 16-byte block with the needle in one instruction. Then reduce the lane mask to a count:

NEON: vceqq_u8 gives 0xFF in each lane that matches. vandq_u8 masks each lane to 1, and vaddvq_u8 adds the lanes horizontally.
SSE2: _mm_cmpeq_epi8 makes the same mask. _mm_movemask_epi8 compresses it to a 16-bit integer, and count_ones() counts the matches.

The scalar loop processes one byte in each iteration. The vector loop processes sixteen. The second part is the chunking pattern. slice::as_chunks::<16>() (stabilized 1.88) divides a slice into full 16-byte blocks for the SIMD path and a tail for the scalar code:

// count_scalar(hay: &[u8], needle: u8) -> u32 counts the matches byte by byte.
// count_block_neon(block: &[u8; 16], needle: u8) -> u32 is the NEON kernel.
// It is a safe fn with #[target_feature(enable = "neon")].
// Returns the count and the name of the path that ran.
fn count_fast(hay: &[u8], needle: u8) -> (u32, &'static str) {
    // blocks: &[[u8; 16]]. tail: &[u8] with fewer than 16 bytes.
    let (blocks, tail) = hay.as_chunks::<16>();
    #[cfg(target_arch = "aarch64")]
    if std::arch::is_aarch64_feature_detected!("neon") {
        let mut n = 0u32;
        for block in blocks {
            // SAFETY: the `if` above checked at runtime that the CPU supports NEON.
            n += unsafe { count_block_neon(block, needle) };
        }
        return (n + count_scalar(tail, needle), "neon");
    }
    // ... the same block for x86_64 with SSE2, and then the scalar fallback:
    (count_scalar(hay, needle), "scalar")
}

The example checks that the SIMD count equals the scalar count for all 256 possible needle values. This exhaustive check is cheap in a test, and it is very valuable when you port a kernel to a new architecture.

22_08_simd_byte_search.rs counts the byte 'o' in a 43-byte text. The example prints:

dispatched to: neon   # (varies by platform)
counting 'o' in 43 bytes: 4
SIMD/scalar parity verified for all 256 needle values
processed 2 SIMD blocks + 11-byte scalar tail

All assertions passed.

FP16 Intrinsics and Future Work

Rust 1.94 stabilized two large groups of half-precision intrinsics: the AVX-512 FP16 family in std::arch::x86_64 and the NEON FP16 family in std::arch::aarch64. Half-precision floats are important where memory bandwidth is the bottleneck and a precision of 3 decimal digits is sufficient. Examples are neural-network inference, image processing, and audio DSP. The reason is that an element of half the size gives two times the lanes for each instruction.

This tutorial describes these intrinsics but has no example for them. The intrinsics operate on opaque vector types, but work with individual half-precision values needs the f16 primitive. f16 is still unstable on 1.99 (feature gate f16, tracking issue #116909). On stable, let x: f16 = 1.0; fails with "the type f16 is unstable".

Until f16 becomes stable, stable FP16 SIMD code can exist, but it cannot easily splat, extract, or print scalar half-precision values. This difference between stable intrinsics and unstable ergonomics will go away when the type becomes stable.

std::simd is also a future feature. It is portable SIMD: you write f32x4 one time, and the compiler targets NEON, SSE, AVX, or wasm automatically. It stays nightly-only as of 1.99. Thus the std::arch pattern of this tutorial, with one kernel for each architecture, is the current solution for production code.

For WebAssembly, std::arch::wasm32 gives you the fixed-width v128 SIMD proposal. You select wasm features at compile time (#[cfg(target_feature = "simd128")]), because a wasm module with SIMD instructions either passes validation or does not.

Summary

Concept	Key point
`std::arch::{x86_64, aarch64, wasm32, ...}`	Vendor intrinsics for each architecture, with the names from the Intel and Arm reference manuals
`#[cfg(target_arch)]`	Compile-time question: for which architecture is this binary?
`is_x86_feature_detected!` / `is_aarch64_feature_detected!`	Runtime question: does this CPU have the extension? (cached, cheap)
Baselines	`sse2` is always true on x86-64, and `neon` is always true on AArch64
`#[target_feature(enable = ...)]`	Compiles one function with the feature on. The body is safe, but a call from a function without the same attribute needs `unsafe` and a detection check (1.99 rules)
Dispatch pattern	scalar reference → kernel for each architecture → runtime dispatch → `assert_eq!(simd, scalar)`
`slice::as_chunks::<N>()`	Divides a slice into full N-element blocks (SIMD) and a tail (scalar). Stabilized in 1.88
Vertical vs horizontal	Operations that are independent for each lane are fast. Reductions across lanes cost more
FP16 intrinsics	AVX-512 FP16 and NEON FP16 stabilized in 1.94. The `f16` type is still unstable on 1.99
`std::simd`	The portable future interface. It is still nightly-only, so use `std::arch` today

Remember this discipline. Apart from the raw-pointer loads and stores, SIMD code is unsafe at exactly one boundary: the call into a #[target_feature] function from a function without the attribute. A runtime detection check makes that call sound. Always keep a scalar version: it is your fallback, your reference result, and your documentation.

Code Examples

File	Description
`22_06_feature_detection.rs`	Compile-time `cfg(target_arch)` and runtime `is_*_feature_detected!`, with probes for NEON/AES/SVE/dot-product and SSE2/AVX/AVX2/AVX-512F
`22_07_simd_add_fallback.rs`	4-lane `f32` addition with runtime dispatch: NEON `vaddq_f32` and SSE2 `_mm_add_ps` kernels. An assertion compares the result with the scalar reference
`22_08_simd_byte_search.rs`	The kernel pattern of `memchr`: a count of byte matches, 16 lanes in each step. It uses `as_chunks`, a scalar tail, and an equality check for all 256 needle values

23.1 · Debug Formatting and Diagnostic Output

Domain 23 — Testing, Debugging, and Pattern Matching Duration: ~15 minutes Library components: std::fmt::Debug, std::fmt::DebugStruct, std::fmt::DebugTuple, std::fmt::DebugList, std::fmt::DebugMap, std::any::type_name, dbg!

Introduction

Debug is the trait that implements the {:?} format. This format shows a value fully, for diagnosis. These items use it:

dbg!
the failure reports of assert_eq!
the panic messages of unwrap
almost all the log lines that you read when you debug a program

Display (Tutorial 3.2) is an interface for end users, and its author selects what it shows. Debug is a diagnostic interface for programmers. Its output should be accurate and complete, and it should be easy to get.

This tutorial shows:

#[derive(Debug)], and the compact {:?} form compared with the pretty {:#?} form.
dbg! for quick print-style debugging, with no formatting code.
How to write a manual Debug implementation with the Formatter builders: debug_struct, debug_tuple, debug_list, debug_map, debug_set. It also shows why the builders are better than direct write! calls.
A Debug implementation that redacts passwords and API keys, so that they do not go into logs.
fmt::Pointer ({:p}) to print addresses, and type_name / type_name_of_val to examine types at run time (Tutorial 17.1 gives more detail).

One rule applies to all of this tutorial: Debug output is not a stable interface. Its exact text may change between compiler releases. Thus, never parse it in production code. Assert on it only in tests that you are ready to update.

derive(Debug): `{:?}` and `{:#?}`

#[derive(Debug)] is valid when each field also implements Debug. Use it as your default. The derived implementation prints the type name, then each field recursively. Thus a full tree of derived types needs no formatting code.

Figure: Choosing between derived and manual Debug

23_1_debug_formatting_diagnostics_1

// Endpoint and Protocol also derive Debug:
//   struct Endpoint { host: String, port: u16 }
//   enum Protocol { Http, Https { hsts: bool }, Custom(u16) }
#[derive(Debug)]
struct Deployment {
    name: String,
    replicas: u32,
    endpoint: Endpoint,     // a nested struct
    protocol: Protocol,     // an enum
    tags: Vec<&'static str>,
}

The same derived implementation gives two formats. {:?} is the compact form: one line, which is good for log records. {:#?} is the alternate form (the # flag). It prints one field per line with an indentation of 4 spaces, and nested values get one more level. The data and the implementation are the same. Only the flag is different:

// `deploy` is a Deployment value. The output below shows its fields.
let compact = format!("{deploy:?}");
assert_eq!(compact.lines().count(), 1);   // all the fields are on one line

let pretty = format!("{deploy:#?}");
assert_eq!(pretty.lines().count(), 15);   // the same value on 15 lines

An enum variant has the same format as the equivalent struct or tuple: Http, Custom(9000), Https { hsts: false }.

23_01_derive_debug_pretty.rs prints to stdout:

compact:
Deployment { name: "billing-api", replicas: 3, endpoint: Endpoint { host: "api.example.dev", port: 8443 }, protocol: Https { hsts: true }, tags: ["prod", "eu-west"] }

pretty:
Deployment {
    name: "billing-api",
    replicas: 3,
    endpoint: Endpoint {
        host: "api.example.dev",
        port: 8443,
    },
    protocol: Https {
        hsts: true,
    },
    tags: [
        "prod",
        "eu-west",
    ],
}

All assertions passed.

dbg! for Quick Diagnostics

dbg!(expr) prints [file:line:column] expr = value to stderr in the {:#?} format. Then it returns the value. Thus you can put it around a subexpression, and the adjacent code does not change:

// base_replicas: u32 = 3 (a copy of deploy.replicas)
let scaled = dbg!(base_replicas * 2) + 1;  // prints the line, then gives 6 + 1
assert_eq!(scaled, 7);

let tags = dbg!(&deploy.tags);  // a reference, so dbg! does not move deploy.tags
// tags: &Vec<&'static str>

23_01_derive_debug_pretty.rs prints these lines to stderr. The path and the line numbers identify each dbg! call in the source file:

[domain-23-testing-debugging-patterns/examples/src/bin/23_01_derive_debug_pretty.rs:101:18] base_replicas * 2 = 6   # (stderr)
[domain-23-testing-debugging-patterns/examples/src/bin/23_01_derive_debug_pretty.rs:103:16] &deploy.tags = [   # (stderr)
    "prod",   # (stderr)
    "eu-west",   # (stderr)
]   # (stderr)

Two cautions apply:

dbg! moves its argument. For a value that is not Copy and that you use again, pass &value.
dbg! is a debugging tool, not a logging tool. The compiler does not remove it in release builds. Remove the calls before you release the code. The dbg_macro restriction lint of clippy can enforce this in CI.

Manual Debug: debug_struct and debug_tuple

Sometimes the derived output is misleading, not only long. This occurs when the physical storage of a type has little relation to its logical meaning. Then you implement Debug manually. The Formatter builders let you do this and keep the {:#?} form.

Figure: The Formatter builder family

23_1_debug_formatting_diagnostics_2

debug_struct makes output with named fields. Each .field() call takes a name and a &dyn Debug. finish_non_exhaustive() ends the output with .., which is the conventional sign that the implementation omits fields on purpose:

use std::fmt;

// struct Connection { addr: &'static str, retries: u32, recv_buffer: Vec<u8> }
impl fmt::Debug for Connection {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_struct("Connection")           // the type name in the output
            .field("addr", &self.addr)         // a field name and a &dyn Debug value
            .field("retries", &self.retries)
            .finish_non_exhaustive()   // prints `..` and hides the 4 KiB recv_buffer
    }
}

The primary advantage: the builders obey the alternate flag automatically. With {:?}, the same implementation prints Connection { addr: "10.0.0.1:5432", retries: 3, .. }. With {:#?}, it prints the indented form on many lines. A manual write!("Connection {{ ... }}") call must implement that logic again. For this reason, the builders are the correct default for a manual implementation.

debug_tuple is the equivalent builder for positional fields. It is a good fit for a newtype: f.debug_tuple("Celsius").field(&self.0).finish() prints Celsius(21.5).

debug_list, debug_map, debug_set

The other three builders make output in the form of a collection. Each one accepts a full iterator through .entries(...). They are most useful when you print the logical view of a type:

debug_list: a ring buffer keeps its elements in a physical order that is not the insertion order. Its Debug goes through the ring in insertion order. It prints [10, 20, 30, 40], and the position of head has no effect.
debug_map: a sparse vector that keeps (index, value) pairs prints as {2: 9.5, 7: 1.25}. This is much easier to read than a Vec of tuples. entries takes (key, value) pairs. key() and value() add one half of an entry at a time.
debug_set: a wrapper for bit flags prints {READ, WRITE} for the value 0b011, which has no clear meaning as a number. A small helper type keeps the flag names without quotation marks, because its Debug writes the string directly.

// struct RingBuffer { slots: [i32; 4], head: usize, len: usize }
// `head` is the index of the oldest element.
impl fmt::Debug for RingBuffer {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_list()
            // Start at `head` and wrap at the end of `slots`: this is the insertion order.
            .entries((0..self.len).map(|i| self.slots[(self.head + i) % self.slots.len()]))
            .finish()
    }
}
// slots = [30, 40, 10, 20] with head = 2 and len = 4 prints [10, 20, 30, 40].

23_02_manual_debug_builders.rs prints:

Connection { addr: "10.0.0.1:5432", retries: 3, .. }
Connection {
    addr: "10.0.0.1:5432",
    retries: 3,
    ..
}
Celsius(21.5)
[10, 20, 30, 40]
{2: 9.5, 7: 1.25}
{READ, WRITE}
All assertions passed.

Redacting Debug: Keeping Secrets Out of Logs

The configuration of a service contains a database password. One println!("{config:?}") in a handler must not leak it. A crash reporter that prints the program state must not leak it. Put the redaction in the type. Do not rely on each caller to be careful. There are two strategies.

Strategy 1: a Secret<T> wrapper (the secrecy crate uses this pattern). The Debug of the wrapper prints a constant marker. Each container that holds the wrapper gets the redaction, because a derived Debug only calls the implementation of each field. The implementation needs no T: Debug bound, because it never formats the inner value. To read the secret, you call an explicit method that a text search finds easily:

struct Secret<T>(T);

impl<T> Secret<T> {
    // The only way to read the secret. A search for `expose()` finds each use.
    fn expose(&self) -> &T {
        &self.0
    }
}

// No `T: Debug` bound: the implementation does not format the inner value.
impl<T> fmt::Debug for Secret<T> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("Secret([REDACTED])")   // the same text for each secret
    }
}

#[derive(Debug)]            // safe: the derive calls the Debug of each field
struct DbConfig {
    host: String,
    port: u16,
    username: String,
    password: Secret<String>,   // prints as Secret([REDACTED])
}

Strategy 2: a manual implementation on the struct itself. It prints a placeholder for the sensitive field: .field("api_key", &"[REDACTED]"). This gives you control of one type. But it silently omits a field that you add later, unless you remember to update fmt().

23_03_redacting_debug.rs prints:

DbConfig { host: "db.internal", port: 5432, username: "app_rw", password: Secret([REDACTED]) }
ApiClient { endpoint: "https://api.example.dev/v2", api_key: "[REDACTED]" }
All assertions passed.

The example proves the important property with assertions: the secret text is not in the {:?} output or the {:#?} output. Thus it is not in dbg! output, because dbg! uses {:#?} internally. But config.password.expose() still gives the real value to the program when the program needs it. Tutorial 26.1 shows this again as part of the newtype pattern.

fmt::Pointer: Printing Addresses

{:p} formats a value through the fmt::Pointer trait. It prints the address that the value points to as hexadecimal digits with a 0x prefix. The standard library implements the trait for &T, &mut T, Box, Rc, Arc, raw pointers, and function pointers. You can implement it for your own smart pointer: call the fmt::Pointer implementation of the inner pointer.

Addresses change from run to run (ASLR, allocator state). Thus correct code asserts on their shape and equality, and never on their actual values:

// reference: &u64, a reference to a local variable
let addr = format!("{reference:p}");                   // "0x" and then hex digits
assert!(addr.starts_with("0x"));                       // shape, not value

// first: Rc<String>. third: a separate Rc<String> with equal contents.
let second = Rc::clone(&first);                        // shares the allocation of `first`
assert_eq!(format!("{first:p}"), format!("{second:p}")); // same allocation
assert_ne!(format!("{first:p}"), format!("{third:p}"));  // equal contents, different allocation

The last two assertions show the practical use. When you look for an aliasing bug, a comparison of {:p} output tells you immediately if two handles refer to the same allocation. (std::ptr::eq and Rc::ptr_eq answer the same question without strings.)

type_name and type_name_of_val

std::any::type_name::<T>() returns the name that the compiler gives to a type, as a &'static str. It is useful in error messages, test diagnostics, and generic trace code. type_name_of_val(&expr) infers T from an expression. This is important for types that you cannot name or that are long to write: closures and long chains of iterator adaptors.

use std::any::{type_name, type_name_of_val};

assert_eq!(type_name::<i32>(), "i32");
let doubler = |x: i32| x * 2;
// You cannot write the type of a closure. Its type name ends with `{{closure}}`.
assert!(type_name_of_val(&doubler).ends_with("{{closure}}"));

23_04_pointer_type_name.rs prints:

{:p} output shape verified (addresses omitted: they vary per run)
type_name::<i32>()          = i32
type_name::<Vec<String>>()  = alloc::vec::Vec<alloc::string::String>
type_name::<Option<&str>>() = core::option::Option<&str>
type_name::<fn(i32) -> i32>() = fn(i32) -> i32
type_name_of_val(&ratio)    = f64
All assertions passed.

Look at the alloc:: and core:: prefixes. The documentation states that the exact string is not guaranteed to be stable across compiler versions. Use it as diagnostic output only. Tutorial 17.1 gives the details of type_name, TypeId, and Any.

Summary

Concept	Key point
`#[derive(Debug)]`	The default. It is recursive, needs no code, and is valid when all fields are `Debug`.
`{:?}` vs `{:#?}`	The same implementation. The `#` flag selects the alternate form on many lines.
`dbg!`	Prints `[file:line:column] expr = value` to stderr and returns the value. Release builds keep it.
`debug_struct` / `debug_tuple`	Manual output with named or positional fields. `finish_non_exhaustive()` prints `..`.
`debug_list` / `debug_map` / `debug_set`	Output in the form of a collection, from an iterator through `entries`.
Builders vs direct `write!`	The builders keep the pretty `{:#?}` form at no cost.
Redacting Debug	A `Secret<T>` wrapper (the redaction goes with the value) or a manual implementation (for one type).
`fmt::Pointer` (`{:p}`)	Prints addresses. Assert on shape and equality only, because the values change from run to run.
`type_name` / `type_name_of_val`	The name that the compiler gives to a type, for diagnostics. The string is not stable across versions.
Stability	Debug output is not a stable interface. Never parse it in production.

Code Examples

File	Description
`23_01_derive_debug_pretty.rs`	`#[derive(Debug)]`, `{:?}` and `{:#?}`, the format of enum variants, `dbg!`
`23_02_manual_debug_builders.rs`	Manual implementations with `debug_struct`, `debug_tuple`, `debug_list`, `debug_map`, `debug_set`, `finish_non_exhaustive`
`23_03_redacting_debug.rs`	Redacting Debug: a `Secret<T>` wrapper and a manual implementation. Assertions prove that the secrets are not in the output.
`23_04_pointer_type_name.rs`	`fmt::Pointer` (`{:p}`): the shape and equality of addresses. `type_name` and `type_name_of_val`.

23.2 · Assertions, Panics, and Test Harness Integration

Domain 23 — Testing, Debugging, and Pattern Matching Duration: ~15 minutes Library components: assert!, assert_eq!, assert_ne!, debug_assert!, std::panic::catch_unwind, #[test], #[should_panic]

Introduction

Assertions are how Rust code states its invariants explicitly. The test harness runs those statements automatically. The two use the same primitive, the panic. An assertion that fails panics. The harness catches that panic, records the test as failed, and continues with the next test.

This tutorial shows:

assert!, assert_eq!, assert_ne!, and why custom messages are important.
assert_matches! (stabilized in 1.96), which asserts on the shape of a value.
The debug_assert_* variants, which are not active in release builds.
#[test], #[cfg(test)], #[should_panic], and test functions that return Result.
How to catch panics with std::panic::catch_unwind, and how to use it to write a small custom test framework in approximately sixty lines.

A note about the examples in this repository: a binary runs main(), not #[test] functions. Thus example 23_06 has two parts. Its main uses the assertion macros, and the file contains a real #[cfg(test)] module. You can run that module with cargo test -p domain-23-testing-debugging-patterns.

The assert Family and Custom Messages

Three macros are sufficient for almost all checks. assert!(cond) takes a boolean. assert_eq!(a, b) and assert_ne!(a, b) compare two values. When they fail, they print the two operands with {:?}. For this reason they are better than assert!(a == b), which can only report that the condition is false.

Figure: Which assertion macro?

23_2_assertions_panics_test_harness_1

Each macro accepts an optional format string after its other arguments. Prefer a message that states the expectation and the actual value:

// In the example, cart_items = 3 and subtotal_cents = 4.
assert!(
    cart_items <= 100,
    "cart exceeds line-item limit: {cart_items} > 100"   // the macro prints this on failure
);
// The message comes after the two values that the macro compares.
assert_ne!(subtotal_cents, 0, "an empty order must not reach checkout");

assert_matches! (stable since 1.96) asserts that a value matches a pattern, and the pattern can have a guard. The macro is not in the prelude. Use the path std::assert_matches!, or import the macro from the std root. The module path std::assert_matches::assert_matches, from before the stabilization, does not exist now. The macro is a good fit for enums, when the variant is important and full equality is not:

// enum PaymentState { Pending, Captured { amount_cents: u64 }, Declined(String) }
// state = PaymentState::Captured { amount_cents: 1299 }
std::assert_matches!(state, PaymentState::Captured { .. });   // checks only the variant
std::assert_matches!(
    state,
    PaymentState::Captured { amount_cents } if amount_cents > 0,   // a pattern with a guard
    "captured payments must have a positive amount"                // the optional message
);

debug_assert_*: Checks That Do Not Run in Release

Each macro above has a debug_ variant:

debug_assert!
debug_assert_eq!
debug_assert_ne!
std::debug_assert_matches!

A debug_ variant does its check only when debug_assertions is on (the dev profile). In --release builds, the check does not run, and the program does not evaluate its operands.

debug_assert!(cart_items > 0);          // no cost in release builds
// cfg!(debug_assertions) is true with `cargo run` and false with `--release`.
println!("debug_assertions active: {}", cfg!(debug_assertions));

The general rule:

Internal consistency checks that cost too much time in production code: use debug_assert_*.
Checks of untrusted input or of a public API contract: always use the regular assert_*. Release builds skip debug_assert_* fully. A security check in a debug_assert_* does not exist in production.

23_05_assert_macros.rs prints the lines below. The section about catch_unwind explains the second line.

debug_assertions active: true
payload carried the custom assertion message: verified
All assertions passed.

#[test], #[cfg(test)], and #[should_panic]

cargo test does three things:

It compiles each target with --cfg test.
It replaces main with the libtest harness.
It runs each #[test] function in its own thread, so one panic cannot stop the other tests.

A plain #[test] passes if it returns and does not panic.

Figure: What cargo test does

23_2_assertions_panics_test_harness_2

The idiomatic layout for unit tests puts the tests in the same file as the code, in a #[cfg(test)] module. The tests can then use private items (use super::*), and normal builds have no cost. The compiler removes the module unless the build sets --cfg test:

// The same file defines two private functions:
//   fn parse_percent(input: &str) -> Result<u8, String>       // "42%" gives Ok(42)
//   fn apply_discount(price_cents: u64, percent: u8) -> u64   // panics if percent > 100
#[cfg(test)]
mod tests {
    use super::*;   // brings the private items of the parent module into scope

    #[test]
    fn parses_plain_and_suffixed_percentages() {
        assert_eq!(parse_percent("42%"), Ok(42));
    }

    #[test]
    #[should_panic(expected = "exceeds 100%")]   // a substring of the panic message
    fn discount_above_100_percent_is_a_bug() {
        // Panics with "discount of 130% exceeds 100%".
        let _ = apply_discount(1000, 130);
    }
}

#[should_panic] inverts the pass condition: the test fails unless the body panics. Always supply expected = "...", which must be a substring of the panic message. Without it, an accidental panic (for example, an index out of bounds) also counts as a pass, with no warning.

Run the module with cargo test -p domain-23-testing-debugging-patterns. The harness of 23_06_test_harness_integration.rs prints:

running 4 tests
test tests::discount_round_trip ... ok   # (order varies)
test tests::parses_plain_and_suffixed_percentages ... ok   # (order varies)
test tests::rejects_out_of_range_and_garbage ... ok   # (order varies)
test tests::discount_above_100_percent_is_a_bug - should panic ... ok   # (order varies)

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s   # (time varies)

Result-Returning Tests

A test function may return Result<(), E> where E: Debug. If the function returns Err, the test fails and the harness prints the error. Thus you can use the ? operator. Fallible preparation code becomes a simple sequence of lines, without a series of unwrap() calls:

#[test]
fn discount_round_trip() -> Result<(), String> {
    let percent = parse_percent("15%")?;   // an Err here fails the test
    assert_eq!(apply_discount(10_000, percent), 8_500);   // 10_000 less 15 percent
    Ok(())   // the test passes
}

This style is most useful when a test needs several fallible steps (open a file, parse it, connect a socket) before the assertion. One limit applies: you cannot use #[should_panic] on a test that returns Result. An Err return value is a failure, not a panic.

catch_unwind and Panic Payloads

std::panic::catch_unwind(closure) runs the closure. It converts an unwinding panic into Err(payload), and the thread does not stop. The payload is a Box<dyn Any + Send>. In practice it contains one of two types:

a &'static str, from a literal panic message
a String, from a formatted message, which includes each message of assert_eq!, assert_ne!, and assert_matches!

A complete helper function tries a downcast to each type:

use std::any::Any;

fn payload_message(payload: &(dyn Any + Send)) -> &str {
    payload
        .downcast_ref::<&str>()     // Some if the payload is a literal message
        .copied()                   // Option<&&str> becomes Option<&str>
        // If not, try a String (a formatted message).
        .or_else(|| payload.downcast_ref::<String>().map(String::as_str))
        .unwrap_or("<non-string panic payload>")   // for example, from panic_any(42)
}

Example 23_05 uses this helper to prove that a custom assertion message is in the payload. It has three steps:

It replaces the default panic hook (Tutorial 2.3) with a hook that prints nothing.
It causes an intended assert_eq! failure in catch_unwind.
It asserts on the text of the payload.

// attempts = 3. A silent panic hook is active.
let err = panic::catch_unwind(|| {
    assert_eq!(attempts, 5, "retry budget mismatch: saw {attempts}");   // fails: 3 != 5
})
.expect_err("the assertion above must fail");   // err: Box<dyn Any + Send>

let msg = payload_message(err.as_ref());        // msg: &str
assert!(msg.starts_with("assertion `left == right` failed"));
assert!(msg.contains("retry budget mismatch: saw 3"));   // the custom message
assert!(msg.contains("left: 3") && msg.contains("right: 5"));   // the two operands

catch_unwind is not a general tool for error handling. Use Result for that (Tutorial 2.1). catch_unwind is for boundaries: test harnesses, FFI boundaries, and thread pools that must continue after a task panics. Tutorial 2.3 explains unwind safety and the UnwindSafe bound.

A Custom Test Harness in Sixty Lines

catch_unwind is exactly the primitive that the real libtest harness uses. Thus a small harness shows clearly what cargo test does with your #[test] functions. Example 23_07 registers plain functions as named test cases. It runs each one in catch_unwind and prints a report in the libtest style:

struct TestCase {
    name: &'static str,
    run: fn(),          // a plain function pointer: the test body
}

// tests: &[TestCase]. A silent panic hook is active during the loop.
for test in tests {
    // A function pointer is unwind-safe, so AssertUnwindSafe is not necessary.
    match panic::catch_unwind(test.run) {
        Ok(()) => println!("test {} ... ok", test.name),   // the function returned
        Err(payload) => {                                   // the function panicked
            println!("test {} ... FAILED", test.name);
            // payload_message is the helper from the previous section.
            println!("    {}", payload_message(payload.as_ref()));
        }
    }
}

23_07_custom_test_harness.rs prints:

running 3 tests
test empty_slice_checksums_to_zero ... ok
test checksum_wraps_instead_of_overflowing ... ok
test deliberately_failing_expectation ... FAILED
    assertion `left == right` failed: checksum spec changed?
  left: 6
 right: 7
result: 2 passed; 1 failed
All assertions passed.

The harness continues after the test that fails, which is the purpose of catch_unwind here. The real harness adds these features to this small version:

one thread per test
output capture
a filter by name
#[ignore]
timing
reporters

With the unstable custom_test_frameworks feature, a no_std target can use this type of manual runner in place of libtest.

Summary

Concept	Key point
`assert!`	A boolean invariant. Always add a message that states the expectation and the actual value.
`assert_eq!` / `assert_ne!`	They print the two operands on failure. Prefer them to `assert!(a == b)`.
`assert_matches!` (1.96)	Asserts on the shape of a value (a pattern and an optional guard). Import it from the std root.
`debug_assert_*`	Active only with `debug_assertions`. Never use it for untrusted input.
`#[test]`	Passes if it returns and does not panic. Each test runs in its own thread.
`#[cfg(test)] mod tests`	Unit tests in the same file, with access to private items. Normal builds do not contain the module.
`#[should_panic(expected)]`	Inverts the pass condition. `expected` is a substring of the panic message.
Tests that return `Result`	`Err` fails the test. You can use `?` in fallible preparation code.
`catch_unwind`	Converts a panic into `Err(payload)`. The payload downcasts to `&str` or `String`.
Custom harness	`catch_unwind` for each test and a report loop are the skeleton of libtest.

Code Examples

File	Description
`23_05_assert_macros.rs`	`assert!`/`assert_eq!`/`assert_ne!` with custom messages, `assert_matches!` (1.96), `debug_assert_*`, panic payloads that `catch_unwind` captures
`23_06_test_harness_integration.rs`	`#[test]`, `#[cfg(test)]`, `#[should_panic(expected)]`, tests that return `Result`. Run it with `cargo test -p domain-23-testing-debugging-patterns`.
`23_07_custom_test_harness.rs`	A small libtest: named test cases run in `catch_unwind`. The harness reports each failure and continues.

23.3 · matches!, if let Chains, and Exhaustive Matching Strategies

Domain 23 — Testing, Debugging, and Pattern Matching Duration: ~15 minutes Library components: matches!, std::option::Option, std::result::Result, language-level pattern syntax

Introduction

Pattern matching is the primary control-flow concept of Rust, and the standard library uses it in all its parts. Option and Result are enums that you match. This tutorial shows the full set of pattern constructs:

the matches! macro, for boolean shape tests
if let and let else, to extract a value or leave
let chains (stabilized in 1.88, edition 2024), to combine several conditions
if-let guards in match arms (stabilized in 1.95)
the exhaustiveness check, which makes match a protection when you refactor code

Each construct answers a different question:

matches!: "does this value have this shape?" (yes or no)
if let: "act on this one shape."
let else: "extract this shape or leave."
let chains: "all these shapes and facts together."
match: "each shape, with a different action for each."

Figure: The pattern-syntax family

23_3_matches_let_chains_exhaustive_1

matches!: Boolean Shape Tests

matches!(expr, pattern) expands to a match that returns true or false. Its advantage over == is that it needs no PartialEq, because it tests shape, not equality. It also accepts the full pattern grammar. A typical use is a log ingestion pipeline that classifies events:

enum LogEvent {                       // note: no #[derive(PartialEq)]
    Http { status: u16, path: String },
    Metric { name: String, value: f64 },
    Heartbeat,
}

fn is_server_error(event: &LogEvent) -> bool {
    // Exclusive range pattern (1.80): 500..600 matches 500 through 599.
    // It has the same form as the usual statement of the limit ("below 600").
    // `..` ignores the other fields of the variant.
    matches!(event, LogEvent::Http { status: 500..600, .. })
}

You can combine all the pattern operators in the macro: | alternatives, guards, and @ bindings. An @ binding constrains a value and also names it for the guard:

// event: &LogEvent. Each line is the body of one function of the example.

// is_operational_noise, with | alternatives: one of the two patterns is sufficient.
matches!(event, LogEvent::Heartbeat | LogEvent::Metric { value: 0.0, .. })

// is_slow_endpoint_hit, with a guard: a boolean condition on the binding `path`.
matches!(event, LogEvent::Http { path, .. } if path.starts_with("/reports/"))

// is_retryable_client_error, with an @ binding and a guard:
// `s` is the status, and the pattern already limits it to 400 through 499.
matches!(event, LogEvent::Http { status: s @ 400..500, .. } if *s == 408 || *s == 429)

matches! is easy to read in an iterator chain, where a boolean shape test is the correct tool: events.iter().filter(|e| is_operational_noise(e)).count(). It has one limit: it cannot bind variables for use outside the macro. To extract the matched data, use if let or a full match.

23_08_matches_macro.rs prints:

events: 6 total, 2 noise, 4 actionable
All assertions passed.

if let and let else

if let runs a block when one pattern matches, and it binds the variables of the pattern in that block. It is the correct tool when only one shape is important and there is no early return:

// directive: Option<&Directive>, where
// struct Directive { section: String, key: String, value: u32 }
if let Some(d) = directive {
    // d: &Directive, in scope only in this block
    println!("loaded {}.{} = {}", d.section, d.key, d.value);
}

let else is the opposite construct, for code that must extract a value or leave. Its form is let PATTERN = EXPR else { DIVERGE };. When the refutable pattern matches, it binds as a plain let does. If the pattern does not match, the else block runs and must diverge (return, break, continue, panic!).

The important difference is that the bindings go into the enclosing scope. if let keeps them in its body, and each subsequent step gets one more level of indentation. A parser consists almost fully of steps that extract a value or leave. let else keeps those steps at one indentation level:

// Parses "section.key=value", for example "net.max_connections = 250".
fn parse_directive(line: &str) -> Option<Directive> {
    // split_once returns Option<(&str, &str)>: the text before and after the first '='.
    let Some((path, raw_value)) = line.split_once('=') else {
        return None;                        // not a key=value line
    };
    let Some((section, key)) = path.trim().split_once('.') else {
        return None;                        // no section prefix
    };
    let Ok(value) = raw_value.trim().parse::<u32>() else {
        return None;                        // a Result pattern is also permitted
    };
    // Success path: all the bindings are in scope, and there is no nesting.
    Some(Directive { section: section.to_string(), key: key.to_string(), value })
}

(When the else block contains only return None, clippy tells you that ? on split_once does the same thing. The full form is here to show the syntax. Real code uses the two forms.)

Let Chains (1.88, Edition 2024)

A let chain joins let bindings and plain boolean expressions with && in one if (or while). A later link may use the bindings of an earlier link. This scoping from left to right is the purpose of the construct. Before 1.88, this logic needed nested if let blocks or a sequence of combinator calls:

fn resolve_limit(cli_flag: Option<&str>, config: Option<&Directive>) -> u32 {
    // The flag is present AND it parses AND it is positive: three conditions, one `if`.
    if let Some(raw) = cli_flag                // raw: &str
        && let Ok(n) = raw.parse::<u32>()      // uses `raw` from the first link
        && n > 0                               // uses `n` from the second link
    {
        return n;
    }
    // The two boolean links read `d` (a &Directive), which the first link binds.
    if let Some(d) = config
        && d.key == "max_connections"
        && d.value > 0
    {
        return d.value;
    }
    64 // default
}

The chain has the same structure as the requirement: "if the flag is present, and it parses, and it is positive, use it". Let chains are a feature of edition 2024. In an older edition, the compiler rejects the chain with the error "let chains are only allowed in Rust 2024 or later".

23_09_let_else_let_chains.rs prints:

parse_directive: 1 accepted, 3 rejected
loaded net.max_connections = 250
resolve_limit: flag > config > default fallback verified
All assertions passed.

match Guards and if-let Guards (1.95)

In a match, a guard (pattern if condition) adds a logical condition to the structural condition of an arm. Since Rust 1.95, the guard can be a let binding (an if-let guard), and the binding is available in the arm body. The match selects the arm only when the variant matches and the inner pattern matches:

// command: &Command, store: &mut BTreeMap<String, i64>
// The variants of Command: Get { key }, Set { key, raw_value }, Delete { key }, Quit.
// Each field is a String.
match command {
    // Selected only when raw_value parses. `n` (an i64) is available in the body.
    Command::Set { key, raw_value } if let Ok(n) = raw_value.parse::<i64>() => {
        store.insert(key.clone(), n);
        Reply::Stored
    }
    // The same variant, but the guard was false. The order is important:
    // when a guard is false, the match tries the subsequent arms.
    Command::Set { .. } => Reply::Rejected("value must be an integer"),
    // ... the arms for Get, Delete, and Quit
}

Before 1.95, this logic needed a nested match or a second parse in the arm. Two more pattern tools complete the arm syntax. An @ binding with a range constrains the value and also names it. An or-pattern can share one binding between its alternatives:

// delta: i64. Each arm gives a String.
match delta {
    0 => "unchanged".to_string(),
    d @ 1..=9 => format!("nudged up by {d}"),          // binds d and limits it to 1..=9
    d @ -9..=-1 => format!("nudged down by {}", -d),
    // An or-pattern: the two ranges share the binding `d`.
    d @ (i64::MIN..=-10 | 10..=i64::MAX) => format!("jumped by {d}"),
}

That last match has no _ arm, but it compiles: the compiler proves that the four arms include each i64 value. This property is exhaustiveness, the subject of the next section.

Exhaustive Matching as a Refactoring Safeguard

For an enum that you own, list each variant and do not write the _ catch-all arm. If you add a Command::Rename variant later, each exhaustive match fails to compile:

error[E0004]: non-exhaustive patterns: `&Command::Rename { .. }` not covered

That error is useful. A final _ => ... arm accepts the new variant, and the compiler gives no warning. The best example in the standard library is std::cmp::Ordering (Tutorial 13.1). It has three variants and you write three arms. The compiler proves that the code handles each comparison result:

// In the example, guess = 37 and target = 42.
let hint = match guess.cmp(&target) {   // cmp returns an Ordering
    Ordering::Less => "too small",      // this arm matches: 37 < 42
    Ordering::Greater => "too big",
    Ordering::Equal => "spot on",
};   // no `_` arm: the three arms are all the variants

Figure: Which construct for which job?

23_3_matches_let_chains_exhaustive_2

Use _ only for sets that are really open:

integers and chars, where you cannot list each value
#[non_exhaustive] enums from other crates, where the compiler requires a catch-all arm, so that the upstream crate can add variants and your code still compiles

In your own crate, exhaustiveness is a protection that costs nothing.

23_10_match_guards_exhaustive.rs prints:

set retries=5 stored; set retries=many rejected
describe_change buckets: 0 / ±1..=9 / everything else
All assertions passed.

Summary

Construct	Question it answers	Since
`matches!(v, pat)`	"Does this value have this shape?" No `PartialEq` is necessary.	1.42
Exclusive range patterns `a..b`	Match `a..=b-1`, the same form as a "below b" limit	1.80
`\|` alternatives	One of the patterns is sufficient. You can nest them in variants.	—
`@` bindings	Constrain a value and name it (`s @ 400..500`)	—
Guards `pat if cond`	Add a boolean condition to a structural match	—
`if let`	Act on one shape. The bindings stay in the block.	—
`let else`	Extract or diverge. The bindings go into the enclosing scope.	1.65
Let chains `if let … && let … && cond`	Several shapes and facts in one `if`, with scoping from left to right	1.88 (edition 2024)
If-let guards in `match` arms	The guard binds through a nested pattern. The binding is available in the arm.	1.95
Exhaustive `match`, no `_`	The compiler reports each unhandled variant when you refactor	—

Code Examples

File	Description
`23_08_matches_macro.rs`	`matches!` with `\|` alternatives, guards, `@` bindings, and exclusive ranges. It classifies log events without `PartialEq`.
`23_09_let_else_let_chains.rs`	`if let`, `let else` for a parser with early returns, let chains (1.88) that try a flag, then a config value, then a default
`23_10_match_guards_exhaustive.rs`	If-let guards (1.95), `@` bindings with ranges, nested patterns, exhaustive matching on `Command` and `Ordering`

24.1 · dyn Trait, Vtables, and Object Safety

Domain 24 — Dynamic Dispatch and Trait Objects Duration: ~15 minutes Library components: std::any::Any, dyn keyword, std::boxed::Box, std::ops::Fn* traits

Introduction

Generics accept any type, and the compiler decides the type at compile time. Trait objects also accept any type, but the program decides the type at run time. Use dyn Trait in cases such as these:

A Vec must hold handlers of different concrete types.
A function must return one of several closures.
A plugin registry cannot know its plugins in advance.

A trait object is a type-erased value together with a table of function pointers. The table records how the concrete type behaves.

This tutorial shows:

The structure of a fat pointer: why size_of::<&dyn Trait>() equals 2 * size_of::<usize>(), and what the vtable contains (destructor, size, alignment, method pointers).
Dyn compatibility: which traits can become dyn Trait, the common violations, and the where Self: Sized exclusion.
The three pointer types &dyn Trait, Box<dyn Trait>, and Arc<dyn Trait + Send + Sync>, and the implicit 'static bound.
How to recover the concrete type with std::any::Any: downcast_ref, Box::downcast, the as_any pattern, and trait upcasting (stable since 1.86).
dyn Fn, dyn FnMut, and dyn FnOnce as callable trait objects.
The trade-off against monomorphization: one shared function body, or one specialized copy for each type.

Fat Pointers and the Vtable

A reference to a concrete type is one machine word: an address. A reference to dyn Trait must also record which concrete type it points to, so it is two words:

The data pointer is the address of the value. The coercion does not change this address.
The vtable pointer is the address of a static table that the compiler builds at compile time. There is a table for each (concrete type, trait) pair.

Figure: Fat pointer and vtable layout for &dyn Shape pointing at a Circle

Fat pointer and vtable layout

The first slots of the vtable are not methods:

Slot 0 is drop_in_place for the concrete type. Through this slot, drop(Box<dyn Trait>) finds the destructor of the concrete type, although it does not know that type.
Slots 1 and 2 are the size and the alignment of the concrete type. mem::size_of_val and mem::align_of_val read these slots at runtime when you give them an erased reference.
The function pointers of the methods follow.

This layout is an implementation detail of rustc, not a stable ABI. However, the operations that the language must do on an erased value guarantee the contents.

// Sensor is a trait with one method: fn sample(&self) -> i32.
// Thermometer { celsius: i32 } and Barometer { readings: [i32; 16] } implement Sensor.
let word = mem::size_of::<usize>();                       // 8 on a 64-bit target
assert_eq!(mem::size_of::<&dyn Sensor>(), 2 * word);      // data ptr + vtable ptr
assert_eq!(mem::size_of::<Box<dyn Sensor>>(), 2 * word);  // owning pointers are fat too
assert_eq!(mem::size_of::<Option<Box<dyn Sensor>>>(), 2 * word); // niche: None is the null data ptr

// The two boxes have the SAME static type, but size_of_val gives different sizes.
// It reads the size from the vtable of each value at runtime.
let small: Box<dyn Sensor> = Box::new(Thermometer { celsius: 20 }); // 4 bytes
let large: Box<dyn Sensor> = Box::new(Barometer { readings: [1013; 16] }); // 64 bytes
assert_eq!(mem::size_of_val(&*small), 4);   // &*small is a &dyn Sensor
assert_eq!(mem::size_of_val(&*large), 64);

Slices are the other type of fat pointer. &[T] and &str are also two words, but their metadata is a length, not a vtable pointer. The layout is the same, and only the second word is different.

24_02_fat_pointer_anatomy.rs prints these lines on a 64-bit target:

size_of::<&Thermometer>()        = 8 bytes (thin)
size_of::<&dyn Sensor>()         = 16 bytes (fat)
size_of::<Box<dyn Sensor>>()     = 16 bytes (fat)
size_of::<Option<Box<dyn ..>>>() = 16 bytes (niche)
size_of::<&[i32]>()              = 16 bytes (ptr + len)

data pointer == &Thermometer:      true
size_of_val(&*small) =  4 bytes (Thermometer, via vtable)
size_of_val(&*large) = 64 bytes (Barometer, via vtable)
small.sample() = 20, large.sample() = 1013

Probe::drop reached through the vtable: true

All assertions passed.

Dyn Compatibility (Object Safety): The Rules

Not every trait can become dyn Trait. Use this model: a vtable is a fixed, finite table of plain function pointers, and the compiler makes its layout at compile time. Anything that the compiler cannot reduce to such a table makes the trait not dyn compatible. The Rust Reference calls these rules dyn compatibility. Older books and older compiler error messages call the same concept object safety.

Figure: Is this method dispatchable through dyn Trait?

24_1_dyn_trait_vtables_object_safety_1

A trait that breaks these rules is legal to define. The compiler reports the error (E0038) when you form the dyn type. The snippet shows each violation and why the vtable model rejects it:

// Each trait below compiles when you only define it. The error appears when code
// names the `dyn` type, for example in `fn take(_: &dyn Comparator) {}`.

// 1. Generic method. The vtable would need one slot for each instantiation of T.
//    The set of instantiations is open-ended, and the compiler cannot list it.
// trait Comparator { fn best<T: PartialOrd>(&self, a: T, b: T) -> bool; }
// error[E0038]: the trait `Comparator` is not dyn compatible
//   note: ...because method `best` has generic type parameters

// 2. Method that returns Self. `dyn` erases the concrete type, so the caller does
//    not know the size of the returned value. For this reason, Clone is not dyn
//    compatible.
// trait Duplicate { fn duplicate(&self) -> Self; }
// error[E0038]: the trait `Duplicate` is not dyn compatible
//   note: ...because method `duplicate` references the `Self` type in its return type

// 3. No self receiver. There is no value from which to get a vtable.
// trait Spawner { fn spawn() -> Self; }
// error[E0038]: the trait `Spawner` is not dyn compatible
//   note: ...because associated function `spawn` has no `self` parameter

// 4. Associated const. The compiler resolves it at compile time, and no vtable
//    slot exists for it.
// trait Bounded { const MAX: u32; }
// error[E0038]: the trait `Bounded` is not dyn compatible
//   note: ...because it contains associated const `MAX`

// 5. `Self: Sized` supertrait. `dyn Trait` is unsized, which contradicts the bound.
// trait OnlyConcrete: Sized { fn id(&self) -> u32; }
// error[E0038]: the trait `OnlyConcrete` is not dyn compatible
//   note: ...because it requires `Self: Sized`

async fn methods and -> impl Trait methods in traits (stable since 1.75) also make a trait not dyn compatible. The return type depends on the erased Self, which is the same cause as in violation 2.

The Opt-Out Mechanisms

Two patterns help a trait that is almost dyn compatible.

where Self: Sized on a method excludes that one method from the vtable, and the trait stays dyn compatible. Iterator stays dyn compatible in this way, although it has many generic adapters. map, filter, and collect each have a where Self: Sized bound, so Box<dyn Iterator<Item = u32>> is a valid type. You cannot call these adapters on the unsized dyn Iterator itself. You can call them on the box, because the box is a sized type that also implements Iterator.

The Shape trait of 24_01_dyn_compatibility.rs uses the same bound:

trait Shape {
    fn area(&self) -> f64;               // gets a vtable slot
    fn scaled(self, factor: f64) -> Self
    where
        Self: Sized;                     // no vtable slot: callable only on a concrete type
}

// On a concrete type, the call is valid:
//     Circle { radius: 2.0 }.scaled(3.0)     // a Circle with radius 6.0
// On a trait object `s: &dyn Shape`, the call `s.scaled(2.0)` does not compile:
//     error: the `scaled` method cannot be invoked on a trait object

The clone_box pattern is the substitute for Clone, whose clone method returns Self. Box<dyn Shape> is a concrete type with a fixed size, so a method that returns it is dispatchable. The Service stacks of the tower crate and many plugin systems use this pattern:

trait Shape {
    fn clone_box(&self) -> Box<dyn Shape>;   // dyn-compatible substitute for Clone
}

// Circle is a struct with #[derive(Clone)].
impl Shape for Circle {
    // self.clone() makes a Circle. The return coerces Box<Circle> to Box<dyn Shape>.
    fn clone_box(&self) -> Box<dyn Shape> { Box::new(self.clone()) }
}

Trait Objects in Practice: `Box`, `&dyn`, `Arc`, and the Bounds

dyn Trait itself is an unsized type, so a trait object is always behind a pointer. The pointer type that you select sets the ownership:

Pointer type	Ownership	Typical use
`&dyn Trait`	Borrowed	One polymorphic call with zero allocation
`Box<dyn Trait>`	Owned, unique	Heterogeneous collections, returned values
`Rc<dyn Trait>`	Shared, single-thread	Shared handlers in single-threaded code
`Arc<dyn Trait + Send + Sync>`	Shared, cross-thread	A handler that threads from `thread::spawn` share

The usual reason to use trait objects is the heterogeneous collection. A Vec<Plain> can hold only Plain values. If you erase each handler behind Box<dyn EventHandler>, one Vec can hold all of them. Each call goes through the vtable of the concrete type:

// EventHandler is a trait with two methods:
//     fn name(&self) -> &'static str;
//     fn handle(&self, event: &str) -> String;
// Plain, Shouter, Redactor, and Counter are four different structs that implement it.
let pipeline: Vec<Box<dyn EventHandler>> = vec![
    Box::new(Plain),                                        // returns the event unchanged
    Box::new(Shouter),                                      // returns the event in uppercase
    Box::new(Redactor { secret: "hunter2".to_string() }),   // replaces the secret with "***"
    Box::new(Counter { seen: AtomicUsize::new(0) }),        // adds the event number
];
// event: &str is "login with password hunter2".
for handler in &pipeline {
    // handler: &Box<dyn EventHandler>. The vtable selects the method at runtime.
    println!("[{}] {}", handler.name(), handler.handle(event));
}

Two bounds are easy to miss:

'static is implicit in Box<dyn Trait>. The full type is Box<dyn Trait + 'static>, so the erased value may not borrow local variables. For a reference, the default bound is the lifetime of the reference: &'a dyn Trait is dyn Trait + 'a.
You must write auto traits in the type. The compiler cannot examine an erased type to check thread safety. Thus a trait object that crosses threads must promise thread safety in its type: Arc<dyn EventHandler + Send + Sync>. Without + Send + Sync, the compiler rejects the closure that you give to thread::spawn.

24_03_heterogeneous_dispatch.rs prints the pipeline lines first. Then it prints the result of one &dyn EventHandler call and the count from an Arc handler that three threads share:

[plain] login with password hunter2
[shouter] LOGIN WITH PASSWORD HUNTER2
[redactor] login with password ***
[counter] event #1: login with password hunter2

[redactor] token=***
event #7: shutdown

All assertions passed.

Downcasting with `Any`

Type erasure is not reversible, unless you use std::any::Any. Every T: 'static implements Any. Any identifies each value with a TypeId, so a later downcast can check at runtime whether the value really is a Position.

let mut value: Box<dyn Any> = Box::new(42_i32);
assert!(value.is::<i32>());                          // compares the TypeId
assert_eq!(value.downcast_ref::<i32>(), Some(&42));  // borrows the value as &i32
assert_eq!(value.downcast_ref::<String>(), None);    // incorrect type: None, no panic

// downcast_mut gives a mutable borrow of the concrete value.
if let Some(n) = value.downcast_mut::<i32>() {
    *n += 100;                                       // the boxed value is now 142
}

// Box::downcast gives OWNERSHIP back. A failed downcast returns the same box in Err.
let boxed: Box<dyn Any> = Box::new("hello".to_string());
let wrong = boxed.downcast::<i32>().unwrap_err();    // wrong: Box<dyn Any>, nothing is lost
let right: Box<String> = wrong.downcast().expect("it really is a String"); // *right == "hello"

A domain trait can support downcasting with the as_any pattern. Add Any as a supertrait, which requires Self: 'static. Then add an accessor method that returns the value as &dyn Any:

trait Component: Any {
    fn kind(&self) -> &'static str;
    fn as_any(&self) -> &dyn Any;               // each impl returns `self`
    fn as_any_mut(&mut self) -> &mut dyn Any;   // each impl returns `self`
}

// Recover THE Position from a Vec<Box<dyn Component>>.
// component: &mut Box<dyn Component> is one element of that Vec.
// Position { x: i32, y: i32 } is one of the structs that implement Component.
if let Some(pos) = component.as_any_mut().downcast_mut::<Position>() {
    pos.x += 10;   // pos: &mut Position. A component of a different type gives None.
}

Since Rust 1.86, trait upcasting makes the accessor methods optional. Because Component: Any, a &dyn Component coerces directly to &dyn Any. The compiler substitutes the vtable of the supertrait, and no method call is necessary. The manual as_any method stays very common in codebases written before 1.86, so learn to recognize the two forms.

Tutorial 2.3 already used Any. catch_unwind returns Err(Box<dyn Any + Send>) because a panic payload can be a value of any type. To read the message, downcast the payload to &str (for a panic with a string literal) or to String (for a panic with format arguments).

`dyn Fn`, `dyn FnMut`, `dyn FnOnce`: Callable Trait Objects

Every closure has its own anonymous type. Thus a collection of callbacks, or a function that returns one of several closures, requires erasure behind the Fn* traits. The exception is a closure that captures nothing, which can also coerce to a fn pointer. Tutorial 14.3 covers the hierarchy itself.

Figure: The Fn hierarchy — capability to call vs. freedom to implement

24_1_dyn_trait_vtables_object_safety_2

type Callback = Box<dyn Fn(&str) -> String>;   // an alias keeps the long type readable

// punctuation: String is "?".
// shout is a plain function: fn shout(s: &str) -> String (uppercase plus "!").
let callbacks: Vec<Callback> = vec![
    Box::new(|s| format!("({s})")),                 // capture-less closure
    Box::new(move |s| format!("{s}{punctuation}")), // capturing closure
    Box::new(shout),                                // plain fn item
];
// Three different concrete types, one element type.
// With "hello", the three callbacks return "(hello)", "hello?", and "HELLO!".

// A function that returns one of several closures. Each closure has its own
// anonymous type, and `-> impl Fn(i32) -> i32` permits only ONE concrete type.
// With closures that capture, the arms do not unify (E0308), so you box them.
// (Capture-less closures such as these can also coerce to one fn(i32) -> i32 pointer.)
fn make_op(kind: &str) -> Box<dyn Fn(i32) -> i32> {
    match kind {
        "double" => Box::new(|x| x * 2),
        "negate" => Box::new(|x| -x),
        _ => Box::new(|x| x),
    }
}
// make_op("double")(21) returns 42, and make_op("negate")(5) returns -5.

Box<dyn FnMut> can hold private mutable state, for example a running total that you move into the closure. A call to it needs &mut access.

Box<dyn FnOnce> is directly callable. The call moves the closure out of the box and consumes the closure and the box. This direct call is possible since Rust 1.35. A second call gives error E0382 (use of moved value). The usual pattern is a Vec<Box<dyn FnOnce()>> of deferred cleanup actions that each run one time, in LIFO order:

// log: Rc<RefCell<Vec<&str>>> records each action that runs.
let mut deferred: Vec<Box<dyn FnOnce()>> = Vec::new();
deferred.push(Box::new(move || log.borrow_mut().push("close file")));
// pop() removes the newest action first (LIFO).
while let Some(action) = deferred.pop() {
    action();   // a call by value: each action runs exactly one time
}

Monomorphization vs. Dynamic Dispatch

The snippet shows the same function with two compilation strategies:

// MONOMORPHIZED: the compiler makes one specialized copy for each distinct F.
// The calls are direct and often inlined, which is fastest at runtime.
// N closure types give N copies of the code.
fn apply_generic<F: Fn(i32) -> i32>(xs: &[i32], f: F) -> Vec<i32> {
    xs.iter().map(|&x| f(x)).collect()
}

// DYNAMIC: exactly one compiled copy. Each f(x) is an indirect call through the
// vtable. The binary is smaller and the compile is faster, but each call has a
// runtime cost.
fn apply_dyn(xs: &[i32], f: &dyn Fn(i32) -> i32) -> Vec<i32> {
    xs.iter().map(|&x| f(x)).collect()
}

// The two functions give the same result. With doubler = |x: i32| x * 2:
//     apply_generic(&[1, 2, 3, 4], doubler)   // [2, 4, 6, 8]
//     apply_dyn(&[1, 2, 3, 4], &doubler)      // [2, 4, 6, 8]

	Generic `<T: Trait>`	`dyn Trait`
Dispatch	Static: a direct call that the compiler can inline	Indirect: a jump through the vtable
Code size	One copy for each instantiation	One copy total
Compile time	Grows with the number of instantiations	Constant
Heterogeneous `Vec`	No: one `T` for each instance	Yes
Return "one of several types"	No (`impl Trait` = one type)	Yes
Optimizer visibility	Full (can inline, vectorize)	The call boundary is opaque

General rules from real codebases:

Keep hot inner loops and iterator chains generic. The iterator adapters of the standard library are the standard example.
Use dyn for plugin registries, GUI callbacks, and all code that crosses a "compiled separately" boundary.

The indirect call itself costs a few nanoseconds. The larger cost is usually the lost inlining. Measure before you assume one or the other. cargo, rustc, and serde each use the two strategies together.

Summary

Concept	Key point
Fat pointer	`&dyn Trait` = data ptr + vtable ptr = `2 * size_of::<usize>()`
Vtable contents	`drop_in_place`, size, alignment, then one function pointer for each dispatchable method
Dyn compatibility	No generic methods, no `Self` in signatures, no functions without a receiver, no associated consts, no `Self: Sized` supertrait
`where Self: Sized`	Excludes one method from the vtable. `Iterator` stays dyn compatible in this way
`clone_box`	Dyn-compatible substitute for `Clone`: return `Box<dyn Trait>`, not `Self`
Pointer types	`&dyn` (borrowed), `Box<dyn>` (owned), `Arc<dyn + Send + Sync>` (shared across threads)
Implicit bounds	`Box<dyn Trait>` means `+ 'static`. You must write auto traits in the type
`Any`	`is`, `downcast_ref`, `downcast_mut`, `Box::downcast`, the `as_any` pattern, and upcasting since 1.86
`dyn Fn*`	`Fn` (call via `&self`), `FnMut` (`&mut self`), `FnOnce` (by value). `Box<dyn FnOnce>` is directly callable
Generic vs `dyn`	Monomorphization gives speed and inlining. Dynamic dispatch gives one copy, heterogeneity, and a selection at runtime

Code Examples

File	Description
`24_01_dyn_compatibility.rs`	Dyn-compatibility rules with the five violations as commented compile errors, the `where Self: Sized` exclusion, and the `clone_box` pattern
`24_02_fat_pointer_anatomy.rs`	Thin and fat pointers, `size_of_val`/`align_of_val` that read the vtable, and a drop flag that proves the `drop_in_place` call
`24_03_heterogeneous_dispatch.rs`	`Vec<Box<dyn Trait>>` event pipeline, `&dyn Trait` without allocation, and `Arc<dyn Trait + Send + Sync>` across threads
`24_04_any_downcasting.rs`	`Any` basics, `Box::downcast`, `TypeId`, the `as_any` pattern, trait upcasting (1.86), and the downcast of a panic payload
`24_05_fn_trait_objects.rs`	`dyn Fn`/`FnMut`/`FnOnce` callbacks, callable `Box<dyn FnOnce>`, a function that returns closures, and a generic function compared with its `dyn` form

25.1 · Vec, Boxed Slices, and Arrays: Choosing the Right Container

Domain 25 — The Contiguous Memory Triumvirate Duration: ~15 minutes Library components: std::vec::Vec, std::boxed::Box, primitive [T; N], std::array::from_fn, std::array::try_from_fn, std::boxed::BoxedArrayIntoIter

Introduction

Rust has exactly three ways to own a contiguous sequence of T:

Vec<T>, which can grow.
Box<[T]>, which is on the heap and has a fixed length.
The array [T; N], which has a length that the compiler knows.

All three give the same borrowed view, &[T], so code that only reads the elements can use each of them. For an owner, they differ in exactly two properties: the location of the elements, and the quantity of bookkeeping data in the handle.

Most Rust developers use Vec by default and do not consider the alternatives. This tutorial gives the reasons to use the other two containers. A Box<[T]> is one machine word smaller for each instance, and its type proves that the sequence will never grow. A [T; N] has no cost other than its elements, and it can be entirely in registers. A deliberate choice is most important in long-lived structs, large collections of collections, and hot paths.

This tutorial shows:

The memory layout of the three containers (3 words, 2 words, and inline elements), with size_of assertions as proof.
Capacity slack: the memory that a grown Vec wastes, and how into_boxed_slice() releases it.
The conversion cycle: Vec::into_boxed_slice(), into_vec() on a Box<[T]>, as_slice() on an array, and TryFrom<&[T]> for arrays.
How to construct arrays from closures with array::from_fn, and the fallible counterpart array::try_from_fn, which is still unstable.
How to iterate a boxed array Box<[T; N]> directly (stabilized in 1.99).
A decision framework: which container to use, and when.

The Triumvirate Layout

A Vec<T> is three machine words on the stack: pointer, length, and capacity. A Box<[T]> is a fat pointer of two words: pointer and length. It has no capacity word, because a sequence that cannot grow has no capacity to record. A [T; N] is only its elements. The length N is part of the type, so it uses zero bytes at runtime.

Figure: Vec<T> vs Box<[T]> vs [T; N] memory layout

Vec vs boxed slice vs array layout comparison

use std::mem;

// One machine word: 8 bytes on a 64-bit target.
let word = mem::size_of::<usize>();
assert_eq!(mem::size_of::<Vec<u64>>(), 3 * word);    // ptr + len + cap
assert_eq!(mem::size_of::<Box<[u64]>>(), 2 * word);  // ptr + len (fat pointer)
assert_eq!(mem::size_of::<&[u64]>(), 2 * word);      // the borrowed view is also a fat pointer
assert_eq!(mem::size_of::<[u64; 4]>(), 4 * 8);       // only the 4 elements, inline

Two consequences are important:

Box<[T; N]> is a thin pointer. When the length is in the type, the runtime pointer needs only one word. On a 64-bit target, Box<[u64; 4]> is 8 bytes and Box<[u64]> is 16 bytes.
Niche optimization has no cost. Option<Vec<T>> and Option<Box<[T]>> have the same size as their payload, because the null pointer encodes None.

size_of_val shows where the elements are. For a Vec or Box<[T]> binding, it reports the size of the handle on the stack (24 or 16 bytes). For an array, it reports the size of the elements, because the elements are the value.

The shared &[T] view gives a design rule: accept &[T] in function signatures, never &Vec<T>. One function then serves all three containers, and each other contiguous type.

25_01_triumvirate_layout.rs prints:

size_of::<Vec<u64>>()      = 24 bytes (3 words)
size_of::<Box<[u64]>>()    = 16 bytes (2 words)
size_of::<&[u64]>()        = 16 bytes (2 words)
size_of::<Box<[u64; 4]>>() = 8 bytes (1 word — N is in the type)
size_of::<[u64; 4]>()      = 32 bytes (4 × 8, elements inline)

Option<Box<[u64]>> is still 16 bytes (niche optimization)

size_of_val(&vec_data)   = 24 bytes (handle only — elements on heap)
size_of_val(&boxed_data) = 16 bytes (handle only — elements on heap)
size_of_val(&array_data) = 32 bytes (the elements themselves)

checksum agrees across Vec, Box<[T]>, and [T; N]: 10

All assertions passed.

Capacity Overhead and into_boxed_slice

A Vec doubles its capacity when it grows (Tutorial 4.1). This amortized doubling makes push O(1), but it leaves slack. After 100 push calls of one element each, a Vec<u64> typically has capacity 128. The 28 unused slots are 224 bytes of heap that the program cannot use while the Vec exists.

When you build a sequence one time and then only read it, into_boxed_slice() is the step that freezes it. A frozen sequence has a permanent length:

let mut readings: Vec<u64> = Vec::new();
for i in 0..100 {
    readings.push(i * i);          // amortized doubling leaves slack
}
// The capacity is now typically 128, which is 28 slots more than the length.
let frozen: Box<[u64]> = readings.into_boxed_slice();   // consumes the Vec and releases the slack
assert_eq!(frozen.len(), 100);     // exactly 100 elements, and no capacity field

The conversion has three properties:

If capacity > len, the method shrinks the buffer, so the boxed slice owns exactly len elements. The shrink may reallocate and copy.
If capacity == len, the conversion has no cost. The boxed slice uses the same allocation, and the pointer does not move.
The conversion back to a Vec never has a cost. into_vec() on a Box<[T]> uses the same buffer and sets capacity == len, which guarantees zero slack.

If an iterator pipeline must give a frozen sequence, you can also collect directly into a boxed slice:

// collect makes the Box<[u64]> directly: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81].
let squares: Box<[u64]> = (0..10).map(|i| i * i).collect();

The general rule is: build with Vec, freeze with into_boxed_slice, and lend &[T]. The ability to grow is necessary while you build the sequence, not while you keep it.

25_02_capacity_overhead.rs prints:

after building: len=100, cap=128    # (cap varies by allocator/strategy)
slack: 28 unused slots = 224 bytes of dead heap    # (varies with cap)

frozen: Box<[u64]> with len=100 — no capacity field exists
thawed back into Vec: len=100, cap=100

collected Box<[u64]>: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
exact-capacity Vec froze in place (buffer did not move)

All assertions passed.

Constructing Arrays with array::from_fn

Array literals have limits. The repeat form, for example [0u8; 32], gives one repeated value, and it needs T: Copy or a constant value. std::array::from_fn constructs a [T; N] from a closure. It calls the closure with each index in 0..N, which is equivalent to [f(0), f(1), …, f(N-1)]:

use std::array;

// The type of the binding sets N = 8. The closure gets each index i in 0..8.
let squares: [usize; 8] = array::from_fn(|i| i * i);
assert_eq!(squares, [0, 1, 4, 9, 16, 25, 36, 49]);

// A lookup table: popcount[i] is the number of 1 bits in i.
// Compute the table one time at startup, then index it in hot loops.
let popcount: [u32; 16] = array::from_fn(usize::count_ones);
assert_eq!(popcount[0b1011], 3);   // 0b1011 has three 1 bits

The documentation guarantees ascending index order, so stateful construction is sound. The closure can keep state between calls, for example to build the Fibonacci sequence:

// `pair` holds two adjacent Fibonacci numbers. The closure changes it on each call.
let mut pair = (0u64, 1u64);
let fib: [u64; 10] = array::from_fn(|_| {    // the closure ignores the index
    let next = pair.0;                       // the value of this element
    pair = (pair.1, pair.0 + pair.1);        // move one step for the next call
    next
});
assert_eq!(fib, [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]);

Fallible Construction: try_from_fn and the Stable Recipe

The curriculum lists std::array::try_from_fn. The closure of this function returns Result<T, E> or Option<T>, and the full construction stops at the first failure. It is still unstable in Rust 1.99 (feature array_try_from_fn, tracking issue #89379). Example 25_05 shows it behind the nightly feature gate. The example also makes the short-circuit visible: after a failure at index 2, the function never calls the closure for index 3:

#![feature(array_try_from_fn)]  // nightly only: stable Rust rejects this attribute (E0554)

use std::array;
use std::num::ParseIntError;

let fields = ["10", "20", "30", "40"];
// Each closure call returns Result<i32, ParseIntError>. The first Err stops the construction.
let parsed: Result<[i32; 4], ParseIntError> =
    array::try_from_fn(|i| fields[i].parse::<i32>());
assert_eq!(parsed, Ok([10, 20, 30, 40]));

The stable recipe collects into a Vec and then converts with TryFrom<&[T]> for [T; N]. This conversion succeeds only when the length matches:

// This code is in a function that returns Result<_, Box<dyn std::error::Error>>,
// so `?` can return the two different error types. Example `25_03` uses `expect`.
let good = ["10", "20", "30", "40"];
let parsed: Vec<i32> = good.iter()
    .map(|s| s.parse::<i32>())
    .collect::<Result<_, _>>()?;     // stops at the first field that does not parse
// The conversion copies the elements, so it needs T: Copy. It checks the length at runtime.
let arr: [i32; 4] = parsed.as_slice().try_into()?;   // [10, 20, 30, 40]

The Decision Framework

When you know the layouts and the conversions, the choice is mechanical:

Figure: Choosing a contiguous container

25_1_vec_boxed_slices_arrays_1

Example 25_04 is a color-palette pipeline. It uses each of the three containers where that container is optimal:

[u8; 3] for an RGB color: the compiler knows the size, the value is small, and the type is Copy.
Vec<Rgb> while the program parses an unknown number of colors.
Box<[Rgb]> as the frozen long-term owner.

One caveat is important: an array is in the same memory as its owner. A local [u64; 1_000_000] is 8 MB on the stack, which is more than the default stack of a spawned thread on many platforms. For large fixed-size buffers, build on the heap with vec![0; N].into_boxed_slice(). Do not use Box::new([0u64; N]). In debug builds, that expression first makes the array as a temporary on the stack.

25_04_choosing_container.rs prints:

parsed 4 colors into a Vec (cap=4)    # (cap varies by allocator/strategy)
palette handle: 16 bytes on the stack, payload: 12 bytes on the heap
brightest color: [f2, a9, 00]

[u64; 1_000_000] would be 8000000 bytes of stack — don't.
heap-built Box<[u64]> of the same size: fine.

All assertions passed.

When the type must record the length, convert the boxed slice to a boxed array. try_into() converts a Box<[T]> to a Box<[T; N]>.

Since Rust 1.99, you can iterate a boxed array directly. Box<[T; N]>, &Box<[T; N]>, and &mut Box<[T; N]> implement IntoIterator. Before 1.99, for item in boxed_array did not compile, and boxed_array.into_iter() first moved the full array to the stack. The by-value iterator, std::boxed::BoxedArrayIntoIter, takes the elements directly from the heap:

let mut levels: Box<[u32; 4]> = Box::new([10, 20, 30, 40]);

for level in &levels { /* level: &u32, and the loop only borrows the box */ }
for level in &mut levels { *level *= 2; }   // level: &mut u32, and the change is in place

// By value: into_iter consumes the box and returns a BoxedArrayIntoIter<u32, 4>.
// The iterator yields each u32 from the heap.
let doubled: Vec<u32> = levels.into_iter().collect();
assert_eq!(doubled, [20, 40, 60, 80]);

25_13_boxed_array_into_iter.rs also iterates a 1 MiB boxed array by value, which is safe for the stack:

sum through &Box<[u32; 4]>: 100
after doubling through &mut Box: [20, 40, 60, 80]
lengths of the moved Strings: [5, 4, 5]
remaining after one item from each end: [2, 3, 4]
the remainder in reverse order: [4, 3, 2]
iterated a 1048576-byte boxed array by value

All assertions passed.

Summary

Concept	Key point
`Vec<T>` layout	3 words: pointer, length, and capacity. It can grow, and it may have slack
`Box<[T]>` layout	2 words: pointer and length (a fat pointer). It is frozen and has zero slack
`[T; N]` layout	The elements are inline. `N` is part of the type and uses 0 bytes at runtime
`Box<[T; N]>`	A thin pointer (1 word), because the length is in the type. You can iterate it by value and by reference since 1.99
Niche optimization	`Option<Vec<T>>` and `Option<Box<[T]>>` have no extra cost
`into_boxed_slice()`	Freezes a `Vec`: it shrinks the buffer to the exact size (no cost if `cap == len`)
`into_vec()`	Converts a `Box<[T]>` back to a `Vec`: it never has a cost, and it guarantees `capacity == len`
`array::from_fn`	Builds `[T; N]` from a closure. The documentation guarantees ascending index order
`array::try_from_fn`	The fallible counterpart. It is still unstable in 1.99 (nightly example `25_05`)
Stable fallible recipe	`collect::<Result<Vec<_>, _>>()`, then `TryFrom<&[T]> for [T; N]`
Stack limit caveat	For large fixed-size buffers, use `vec![…].into_boxed_slice()`, not a local array
API design	Accept `&[T]`. Each contiguous container coerces to it

Code Examples

File	Description
`25_01_triumvirate_layout.rs`	`size_of` proofs of the three layouts (3 words, 2 words, inline), niche optimization, and one `&[T]` function for all three containers
`25_02_capacity_overhead.rs`	Capacity slack after growth, the shrink in `into_boxed_slice`, and the `into_vec` round trip with `capacity == len`
`25_03_array_from_fn.rs`	`array::from_fn` (index-driven, lookup tables, stateful), and stable fallible construction with `TryFrom<&[T]>`
`25_04_choosing_container.rs`	The decision framework in one pipeline with `[u8; 3]`, `Vec`, and `Box<[Rgb]>`, and the stack-limit caveat
`25_05_array_try_from_fn.rs`	Nightly: `array::try_from_fn` with `Result` and `Option` closures, and a visible short-circuit
`25_13_boxed_array_into_iter.rs`	`IntoIterator` for `Box<[T; N]>` (1.99): by reference, by mutable reference, and by value with `BoxedArrayIntoIter`

25.2 · Slice Algorithms: Advanced Sorting, Searching, and Splitting

Domain 25 — The Contiguous Memory Triumvirate Duration: ~15 minutes Library components: Primitive [T] methods, std::slice

Introduction

Each contiguous container from Tutorial 25.1 dereferences or coerces to [T], and [T] has one of the largest method sets in the standard library. Tutorial 4.5 showed the common operations. This tutorial shows the algorithmic methods. These methods replace manual loops (and their off-by-one bugs) with named and tested operations, which often have better asymptotic complexity.

This tutorial shows:

The binary-search family on sorted data: binary_search_by and binary_search_by_key, and why partition_point is usually the method that you need to find a boundary.
Safe structural decomposition: split_at / split_at_mut, the split_first / split_last family, and subslice_range and strip_circumfix (1.98).
Fixed-size views: chunks_exact, the as_chunks family, and array_windows for pairwise processing.
chunk_by, which splits a slice with a predicate on adjacent elements (the earlier name of this API was group_by).
Key-based sorting (sort_by_key, sort_by_cached_key, and sort_unstable_by_key) and O(n) selection with select_nth_unstable.

The curriculum names two APIs that changed before stabilization:

group_by was renamed. It became stable as chunk_by in Rust 1.77.
array_chunks is still unstable in 1.99. It is the iterator method Iterator::array_chunks, and [T] has no method with this name. The stable replacements for slices are chunks_exact and the as_chunks family (stable since 1.88).

This tutorial shows the stable form of each one.

The Binary-Search Family

On sorted data, binary_search and its variants find an element in O(log n). The by variant takes a comparator closure that returns Ordering. The by_key variant extracts a key and compares it with Ord, which is shorter when one field gives the sort order:

// struct Event { ts: u64, msg: &'static str }
// log: [Event; 6] is an event log, sorted by timestamp: ts = 100, 250, 400, 400, 400, 640.
let hit = log.binary_search_by(|e| e.ts.cmp(&250));      // Ok(1): found at index 1
let miss = log.binary_search_by_key(&500, |e| e.ts);     // Err(5): the insertion point

The Result return value gives the full answer. Ok(i) means that the element is at index i. Err(i) means that the element is not in the slice, and i is the index where an insertion keeps the slice sorted. Thus a short match inserts a value only if it is absent, and keeps the order:

// sorted: Vec<i32> = [10, 20, 40, 50], and value = 30.
match sorted.binary_search(&value) {
    Ok(_) => {}                              // already present: do nothing
    Err(pos) => sorted.insert(pos, value),   // pos is 2: insert before 40
}
// sorted is now [10, 20, 30, 40, 50]

The duplicate-key caveat: when several elements compare equal, binary search may return any index that matches. The standard library does not specify which one. Do not use this index to find a range. Use partition_point for that task.

25_06_partition_point_binary_search.rs prints these results. The log has three events with ts == 400:

binary_search_by(ts=250)  = Ok(1)  ("listen")
binary_search_by_key(500) = Err(5)  (insertion point)
binary_search_by_key(400) = Ok(4)  (any of 2..=4 — unspecified!)

partition_point: The Deterministic Boundary Finder

partition_point(pred) works on each slice that the predicate partitions. In such a slice, all the elements for which the predicate returns true come before all the elements for which it returns false. The method returns the index of the first false element in O(log n). On sorted data, each monotone predicate qualifies. binary_search can return one of several indexes that match, but partition_point always returns the boundary.

Figure: partition_point returns the first index where the predicate flips to false

25_2_slice_algorithms_1

Two partition points extract a full range of equal elements in O(log n). This is the usual lower-bound and upper-bound idiom:

// `log` is the sorted event log from the previous section: ts = 100, 250, 400, 400, 400, 640.
let lo = log.partition_point(|e| e.ts < 400);   // 2: the first index with ts >= 400
let hi = log.partition_point(|e| e.ts <= 400);  // 5: the first index with ts >  400
let burst = &log[lo..hi];                       // log[2..5]: each event with ts == 400

The same idiom finds all the events between two timestamps in a sorted, append-mostly log. It does not scan the log, and duplicate keys cause no ambiguity. The edge cases are well-defined. On an empty slice, or when the predicate is false for each element, the method returns 0. When the predicate is true for each element, the method returns len.

The range queries of the same example print:

events with ts == 400: log[2..5] = ["connect", "connect", "auth"]
events in 200..600:    log[1..5] = ["listen", "connect", "connect", "auth"]

The Splitting Family

split_at(mid) divides one borrow into two views that do not overlap, and it copies nothing. The _mut variant is the important one. The borrow checker rejects &mut frame[..4] and &mut frame[4..] when both are live, because they are two mutable borrows of frame (error E0499). split_at_mut contains the proof that the two parts are disjoint, and it returns the two &mut halves together:

// frame: [u8; 12] has a 4-byte header, the 7-byte payload "rustace", and 1 checksum byte.
let (header_mut, body_mut) = frame.split_at_mut(4);   // two &mut [u8]: 4 bytes and 8 bytes
header_mut[1] |= 0b1000_0000;          // set a flag bit in the header
for b in &mut body_mut[..7] {
    *b = b.to_ascii_uppercase();       // change the payload while header_mut is also live
}
// The payload is now "RUSTACE".

split_at panics if mid > len. split_at_checked returns an Option instead.

To process the head or the tail of a slice, use split_first and split_last. They return Option<(&T, &[T])>: one element and the remainder. Their _mut variants return mutable references. You do not need len() - 1 arithmetic, and empty input does not cause a panic. These methods are useful for framed data. After you remove a version byte from the front and a checksum byte from the back, the remainder is the payload:

// This check runs before the split_at_mut edits in the previous snippet.
// rest: &[u8] is the frame after the 4-byte header: the payload, then 1 checksum byte.
// checksum(&[u8]) -> u8 is the wrapping sum of the bytes.
let (stored_sum, payload) = rest.split_last().expect("frame has a checksum byte");
assert_eq!(*stored_sum, checksum(payload));   // stored_sum: &u8, payload: &[u8]

// After the edits, the checksum byte is stale. body: &mut [u8] is the same region of the frame.
// split_last_mut gives (&mut last, &mut remainder): read the payload and write the last byte.
let (sum_byte, payload) = body.split_last_mut().expect("non-empty body");
*sum_byte = checksum(payload);

25_07_split_family.rs prints:

header  = [1, 0, 0, 7]
payload = "rustace" (checksum 247 verified)
after split_at_mut edits: flags=0b10000000, payload="RUSTACE"
checksum re-patched in place: 23

Two methods from 1.98 complete the set of structural tools:

subslice_range returns the index range that a subslice occupies. It uses pointer arithmetic, not a search, so an independent slice with the same values gives None. Code that splits a slice can use it to compute diagnostic spans.
strip_circumfix removes a prefix and a suffix in one call. For a frame [STX][payload][ETX], it returns the payload, or None if one of the two sentinels is missing.

Chunks and Windows

There are two types of fixed-size views: disjoint chunks that tile the slice, and overlapping windows that move across it.

Figure: disjoint chunks vs overlapping windows on 6 elements

25_2_slice_algorithms_2

chunks_exact(n) / chunks_exact_mut(n): the two methods guarantee that each chunk has length n, so the optimizer can remove bounds checks. .remainder() returns the elements at the end that do not fill a chunk.
as_chunks::<N>() (stable since 1.88): one call returns (&[[T; N]], &[T]), a slice of arrays and the remainder. Each element is a [T; N], so a pattern can destructure a chunk: .map(|&[r, g, b]| …). as_chunks_mut gives mutable array chunks. as_rchunks puts the remainder at the front. This family is the stable alternative to the array_chunks iterator. The curriculum lists array_chunks, but it is still unstable in 1.99.
array_windows::<N>() (stable since 1.94): overlapping &[T; N] windows, the usual tool for pairwise processing. Tutorial 4.5 introduces it. windows(2) yields &[T], so you must index each window. With array_windows, the closure destructures the window directly:

let temps: [i32; 6] = [18, 21, 25, 24, 20, 22];
// Each window is a &[i32; 2]. The pattern [a, b] sets N = 2.
let deltas: Vec<i32> = temps.array_windows().map(|[a, b]| b - a).collect();
assert_eq!(deltas, [3, 4, -1, -4, 2]);   // 6 elements give 5 windows

Example 25_08 uses all three. It processes an RGB pixel buffer with 3 bytes for each pixel and one extra padding byte, which makes the remainder visible. It also processes a sensor series in pairs. The example prints:

chunks_exact(3): 3 bright pixels, remainder [238]
after chunks_exact_mut red-halving: first pixel = [127, 0, 0]
as_chunks::<3>: 4 pixels, brightest channel Some(255)

temps  = [18, 21, 25, 24, 20, 22]
deltas = [3, 4, -1, -4, 2] (via array_windows::<2>)
6 elements: 5 overlapping windows vs 3 disjoint chunks

chunk_by: Splitting on Adjacent-Element Predicates

Some group boundaries are not at fixed sizes. They are at each position where a relation between adjacent elements stops. chunk_by(pred) yields the maximal runs of adjacent elements for which the pairwise predicate is true. The closure gets (previous, current). true keeps the two elements in the same chunk, and false starts a new chunk. The chunks are non-empty subslices that tile the original slice exactly.

Naming history: for years, this API was unstable with the name group_by. The curriculum and older blog posts use that name. The API was renamed to chunk_by when it became stable in Rust 1.77. If you search for group_by on [T] today, chunk_by is the method that you need.

Three predicates give three algorithms:

// struct Reading { day: u32, value: i32 }
// readings: [Reading; 6] is sorted by day. Like dedup, chunk_by compares only ADJACENT pairs.
// The result has one slice for each day.
let per_day: Vec<&[Reading]> = readings.chunk_by(|a, b| a.day == b.day).collect();

// Equality gives run-length encoding. dna is b"AAACCGGGGT".
let rle: Vec<(u8, usize)> = dna.chunk_by(|a, b| a == b)
    .map(|run| (run[0], run.len())).collect();   // [(b'A', 3), (b'C', 2), (b'G', 4), (b'T', 1)]

// a <= b gives the maximal ascending runs (the first step of many merge-based algorithms).
// series = [1, 4, 9, 2, 3, 8, 8, 5] gives [1, 4, 9], [2, 3, 8, 8], and [5].
let runs: Vec<&[i32]> = series.chunk_by(|a, b| a <= b).collect();

chunk_by_mut gives mutable access to one group at a time. Example 25_09 uses it to subtract the first value of each day from all the readings of that day, in place. Like dedup, chunk_by needs data that is already grouped. It never reorders elements, and it only splits the slice.

Key-Based Sorting and O(n) Selection

Choosing a sort variant

The three key-based sorts give the same order, except for ties (elements with equal keys). They differ in stability, in allocation, and in the number of key closure calls:

Variant	Stable	Key evaluations	When to use
`sort_by_key`	Yes: equal keys keep their original order	O(n log n): one for each comparison	A cheap key, when tie order is important
`sort_by_cached_key`	Yes	At most one for each element (the sort caches the keys)	An expensive key (one that allocates or reads a full string)
`sort_unstable_by_key`	No	O(n log n)	A cheap key, when tie order is not important. The fastest variant, with no allocation

Example 25_10 counts the key calls to make the difference visible. It sorts 8 words without regard to case, and the key allocates a String. sort_by_key calls the key function 36 times, but sort_by_cached_key calls it exactly 8 times.

// crates: [(&str, i32); 4] holds (name, downloads). Reverse puts the largest count first.
// The sort is stable: crates with equal counts keep their original order.
crates.sort_by_key(|&(_, downloads)| std::cmp::Reverse(downloads));
// latencies_us: [i32; 6] holds microsecond values. The key is the value itself.
// The unstable sort is the fastest choice for Copy scalars, and it does not allocate.
latencies_us.sort_unstable_by_key(|&us| us);

select_nth_unstable: order statistics without sorting

When you need the median or the top-k elements, and not a full order, a sort does more work than necessary. select_nth_unstable(n) puts at index n the element that a full sort would put there. All the elements before it are ≤ that element, and all the elements after it are ≥ that element. The operation is O(n). The method does not sort the two sides. It returns (&mut [T], &mut T, &mut [T]): the elements below, the pivot, and the elements above.

// response_ms: [i32; 7] = [402, 88, 273, 91, 1_180, 33, 250], not sorted.
let mid = response_ms.len() / 2;                                    // 3
let (below, median, above) = response_ms.select_nth_unstable(mid);  // *median is 250, in O(n)

// scores: [i32; 8] and k = 3. The descending comparator puts the 3 largest scores first.
// Sort only those k scores after this call: top-k in O(n + k log k).
let (top, kth, _rest) = scores.select_nth_unstable_by(k - 1, |a, b| b.cmp(a));
// *kth is 88, the 3rd-largest score. `top` holds the 2 larger scores, not sorted.

select_nth_unstable_by_key completes the family for struct fields. For example, one call finds the file with the 2nd-smallest line count.

25_10_sort_select.rs prints:

stable sort, ties preserved: [("serde", 480), ("libc", 480), ("rand", 320), ("cfg-if", 210)]

sort_by_key ran the key fn 36 times; sort_by_cached_key exactly 8    # (36 varies with the sort implementation)
sort_unstable_by_key: [12, 12, 47, 340, 830, 9800]

median latency: 250 ms (found in O(n), slice NOT sorted)
top-3 of 8 scores: [99, 94, 88]
2nd-smallest file: ("lib.rs", 90)

All assertions passed.

Summary

Concept	Key point
`binary_search_by` / `_by_key`	O(log n) on sorted data. `Err(i)` is the insertion point
Duplicate-key caveat	Binary search may return any index that matches. The library does not specify which one
`partition_point`	The deterministic boundary: the first index where the predicate becomes `false`
Lower/upper bound idiom	Two partition points extract a range of equal elements in O(log n)
`split_at` / `split_at_mut`	Two disjoint views. `_mut` safely gives the two `&mut` halves that the borrow checker rejects for manual slicing
`split_first` / `split_last` (+`_mut`)	`Option<(&T, &[T])>`: one element and the remainder, with no index arithmetic and no panics
`subslice_range` (1.98)	Converts a subslice to its index `Range` by pointer arithmetic
`strip_circumfix` (1.98)	Removes a prefix and a suffix in one call, or returns `None`
`chunks_exact` (+`_mut`)	`&[T]` chunks with a guaranteed length. `.remainder()` returns the remainder
`as_chunks` family (1.88)	`(&[[T; N]], &[T])`: arrays that a pattern can destructure. The stable alternative to the unstable `array_chunks`
`array_windows` (1.94)	Overlapping `&[T; N]` windows for pairwise processing with destructuring
`chunk_by` (1.77)	Runs of adjacent elements that satisfy a predicate. Renamed from `group_by` at stabilization
`sort_by_key` vs `_cached_key` vs `_unstable_by_key`	Choose by tie order, key cost, and speed (see the table above)
`select_nth_unstable` family	O(n) median and top-k: order statistics without a full sort

Code Examples

File	Description
`25_06_partition_point_binary_search.rs`	`binary_search_by`/`_by_key`, the duplicate-key caveat, and range extraction with `partition_point` (lower and upper bound)
`25_07_split_family.rs`	Binary-frame parsing with `split_at`/`_mut` and `split_first`/`split_last` (+`_mut`)
`25_11_subslice_range.rs`	`[T]::subslice_range` (1.98): find the index range of each piece of a split
`25_12_slice_strip_circumfix.rs`	`[T]::strip_circumfix` (1.98): remove the two ends of a framed slice
`25_08_chunks_windows.rs`	`chunks_exact`/`_mut` and `as_chunks`/`as_chunks_mut`/`as_rchunks` (1.88) on a pixel buffer, and `array_windows` (1.94) on a sensor series
`25_09_chunk_by.rs`	`chunk_by`/`chunk_by_mut` (earlier name: `group_by`): per-day grouping, run-length encoding, and monotone runs
`25_10_sort_select.rs`	`sort_by_key`, `sort_by_cached_key`, and `sort_unstable_by_key` with counted key calls, and median and top-k with `select_nth_unstable`

26.1 · The Newtype Pattern with Standard Library Traits

Domain 26 — Cross-Cutting Patterns Duration: ~15 minutes Library components: std::ops::Deref, std::fmt::Debug, std::fmt::Display, std::cmp::Ord, std::convert::From

Introduction

A newtype is a tuple struct with one field, for example struct Meters(f64). It wraps an existing type and gives it a new identity. At runtime, a newtype has no cost: Meters has the same representation in memory as an f64. At compile time, the newtype is a different type:

A Meters is not a Feet.
A UserId is not an order id.
A Password is not a String that you can print accidentally.

This is the first of three capstone tutorials. They use mechanisms from earlier domains again: trait derivation (13.1), operator overloading (14.1), Deref (14.2), and From/Into (1.3). They assemble these mechanisms into patterns that you will find in almost every production codebase.

This tutorial shows:

How to derive or manually implement standard traits on a newtype, and how each selection changes the behavior.
A Password(String) that redacts itself in Debug. Thus each log site is safe by construction.
Meters(f64) and Feet(f64) with typed arithmetic through std::ops and explicit From conversions.
SortByName(Person), which carries an alternative Ord into BinaryHeap and BTreeSet, and std::cmp::Reverse.
A judicious use of Deref, and the three problems that Deref abuse causes.

One Wrapper, Three Trait Strategies

The wrapper itself is simple. The design work is to decide, trait by trait, what the newtype must do. For each trait, the standard library gives you three options:

Figure: Choosing a trait strategy for each trait on a newtype

26_1_newtype_pattern_1

UserId(u64) uses the first branch for every trait. It derives Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, and Ord. Thus it sorts, hashes, and deduplicates exactly as the u64 that it wraps. But assert_eq!(user_id, 4_u64) stays a type error.

The omit branch is a design tool of the same value as the other two branches. If you do not implement a trait, a category of misuse becomes a compile error.

// Each derived trait forwards to the inner u64.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
struct UserId(u64);

let mut ids = vec![UserId(30), UserId(4), UserId(17)];
ids.sort();                       // Ord forwards to u64
// ids is now [UserId(4), UserId(17), UserId(30)]

// A raw u64 is not a UserId, so these two lines do not compile:
// let raw: u64 = 4;
// assert_eq!(ids[0], raw);       // error[E0308]: mismatched types (expected `UserId`, found `u64`)

Redacting Debug: `Password(String)`

A derived Debug on a secret prints the secret. The solution uses two branches of the decision tree. Implement Debug manually (the second branch) to redact the secret. Do not implement Display at all (the omit branch):

use std::fmt;

// There is no derive here: a derived Debug prints the inner String.
struct Password(String);

impl fmt::Debug for Password {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("Password(<redacted>)")   // shows no secret and no length
    }
}

// For a `pw: Password` that holds "hunter2":
// format!("{pw:?}") gives "Password(<redacted>)"
// println!("{pw}");   // error[E0277]: `Password` doesn't implement `std::fmt::Display`

The redaction also applies to outer types. A DbConfig struct that derives Debug calls the Debug implementation of each field. Thus the redaction applies at each log site, present and future. Nobody must remember to mask a field.

One method with a deliberate name, expose_secret, gives access to the secret. A text search then finds each use in a code review. The secrecy crate made this convention popular.

A related type, ApiToken, shows only its last four characters: ApiToken(…5b4a). That suffix is sufficient to correlate a support ticket with a token. It is not sufficient to steal the token.

26_01_newtype_redacting_debug.rs prints:

sorted ids: [UserId(4), UserId(17), UserId(30)]
debug of password: Password(<redacted>)
debug of token:    ApiToken(…5b4a)
config as logged:  DbConfig { host: "db.internal:5432", user: "app_rw", password: Password(<redacted>) }

All assertions passed.

Typed Units: `Meters`, `Feet`, and `From`

In 1999, NASA lost the Mars Climate Orbiter because two subsystems did not agree about units. With one newtype for each unit, that bug does not compile:

// Each wrapper is one f64 at runtime. The unit exists only at compile time.
#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]
struct Meters(f64);
#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]
struct Feet(f64);

// Add, Mul, and Div are the operator traits from std::ops.
// The method bodies are not shown. Each one operates on the inner f64.
impl Add for Meters { /* Meters + Meters = Meters */ }
impl Mul<f64> for Meters { /* Meters * f64 = Meters: a scale by a ratio is valid */ }
impl Div for Meters { type Output = f64; /* Meters / Meters = f64: the units cancel */ }

// There is no `impl Add<Feet> for Meters`, so this line does not compile:
// let bad = Meters(11_000.0) + Feet(1_200.0);
// error[E0308]: mismatched types (expected `Meters`, found `Feet`)

Three details are important for the design. All three come from the operator tutorials of Domain 14:

Only operations with a physical meaning exist. There is no Add<Feet> for Meters. That omission is the safety feature.
The output types document the physics. Meters / Meters returns a plain f64 because the units cancel. The signature says so.
From is the only conversion point. Meters::from(feet) and .into() make the conversion explicit. You can audit the conversion in one location, because the conversion constant is in no other location.

The example also implements Sum. Then the type operates directly in iterator pipelines: legs.iter().copied().sum::<Meters>(). Tutorial 26.3 shows such pipelines.

Ordering Wrappers: `SortByName` and `Reverse`

For a single sort, sort_by_key takes a closure, and you do not need a wrapper. But BinaryHeap<T> and BTreeSet<T> use T: Ord and accept no closure at all. A newtype is the only way to carry an alternative ordering into an ordered container:

Figure: The wrapper carries a different Ord into the container

26_1_newtype_pattern_2

// Person { id: u32, name: String } derives Ord, so its natural order is by id.
struct SortByName(Person);

impl Ord for SortByName {
    fn cmp(&self, other: &Self) -> Ordering {
        // Compare the names first. If the names are equal, compare the ids.
        self.0.name.cmp(&other.0.name).then_with(|| self.0.id.cmp(&other.0.id))
    }
}
// PartialEq, Eq, and PartialOrd for SortByName are not shown. They call cmp.
// For the people Miriam, Ana, Zoe, and Kai, a BinaryHeap<SortByName> pops Zoe first.

Two points of the contract from tutorial 13.1 are important here:

The tie-break on id keeps the ordering total: two different people never compare equal.
The example writes PartialEq and PartialOrd in terms of cmp. Thus all four comparison traits agree because of the structure of the code, not by coincidence.

The standard library has this same pattern as std::cmp::Reverse: a newtype whose Ord reverses the order of the inner type. BinaryHeap<Reverse<T>> is the idiomatic min-heap, and sort_by_key(|p| Reverse(p.id)) is the idiomatic descending sort. The removal of the wrapper has no runtime cost, because the wrapper has the same layout as its payload.

`Deref` Used Judiciously

When a newtype implements Deref, a method call on the newtype can reach the methods of the inner type. With judicious use, Deref gives a large read-only API and an invariant stays protected. The Tags wrapper of the example guarantees that its tags are sorted, deduplicated, and lowercase. Each mutation goes through insert, which keeps the invariant. The Deref implementation is:

// Tags is `struct Tags(Vec<String>)`. Only `Tags::insert` changes the Vec.
impl Deref for Tags {
    type Target = [String];          // the SLICE, not the Vec
    fn deref(&self) -> &[String] { &self.0 }   // &Vec<String> coerces to &[String]
}
// tags.len(), tags.first(), and tags.join(", ") now compile for a `tags: Tags`.

The implementation makes two deliberate decisions:

The target is the slice view, not the owned container. A shared &[String] gives len, iter, binary_search, join, and more. A slice has no push, so tags.push(…) is a compile error (E0599).
There is no DerefMut. A slice method that takes &mut self, such as sort or reverse, is a compile error (E0596). A mutation must go through insert.

Deref coercion also applies at call sites: a function that takes &[String] accepts &tags directly. The same mechanism lets &Vec<T> coerce to &[T] (tutorials 4.1 and 14.2).

When `Deref` Misuse Causes Problems

The example shows three failure modes. You can observe each one when the code runs:

A leaked secret. LeakyPassword with Deref<Target = str> makes the full &str API available again: pw.len(), pw.contains(…), and format!("{}", &*pw) all work. Redaction and Deref cannot exist together. The purpose of the newtype is to restrict the inner API, and Deref removes all those restrictions.
An invariant that DerefMut breaks. SortedScores implements DerefMut with the target Vec<i32>, so callers can push elements that are not in order. After such a push, binary_search(&5) does not find an element that is clearly in the vector. The invariant became false, and no line of unsafe code was necessary.
A false appearance of inheritance. Method calls use auto-deref, but trait implementations do not pass through Deref. &Vec<String> implements IntoIterator and &Tags does not, so for t in &tags does not compile. Deref is not subtyping.

In short form, the rule has two parts. Deref is for smart pointers (Box and Rc, tutorials 7.1 and 7.2) and for read-only wrappers that are a view of the inner type. Do not implement Deref on a type whose purpose is to restrict the inner API.

26_04_newtype_deref_tradeoffs.rs prints:

tags.join → cli, parser, rust
describe(&tags) → 3 tag(s): cli, parser, rust
LeakyPassword leaks through Deref: logged: hunter2
after leaked push: [10, 25, 40, 5] — binary_search(&5) fails

All assertions passed.

Summary

Concept	Key point
Newtype	A tuple struct with one field. It has a new identity at compile time and no cost at runtime.
Derive	Forwards the behavior of the inner type. This is correct when you want that behavior.
Implement manually	Changes the behavior: a `Debug` that redacts, a `Display` with units, an alternative `Ord`.
Omit	The absence of a trait is a design tool. No `Display` on `Password` makes a print a compile error.
`Password(String)`	The redaction is in the type. Each log site is safe, and this includes the derived `Debug` of outer structs.
`Meters`/`Feet`	Only operations with a meaning exist. `From` is the one explicit, auditable conversion point.
`SortByName(Person)`	The only way to carry an alternative `Ord` into `BinaryHeap`/`BTreeSet`.
`std::cmp::Reverse`	The ordering newtype of the standard library. `BinaryHeap<Reverse<T>>` is the idiomatic min-heap.
`Deref` (judicious)	A read-only view of a wrapper that keeps an invariant. Use the slice as the target. Do not implement `DerefMut`.
`Deref` (abuse)	Leaks secrets, lets `DerefMut` break invariants, and is not inheritance.

Code Examples

File	Description
`26_01_newtype_redacting_debug.rs`	`UserId` derives every trait, `Password` redacts its `Debug` and has no `Display`, and `ApiToken` shows only a suffix. A derived outer `Debug` keeps the redaction.
`26_02_newtype_units_arithmetic.rs`	`Meters`/`Feet` with typed `Add`/`Sub`/`Mul`/`Div`, `From` conversions in the two directions, `Display` with units, and `Sum` for pipelines.
`26_03_newtype_ord_wrappers.rs`	`SortByName(Person)` carries an alternative `Ord` into `BinaryHeap`. `std::cmp::Reverse` makes min-heaps and descending sorts.
`26_04_newtype_deref_tradeoffs.rs`	`Tags` uses `Deref<Target = [String]>` judiciously. `LeakyPassword` and `SortedScores` show the abuse cases. `Deref` is not inheritance.

26.2 · Builder Patterns with Default and Option

Domain 26 — Cross-Cutting Patterns Duration: ~15 minutes Library components: std::default::Default, std::option::Option, std::mem::take, std::mem::replace

Introduction

Rust has no named function arguments and no optional function arguments. The Rust ecosystem uses a pair of patterns as the alternative. The two patterns use only standard library parts:

Default supplies the baseline values.
Builders override a small number of settings and then assemble the value.

You already used the two patterns. std::process::Command, std::fs::OpenOptions, and std::thread::Builder are builders. ..Default::default() occurs in almost every codebase that has much configuration.

This capstone tutorial assembles parts from tutorial 17.3 (Default), tutorial 2.1 (Option combinators), and the ownership mechanics of Domain 1. These parts make the two canonical builder forms. The tutorial also shows exactly when each form is the better selection.

This tutorial shows:

Three ways to supply a Default: a derive, #[default] on an enum, and a manual implementation. It also shows when zero values are wrong.
The struct update idiom ..Default::default() and layered Option chains (or, unwrap_or, unwrap_or_default).
A consuming HttpRequestBuilder (it takes self by value) with Option fields and a fallible build().
The &mut self builder that uses mem::take, and the related Option::take and mem::replace.
The trade-off between misuse safety at compile time and ergonomic control flow.

`Default`, Three Ways

Default supplies one thing for a type: its baseline value. The standard library gives you three ways to supply that value:

// 1. Derived: each field gets the default of its type (0, "", empty Vec, None).
//    Correct when "all zeroes" really means "new": counters, accumulators.
#[derive(Default)]
struct RequestStats { served: u64, errors: u64 }

// 2. Enums: #[default] marks the baseline variant.
#[derive(Default)]
enum LogFormat { #[default] Text, Json }

// 3. Manual: when the zero values are wrong. A server configuration of all
//    zeroes (empty host, port 0, 0 workers) is not a usable server.
impl Default for ServerConfig {
    fn default() -> Self {
        // Not shown: log_format (LogFormat::default()) and tls_cert (None).
        Self { host: "127.0.0.1".into(), port: 8080, workers: 4, /* … */ }
    }
}

// RequestStats::default() is RequestStats { served: 0, errors: 0 }
// LogFormat::default() is LogFormat::Text

The manual implementation encodes the baseline of the domain. It is the one source of truth for the mechanisms that follow: struct update, unwrap_or_default, and mem::take.

Struct Update: `..Default::default()`

The struct update syntax takes each field that you did not name from a different instance. With Default::default() as the base, a configuration site states only its overrides:

// ServerConfig has five fields: host, port, workers, log_format, and tls_cert.
let prod = ServerConfig {
    host: String::from("0.0.0.0"),
    port: 443,
    tls_cert: Some(PathBuf::from("/etc/ssl/fullchain.pem")),
    ..Default::default()          // workers, log_format: inherited
};
// prod.workers is 4 and prod.log_format is LogFormat::Text

Read .. as "and the remaining fields from". Two properties make this idiom very common:

The overrides are explicit, and a text search finds them. The inherited values have one source of truth in the Default implementation.
The base can be any value of the type, not only the default. A staging configuration that you derive from prod uses the same syntax: ServerConfig { port: 8443, ..prod.clone() }.

Layered configuration has a precedence: a CLI flag overrides an environment variable, and an environment variable overrides the built-in value. Option combinators express the same idea, and the first Some is the result:

// cli_port: Option<u16> is None (no --port flag)
// env_port: Option<u16> is Some(9000)
let port = cli_port.or(env_port).unwrap_or(ServerConfig::default().port);
// port is 9000. If the two options are None, port is the default 8080.

When "absent" must only mean "empty", unwrap_or_default() converts an Option<T> directly to T::default(). There is no match and no panic path. A banner setting that is absent becomes "", and a list that is absent becomes an empty Vec.

The Consuming Builder: `self` by Value

The builder has the same fields as its product, but with relaxed types: each field is optional, and each field has a default. For that reason, #[derive(Default)] on the builder needs no other code. Each setter takes mut self by value and returns it, so the ownership moves through the chain:

// Method is an enum with the variants Get (the #[default]) and Post.
#[derive(Default)]
struct HttpRequestBuilder {
    method: Option<Method>,
    url: Option<String>,          // the one required field
    headers: Vec<(String, String)>,
    body: Option<Vec<u8>>,
    timeout: Option<Duration>,
}

impl HttpRequestBuilder {
    // The setters for method, header, body, and timeout have the same form as `url`.
    #[must_use]   // warns if the caller discards the returned builder
    fn url(mut self, url: impl Into<String>) -> Self {
        self.url = Some(url.into());
        self                                        // gives the builder back to the caller
    }

    // `self` by value: build consumes the builder and moves its fields into the product.
    fn build(self) -> Result<HttpRequest, MissingUrl> {
        Ok(HttpRequest {
            method: self.method.unwrap_or_default(),    // None becomes Method::Get
            url: self.url.ok_or(MissingUrl)?,           // required → Err, not panic
            headers: self.headers,
            body: self.body,                            // stays an Option in the product
            timeout: self.timeout.unwrap_or(Duration::from_secs(30)),
        })
    }
}

Figure: Option fields collapse exactly once, inside build()

26_2_builder_default_option_1

The product contains no remaining Options, except for data that is really optional, such as a body. build() resolves each default exactly one time. Thus a consumer never needs to examine again if the caller configured a timeout.

build(self) also consumes the builder. Thus a second use of the builder after build is a compile error, which is the strongest misuse guarantee available.

The Limitation of the Consuming Builder

Each setter moves the builder. Thus conditional configuration needs a rebinding:

// want_auth: bool, a condition that the program knows only at runtime.
let mut builder = HttpRequestBuilder::new().url("https://api.example.com/v1/me");
if want_auth {
    // `header` consumes `builder`. Assign the returned builder to the variable again.
    builder = builder.header("authorization", "Bearer <token>");   // rebind!
}

One if is acceptable. But add a loop over override pairs, three feature flags, and an early return, and the rebindings hide the logic. The second builder form is for code with that quantity of control flow.

26_06_consuming_request_builder.rs prints:

POST request: https://api.example.com/v1/orders (2 headers)
minimal request: https://api.example.com/health Get
no url → request builder finished without a URL

All assertions passed.

The `&mut` Builder and `mem::take`

Setters that take &mut self and return &mut Self chain equally well. But the builder stays a plain mutable local variable, so loops and branches need no rebinding. std::process::Command has this form:

// overrides: an array of (&str, &str) pairs, the connection parameters to add.
let mut builder = DbOptionsBuilder::default();
builder.host("replica.internal").database("orders");   // each setter returns &mut Self
for (key, value) in overrides {
    builder.param(key, value);     // call in a loop: no move, no rebind
}

The difficult part is finish(&mut self). You cannot move self.params out of a mutable reference: that is error E0507. std::mem::take solves the problem for the full builder in one call:

fn finish(&mut self) -> DbOptions {
    // `params: self.params` is error[E0507] here: `self` is only a mutable reference.
    let taken = mem::take(self);   // put Default in *self, get the old value OWNED
    DbOptions {
        host: taken.host.unwrap_or_else(|| String::from("localhost")),
        params: taken.params,      // moved, not cloned: `taken` is an owned value
        // … port, database, and connect_timeout use the same pattern as host
    }
}

mem::take(self) puts a DbOptionsBuilder::default() in the location of the builder and returns the previous value. That value is owned and movable, and there is no clone. This is why a &mut builder that moves its state out requires Default. Two related functions do similar work:

Option::take() is the version for one field. It moves the value out through a reference and leaves None. It needs no Default bound, because None is the empty state.
mem::replace(&mut slot, new) operates as mem::take, but you select the replacement. For example, you move a full buffer out for processing and put an empty buffer with reserved capacity in its location.

Choosing Between the Two

The two forms differ in exactly one mechanical detail: the type of self. That detail decides all the other properties:

Figure: Consuming vs &mut builder decision

26_2_builder_default_option_2

The reset is an important detail. After finish(), mem::take leaves a default builder in the variable. A second call of finish() compiles and runs, and it silently returns options that are all defaults. With the consuming builder, that same mistake is a compile error. That is the real trade-off: misuse safety at compile time against ergonomic control flow.

26_07_mutref_builder_mem_take.rs prints:

quick: db.internal:6432/orders
replica: replica.internal:5432/orders with 3 params
finish() again → host=localhost (builder had been reset)
password consumed once; second take → None
rotated 2 conns out; buffer ready for more

All assertions passed.

Summary

Concept	Key point
Derived `Default`	Correct when all-zero or empty really means "new".
`#[default]`	Marks the baseline enum variant.
Manual `Default`	Encodes the baseline of the domain. It is the one source of truth.
`..Default::default()`	State only the overrides. The remaining fields come from the base, and the base can be any instance.
`or` / `unwrap_or` chains	Layered precedence: the first `Some` is the result.
`unwrap_or_default`	Converts `Option<T>` to `T::default()`: absent means empty.
Consuming builder	`mut self → Self`. The chain is one expression. A second use after `build()` is a compile error.
`Option` fields + `build()`	The defaults resolve exactly one time. The product has no remaining `Option`s.
`&mut` builder	`&mut self → &mut Self`. Loops and branches need no rebinding (the form of `Command`).
`mem::take`	Moves owned state out of `&mut` and puts `Default` in its location. The `finish` of the `&mut` builder uses it.
`Option::take` / `mem::replace`	`Option::take` extracts one field. `mem::replace` uses a replacement that you select.

Code Examples

File	Description
`26_05_default_struct_update.rs`	`Default` (derived, `#[default]`, and manual), `..Default::default()` and `..prod.clone()`, and `or`/`unwrap_or`/`unwrap_or_default` chains.
`26_06_consuming_request_builder.rs`	`HttpRequestBuilder`: setters that take `self` by value, `#[must_use]`, a fallible `build()` with `ok_or`, and the rebinding problem.
`26_07_mutref_builder_mem_take.rs`	`DbOptionsBuilder`: `&mut` setters, `finish()` with `mem::take`, the silent reset trade-off, `Option::take`, and `mem::replace`.

26.3 · The Iterator + Collect Pattern as a Data Pipeline

Domain 26 — Cross-Cutting Patterns Duration: ~15 minutes Library components: std::iter::Iterator, std::iter::FromIterator, std::iter::Extend

Introduction

Domain 5 showed the mechanisms: lazy adapters (5.1), and consumers and the FromIterator/Extend traits (5.2). This capstone tutorial assembles them into the form that is most common in production Rust codebases: the data pipeline. A data pipeline has four steps: parse, transform, aggregate, and render. It has no intermediate for loops, and no partially initialized collection is in scope.

The examples send a small embedded CSV and an access log through pipelines, as an ETL job or a metrics collector does.

This tutorial shows:

One pipeline with many destinations: collect() into Vec, HashMap, HashSet, and String, and fold into BTreeMap. The target type selects the FromIterator implementation.
Errors in pipelines: the fail-fast collect::<Result<Vec<_>, _>>(), a partition that collects every error, and the related Option<Vec<_>>.
unzip: one pass, two collections.
Performance: Extend appends and does not collect again, and chain concatenates with no allocation at all.
How to implement FromIterator and Extend for your own Histogram type, so that .collect() can build a Histogram.

The Pipeline Shape

Each pipeline has three zones:

A source. Here the source is const text. In production, it is a file (tutorial 9.1) or a socket (tutorial 10.1).
A sequence of lazy adapters.
One consumer that pulls all the items through the adapters.

collect() is the most versatile consumer, because it delegates the construction to the destination:

Figure: The same pipeline, fanned out by target type

26_3_iterator_collect_pipeline_1

// ORDERS_CSV: &str, a header line and six lines "order_id,customer,region,amount_cents".
// parse_order: fn(&str) -> Order. An Order has the fields id, customer, region, amount_cents.
let orders: Vec<Order> = ORDERS_CSV.lines().skip(1).map(parse_order).collect();   // 6 orders
// An iterator of (key, value) pairs collects into a map.
let by_id: HashMap<u32, &Order> = orders.iter().map(|o| (o.id, o)).collect();
// A set keeps each different value one time: 3 customers from 6 orders.
let customers: HashSet<&str> = orders.iter().map(|o| o.customer.as_str()).collect();

collect() is a short form of FromIterator::from_iter. The type annotation (or the turbofish) selects the implementation. Three behaviors that depend on the target are important to remember:

Maps collect from pairs. For a duplicate key, the last value replaces the earlier values. The result is a "latest snapshot", not a merge. The map silently drops the earlier values.
Aggregation (many rows for each key) is not plain collect. Use fold with the entry API (tutorial 4.3). A BTreeMap keeps its keys sorted, so the order of the report is deterministic.
String is also a collection. An iterator of char or &str collects into one String. For formatted text, fold with write! appends to one String, with no temporary allocation for each line.

The first lines that 26_08_pipeline_collect_targets.rs prints are:

Vec        → 6 orders parsed
HashMap    → lookup 1004 = initech
BTreeMap   → apac total =   1320.00
BTreeMap   → eu   total =    349.00
BTreeMap   → us   total =    825.75
HashSet    → 3 distinct customers
String     → initials: agi
String     → roster: acme, globex, initech

Fail-Fast: Collecting into `Result<Vec<T>, E>`

When you cannot trust the input, the pipeline carries Result items. You can collect an iterator of Result<T, E> into a Result<Vec<T>, E>. The result holds all the Ok values, or the collection stops at the first Err and discards all that it built before. A counter in the pipeline proves the short-circuit:

// READINGS: &str, six lines "sensor,value". Lines 3 and 5 are malformed.
// parse_reading(line_no, line) returns Result<Reading, ParseError>.
let mut attempts = 0;                             // counts the lines that the pipeline parses
let all: Result<Vec<Reading>, ParseError> = READINGS
    .lines()
    .enumerate()                                  // gives (index from 0, line)
    .map(|(n, line)| { attempts += 1; parse_reading(n + 1, line) })
    .collect();                                   // stops at the first Err

assert!(all.is_err());
assert_eq!(attempts, 3);        // line 3 failed, and lines 4–6 were never parsed

This pattern operates together with ? at the call site. A function that returns Result can apply ? to …collect::<Result<_, _>>(). Then the first error of the pipeline becomes the error of the function.

The related Option<Vec<T>> target short-circuits in the same way, but it gives a plain None. The result is all or nothing, and it discards the reason.

Note that the error type must carry its own context (the line number, the incorrect text). The pipeline replaces the loop, so the error value is the only location for position data.

Keep Every Error: `partition`

Fail-fast is the correct behavior for a configuration file, because one bad line makes the full file suspect. A nightly batch needs the opposite: load the good rows and report all the bad rows together. Then the operator corrects them in one pass, not one row for each new run.

Figure: Fail-fast vs. partition

26_3_iterator_collect_pipeline_2

// The same source and the same parser as in the fail-fast snippet.
let (oks, errs): (Vec<_>, Vec<_>) = READINGS
    .lines()
    .enumerate()
    .map(|(n, line)| parse_reading(n + 1, line))
    .partition(Result::is_ok);    // true goes to `oks`, false goes to `errs`

// oks and errs are each Vec<Result<Reading, ParseError>>.
// oks holds only Ok values and errs holds only Err values, so these calls cannot panic.
let readings: Vec<Reading> = oks.into_iter().map(Result::unwrap).collect();        // 4 readings
let errors: Vec<ParseError> = errs.into_iter().map(Result::unwrap_err).collect();  // 2 errors

The predicate proved which variant each element holds, but the type system does not record that proof. The two sides are still Vec<Result<…>>. Thus the code needs the unwrap pass, which is safe by construction. The cost is the opposite of fail-fast: partition visits each item and builds the two collections. Select by intent:

For a validation gate, use fail-fast.
For a batch report, use partition.

26_09_result_collect_partition.rs prints:

fail-fast: aborted after 3 lines — line 3: "not-a-number" is not a number
Option collect: good=Some([3, 14, 15]) bad=None
partition: 4 readings loaded, 2 rejected:
  line 3: "not-a-number" is not a number
  line 5: missing field

All assertions passed.

`unzip`: One Pass, Two Collections

unzip is a variant of collect with two targets. It consumes an iterator of pairs. It puts the left elements into one collection and the right elements into a second collection, in a single traversal. Two passes of map and collect would traverse the data two times:

// WARM_RUNS: [(&str, u64); 4], pairs of (benchmark name, time in microseconds).
let (names, times): (Vec<&str>, Vec<u64>) = WARM_RUNS.into_iter().unzip();
// names is ["parse", "plan", "exec", "render"] and times is [120, 30, 480, 70]

The signature of unzip shows its requirements: each of the two targets needs only Default + Extend. This tutorial uses these same two traits in other sections too. Any pair of collections is valid, and you select each one independently:

// COLD_RUNS has the same type as WARM_RUNS. The set keeps the 4 different names.
let (distinct, cold_times): (HashSet<&str>, Vec<u64>) = COLD_RUNS.into_iter().unzip();
// cold_times is [340, 90, 610, 95]

`Extend` and `chain`: Avoiding Intermediate Allocations

collect() always builds a new collection, and a new collection usually needs a new allocation. In a loop that merges many batches, that is O(batches) allocations for a job that needs zero. When a buffer already exists, Extend appends in place. If you reserve the capacity first, the pointer never moves (tutorial 4.1 uses the same proof technique):

// times: Vec<u64> and cold_times: Vec<u64> come from the two `unzip` calls above.
let mut all_times: Vec<u64> = Vec::with_capacity(WARM_RUNS.len() + COLD_RUNS.len());
all_times.extend(times);                 // the first batch moves in
let ptr_before = all_times.as_ptr();     // the address of the heap buffer
all_times.extend(cold_times);            // the second batch appends in place
assert!(std::ptr::eq(ptr_before, all_times.as_ptr()));   // no reallocation

// The anti-pattern that `extend` replaces makes a new allocation for each merge:
// all_times = all_times.into_iter().chain(cold_times).collect();

Extend is a trait, not a Vec method. On HashMap, it inserts or updates each key, with the same rule for duplicate keys as collect. On String, it consumes an iterator of char or &str.

When the program consumes the merged sequence only once, a collection of any type is waste. chain concatenates lazily and without an allocation:

// `chain` gives the items of WARM_RUNS, then the items of COLD_RUNS. It builds no collection.
let total: u64 = WARM_RUNS.iter().chain(COLD_RUNS.iter()).map(|&(_, t)| t).sum();   // 1835

The general rule for the end of a pipeline:

If you need the sequence one time, use chain.
If you have a buffer, use extend.
If you need a new owner, use collect.

`FromIterator` for Your Own Type: `Histogram`

A custom type can be a collect target with the same status as a standard collection. A latency Histogram holds five bucket counters, not a Vec of samples. Two implementations, in the idiomatic order, make it a full pipeline consumer:

// Histogram derives Default: five bucket counters and a total, all 0.
// Histogram::record(&mut self, latency_ms: u64) adds 1 to the bucket of the sample.
impl Extend<u64> for Histogram {
    fn extend<I: IntoIterator<Item = u64>>(&mut self, iter: I) {
        for sample in iter { self.record(sample); }
    }
}

impl FromIterator<u64> for Histogram {
    fn from_iter<I: IntoIterator<Item = u64>>(iter: I) -> Self {
        let mut hist = Self::default();   // Default + Extend: the canonical pair
        hist.extend(iter);                // calls the Extend implementation above
        hist
    }
}

Now the text of the pipeline agrees with the requirement, and the pipeline aggregates while it iterates. For 8 samples or for 8 billion samples, the memory use is the same five bucket counters and one total:

// ACCESS_LOG: &str, eight lines "method path status latency_ms".
// parse_line: fn(&str) -> (u16, u64), the pair (status, latency_ms).
let hist: Histogram = ACCESS_LOG
    .lines()
    .map(parse_line)
    .filter(|&(status, _)| status < 500)      // drops the one line with status 500
    .map(|(_, latency_ms)| latency_ms)        // keeps only the latency
    .collect();                               // calls Histogram::from_iter
// hist holds 7 samples in the buckets [3, 1, 2, 1, 0]

Extend then adds later batches to the same buckets. There is no new histogram, and the histogram loses no counts. Because Histogram implements FromIterator<u64>, it also operates with the patterns of the earlier sections. A fallible pipeline can collect into Result<Histogram, E> exactly as into Result<Vec, E>, and this includes fail-fast.

26_11_fromiterator_histogram.rs prints:

after first batch (7 samples):
 <10 ms │███
 <50 ms │█
<100 ms │██
<500 ms │█
500+ ms │

after extend with 3 more:
 <10 ms │████
 <50 ms │██
<100 ms │██
<500 ms │█
500+ ms │█

Result<Histogram, _> from fallible samples: works like Result<Vec, _>

All assertions passed.

Summary

Concept	Key point
`collect()`	A short form of `FromIterator::from_iter`. The target type supplies the construction.
Map targets	Collect from `(K, V)` pairs. For a duplicate key, the last value stays (a snapshot, not a merge).
Aggregation	Many rows for each key need `fold` and the entry API. Use `BTreeMap` for a deterministic order.
`String` target	Iterators of `char`/`&str` collect into one `String`. Use `fold` and `write!` for formatted text.
`Result<Vec<T>, E>`	Fail-fast: all the `Ok` values or the first `Err`. It short-circuits and operates with `?`.
`Option<Vec<T>>`	The same short-circuit, but it discards the reason.
`partition(Result::is_ok)`	Visits each item. Keeps every success and every error, in input order.
`unzip`	One pass, two collections. The targets need only `Default + Extend`.
`Extend`	Appends to existing storage. With reserved capacity, there are zero reallocations.
`chain`	Lazy concatenation. It allocates nothing for a sequence that you consume one time.
Custom `FromIterator`	Implement `Extend`, then `FromIterator` with `Default` and `extend`. Then `.collect()` can build the type.

Code Examples

File	Description
`26_08_pipeline_collect_targets.rs`	One embedded CSV goes into `Vec`, `HashMap` (the last duplicate value stays), `BTreeMap` (`fold` and the entry API), `HashSet`, and `String`.
`26_09_result_collect_partition.rs`	The fail-fast `collect::<Result<Vec<_>, _>>()` with a proof of the short-circuit, `Option<Vec<_>>`, and a `partition` that collects every error.
`26_10_unzip_extend_chain.rs`	`unzip` into independent targets, `Extend` on `Vec`/`HashMap`/`String` with a proof of no reallocation, and lazy `chain`.
`26_11_fromiterator_histogram.rs`	`Extend` and `FromIterator` for a custom `Histogram`: collect from an access log, extend with later batches, and collect into `Result<Histogram, _>`.

Keyboard shortcuts

Rust Standard Library Tutorials