5.3 · Advanced Iterators: Peekable, Chain, Scan, and Inspect

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::Peekable, std::iter::Scan, std::iter::Inspect, std::iter::Intersperse

Introduction

The adapters in Tutorial 5.1 are stateless. This tutorial adds the stateful and structural tools:

• lookahead with Peekable
• running state with scan
• observation of a pipeline with inspect
• consumption in stages with by_ref
• cloned and copied, which change references into owned values

The last section shows two adapters that are still nightly-only on Rust 1.99: intersperse and array_chunks. They are useful to know before they become stable.

This tutorial shows:

• Peekable: peek, peek_mut, next_if, next_if_eq, and next_if_map (stable since 1.94). The example is a lexer that you write manually.
• scan for stateful transformations: running balances, circuit breakers, and prefix sums for sliding windows.
• inspect to debug a pipeline without a change to its structure.
• by_ref to consume an iterator in stages, and cloned compared with copied.
• A nightly preview: intersperse/intersperse_with and array_chunks.

Peekable: One-Item Lookahead

iterator.peekable() wraps an iterator and adds a one-slot buffer. peek() pulls one item from the inner iterator into the slot and lends you a reference. The item stays available for the subsequent next(). That one slot is sufficient for a very large class of parsing problems. For example, a lexer uses it to make the decision between = and ==.

Figure: The peek slot — state machine

// it: Peekable<array::IntoIter<i32, 3>>
let mut it = [10, 20, 30].into_iter().peekable();
assert_eq!(it.peek(), Some(&10));   // peek returns a reference, and 10 stays in the slot
assert_eq!(it.peek(), Some(&10));   // a second peek gives the same item
assert_eq!(it.next(), Some(10));    // next takes the item from the slot, by value

peek_mut() lends the slot mutably. You can change the next item before a consumer takes it. The example increases the priority of the next task in place.

A peek does move the inner iterator forward. The item is no longer in the source: it is in the slot. If only your code uses the Peekable, you cannot see this difference. It is important only if you plan to use the inner iterator directly again.

Conditional Consumption: next_if, next_if_eq, next_if_map

The next_if family solves the take_while problem from Tutorial 5.1. take_while loses the first item that fails the predicate. These methods test the item and put it back:

• next_if(pred) consumes the next item only if the predicate is true. If not, the item stays in the slot.
• next_if_eq(&expected) consumes the next item only if it is equal to a value. The lexer uses it to find a second = after the first.
• next_if_map(f) is stable since 1.94. The closure takes the item by value. It returns Ok(mapped) to consume and transform the item, or Err(item) to put the item back in the slot without a change. Thus it consumes and converts in one step:

// chars: Peekable<Chars>, the characters of the source text. The next char is a digit.
// to_digit(10) returns Some(digit) for '0'..='9', and ok_or(c) changes a None into Err(c).
// Ok(d) consumes the char. Err(c) puts the char back, and the loop stops.
let mut value = 0_u32;
while let Some(d) = chars.next_if_map(|c| c.to_digit(10).ok_or(c)) {
    value = value * 10 + d;   // for "42": value is 4, then 42
}

05_08_peekable_token_stream.rs uses all five methods. Four of them (peek, next_if, next_if_eq, next_if_map) make a lexer for total = 42 + 7. The lexer of rustc and many parser crates have the same structure:

// One arm of the lexer's `match` on the peeked char. tokens: Vec<Token>.
'=' => {
    chars.next();   // consume the first '='
    // next_if_eq consumes the second '=' only if it is the next char.
    if chars.next_if_eq(&'=').is_some() { tokens.push(Token::Eq) }   // the text was "=="
    else { tokens.push(Token::Assign) }                              // the text was "="
}

05_08_peekable_token_stream.rs prints:

consumed while small: [1, 2, 3]; 40 stayed in the stream
tokens: [Ident("total"), Assign, Number(42), Plus, Number(7)]
comparison: [Ident("count"), Eq, Number(10)]

All assertions passed.

scan: Stateful Transformation

scan(init, f) is the lazy form of fold. The state goes through the closure as in fold. But scan yields a value for each item and stays lazy. The closure can also stop the stream early: it returns None.

Figure: fold vs scan

05_09_scan_stateful.rs shows four uses of scan, from simple to complex. The first two use a transaction feed:

// Each fragment continues `transactions.iter()`.
// transactions: [i64; 6] is [500, -120, -80, 300, -900, 250].
// balance: &mut i64 is the state that scan keeps between items. It starts at 0.

// Running balance: a fold that yields each intermediate value.
.scan(0_i64, |balance, &tx| { *balance += tx; Some(*balance) })
// yields 500, 380, 300, 600, -300, -50

// Circuit breaker: the first overdraft STOPS the stream.
.scan(0_i64, |balance, &tx| {
    let next = *balance + tx;
    if next < 0 { None } else { *balance = next; Some(tx) }   // None stops the stream
})
// yields 500, -120, -80, 300. The -900 stops the stream, and scan does not read 250.

The third use is the sliding-window method from real metrics pipelines. One scan pass calculates prefix sums. After that, any window sum is prefix[i + w] - prefix[i]. You do not add the items of each window again. This method fits well with slice::windows (Tutorial 4.5).

The state and the yielded value do not need to have the same type. In the fourth use, a status feed has consecutive duplicates. The scan keeps the previous item as state and yields only the changes (ok → degraded → ok).

inspect: Debugging a Pipeline

When a chain gives the wrong output, you must find the stage that lost the item. inspect(f) passes each item through without a change and runs a closure on a reference to it. This is printf debugging that you can add to any pipeline, and the structure of the pipeline stays the same:

// readings: [&str; 4] is ["42", "-3", "oops", "17"].
let valid: Vec<u32> = readings.iter()
    .inspect(|raw| println!("  raw:      {raw:?}"))       // each item, before the parse
    .filter_map(|raw| raw.parse::<u32>().ok())            // drops "-3" and "oops"
    .inspect(|parsed| println!("  survived: {parsed}"))   // only the items that parsed
    .collect();
// valid is [42, 17]

05_10_inspect_by_ref.rs prints:

  raw:      "42"
  survived: 42
  raw:      "-3"
  raw:      "oops"
  raw:      "17"
  survived: 17
valid readings: [42, 17]
parsed 3 header lines + 1 body line(s)
cloned() performed 2 clones, lazily

All assertions passed.

The important point is the order of the lines. Items go through the chain one at a time. The pipeline inspects "42", parses it, and inspects it again before it reads "-3". This output makes laziness visible.

by_ref and cloned vs copied

by_ref. Adapters take self by value, so lines.take_while(...) moves lines. Then you cannot read the remainder of the document. by_ref() lends &mut I instead, and &mut I is also an Iterator. Thus a stage can consume only the items that it needs and then return the cursor:

// request: &str holds 3 header lines, one empty line, and the body line "run=nightly".
let mut lines = request.lines();
// by_ref lends `lines`. take_while stops at the empty line and consumes it.
let head: Vec<&str> = lines.by_ref().take_while(|l| !l.is_empty()).collect();   // 3 lines
let body: Vec<&str> = lines.collect();   // continues after the empty line: ["run=nightly"]

This is one pass with no index arithmetic. It is the standard structure for protocols with a header and a body, and for the batching pattern in Tutorial 5.5.

cloned vs copied. Slice iterators yield &T. These adapters change the references into owned values:

• copied() makes a bitwise copy and compiles only for T: Copy. When you use it, the code documents and enforces that the operation is trivial.
• cloned() calls clone() and accepts any T: Clone, at any cost. 05_10_inspect_by_ref.rs proves the cost and the laziness with a type that counts its clones. The count is zero after the construction of the pipeline, and exactly one for each item after consumption.

// names: [String; 1]. String is not Copy, so copied() is a compile error. This is intended:
//     let _: Vec<String> = names.iter().copied().collect();
// error[E0277]: the trait bound `String: Copy` is not satisfied

Nightly Preview: intersperse and array_chunks

These two adapters are not available on stable. On Rust 1.99, each one still requires nightly. Thus 05_11_intersperse_array_chunks.rs has required-features = ["nightly"] in the package manifest. Run it with this command:

cargo +nightly run -p domain-05-iterators --features nightly --bin 05_11_intersperse_array_chunks

• intersperse(sep) (feature iter_intersperse, issue #79524) puts a separator between items. It is a lazy join for any iterator. intersperse_with(f) calculates each separator on demand. A method-resolution conflict prevents the stabilization, and this conflict is years old. The widely used itertools crate already has an intersperse method. A stable std method would change the method that existing code calls.
• array_chunks::<N>() (feature iter_array_chunks, issue #100450) yields [T; N] by value, and the compiler checks N at compile time. slice::chunks (Tutorial 4.5) is different: it yields &[T], and the length of each slice is a runtime value. Destructuring needs no bounds checks. APIs such as u16::from_be_bytes accept the arrays directly. You can get the remaining items with into_remainder().

// stream: [u8; 5] is [0x12, 0x34, 0xAB, 0xCD, 0xEF].
let words: Vec<u16> = stream.into_iter()
    .array_chunks::<2>()           // yields [0x12, 0x34], then [0xAB, 0xCD]
    .map(u16::from_be_bytes)       // [u8; 2] -> u16, no try_into().unwrap()
    .collect();
// words is [0x1234, 0xABCD]. The last byte, 0xEF, is the remainder.

On stable, the alternative to intersperse is slice::join or manual separator logic. The alternative to array_chunks is slice::chunks_exact.

Summary

Code Examples