5.2 · Consumers, Collectors, and the FromIterator/Extend Traits

Domain 5 — Iterators and Lazy Computation Duration: ~15 minutes Library components: std::iter::FromIterator, std::iter::Extend, std::iter::Sum, std::iter::Product

Introduction

Tutorial 5.1 showed how to make pipelines. This tutorial shows how to end them. Consumers (also known as terminal operations) are the methods that call next() until they have an answer. The answer can be a number, a boolean, a position, or a new collection. The most important consumer is collect. Two traits work with collect, and it helps to know their names:

• FromIterator makes a collection from an iterator.
• Extend appends an iterator to an existing collection.

This tutorial shows:

• Reductions: fold, reduce, sum, product, count, for_each, unzip.
• Searches: any, all, find, position, nth, last.
• Extremes: min, max, min_by, max_by, min_by_key, max_by_key, and the rules for ties.
• collect, FromIterator, and the collect::<Result<Vec<_>, _>>() fail-fast pattern.
• Extend to append items, and when it is better than collect.

Choosing a Consumer

Each consumer answers a question. To select the correct consumer, you mostly need to state the question precisely:

Figure: Consumer decision map

All of these are provided methods. The implementation of each one calls next() in a loop (or try_fold, see Tutorial 5.6). This fact explains their behavior. count pulls each remaining item. any stops at the first true.

fold, reduce, and Related Methods

fold(init, f) passes an accumulator through each item. It is the universal reduction: you could write all the other reductions with it. 05_04_terminal_consumers.rs uses it to calculate latency statistics in one pass:

// requests: [(&str, u64); 6] holds (route, latency in ms) pairs.
// The accumulator is a (min, max, sum) tuple. The first argument of fold is its start value.
let (min, max, sum) = requests.iter().fold(
    (u64::MAX, u64::MIN, 0_u64),
    |(min, max, sum), &(_, ms)| (min.min(ms), max.max(ms), sum + ms),
);
// (min, max, sum) is (2, 340, 791)

reduce(f) is fold without an initial value. The first item becomes the initial accumulator, so the result is an Option. An empty stream has no first item, and then the result is None. Prefer reduce when there is no natural identity value. For example, a route name has no "zero".

sum() and product() use traits: std::iter::Sum and std::iter::Product. The standard library implements these traits for each numeric type, and for Option and Result of numeric types. For this reason, sum usually needs a type annotation:

// requests is the array of (route, latency in ms) pairs from the previous snippet.
let total: u64 = requests.iter().map(|(_, ms)| ms).sum();    // 791: the annotation selects u64
let compounded: f64 = [1.10, 0.95, 1.20].iter().product();  // 1.254: the three rates multiplied

for_each(f) runs a side effect for each item. It is equivalent to a for loop, but it fits at the end of a long pipeline. The rustc and cargo code uses this style convention: for for blocks of logic, and for_each for a one-line last step.

unzip() is the inverse of zip. It makes one pass through (A, B) pairs and gives two collections.

05_04_terminal_consumers.rs prints:

6 requests, 791 ms total, mean 131 ms
fold in one pass: min=2 max=340 sum=791
slowest route: Some(("/search", 340))
compounded throughput factor: 1.254
/search: 340 ms
/search: 295 ms
unzipped into 6 routes + 6 latencies

All assertions passed.

Searching: any, all, find, position, nth, last

05_05_searching_extremes.rs examines a server fleet during an incident:

// FLEET: [Server; 5]. A Server has the fields name, latency_ms, error_rate, and healthy.
// Two servers are not healthy: web-2 (index 1) and cache-1 (index 4).
FLEET.iter().any(|s| !s.healthy)              // -> bool (true), stops at the first true
FLEET.iter().all(|s| s.latency_ms < 1000)     // -> bool (true), stops at the first false
FLEET.iter().find(|s| !s.healthy)             // -> Option<&Server>, the ITEM (web-2)
FLEET.iter().position(|s| !s.healthy)         // -> Option<usize>, the INDEX (Some(1))

any and all replace the usual loop with a boolean flag. find and position replace the loop that counts an index. All four methods short-circuit.

Two positional consumers complete the set:

• nth(n) skips n items and returns the next one. It consumes all the items up to and including that one. A call to nth(2) and then next() gives items 2 and 3, not items 2 and 0.
• last() pulls the full iterator to get to the end. On a double-ended iterator, next_back() gets the last item in one step (Tutorial 5.4). Clippy tells you about this.

Extremes and Tie-Breaking

For items that have a natural order, min() and max() are sufficient. For structs, extract a key or supply a comparator:

// FLEET is the [Server; 5] array from the previous snippet.
// max_by_key compares the key but returns the item: Option<&Server>, here web-2 (340 ms).
let worst = FLEET.iter().max_by_key(|s| s.latency_ms);
// error_rate is an f64, and f64 is not Ord. total_cmp gives the comparator a total order.
let flakiest = FLEET.iter()
    .max_by(|a, b| a.error_rate.total_cmp(&b.error_rate));
// flakiest is cache-1. It ties with web-2 at 0.043, and max_by keeps the last maximum.

f64 does not implement Ord, because NaN prevents a total order. Thus max_by_key with a float key does not compile. The standard solution is f64::total_cmp in max_by.

Memorize the rule for ties: min* returns the first minimum, and max* returns the last maximum. In the example fleet, two servers have the same error rate. max_by selects the later server. min_by with a reversed comparator selects the earlier server. Thus, when ties exist, a reversed comparator is not equivalent to an exchange of min and max.

05_05_searching_extremes.rs prints:

fleet degraded: true
first unhealthy: web-2 at index 1
latency range: 5 ms .. 340 ms
flakiest (max_by, last tie wins): cache-1

All assertions passed.

collect and FromIterator

collect is one method with many possible results. It calls FromIterator::from_iter on the target type. Your type annotation (or turbofish) selects the implementation:

// words: &str is "the quick brown fox the lazy dog the end" (9 words, "the" occurs 3 times).
let list:   Vec<&str>           = words.split(' ').collect();   // 9 items
let unique: HashSet<&str>       = words.split(' ').collect();   // 7 items: no duplicates
// The first letter of each word: "tqbftldte"
let text:   String              = words.split(' ').filter_map(|w| w.chars().next()).collect();
// Word -> position. A later pair replaces an earlier pair, so index["the"] is 7.
let index:  BTreeMap<&str, usize> = words.split(' ').enumerate().map(|(i, w)| (w, i)).collect();

You can implement FromIterator for your own types. 05_06_collect_fromiterator.rs implements FromIterator<u64> for a LatencyStats struct. Thus a pipeline collects directly into the statistics, with no intermediate Vec and no second pass:

// LatencyStats has the fields count: usize, total: u64, min: u64, and max: u64.
impl FromIterator<u64> for LatencyStats {
    // collect() calls this function and gives it the pipeline as `iter`.
    fn from_iter<I: IntoIterator<Item = u64>>(iter: I) -> Self {
        let mut stats = LatencyStats { count: 0, total: 0, min: u64::MAX, max: 0 };
        for ms in iter { /* add ms to count and total, update min and max */ }
        stats
    }
}

// The annotation selects the implementation above.
let stats: LatencyStats = [12_u64, 340, 5, 7, 210].into_iter().collect();
// stats is LatencyStats { count: 5, total: 574, min: 5, max: 340 }

Figure: How collect dispatches

Collecting into Result and Option

The FromIterator implementation of Result in std is especially useful. When you collect an iterator of Result<T, E> into Result<Vec<T>, E>, the collection stops at the first Err and returns it. You get the vector only if each item is Ok. This is the standard pattern to parse all the items or report the item that failed:

let good = ["8080", "9090", "9091"];
// s.parse() returns Result<u16, ParseIntError>. The annotation on `ports` sets these types.
let ports: Result<Vec<u16>, ParseIntError> = good.iter().map(|s| s.parse()).collect();
assert_eq!(ports, Ok(vec![8080, 9090, 9091]));   // each item is Ok

let bad = ["8080", "not-a-port", "9091"];
let ports: Result<Vec<u16>, ParseIntError> = bad.iter().map(|s| s.parse()).collect();
assert!(ports.is_err());   // collect stops at "not-a-port" and does not parse "9091"

Option has the same behavior: one None makes the full result None. The two implementations stop early. The fail-fast check is in the loop of from_iter, so no work occurs after the first error. Internally, the short-circuit mechanism is ControlFlow (Tutorial 5.6).

Extend: Appending Instead of Rebuilding

collect always makes a new collection. When a collection already exists (an accumulation buffer, a cache, a config map), Extend appends an iterator to it:

let mut buffer: Vec<u32> = Vec::with_capacity(16);
buffer.extend([1, 2, 3]);                    // buffer is [1, 2, 3]
buffer.extend((10..13).map(|x| x * x));      // any IntoIterator is valid: adds 100, 121, 144

let mut settings = HashMap::from([("timeout", 30), ("retries", 3)]);
settings.extend([("retries", 5), ("workers", 8)]);  // a later key replaces the earlier value
assert_eq!(settings["retries"], 5);                 // 3 became 5, "timeout" is still 30

These details are important in practice (05_07_extend_trait.rs):

• Extend reads size_hint to reserve space before it appends. A batch append is one reservation and N writes, not N possible reallocations (see Tutorial 5.4 for size_hint).
• One collection can implement Extend<A> for several item types: String accepts char, &str, and String items.
• If you implement Extend on your own type, callers can append to it from any iterator, and the type enforces its own invariants. The EventLog in the example has a size limit. It counts the events that it drops, and it does not grow without limit.
• In a loop that accumulates batches, extend on one Vec uses the same allocation again. A collect for each batch allocates each time. For slices of Copy types specifically, extend_from_slice is the fastest method (Tutorial 4.1).

05_07_extend_trait.rs prints:

buffer after two batches: [1, 2, 3, 100, 121, 144]
len=14, capacity=16 (no reallocation)
hello, iterators
settings after override batch: 3 keys
event log kept 4 events, dropped 2

All assertions passed.

Summary

Code Examples