8.2 · Cursor, Sink, Empty, Repeat: In-Memory I/O and Binary Parsing

Domain 8 — I/O System Duration: ~15 minutes Library components: std::io::Cursor, std::io::Empty, std::io::Sink, std::io::Repeat, std::io::empty, std::io::sink, std::io::repeat

Introduction

The traits from tutorial 8.1 give their full benefit only if you can use them without real devices. std::io has four in-memory types for this purpose:

• Cursor<T> makes a byte buffer into a full Read + Write + Seek stream.
• io::empty() is a reader that is always at EOF.
• io::sink() is a writer that discards all bytes.
• io::repeat(byte) is an infinite reader of one byte.

For the I/O traits, these types do the work of /dev/null and /dev/zero. They are also the base of deterministic I/O tests.

This tutorial shows:

• Cursor<T>, and how the backing store (&[u8], Vec<u8>, &mut [u8]) sets its capabilities.
• empty, sink, and repeat as test doubles and data generators.
• The primary use case, a parser for a binary file:
  • It parses a header with read_exact and from_le_bytes.
  • It uses Seek to go through an offset table.
  • It uses Cursor<Vec<u8>> to write patches.

Cursor: A Position Over a Buffer

A Cursor<T> has only two fields: the wrapped buffer and a u64 position. A bare &[u8] reader does not have that position. A slice reader can only move forward: the slice is its own position, and it becomes shorter with each read. A cursor can go to any position and then return.

Figure: Cursor position and offset-table navigation over a 52-byte container

// These lines are in a function that returns io::Result<()>, so `?` is valid.
let mut cur = Cursor::new(&b"ABCDEFGHIJ"[..]);   // Cursor<&[u8]>, position 0

let mut chunk = [0u8; 3];
cur.read_exact(&mut chunk)?;        // reads "ABC" and moves the position forward
assert_eq!(cur.position(), 3);

cur.set_position(0);                // back to the start: a bare &[u8] cannot do this
cur.read_exact(&mut chunk)?;        // reads the same 3 bytes again
assert_eq!(&chunk, b"ABC");

cur.seek(SeekFrom::End(-2))?;       // position 8, 2 bytes before the end (tutorial 8.3)

Cursor also has these methods:

• position() and set_position() read and set the position at low cost. They do not return an io::Result, and they do not need a trait import.
• get_ref() and get_mut() give access to the whole underlying buffer, independently of the position.
• into_inner() returns the buffer when you finish with the stream.

Choosing the Backing Store

Cursor<T> is a reader for any T: AsRef<[u8]>. Only the owned and mutable variants implement Write. The selection of T is the API design decision:

Figure: Which Cursor backing store fits the job

Two behaviors of the variants that can write are not obvious:

• Cursor<Vec<u8>> fills gaps with zeros. If you seek past the end and then write, the bytes in the gap become zeros. This is the in-memory equivalent of a sparse file:

  let mut cur = Cursor::new(Vec::new());   // Cursor<Vec<u8>>, empty
  cur.write_all(b"header")?;               // 6 bytes, the position is now 6
  cur.set_position(9);                     // 3 bytes past the end
  cur.write_all(b"!")?;                    // the Vec grows to 10 bytes
  assert_eq!(cur.get_ref(), b"header\0\0\0!");   // bytes 6, 7, and 8 are zeros

• Cursor<&mut [u8]> cannot grow. A write at the end accepts zero bytes, so write_all reports WriteZero. Thus this variant is the appropriate selection to patch fixed-size regions such as packet headers.

The pattern that results is this: write functions that are generic over Read + Seek (or Write). Pass a File in production and a Cursor in tests. The code is the same, the tests use no disk, and the tests are fully deterministic.

empty, sink, and repeat

Three zero-cost utility streams complete the set of in-memory types:

• io::empty(): each read returns Ok(0). It is the standard "no input" value for code that requires impl Read. It also implements BufRead, so empty().lines() gives no items.
• io::sink(): it accepts and discards any number of bytes. Use it as a test double when you only need to know that serialization runs. You can also use it for a mandatory output that you want to ignore.
• io::repeat(b): an infinite reader of one byte. Always limit it with take or a fixed-size read_exact. A read_to_end call on the bare reader cannot complete: std returns an OutOfMemory error immediately.

When you compose them with io::copy (tutorial 8.1), one line does the work that otherwise needs a loop:

// reader: any value that implements Read.
// Measure the size of a stream and store nothing:
let size = io::copy(&mut reader, &mut io::sink())?;   // size: u64, the byte count

// Make a 16-byte 0xAB fixture without a loop:
let mut fixture = Vec::new();
io::repeat(0xAB).take(16).read_to_end(&mut fixture)?;   // fixture == [0xAB; 16]

// Send 10 KiB of synthetic input through a pipeline, with no allocation:
let n = io::copy(&mut io::repeat(b'x').take(10 * 1024), &mut io::sink())?;   // n == 10240

Binary Parsing: Header and Offset Table

The primary use case of Cursor is a parser for a binary container. The example defines a 52-byte "RIMG" format as a const array. The array contains a magic number, little-endian header fields, and then an offset table that points to tagged chunks (see the figure on slide 2). Real formats (BMP, PNG, WAV, TTF) have this same structure.

The usual pattern for a fixed-size field is read_exact into a stack array, then from_le_bytes. Tutorial 18.1 shows the endian methods:

// Reads 4 bytes and decodes them as a little-endian u32.
fn read_u32_le<R: Read>(r: &mut R) -> io::Result<u32> {
    let mut buf = [0u8; 4];
    r.read_exact(&mut buf)?;        // UnexpectedEof if fewer than 4 bytes remain
    Ok(u32::from_le_bytes(buf))
}

// Header is a struct that the example defines, with the four fields below.
// read_u16_le is the same as read_u32_le, with a 2-byte buffer and u16.
fn parse_header<R: Read>(r: &mut R) -> io::Result<Header> {
    let mut magic = [0u8; 4];
    r.read_exact(&mut magic)?;
    if &magic != b"RIMG" {
        // The input is corrupt: return an error, do not panic.
        return Err(io::Error::new(io::ErrorKind::InvalidData, "bad magic"));
    }
    // Rust evaluates the field expressions in the order written, which is the file order.
    Ok(Header {
        version: read_u16_le(r)?,       // bytes 4 and 5
        chunk_count: read_u16_le(r)?,   // bytes 6 and 7
        width: read_u32_le(r)?,         // bytes 8 to 11
        height: read_u32_le(r)?,        // bytes 12 to 15
    })
}

Obey this rule: malformed input is data corruption, not a bug. A parser returns InvalidData or UnexpectedEof errors (tutorial 8.4). It never panics. You need no more code for truncated input, because read_exact already reports UnexpectedEof.

After the sequential parse of the header, the offset table makes random access possible. With Read + Seek, a Cursor can do what a plain slice reader cannot:

// Chunk is a struct that the example defines: a 4-byte tag and a u32 value.
fn read_chunk<S: Read + Seek>(s: &mut S, offset: u32) -> io::Result<Chunk> {
    s.seek(SeekFrom::Start(u64::from(offset)))?;   // go to the absolute offset of the chunk
    let mut tag = [0u8; 4];
    s.read_exact(&mut tag)?;
    Ok(Chunk { tag, value: read_u32_le(s)? })
}

// cur: a Cursor<&[u8]> on the RIMG bytes. offsets: Vec<u32> == [28, 36, 44], from the table.
let dpi_y = read_chunk(&mut cur, offsets[2])?;   // read the LAST chunk first

Writing Back: Patching Bytes In Place

A write uses the same offsets. Do these steps:

1. Copy the container into a Cursor<Vec<u8>>.
2. Seek to the absolute position of a field.
3. Write exactly as many bytes as the field has.
4. Parse the container again to verify the patch.

// RIMG is the 52-byte const array. HEIGHT_FIELD_OFFSET is 12.
let mut cur = Cursor::new(RIMG.to_vec());          // a mutable copy: Cursor<Vec<u8>>

cur.seek(SeekFrom::Start(HEIGHT_FIELD_OFFSET))?;   // byte 12
cur.write_all(&720u32.to_le_bytes())?;             // 480 -> 720

cur.seek(SeekFrom::Start(28 + 4))?;                // GAMA chunk at 28, value after the 4-byte tag
cur.write_all(&2400u32.to_le_bytes())?;            // 2200 -> 2400

cur.rewind()?;                                     // back to byte 0, to parse again
let header = parse_header(&mut cur)?;
assert_eq!(header.height, 720);                    // the patch is in the bytes
assert_eq!(header.width, 640);                     // the adjacent field did not change
assert_eq!(cur.get_ref().len(), RIMG.len());       // the length is the same: no append

All bytes of the input are in a const array, so each step of the cycle (parse, patch, parse again) is reproducible. 08_07_binary_header_parsing.rs prints:

parsed: 640x480 v1, chunks at [28, 36, 44], GAMA=2200 DPIY=300
rejected: bad magic = InvalidData, truncated = UnexpectedEof
patched: height=720 GAMA=2400 (re-parsed from the rewritten bytes)

All assertions passed.

Summary

Code Examples