6.7 · Barrier and Rendezvous Synchronization

Domain 6 — Concurrency and Parallelism Duration: ~15 minutes Library components: std::sync::Barrier, std::sync::BarrierWaitResult

Introduction

A lock controls which thread may access the data. A channel controls where the data goes. A Barrier does a third job: it makes sure that all the threads are ready. A barrier is a checkpoint for a fixed party of threads. Each call to wait blocks until the last member of the party arrives. Then the barrier releases all the threads together.

This tutorial shows:

• Barrier::new(n) and wait: the rendezvous where each thread waits for all the other threads.
• BarrierWaitResult::is_leader: how to select exactly one thread for the work that follows a rendezvous.
• Phased algorithms: how to use one barrier again in each round (map, barrier, reduce, barrier, repeat).
• Barrier compared with Condvar (6.2) and with channels (6.5): the same rendezvous in three versions, and when to use each one.

Barrier::wait: All Wait for All

Figure: Four Threads Meet at a Barrier

You create a Barrier for a fixed party size n. Each thread calls wait(). The first n - 1 calls block. The n-th call releases all the threads:

const WORKERS: usize = 4;

// The party size is fixed: WORKERS threads must call `wait`.
let barrier = Arc::new(Barrier::new(WORKERS));
// A shared counter of the workers that completed phase 1. It starts at 0.
let setups_done = Arc::new(AtomicUsize::new(0));

// Each worker thread gets a clone of the two Arcs and runs these lines:
setups_done.fetch_add(1, Ordering::Relaxed);                // phase 1: the setup
barrier.wait();                                             // returns when the 4th worker arrives
assert_eq!(setups_done.load(Ordering::Relaxed), WORKERS);   // phase 2: NEVER fails

That assertion shows the guarantee of a barrier. The OS can schedule the threads in any order, and each phase-1 write is still complete before a thread runs phase-2 code.

wait also synchronizes memory. All the operations of each thread before its wait happen-before all the operations of each thread after its wait. This is the Release/Acquire mechanism of Tutorial 6.4, and the barrier includes it.

A typical use is a load-test harness. Each worker must complete its setup before a worker starts to send requests. If not, the workers that start early measure a system that has no load.

is_leader: One Thread for the Follow-Up Work

wait returns a BarrierWaitResult. For each rendezvous, is_leader() returns true for exactly one of the threads that wait. That thread is the leader. The leader is not necessarily the first thread or the last thread that arrives:

// Each worker runs this code. `barrier` is the Barrier from the previous snippet.
let result = barrier.wait();   // result: BarrierWaitResult
if result.is_leader() {
    // Only one thread for each rendezvous runs this block.
    // Do the work that must occur one time for each rendezvous here:
    // aggregate the results, print the banner of the round, reset shared scratch state.
}

Some work must occur only one time for each rendezvous. is_leader selects the thread that does this work. You do not need to elect a coordinator thread, and the threads do not race on a flag. The example binary 06_22_barrier_rendezvous counts the leaders of one rendezvous of 4 threads. It asserts that the count is exactly 1.

Two notes about usage:

• A Barrier::new(1) releases immediately, and its only member is always the leader. This is useful in tests and in degenerate configurations.
• After the barrier releases a full party, it resets automatically. It can then serve the next rendezvous of the same size. You do not create a new barrier. Phased algorithms depend on this property.

Phased Algorithms: One Barrier, Many Rounds

Iterative parallel computations (simulation steps, iterative solvers, map-reduce rounds) repeat one sequence:

1. Compute in parallel.
2. Synchronize.
3. Combine the results.
4. Repeat.

The standard structure uses two waits per round:

// barrier: Barrier for WORKERS (4) threads.
// partials: Vec<AtomicUsize>, one slot for each worker.
// round_totals: Mutex<Vec<usize>>, one total for each round.
// Each worker runs this loop. worker_id is 0, 1, 2, or 3. ROUNDS is 5.
for round in 0..ROUNDS {
    // MAP: each worker writes only its own slot, so there is no contention.
    let contribution = (worker_id + 1) * (round + 1);
    partials[worker_id].store(contribution, Ordering::Relaxed);

    // Barrier 1: each slot holds the value of THIS round before a thread reduces.
    let map_done = barrier.wait();

    // REDUCE: only the leader reads all the slots and records the total.
    if map_done.is_leader() {
        let total: usize = partials.iter().map(|p| p.load(Ordering::Relaxed)).sum();
        round_totals.lock().unwrap().push(total);   // round 0 pushes 1 + 2 + 3 + 4 = 10
    }

    // Barrier 2: no worker starts round R+1 (and overwrites a slot)
    // while the leader still reads the slots.
    barrier.wait();
}

If you omit barrier 1, the leader can add partials of round R to partials of round R-1. If you omit barrier 2, a fast worker can overwrite a slot during the reduction. With the two barriers, the total of each round is deterministic. The example asserts [10, 20, 30, 40, 50] for the five rounds, in each run.

06_23_barrier_phases.rs prints:

per-round reduced totals: [10, 20, 30, 40, 50]
one Barrier, 10 rendezvous, zero re-allocation

All assertions passed.

The Same Rendezvous, Three Ways

The example binary 06_24_barrier_vs_channels solves one problem in three versions. The problem is: all the workers must complete phase A before a worker starts phase B.

General rules:

• Barrier: use it for N symmetric peers that all wait for each other, possibly in repeated rounds. If you plan to make a "countdown latch" from a Condvar, a barrier is probably the latch that you need.
• Channels (6.5): use them when data moves between threads, or when the roles are asymmetric (coordinator and worker, producer and consumer). A sync_channel(0) rendezvous is the special case for two parties.
• Condvar (6.2): use it when the release condition is a custom predicate that no standard primitive expresses ("queue non-empty and not paused").

06_24_barrier_vs_channels.rs prints:

barrier version: ok (1 sync object, 1 line per worker)
channel version: ok (N+1 channels + a coordinator loop)
condvar latch version: ok (manual counting + wakeup logic)

All assertions passed.

The Thread Count: The One Way to Misuse a Barrier

The party size is a contract, not a hint. If fewer than n threads call wait, each thread that called wait blocks forever. This is a barrier deadlock: there is no error, no timeout, and no poisoning. In practice, this deadlock has three causes:

1. Two numbers that do not agree. The barrier is Barrier::new(4), but an edit changed the spawn loop to 0..5. The wait of the fifth thread starts a new rendezvous that never completes. Derive the two numbers from one const WORKERS.
2. A worker that exits early. A ? or return path does not reach the wait. Structure each worker so that each exit path reaches the barrier or stops the full party.
3. A worker that panics. A Mutex poisons, but a Barrier does not. The thread that panicked never arrives, and the other threads wait forever. If a worker can panic in the middle of a phase, a protocol that uses channels handles the failure better. With channels, the other threads can detect a dropped Sender (Tutorial 6.5).

The examples in this domain prevent the three causes, and production code should do the same:

• They use one shared constant.
• Each worker has only one path, and that path goes directly through the wait.
• They join the threads at the end, so the parent thread sees a panic.

Summary

Code Examples