Rust Iterators Primer

Rust for-loop implemented with iterators.

Another Take on the Classic grep Program

The code discussed in depth in this post comes from ch09_grepr chapter of the study_rust_cmdline project.

Youens-Clark’s book Command-Line Rust has an interesting implementation of the UNIX grep utility. From a high-level point of view, the program, called grepr, iterates recursively over a specified directory, inspecting text files against a regular expression pattern. It gathers matching lines and, once it’s completed its work, grepr reports the files and the specific lines filtered according to the regex pattern we supply.

One function in this Rust project, find_lines() processes a file’s contents against the regular expression pattern. It accepts a boolean flag that inverts the match logic: either filter for a match; or filter for lines that do not match the pattern. When all goes well, the function returns a collection of filtered lines. In case an error happens, the function discards the lines it’s gathered and returns the error.

Here is Youens-Clark’s original code for find_lines() taken from the book’s Github repo:

fn find_lines<T: BufRead>(
    mut file: T,
    pattern: &Regex,
    invert: bool,
) -> Result<Vec<String>> {
    let mut matches = vec![];
    let mut line = String::new();

    loop {
        let bytes = file.read_line(&mut line)?;
        if bytes == 0 {
            break;
        }
        if pattern.is_match(&line) ^ invert {
            matches.push(mem::take(&mut line));
        }
        line.clear();
    }

    Ok(matches)
}

Rust is famous for its iterator feature. Shown below is an alternate implementation of find_lines() that uses a handful of adaptor methods strung together in an iterator chains pattern.

fn find_lines<T: BufRead>(file: T, pattern: &Regex, invert: bool) -> Result<Vec<String>> {
    file.lines()
        // Convert errors into Anyhow and pass down the pipeline
        .map(|line| line.map_err(|e| anyhow!("{}", e)))
        .filter(|line_res| match line_res {
                // XOR is true only when one of two conditions is true
                Ok(line_string) => pattern.is_match(&line_string) ^ invert,
                // Capture errors in final result
                Err(_) => true
            })
        // Gather iterator into the Result of Vec 
        .collect()
}    

The flow of the logic here is compact and terse. Let’s take a look at each stage in the iterator chain.

file.lines()

The input file is of type BufRead, meaning the operating system manages a memory buffer for reading data, handing it over in tranches to the line() operator. Behind the scenes, the data stream could be sourced from a file on a disk, a pipe connected to the terminal stdin, or a network socket. In any case, the input is noisy: an error may arise as the line() reads batches of the data stream.

Assuming a text file input, file.lines() returns a stream of Result<String, std::io::Error>. That is if the buffered read operation goes well, it passes a String to the next stage of the iterator chain; otherwise, it passes along an std::io::Error.

map()

This is the first adaptor. Here the closure inspects the Result<String, std::io::Error>. If an std::io::Error happens to come along, the closure transforms this into a anyhow::Error type. In any case, map() passes a Result<String, anyhow::Error> down the iterator chain.

filter()

1. This adaptor takes the output of map() and acts as a gating function. The closure operates on boolean logic, filtering out inputs from map() when the closure evaluates to false; otherwise, when it evaluates to true, the closure passes its input along to the next stage in the iterator chain.

We know that map() hands over a Result<String, anyhow::Error> to filter(). In its closure, the match unpacks the Result and applies the closure sieve:

1. If it happens to be anyhow::Error, the closure evaluates to true and the Result is passed on.

2. In case the match encounters a String, it runs this line of text against the XOR logic pattern.is_match(&line_string) ^ invert. When this piece of logic evalutes to true, the Result passes the filter; otherwise, the Result is discarded and filter() processes the next Result<String, anyhow::Error> that map() passes to it.

collect()

Rust iterators are lazy in that an operator such as collect() acts as the engine that drives the elements of the iterator chain. One way of thinking of this is that all the adaptors preceding collect() are idle until draws in its next input.

Because the function find_lines() returns a Result<Vec<String>>, the last operation in the iterator chain, collect() is responsible to assemble the function output. Implicit in that return signature is that when things work out, find_lines() hands an OK(Vec<String>) back to its caller. Otherwise, find_lines() returns an Error type.

Going back to the conceptual view of collect() as the engine of the iterator chain, collect() manages a Vec<String> that accumulates pieces of text from the input data stream. The preceding filter() adaptor hands down a Result<String, anyhow::Error>.

1. Absent any Error, collect() unpacks the String and pushes it onto the Vec<String> it maintains. Once all the lines have been read from the input data stream, collect() bundles this vector in an Ok return value, and its job is done.

2. At any time, should collect() encounter an anyhow::Error from filter(), it aborts its operation and discards the Vec<String> it maintains. This event quickly leads to the end of find_lines(), and the function returns with an Err value that holds information about the anyhow::Error.

As a result, collect() takes a stream of Result<String, anyhow::Error> and produces a single Result<Vec<String>, anyhow::Error>, flipping the data structure.

All together the error handling in find_lines() represents an idiomatic design pattern in Rust. The iterator chain passes a Result along its adaptors, each one ignoring any Error state. Effectively, error handling is postponed until the collect(), which aborts the iterator chain, ends find_lines() and returns an Error to the calling function.

Is it possible to handle the error within an iterator chain? Yes, by means of the ? operator which halts execution and returns control to the enclosing scope. However, the ? would have to be called within a closure. Aborting due to an error would exit the closure but not out of the find_lines(), the enclosing function.

There are a number of iterator adaptors that short-circuit the chain when an error arises. Among these are try_fold() or try_for_each(). Rewriting the iterator chain as a standard for loop is another way. Any ? enclosed within the loop aborts the enclosing function immediately. Unlike find_lines() where we want any errors acknowledged and noted down in the final output, these alternatives are well suited for fallible operations.