A fast line-oriented reader for Rust.

In my tests LineReader is 1.1x-1.6x faster than the typically-recommended fastest stdlib option: BufReader::read_until(). It achieves this by avoiding copying from its own internal buffer, instead returning immutable slices of its own.

Like read_until, it does not perform UTF-8 processing - you get a slice of raw u8's, including the delimiter, and nothing more.

Lines are limited to the size of the internal buffer (default 1MB).

extern crate linereader;
use linereader::LineReader;

// Note BufReader will result in unnecessary copying, so, er, don't do that.
let mut file = File::open(myfile).expect("open");

// or LineReader::with_capacity(usize);
// or LineReader::with_delimiter(u8);
// or LineReader::with_delimiter_and_capacity(u8, usize)
let reader = LineReader::new(file);

while let Some(line) = reader.next_line() {
    let line = line.expect("oh noes, an IO error");
    // line is a &[u8] owned by reader.
}

Lines can also be read in batches for group processing - e.g. in threads:

while let Some(lines) = reader.next_batch() {
    send(&chan, lines.unwrap().to_vec());
}

This should be more efficient than finding each intermediate delimiter in the main thread, and allocating and sending each individual line.

Comparison with using typical BufReader methods against pwned-passwords-2.0.txt:

Westmere Xeon 2.1GHz, FreeBSD/ZFS, 29GB, 501.6 million lines:

Haswell Xeon 3.4GHz, Windows 10 Subystem for Linux, 5.9GB, 100 million lines:

It's also surprisingly fast on debug builds (or stdlib is surprisingly slow):

Hmmm.