The LineReader struct is a byte-delimiter-focused buffered reader meant as a faster, less error-prone alternative to BufRead::read_until.

It provides two main functions:

Returns Option<io::Result<&[u8]>> - None on end-of-file, an IO error from the wrapped reader, or an immutable byte slice ending on and including any delimiter.

Line length is limited to the size of the internal buffer.

In contrast with read_until, detecting end-of-file is more natural with the use of Option; line length is naturally limited to some sensible value without the use of by_ref().take(limit); copying is minimised by returning borrowed slices; you'll never forget to call buf.clear().

Behaves identically to next_line(), except it returns a slice of all the complete lines in the buffer.

extern crate linereader;
use linereader::LineReader;

let mut file = File::open(myfile).expect("open");

// Defaults to a 1 MiB buffer and b'\n' delimiter; change with one of:
//  * LineReader::with_capacity(usize);
//  * LineReader::with_delimiter(u8);
//  * LineReader::with_delimiter_and_capacity(u8, usize)
let mut reader = LineReader::new(file);

while let Some(line) = reader.next_line() {
    let line = line.expect("read error");
    // line is a &[u8] owned by reader.
}

Lines can also be read in batches for group processing - e.g. in threads:

while let Some(lines) = reader.next_batch() {
    send(&chan, lines.expect("read error").to_vec());
}

This should be more efficient than finding each intermediate delimiter in the main thread, and allocating and sending each individual line. Any line fragments at the end of the internal buffer will be copied to the start in the next call.

Tests performed using 'Dickens_Charles_Pickwick_Papers.xml', concatinated to itself 480 times. The resulting file is 976 MB and 10.3 million lines long.

Buffers in each test are set to 1 MiB.

It's also surprisingly fast on debug builds (or stdlib is surprisingly slow):