Pipeviewer: A small "pv" Rewrite

Question

If you ever had to pipe a large amount of data from some program foo into another program bar, you're probably familiar with the pipeviewer application. In case you're not familiar with the application, here's a small demonstration of a minimal rewrite:

Essentially, pv will take any input from stdin (or files) and forward it to stdout while showing progress on stderr. Note that the original pv has some more features that were non-goals of this project.

Cargo.toml (dependencies)

[dependencies]
anyhow = "1.0"
indicatif = "0.15.0"
structopt = "0.3"

main.rs

use anyhow::Result;
use indicatif::{ProgressBar, ProgressStyle};
use std::fs::File;
use std::io::{self, ErrorKind, Read};
use std::path::PathBuf;
use structopt::StructOpt;

#[derive(Debug, StructOpt)]
#[structopt(name = "pipeviewer", about = "A pipe inspecting application.")]
struct Opt {
    /// Input file, stdin if not specified
    #[structopt(parse(from_os_str))]
    input: Option<PathBuf>,
}

fn main() -> Result<()> {
    let opts = Opt::from_args();

    let (mut input, len): (Box<dyn Read>, Option<u64>) = if let Some(file) = opts.input {
        let file = File::open(file)?;
        let len = file.metadata()?.len();
        (Box::new(file), Some(len))
    } else {
        (Box::new(io::stdin()), None)
    };

    let pb = if let Some(len) = len {
        let pb = ProgressBar::new(len);
        pb.set_style(ProgressStyle::default_bar().template(
            "[{elapsed_precise}] {bar} {bytes_per_sec} [{bytes}/{total_bytes}] ETA: {eta}",
        ));
        pb
    } else {
        let pb = ProgressBar::new_spinner();
        pb.set_style(
            ProgressStyle::default_spinner()
                .template("[{elapsed_precise}] {spinner} {bytes_per_sec} [{bytes}]"),
        );
        pb
    };
    let mut output = pb.wrap_write(io::stdout());

    match io::copy(&mut input, &mut output) {
        Ok(_) => Ok(()),
        Err(e) if e.kind() == ErrorKind::BrokenPipe => Ok(()),
        Err(e) => Err(e.into()),
    }
}

The code is slightly dominated by the styling of the progress bar, unfortunately. However, I didn't want to show a meaningless progress bar on an unknown amount of data (e.g. foo | pv | bar), nor did I want to show no kind of progress if the amount of data is known (e.g. pv somefile).

I'm mostly interested about style issues, hit pitfalls and similar Rust sins. I'm aware that I can add some additional features (e.g. more files as arguments, size hints for STDIN, support for multiple pv's in a single pipe) and will happily take some inspiration from reviews, but I primarily want to improve my Rust skills :).

Some additional remarks

This section only provides some background and is completely optional to read :).

Motivation

This rewrite is inspired by the Udemy course Hands-On System Programming with Rust. However, I deviate from the original course by a lot, as it introduces multithreading (via crossbeam) and other (slightly) overkill features.

Goals of the lazy pipeviewer rewrite

• use already existing crates where possible (don't run into NIH)
• take either a single file name or use stdin as input
• flush everything into stdout (but ignore broken pipes)
• don't overengineer, e.g.
  • no multi-threading if single-core performance is good enough
  • no BufReader or BufWriter if the unbuffered variants seem fast enough
  • no explicit buffering via Read::read(&mut [u8]) if possible, KISS!

Dumbing it down

There was a point while writing this lazy code where a while let Ok(n) = input.read(&mut buf) was introduced, but given that io::copy seemed fast enough, I didn't bother using it in the final version. Same holds for BufReader and BufWriter, as they didn't improve the behaviour on my machines. Further benchmarks might be necessary, but that could more or less defeat the lazy part ;).

Getting to know the crates (for other projects)

This whole project is more or less meant as an exercise for a) searching for appropriate crates and b) using them in the correct manner. Argument parsing? Almost always necessary. Proper error handling? A must. Showing a nice progress bar on the way to completion? A big bonus!

Dependencies

For dependencies, I've opted for

• structopt to parse the single argument (I originally had more arguments)
• anyhow to provide error messages
• indicatif for a progress bar

Answer 1

The main sin (accidental pun) is putting it all in fn main. Prefer fancier but readable code over condensed code. This often boils down to adding a function for some assignments. Indeed, we can add a function for getting the content of let (mut input, len) and another for getting let pb.

A minor sin is failing if fn metadata fails. You only need metadata to get len, which is entirely optional.

I can't find any other sins. You are forgiven for your transgressions on the art of Rust.

• A bug: I get 0B/s on the use case that's in your post, instead of the number in the gif. Can't imagine why.
• I suggest printing a message with the number of bytes copied on exit, like dd does on linux, so that something happens even for tiny files. You could use fn abandon_with_message on pb.

That's all. I found the style ok in case we would be limited to one function. But we aren't, so I made an abstraction to minimize the complexity of fn main at the expense of adding 3 functions:

And last but not least, you could easily avoid dynamic dispatch there, but there's no preference for either type of dispatch in my opinion.

use anyhow::Result;
use indicatif::{ProgressBar, ProgressStyle};
use std::fs::File;
use std::io::{self, ErrorKind, Read};
use std::path::PathBuf;
use structopt::StructOpt;

#[derive(Debug, StructOpt)]
#[structopt(name = "pipeviewer", about = "A pipe inspecting application.")]
struct Opt {
    /// Input file, stdin if not specified
    #[structopt(parse(from_os_str))]
    input: Option<PathBuf>,
}

enum PvSource {
    File {
        file: File,
        size: Option<u64>,
    },
    Stdin,
}

impl PvSource {
    fn with_path(path: PathBuf) -> Result<Self> {
        let file = File::open(path)?;
        Ok(PvSource::File {
            size: file.metadata().ok().map(|metadata| metadata.len()),
            file,
        })
    }

    fn progress_bar(&self) -> ProgressBar {
        if let &PvSource::File { size: Some(len), .. } = self {
            let pb = ProgressBar::new(len);
            pb.set_style(ProgressStyle::default_bar().template(
                "[{elapsed_precise}] {bar} {bytes_per_sec} [{bytes}/{total_bytes}] ETA: {eta}",
            ));
            pb
        } else {
            let pb = ProgressBar::new_spinner();
            pb.set_style(
                ProgressStyle::default_spinner()
                    .template("[{elapsed_precise}] {spinner} {bytes_per_sec} [{bytes}]"),
            );
            pb
        }
    }

    fn readable<'a>(&'a mut self) -> Box<dyn Read + 'a> {
        if let &mut PvSource::File { ref mut file, .. } = self {
            Box::new(file)
        } else {
            Box::new(io::stdin())
        }
    }
}

fn main() -> Result<()> {
    let opts = Opt::from_args();
    let mut source = if let Some(input) = opts.input {
        PvSource::with_path(input)?
    } else {
        PvSource::Stdin
    };
    let progress_bar = source.progress_bar();
    let mut input = source.readable();
    let mut output = progress_bar.wrap_write(io::stdout());

    match io::copy(&mut input, &mut output) {
        Ok(_) => Ok(()),
        Err(e) if e.kind() == ErrorKind::BrokenPipe => Ok(()),
        Err(e) => Err(e.into()),
    }
}