If you ever had to pipe a large amount of data from some program foo
into another program bar
, you're probably familiar with the pipeviewer application. In case you're not familiar with the application, here's a small demonstration of a minimal rewrite:
Essentially, pv
will take any input from stdin
(or files) and forward it to stdout
while showing progress on stderr
. Note that the original pv
has some more features that were non-goals of this project.
Cargo.toml (dependencies)
[dependencies]
anyhow = "1.0"
indicatif = "0.15.0"
structopt = "0.3"
main.rs
use anyhow::Result;
use indicatif::{ProgressBar, ProgressStyle};
use std::fs::File;
use std::io::{self, ErrorKind, Read};
use std::path::PathBuf;
use structopt::StructOpt;
#[derive(Debug, StructOpt)]
#[structopt(name = "pipeviewer", about = "A pipe inspecting application.")]
struct Opt {
/// Input file, stdin if not specified
#[structopt(parse(from_os_str))]
input: Option<PathBuf>,
}
fn main() -> Result<()> {
let opts = Opt::from_args();
let (mut input, len): (Box<dyn Read>, Option<u64>) = if let Some(file) = opts.input {
let file = File::open(file)?;
let len = file.metadata()?.len();
(Box::new(file), Some(len))
} else {
(Box::new(io::stdin()), None)
};
let pb = if let Some(len) = len {
let pb = ProgressBar::new(len);
pb.set_style(ProgressStyle::default_bar().template(
"[{elapsed_precise}] {bar} {bytes_per_sec} [{bytes}/{total_bytes}] ETA: {eta}",
));
pb
} else {
let pb = ProgressBar::new_spinner();
pb.set_style(
ProgressStyle::default_spinner()
.template("[{elapsed_precise}] {spinner} {bytes_per_sec} [{bytes}]"),
);
pb
};
let mut output = pb.wrap_write(io::stdout());
match io::copy(&mut input, &mut output) {
Ok(_) => Ok(()),
Err(e) if e.kind() == ErrorKind::BrokenPipe => Ok(()),
Err(e) => Err(e.into()),
}
}
The code is slightly dominated by the styling of the progress bar, unfortunately. However, I didn't want to show a meaningless progress bar on an unknown amount of data (e.g. foo | pv | bar
), nor did I want to show no kind of progress if the amount of data is known (e.g. pv somefile
).
I'm mostly interested about style issues, hit pitfalls and similar Rust sins. I'm aware that I can add some additional features (e.g. more files as arguments, size hints for STDIN, support for multiple pv
's in a single pipe) and will happily take some inspiration from reviews, but I primarily want to improve my Rust skills :).
Some additional remarks
This section only provides some background and is completely optional to read :).
Motivation
This rewrite is inspired by the Udemy course Hands-On System Programming with Rust. However, I deviate from the original course by a lot, as it introduces multithreading (via crossbeam
) and other (slightly) overkill features.
Goals of the lazy pipeviewer rewrite
- use already existing crates where possible (don't run into NIH)
- take either a single file name or use stdin as input
- flush everything into stdout (but ignore broken pipes)
- don't overengineer, e.g.
- no multi-threading if single-core performance is good enough
- no
BufReader
orBufWriter
if the unbuffered variants seem fast enough - no explicit buffering via
Read::read(&mut [u8])
if possible, KISS!
Dumbing it down
There was a point while writing this lazy code where a while let Ok(n) = input.read(&mut buf)
was introduced, but given that io::copy
seemed fast enough, I didn't bother using it in the final version. Same holds for BufReader
and BufWriter
, as they didn't improve the behaviour on my machines. Further benchmarks might be necessary, but that could more or less defeat the lazy part ;).
Getting to know the crates (for other projects)
This whole project is more or less meant as an exercise for a) searching for appropriate crates and b) using them in the correct manner. Argument parsing? Almost always necessary. Proper error handling? A must. Showing a nice progress bar on the way to completion? A big bonus!
Dependencies
For dependencies, I've opted for
pv
application manages up to 8GB/s onpv < /dev/zero > /dev/null
. NeitherBufReader
norBufWriter
change that behaviour. There might be something amiss inindicatif
but I wasn't able to profile the program yet. \$\endgroup\$pv
'swrite
call is faster on/dev/null
for unknown reasons. On actual in-pipeline usage,pv
yields ~1.2GB/s, which is roughly in the ballpark of the small demo application (~900MB/s). \$\endgroup\$