2
\$\begingroup\$

I have this simple Rust program, which creates a file called numbers.txt and writes the numbers 1 to N (N is here 10000000) to it, separated with newlines:

use std::fs::File;
use std::io::Write;

const N: i32 = 1e7 as i32;

fn main() {
    let mut data = String::new();

    for i in 1..=N  {
        data.push_str(&format!("{0}\n", i));
    }

    let mut f = File::create("numbers.txt").expect("Unable to create file");
    f.write_all(data.as_bytes()).expect("Unable to write data");
}

I've been running and timing the program with:

$ time cargo run --release

numbers.txt after running:

1
2
3
4
...
9999999
10000000

I've timed the execution of the program, and it takes around 4.8 seconds to run on my machine. I tried timing just the string building, by commenting out the last two lines, and the time taken stayed pretty much the same. We can conclude that the bottleneck is the data string building and not writing to the file.

How could I make building data faster? Should I use something else than a String?

\$\endgroup\$
  • \$\begingroup\$ You've run the test in debug mode, don't you? You must benchmark it in release mode. \$\endgroup\$ – Boiethios Sep 4 '19 at 13:37
  • \$\begingroup\$ No, I used cargo run --release to run the code. And actually $ time cargo run --release, when I needed to see the execution time. \$\endgroup\$ – ruohola Sep 4 '19 at 13:41
5
\$\begingroup\$

I improved the performances (and simplicity) like this:

use std::fs::File;
use std::io::Write;
use std::io::BufWriter;

const N: i32 = 1e7 as i32;

fn main() {
    let mut f = BufWriter::new(File::create("numbers.txt").expect("Unable to create file"));
    for i in 1..=N  {
        write!(f, "{0}\n", i);
    }

}

I just don't store the strings in a gigantic buffer but asks the BufWriter to manage the buffering. As a side effect, it's now possible to write bigger files without holding too much RAM.

\$\endgroup\$
  • \$\begingroup\$ Nice! This cut the runtime on my system from 4.8s to 1.4s! \$\endgroup\$ – ruohola Sep 4 '19 at 14:17
0
\$\begingroup\$

This cuts the runtime by about a third (4.8s -> 3.2s):

use std::fs::File;
use std::io::Write;

const N: i32 = 1e7 as i32;

fn main() {
    let data = (1..=N)
        .map(|n| n.to_string())
        .collect::<Vec<String>>()
        .join("\n");

    let mut f = File::create("numbers.txt").expect("Unable to create file");
    f.write_all(data.as_bytes()).expect("Unable to write data");
}
\$\endgroup\$
  • \$\begingroup\$ I'd like to see your benchmark: because that code is actually slower in the playground: play.integer32.com/… \$\endgroup\$ – Boiethios Sep 4 '19 at 13:53
  • \$\begingroup\$ @FrenchBoiethios Sure! I repeated this multiple times and always got practically identical results: i.imgur.com/gJq5Ugx.png \$\endgroup\$ – ruohola Sep 4 '19 at 14:05
0
\$\begingroup\$

How about some threads?

use rayon::prelude::*;
use std::fs::File;
use std::io::Write;

const N: i32 = 1e7 as i32;

fn main() {
    let mut output = File::create("numbers.txt").unwrap();

    let data: Vec<String> = (1..=N)
        .into_par_iter()
        .fold(String::new, |mut lhs, rhs| {
            use std::fmt::Write;
            write!(&mut lhs, "{}\n", rhs).unwrap();
            lhs
        })
        .collect();

    for part in data {
        output.write_all(&part.into_bytes()).unwrap();
    }
}
\$\endgroup\$
  • \$\begingroup\$ Did you benchmark this compared to my original or @DenysSéguret's answer? \$\endgroup\$ – ruohola Sep 6 '19 at 16:38
  • \$\begingroup\$ @ruohola, I did. On my machine mine is much faster. Of course, threads are involved so its going to be very different machine to machine. \$\endgroup\$ – Winston Ewert Sep 6 '19 at 18:00
  • \$\begingroup\$ Ok, would this be possible to be written without the rayon external dependency. \$\endgroup\$ – ruohola Sep 6 '19 at 18:01
  • 1
    \$\begingroup\$ @ruohola, you could, of course, write threading code directly using the stdlib instead of using rayon, but why would you want to? \$\endgroup\$ – Winston Ewert Sep 6 '19 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.