-3
\$\begingroup\$

A Java program that writes each digit on a new line, from 1 to max integer. We should end up with 2.4 Billion lines. After looping to near the maximum integer, the text file becomes really huge. Like 55gb after running for 16 minutes. How can I optimize it to be faster, and also reduce the file size?

Below is the code

import java.io.BufferedWriter;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.UnsupportedEncodingException;
import java.io.Writer;

/**
 *
 * @author kingamada
 */
public class reElegant {

    /**
     * @param args the command line arguments
     * @throws java.io.UnsupportedEncodingException
     */

    public static void main(String[] args) throws UnsupportedEncodingException, IOException {
        // TODO code application logic here
        try (Writer writer = new BufferedWriter(new OutputStreamWriter(
                new FileOutputStream("filename.txt"), "utf-8"))) {
            for (int i = 0; i < Integer.MAX_VALUE - 1; i++) {
                writer.write(i + "   |    " + i*2 + "\n");

            }
        }
    }

}
\$\endgroup\$

closed as unclear what you're asking by 200_success, AJNeufeld, Martin R, Ludisposed, Mast Mar 19 at 12:08

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    \$\begingroup\$ We can't change the file size without changing what the code does. But you haven't specified what the code must achieve. So, how can we possibly help you? \$\endgroup\$ – 200_success Mar 19 at 4:13
  • \$\begingroup\$ @200_success I'm just writing a program that writes new digit on each line, max line of the max integer. I hope that clarifies the question. \$\endgroup\$ – king amada Mar 19 at 4:36
  • 1
    \$\begingroup\$ @kingamada A) Your code doesn't go up to Integer.MAX_VALUE, but "only" Integer.MAX_VALUE - 2. B) You are multiplying by 2, so half way through you are overflowing the integer. C) Finally why do you think a 55 GB file is too big? I roughly estimate that size is to be expected. \$\endgroup\$ – RoToRa Mar 19 at 8:29
  • 1
    \$\begingroup\$ You can't expect a certain output and want the output to be smaller. That's not how it works. \$\endgroup\$ – Mast Mar 19 at 12:08
  • \$\begingroup\$ What is your newline \n setting? Replacing CRLF with LF will save one byte per line. \$\endgroup\$ – rossum Mar 19 at 12:23
2
\$\begingroup\$

I have no idea why you need this text file. Making it faster should be possible though.

Your current code allocates a StringBuilder and a String for every line. You can avoid this by allocating a single StringBuilder and modifying it using append and setSize. This will reduce the time spent in the garbage collector.

Printing an integer involves splitting it into digits, and that involves integer division, one of the slowest opcodes a processor provides. In your case you can avoid integer division by defining a class called StrInt that stores the integer as a character array and as the number itself. You would need to define methods like add(n) and appendTo(StringBuilder) and take care of the wraparound into the negative numbers.

Using these two techniques will make your code much longer (especially the unit test for the StrInt class should be large) and probably also quite fast.

Writing 55 GB in 1000 seconds means the write rate is already 55 MB/s, which is not the worst.

You should also write to a byte stream instead of a character stream to avoid the unnecessary conversion. Simply changing the encoding from UTF-8 to US-ASCII may already speed it up measurably.

\$\endgroup\$
  • 1
    \$\begingroup\$ On my system, using a StringBuilder in the manner described in your first paragraph only slows the program down. I am a little surprised how efficient plain old string addition is in this context. Your second idea is interesting, though I have not implemented it. \$\endgroup\$ – Benjamin Kuykendall Mar 19 at 6:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.