1
\$\begingroup\$

I'm trying to write billions of strings lines to a file, it works for up to 40 million lines, but it's throwing out "java.nio.BufferOverflowException" error for 400 million lines. I also think my solution is slow as it takes 45 seconds to write 40 million lines. Below is the code.

public static void main(String[] args) throws FileNotFoundException, IOException {

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes();
int number_of_lines = 400000000;

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i < number_of_lines; i++) {
    wrBuf.put(buffer);
}
rwChannel.close();
}

How can I make it be faster if possible and also write more than 2 billion lines of string?

\$\endgroup\$

closed as off-topic by esote, yuri, Vogel612 May 10 at 10:01

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Code not implemented or not working as intended: Code Review is a community where programmers peer-review your working code to address issues such as security, maintainability, performance, and scalability. We require that the code be working correctly, to the best of the author's knowledge, before proceeding with a review." – esote, yuri, Vogel612
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    \$\begingroup\$ 40M lines * 46 chars = 1.7GB, but 400M lines = 17 GB, and size "must be non-negative and no greater than Integer.MAX_VALUE". Do you specifically want memory-mapped I/O, or could you just use RandomAccessFile.write? \$\endgroup\$ – Anonymous May 9 at 19:41
  • \$\begingroup\$ @Anonymous I just want to write 2 billion lines of string to a file very fast. \$\endgroup\$ – Hither Joe May 9 at 19:56
  • \$\begingroup\$ For very fast, can you do it in larger chunks than 46 bytes? \$\endgroup\$ – Anonymous May 9 at 20:01
  • \$\begingroup\$ How to go about it, please? \$\endgroup\$ – Hither Joe May 9 at 20:04
  • 2
    \$\begingroup\$ Can you assemble a buffer that contains many lines, and then write that buffer all at once? This may be faster than writing one line at a time. But this is off-topic for codereview. Try a StackOverflow question on this subject. ...why is your code nearly identical to one of the answers there? \$\endgroup\$ – Anonymous May 9 at 23:05
2
\$\begingroup\$

It sure sounds like you're running out of memory as you try to build (or allocate) the buffer.

  • You need to figure out how big a buffer you can (or want to) use. You'll need to ask the system how much RAM it has, and how much is available, and make your buffer that large.
  • You might get a small boost by opening the file as explicitly write-only.
  • You'll need to flush the buffer as you go. Something shaped like this psudo-code might work:
free = system.free_ram();
use = hungry * free;
buffer = new bytes_buffer_to_file("filename", size=use);
try{
  current_buffer = 0;
  while(line = get_bytes_needing_to_be_written()){
    if(current_buffer + line.length > use){
      buffer.flush();
      current_buffer = 0;
    }
    buffer.append(line);
    current_buffer += line.length;
  }
}
finally{
  buffer.flush();
  buffer.destroy();
}
\$\endgroup\$
  • \$\begingroup\$ I don't really understand your code. \$\endgroup\$ – Hither Joe May 9 at 20:38
  • \$\begingroup\$ If this was written in C or C++ the file block size would also be important here as well as the disc cache size. The speed can be running into hardware limitations. \$\endgroup\$ – pacmaninbw May 9 at 21:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.