Writing billions of lines to text file in java [closed]

Question

I'm trying to write billions of strings lines to a file, it works for up to 40 million lines, but it's throwing out "java.nio.BufferOverflowException" error for 400 million lines. I also think my solution is slow as it takes 45 seconds to write 40 million lines. Below is the code.

public static void main(String[] args) throws FileNotFoundException, IOException {

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes();
int number_of_lines = 400000000;

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i < number_of_lines; i++) {
    wrBuf.put(buffer);
}
rwChannel.close();
}

How can I make it be faster if possible and also write more than 2 billion lines of string?

Answer 1

It sure sounds like you're running out of memory as you try to build (or allocate) the buffer.

• You need to figure out how big a buffer you can (or want to) use. You'll need to ask the system how much RAM it has, and how much is available, and make your buffer that large.
• You might get a small boost by opening the file as explicitly write-only.
• You'll need to flush the buffer as you go. Something shaped like this psudo-code might work:

free = system.free_ram();
use = hungry * free;
buffer = new bytes_buffer_to_file("filename", size=use);
try{
  current_buffer = 0;
  while(line = get_bytes_needing_to_be_written()){
    if(current_buffer + line.length > use){
      buffer.flush();
      current_buffer = 0;
    }
    buffer.append(line);
    current_buffer += line.length;
  }
}
finally{
  buffer.flush();
  buffer.destroy();
}