1
\$\begingroup\$

I'm trying to write billions of strings lines to a file, it works for up to 40 million lines, but it's throwing out "java.nio.BufferOverflowException" error for 400 million lines. I also think my solution is slow as it takes 45 seconds to write 40 million lines. Below is the code.

public static void main(String[] args) throws FileNotFoundException, IOException {

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes();
int number_of_lines = 400000000;

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i < number_of_lines; i++) {
    wrBuf.put(buffer);
}
rwChannel.close();
}

How can I make it be faster if possible and also write more than 2 billion lines of string?

\$\endgroup\$
7
  • 2
    \$\begingroup\$ 40M lines * 46 chars = 1.7GB, but 400M lines = 17 GB, and size "must be non-negative and no greater than Integer.MAX_VALUE". Do you specifically want memory-mapped I/O, or could you just use RandomAccessFile.write? \$\endgroup\$
    – Anonymous
    May 9 '19 at 19:41
  • \$\begingroup\$ @Anonymous I just want to write 2 billion lines of string to a file very fast. \$\endgroup\$
    – Hither Joe
    May 9 '19 at 19:56
  • \$\begingroup\$ For very fast, can you do it in larger chunks than 46 bytes? \$\endgroup\$
    – Anonymous
    May 9 '19 at 20:01
  • \$\begingroup\$ How to go about it, please? \$\endgroup\$
    – Hither Joe
    May 9 '19 at 20:04
  • 2
    \$\begingroup\$ Can you assemble a buffer that contains many lines, and then write that buffer all at once? This may be faster than writing one line at a time. But this is off-topic for codereview. Try a StackOverflow question on this subject. ...why is your code nearly identical to one of the answers there? \$\endgroup\$
    – Anonymous
    May 9 '19 at 23:05
2
\$\begingroup\$

It sure sounds like you're running out of memory as you try to build (or allocate) the buffer.

  • You need to figure out how big a buffer you can (or want to) use. You'll need to ask the system how much RAM it has, and how much is available, and make your buffer that large.
  • You might get a small boost by opening the file as explicitly write-only.
  • You'll need to flush the buffer as you go. Something shaped like this psudo-code might work:
free = system.free_ram();
use = hungry * free;
buffer = new bytes_buffer_to_file("filename", size=use);
try{
  current_buffer = 0;
  while(line = get_bytes_needing_to_be_written()){
    if(current_buffer + line.length > use){
      buffer.flush();
      current_buffer = 0;
    }
    buffer.append(line);
    current_buffer += line.length;
  }
}
finally{
  buffer.flush();
  buffer.destroy();
}
\$\endgroup\$
2
  • \$\begingroup\$ I don't really understand your code. \$\endgroup\$
    – Hither Joe
    May 9 '19 at 20:38
  • \$\begingroup\$ If this was written in C or C++ the file block size would also be important here as well as the disc cache size. The speed can be running into hardware limitations. \$\endgroup\$
    – pacmaninbw
    May 9 '19 at 21:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.