# Writing billions of lines to text file in java [closed]

I'm trying to write billions of strings lines to a file, it works for up to 40 million lines, but it's throwing out "java.nio.BufferOverflowException" error for 400 million lines. I also think my solution is slow as it takes 45 seconds to write 40 million lines. Below is the code.

public static void main(String[] args) throws FileNotFoundException, IOException {

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes();
int number_of_lines = 400000000;

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines);
for (int i = 0; i < number_of_lines; i++) {
wrBuf.put(buffer);
}
rwChannel.close();
}


How can I make it be faster if possible and also write more than 2 billion lines of string?

• 40M lines * 46 chars = 1.7GB, but 400M lines = 17 GB, and size "must be non-negative and no greater than Integer.MAX_VALUE". Do you specifically want memory-mapped I/O, or could you just use RandomAccessFile.write? May 9, 2019 at 19:41
• @Anonymous I just want to write 2 billion lines of string to a file very fast. May 9, 2019 at 19:56
• For very fast, can you do it in larger chunks than 46 bytes? May 9, 2019 at 20:01
• How to go about it, please? May 9, 2019 at 20:04
• Can you assemble a buffer that contains many lines, and then write that buffer all at once? This may be faster than writing one line at a time. But this is off-topic for codereview. Try a StackOverflow question on this subject. ...why is your code nearly identical to one of the answers there? May 9, 2019 at 23:05

## 1 Answer

It sure sounds like you're running out of memory as you try to build (or allocate) the buffer.

• You need to figure out how big a buffer you can (or want to) use. You'll need to ask the system how much RAM it has, and how much is available, and make your buffer that large.
• You might get a small boost by opening the file as explicitly write-only.
• You'll need to flush the buffer as you go. Something shaped like this psudo-code might work:
free = system.free_ram();
use = hungry * free;
buffer = new bytes_buffer_to_file("filename", size=use);
try{
current_buffer = 0;
while(line = get_bytes_needing_to_be_written()){
if(current_buffer + line.length > use){
buffer.flush();
current_buffer = 0;
}
buffer.append(line);
current_buffer += line.length;
}
}
finally{
buffer.flush();
buffer.destroy();
}

• I don't really understand your code. May 9, 2019 at 20:38
• If this was written in C or C++ the file block size would also be important here as well as the disc cache size. The speed can be running into hardware limitations. May 9, 2019 at 21:33