Processing a binary file with buffer length tags

Question

I am trying to process a very large binary file using MappedByteBuffer from java.nio package. This is how the data looks like in the file:

[0, 12, 83, 0, 0, 0, 0, 9, -11, -66, -116, -91, 100, 79,
 39, 82, 0, 1, 0, 0, 10, 52, 126, -35, 45, -75, 65, 32, 32, 32, 32, 32, 32, 32, 78, 
 32,0,0, 0, 100, 78, 67, 90, 32, 80, 78, 32, 49, 78, 0, 0, 0, 0, 78....
 ....10 GB of more data]

We skip the first byte (0), the next byte(12) tells us how many bytes to process next. After processing 12 bytes, we see 39 and then we process next 39 bytes and so on. Here is the code that works but it's not the most efficient code. MappedByteBuffers are expensive to create and there is no way to explicitly release them, we let the GC reclaim them when ever it runs. I am creating 2 mappedbytebuffers, one to read the first byte and then another to read the number of bytes the first buffer told me to. I could read let's say 1024 bytes in the first buffer itself but I am not sure how to handle the case where there is not enough number of bytes left to read at the end of the buffer. I am open to using 3rd party libraries, I am already using bytes-java to process my messages later. I tried using Chronicle-Bytes but couldn't understand how to use it in my use case.

public void parse(String filename) throws IOException {
    try (RandomAccessFile file = new RandomAccessFile(new File(filename), "r")) {
        FileChannel inChannel = file.getChannel();

        //skip the first byte, start at 1
        long position = 1;

        long length = inChannel.size();

        //get the number of bytes to read by reading the first byte
        int bytesToRead = 1;

        while(position < length) {

            //first mapped buffer to read the number of bytes to read ahead
            MappedByteBuffer msgTypeBuffer = inChannel.map(FileChannel.MapMode.READ_ONLY, position, bytesToRead);

            bytesToRead = (int) (msgTypeBuffer.get()); //we have the number of bytes to read
            position++; //move cursor to the next position

            //second bytebuffer - reads bytesToRead
            MappedByteBuffer msgBuffer = inChannel.map(FileChannel.MapMode.READ_ONLY, position, bytesToRead);

            byte[] payBytes = new byte[bytesToRead];
            msgBuffer.get(payBytes, 0, bytesToRead);
            Message m = parsers.messageIn(payBytes, stock);
            if (!m.isEmpty()) {
                //process the message
            }

            //move the cursor after the bytes that have been read
            position += bytesToRead + 1;
        }
    }
}

Answer 1

You seem to have taken a very odd starting position here. The data file is in a simple stream format, so I really can't see why you'd bother with RandomAccessFiles and MappedByteBuffers - they add no value.

The processing can be boiled down to this (add error handling as needed) :

1. Open a java.io.BufferedInputStream for the file
2. Read first byte (perhaps check it's actually zero) and discard it
3. Read a length byte
4. Allocate an appropriately sized byte array
5. Read into the array (different form of the read() method)
6. Process the array
7. Repeat from step 3 until finished (let GC deal with the arrays you don't need)

A sketch of the code would be :

public void parse(String filename) throws IOException {
  try (BufferedInputStream inStream = new BufferedInputStream(new FileInputStream(filename))) {

        //skip the first byte
    inStream.read();

    int bytesToRead = 0;
    while((bytesToRead  = inStream.read())!= -1) {
      byte[] payBytes = new byte[bytesToRead];
      // inStream.read(payBytes); 
      // May not read all bytes requested - prefer the loop below
    
      int offset = 0;
      while(bytesToRead > 0) {
        int bytesRead = inStream.read(payBytes, offset, bytesToRead);
        offset += bytesRead;
        bytesToRead -= bytesRead;
      }
      Message m = parsers.messageIn(payBytes, stock);
      if (!m.isEmpty()) {
          //process the message
      }
    }
  }
}