3
\$\begingroup\$

I am trying to process a very large binary file using MappedByteBuffer from java.nio package. This is how the data looks like in the file:

[0, 12, 83, 0, 0, 0, 0, 9, -11, -66, -116, -91, 100, 79,
 39, 82, 0, 1, 0, 0, 10, 52, 126, -35, 45, -75, 65, 32, 32, 32, 32, 32, 32, 32, 78, 
 32,0,0, 0, 100, 78, 67, 90, 32, 80, 78, 32, 49, 78, 0, 0, 0, 0, 78....
 ....10 GB of more data]

We skip the first byte (0), the next byte(12) tells us how many bytes to process next. After processing 12 bytes, we see 39 and then we process next 39 bytes and so on. Here is the code that works but it's not the most efficient code. MappedByteBuffers are expensive to create and there is no way to explicitly release them, we let the GC reclaim them when ever it runs. I am creating 2 mappedbytebuffers, one to read the first byte and then another to read the number of bytes the first buffer told me to. I could read let's say 1024 bytes in the first buffer itself but I am not sure how to handle the case where there is not enough number of bytes left to read at the end of the buffer. I am open to using 3rd party libraries, I am already using bytes-java to process my messages later. I tried using Chronicle-Bytes but couldn't understand how to use it in my use case.

public void parse(String filename) throws IOException {
    try (RandomAccessFile file = new RandomAccessFile(new File(filename), "r")) {
        FileChannel inChannel = file.getChannel();

        //skip the first byte, start at 1
        long position = 1;

        long length = inChannel.size();

        //get the number of bytes to read by reading the first byte
        int bytesToRead = 1;

        while(position < length) {

            //first mapped buffer to read the number of bytes to read ahead
            MappedByteBuffer msgTypeBuffer = inChannel.map(FileChannel.MapMode.READ_ONLY, position, bytesToRead);

            bytesToRead = (int) (msgTypeBuffer.get()); //we have the number of bytes to read
            position++; //move cursor to the next position

            //second bytebuffer - reads bytesToRead
            MappedByteBuffer msgBuffer = inChannel.map(FileChannel.MapMode.READ_ONLY, position, bytesToRead);

            byte[] payBytes = new byte[bytesToRead];
            msgBuffer.get(payBytes, 0, bytesToRead);
            Message m = parsers.messageIn(payBytes, stock);
            if (!m.isEmpty()) {
                //process the message
            }

            //move the cursor after the bytes that have been read
            position += bytesToRead + 1;
        }
    }
}
\$\endgroup\$
3
  • 1
    \$\begingroup\$ Welcome to code review. Unfortunately this site is intended for reviewing working code only. If you need help in figuring out how to ensure the capacity of the MappedByteBuffer, then you should ask for help on stackoverflow.com. \$\endgroup\$ Apr 7 at 4:58
  • \$\begingroup\$ You've got mapped buffers downside up: Map the whole shebang once - you may even close the channel immediately after. If you weren't processing the data sequentially, you could use position(). \$\endgroup\$
    – greybeard
    Apr 7 at 7:20
  • \$\begingroup\$ Yeah, but using MappedByteBuffers smells like premature optimisation... \$\endgroup\$ Apr 7 at 8:21

1 Answer 1

6
\$\begingroup\$

You seem to have taken a very odd starting position here. The data file is in a simple stream format, so I really can't see why you'd bother with RandomAccessFiles and MappedByteBuffers - they add no value.

The processing can be boiled down to this (add error handling as needed) :

  1. Open a java.io.BufferedInputStream for the file
  2. Read first byte (perhaps check it's actually zero) and discard it
  3. Read a length byte
  4. Allocate an appropriately sized byte array
  5. Read into the array (different form of the read() method)
  6. Process the array
  7. Repeat from step 3 until finished (let GC deal with the arrays you don't need)

A sketch of the code would be :

public void parse(String filename) throws IOException {
  try (BufferedInputStream inStream = new BufferedInputStream(new FileInputStream(filename))) {

        //skip the first byte
    inStream.read();

    int bytesToRead = 0;
    while((bytesToRead  = inStream.read())!= -1) {
      byte[] payBytes = new byte[bytesToRead];
      // inStream.read(payBytes); 
      // May not read all bytes requested - prefer the loop below
    
      int offset = 0;
      while(bytesToRead > 0) {
        int bytesRead = inStream.read(payBytes, offset, bytesToRead);
        offset += bytesRead;
        bytesToRead -= bytesRead;
      }
      Message m = parsers.messageIn(payBytes, stock);
      if (!m.isEmpty()) {
          //process the message
      }
    }
  }
}
\$\endgroup\$
7
  • \$\begingroup\$ read(byte[]) does not guarantee that it will fully populate the byte array. It blocks until any data is available, and then reads whatever is available up to the size of the byte array argument. \$\endgroup\$
    – Eric Stein
    Apr 7 at 4:06
  • \$\begingroup\$ My code was a quick sketch, not a fully-worked solution, but yes there probably needs to be a loop on the read to the byte array. In the OP's use case, though, I think there would always be data available - they are reading a complete pre-existing file. \$\endgroup\$ Apr 7 at 5:08
  • \$\begingroup\$ I've updated the example in line with Eric's comment. \$\endgroup\$ Apr 7 at 8:21
  • \$\begingroup\$ Thanks to @greybeard for the edit to tidy up my markdown. \$\endgroup\$ Apr 7 at 11:14
  • 2
    \$\begingroup\$ @MarkBluemel As of Java 11 BufferedInputStream has byte [] readNBytes(int len) which will block until it reads len bytes or hits the end of stream. There is also int readNBytes(byte [] b, int offset, int len) which could easily be used repeatedly with a fixed size array of length 127 (or 256, but the code isn't handling Java's awkward signed bytes) to avoid generating garbage. \$\endgroup\$
    – swpalmer
    Apr 10 at 2:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.