Efficiently packing the header and data in one Byte Array in a single class

Question

I have a header and data which I need to represent in one Byte Array. And I have a particular format for packing the header in a Byte Array and also a different format to pack the data in a Byte Array. After I have these two, I need to make one final Byte Array out of it.

// below is my header offsets layout

// m_off_addressed_center must be the first byte
static constexpr uint32_t  m_off_addressed_center           = 0;
static constexpr uint32_t  m_off_record_version             = m_off_addressed_center + 1;
static constexpr uint32_t  m_off_num_records                = m_off_record_version + 1;
static constexpr uint32_t  m_off_buffer_used                = m_off_num_records + sizeof(uint32_t);
static constexpr uint32_t  m_off_address                    = m_off_buffer_used + sizeof(uint32_t);
static constexpr uint32_t  m_off_address_from               = m_off_address + sizeof(CustomerAddress);
static constexpr uint32_t  m_off_records_partition          = m_off_address_from + sizeof(CustomerAddress);
static constexpr uint32_t  m_off_already_replicated         = m_off_records_partition + 1;

// this is the full size of the header
static constexpr uint32_t m_head_offset = m_off_already_replicated + 1;

And CustomerAddress is a typedef for uint64_t and it is made up like this -

typedef uint64_t   CustomerAddress;

void client_data(uint8_t datacenter, 
                  uint16_t client_id, 
                  uint8_t data_id, 
                  uint32_t data_counter,
                  CustomerAddress& customer_address)
{
    customer_address = (uint64_t(datacenter) << 56)
                    + (uint64_t(client_id) << 40)
                    + (uint64_t(data_id) << 32)
                    + data_counter;
}

And below is my data layout -

// below is my data layout -
//
// key type - 1 byte
// key len - 1 byte
// key (variable size = key_len)
// timestamp (sizeof uint64_t)
// data size (sizeof uint16_t)
// data (variable size = data size)

Now I am trying to represent overall stuff in one particular class in Java so that I can just pass the necessary fields and it can make me a final Byte Array out of it which will have the header first and then the data:

public static void main(String[] args) throws IOException {

    // header layout
    byte addressed_center = 0;
    byte record_version = 1;

    // should packCustomerAddress method be in DataFrame class? 
    // Or we should remove it from there and put it somewhere else?
    long address = DataFrame.packCustomerAddress((byte) 12, (short) 13, (byte) 32, (int) 120);
    long address_from = DataFrame.packCustomerAddress((byte) 21, (short) 23, (byte) 41, (int) 130);

    byte records_partition = 3;
    byte already_replicated = 0;

    // this map will have key as the actual key and value as the actual data, both in byte array
    Map<byte[], byte[]> keyDataHolder = new HashMap<byte[], byte[]>();
    for (int i = 1; i <= 2; i++) {
        keyDataHolder.put(generateKey(), getMyData());
    }

    // this class will make me a final byte array which will have header first and then the data layout
    DataFrame records = new DataFrame(addressed_center, record_version, keyDataHolder, address, address_from,
            records_partition, already_replicated);

    // this will give me final packed byte array
    // which will have header and data in it.
    records.getPackedByteArray();
}

DataFrame class:

public class DataFrame {

    private static final int headerSize = 28; // header offsets

    private static ByteBuffer byteBuffer = ByteBuffer.allocate(64000);

    // this is the final packed byte array which has header and data
    private byte[] packedByteArray;

    /**
     * constructor to pass around the values
     *
     */
    public DataFrame(byte addressed_center, byte record_version, Map<byte[], byte[]> keyDataHolder, long address,
            long address_from, byte records_partition, byte already_replicated) {
        serializeIntoFinalByteArray(addressed_center, record_version, keyDataHolder, address, address_from,
                    records_partition, already_replicated);
    }

    protected void serializeIntoFinalByteArray(byte addressed_center, byte record_version,
            Map<byte[], byte[]> keyDataHolder, long address, long address_from, long address_origin,
            byte records_partition, byte already_replicated) {

        byteBuffer.order(ByteOrder.BIG_ENDIAN);

        // how many records we are packing? In this example, we are packing two records
        int numOfRecords = keyDataHolder.size();
        int bufferUsed = getBufferUsed(keyDataHolder); //28 + dataSize + 1 + 1 + keyLength + 8 + 2;

        // header layout
        byteBuffer.put(addressed_center); // byte
        byteBuffer.put(record_version); // byte
        byteBuffer.putInt(numOfRecords); // int
        byteBuffer.putInt(bufferUsed); // int
        byteBuffer.putLong(address); // long
        byteBuffer.putLong(address_from); // long
        byteBuffer.put(records_partition); // byte
        byteBuffer.put(already_replicated); // byte

        // now the data layout
        for (Map.Entry<byte[], byte[]> entry : keyDataHolder.entrySet()) {
            byte keyType = 0;
            byte keyLength = (byte) entry.getKey().length;
            byte[] key = entry.getKey();
            long timestamp = System.currentTimeMillis();

            byte[] data = entry.getValue();
            short dataSize = (short) data.length;

            byteBuffer.put(keyType);
            byteBuffer.put(keyLength);
            byteBuffer.put(key);
            byteBuffer.putLong(timestamp);
            byteBuffer.putShort(dataSize);
            byteBuffer.put(data);
        }

        packedByteArray = byteBuffer.array();
    }

    // this will return final byte array
    public byte[] getPackedByteArray() {
        return packedByteArray;
    }

    private int getBufferUsed(Map<byte[], byte[]> keyDataHolder) {
        int size = headerSize;
        for (Map.Entry<byte[], byte[]> entry : keyDataHolder.entrySet()) {
            size += 1 + 1 + 8 + 2;
            size += entry.getKey().length;
            size += entry.getValue().length;
        }

        return size;
    }

    private static long packCustomerAddress(byte datacenter, short client_id, byte data_id, int data_counter) {
        return ((long) (datacenter) << 56) | ((long) client_id << 40) | ((long) data_id << 32) | ((long) data_counter);
    }       
}

Is there any better way of representing the above DataFrame class, since I need to make one class which can pack me headers and data in one final byte array?

How do I pack my header and data in one byte array efficiently using ByteBuffer by following the above layout format in DataFrame class? As of now, I am able to pack the header and data in one Byte Array but not sure whether this is the right way to represent it.

Answer 1

The tralnslation layer between C/C++ data formats, and Java formats, are always a problem. You have the right idea with using a ByteBuffer. The alternative without NIO, is using a DataOutputStream.

Your actual implementation is very C like though, and lacks the object nature I would expect from a Java program. I would expect the data frame to contain the data in it's component form, and not in the 'packed' form. The packing would be an on-the-fly thing, and would remove the need for the buffer. You would pack the data as you write it to the stream.

Note that both ByteBuffer and DataOutputStream by default set the format to be Big-Endian, which is also called 'portable', or 'network byte' order. There is no need to re-set it.

Now, the guts of your method is:

    byteBuffer.order(ByteOrder.BIG_ENDIAN);

    // how many records we are packing? In this example, we are packing two records
    int numOfRecords = keyDataHolder.size();
    int bufferUsed = getBufferUsed(keyDataHolder); //28 + dataSize + 1 + 1 + keyLength + 8 + 2;

    // header layout
    byteBuffer.put(addressed_center); // byte
    byteBuffer.put(record_version); // byte
    byteBuffer.putInt(numOfRecords); // int
    byteBuffer.putInt(bufferUsed); // int
    byteBuffer.putLong(address); // long
    byteBuffer.putLong(address_from); // long
    byteBuffer.put(records_partition); // byte
    byteBuffer.put(already_replicated); // byte

    // now the data layout
    for (Map.Entry<byte[], byte[]> entry : keyDataHolder.entrySet()) {
        byte keyType = 0;
        byte keyLength = (byte) entry.getKey().length;
        byte[] key = entry.getKey();
        long timestamp = System.currentTimeMillis();

        byte[] data = entry.getValue();
        short dataSize = (short) data.length;

        byteBuffer.put(keyType);
        byteBuffer.put(keyLength);
        byteBuffer.put(key);
        byteBuffer.putLong(timestamp);
        byteBuffer.putShort(dataSize);
        byteBuffer.put(data);
    }

    packedByteArray = byteBuffer.array();

Here's a quick list of problems I see:

1. The buffer is static, which makes this code non-re-entrant, and not-thread-safe.
2. you do not reset the buffer before using it. A previous call will invalidate subsequent calls. Use-once-only.
3. You get the backing array at the end with the packedByteArray = byteBuffer.array(); call. This returns all 64,000 bytes whether they were used or not.

I would likely recommend you use a simpler DataOutputStream on a ByteArrayOutputStream, and then just get the Output Stream's toByteArray() at the end. That solution will be thread-safe, reentrant, and probably pretty efficient. The resulting array will be the exact right size as well. No need for statics.

If performance really was a problem, then I woud remove the need for the byte array completely, and work directly off whatever stream/channel you have between the C and Java code. Basically, there;s a level missing in your presented code. There's a way to not have the buffer at all. Look for that solution.