F# binary reader

Question

I've written a fairly simple script in F# which reads structured data from binary. I've avoided using the built-in .NET BinaryReader class because it's stateful in that it maintains its index in the stream. Instead I wanted to write a purely functional implementation. I'm doing this for my own personal learning so YAGNI does not apply.

First I defined a binary buffer type and a module of functions to operate on it.

type Buffer = Buffer of uint8 []
type ReadResult<'a> =
  | Ok of 'a*Buffer // (value, remaining bytes)
  | BufferExhausted

module Buffer =
  type ByteOrder =
    | LittleEndian
    | BigEndian

  let private nativeOrder =
    if BitConverter.IsLittleEndian
      then LittleEndian
      else BigEndian

  let private swapOrder (x:ByteOrder) (y:ByteOrder) =
    if x <> y
      then Array.rev
      else id

  let private nativeOfLE = swapOrder LittleEndian nativeOrder
  let private nativeOfBE = swapOrder BigEndian nativeOrder

  let ofHexString (hex:string) =
    if (String.length hex % 2 = 0
      then
        hex.ToCharArray()
        |> Array.chunkBySize 2
        |> Array.map (fun nibbles -> Byte.Parse(String nibbles, NumberStyles.HexNumber))
        |> Buffer
        |> Result.Ok
      else
        Error "hex number length must be multiple of 2"

  let toHexString (Buffer buffer) =
    buffer
    |> Array.map (fun byte -> byte.ToString("x2")
    |> String.concat ""

  let private bytesToInt16 bytes = BitConverter.ToInt16(bytes, 0)
  let private bytesToInt32 bytes = BitConverter.ToInt32(bytes, 0)
  let private bytesToInt64 bytes = BitConverter.ToInt64(bytes, 0)
  let private bytesToUint16 bytes = BitConverter.ToUInt16(bytes, 0)
  let private bytesToUint32 bytes = BitConverter.ToUInt32(bytes, 0)
  let private bytesToUint64 bytes = BitConverter.ToUInt64(bytes, 0)

  let private read size convert (Buffer buffer) =
    if (Array.length buffer) >= size
      then
        let (bytes, remaining) = Array.splitAt size buffer
        Ok (convert bytes, Buffer remaining)
      else
        BufferExhausted

  let readInt16LE = read sizeof<int16> (nativeOfLE >> bytesToInt16)
  let readInt16BE = read sizeof<int16> (nativeOfBE >> bytesToInt16)
  let readInt32LE = read sizeof<int32> (nativeOfLE >> bytesToInt32)
  let readInt32BE = read sizeof<int32> (nativeOfBE >> bytesToInt32)
  let readInt64LE = read sizeof<int64> (nativeOfLE >> bytesToInt64)
  let readInt64BE = read sizeof<int64> (nativeOfBE >> bytesToInt64)
  // repeated for unsigned types...

So when it comes to read a structured message from the binary buffer, I've tried to build up a workflow..

type Message = {
  fieldA: int32
  fieldB: uint64
  fieldC: uint16
}

let bind expr fn =
  match expr with
    | BufferExhausted -> BufferExhausted
    | Ok (value, remaining) -> fn (value, remaining)

let readMessageWorkflow buffer =
  bind (buffer |> Buffer.readInt32LE) (fun (fieldA, buffer) ->
  bind (buffer |> Buffer.readUint64BE) (fun (fieldB, buffer) ->
  bind (buffer |> Buffer.readUint16BE) (fun (fieldC, buffer) ->
    let message =
      {
        fieldA = fieldA
        fieldB = fieldB
        fieldC = fieldC
      }
    ReadResult.Ok (message, buffer)
  )))

This workflow has the signature I want (Buffer->ReadResult<Message>) but the workflow definition becomes quite cumbersome for larger messages. After about a dozen fields it just feels very repetitive and verbose. I'm sure there's a way I could enhance this, maybe as a computation expression looking something like this..

read {
  let! fieldA = Buffer.readInt32LE
  let! fieldB = Buffer.readUint64BE
  let! fieldC = Buffer.readUint16BE

  rtn {
    fieldA = fieldA
    fieldB = fieldB
    fieldC = fieldC
  }
}

Alternatively, could I define a constructor function for the message and somehow thread it through the workflow, so that each step applies the next argument?

let buildMessage fieldC fieldB fieldA =
  {
    fieldA = fieldA
    fieldB = fieldB
    fieldC = fieldC
  }

I would really appreciate any ideas on where I could take this next. I feel like I'm heading in roughly the right direction but there's a whole lot further to go.

I'd also be very appreciative of any feedback on whether my general usage of F# is idiomatic or could be improved.

Answer 1

do you have to deal with little endian and big endian in the same message? Usually, it is one way or the other for the entire message.

If it's a well structured binary format, you may want to look at the way FSPickler does it -- which lends itself well to formulaic situations using a combinator approach.

It all depends on how much control you need.

Computational expressions are an interesting idea for this scenario. How likely is it that the stream will halt midway? I expect that a computational expression, although it'll give you Ok / Exhausted DU you want, will still be somewhat cluttered. There's just not a condensed syntax for them at this stage.

I'd prefer to read something like this given this is the desired approach:

let readMsg b =
  (Buffer.readInt32LE b,Buffer.readUint64BE b, Buffer.readUint16BE b) 
  |> fun a b c-> { fieldA = a; fieldB = b; fieldC = C }