9
\$\begingroup\$

I have a few thousand binary files that have corresponding structs in them, followed by any number of bytes (this exact number of bytes is given in one of the fields in the structs). I read in data from the binary files to a struct, and assign variables from certain fields in those structs. I was wondering if there is any significant improvement I could make to speed things up, specifically regarding the binary file reading?

Note: I'm reading in 256 bytes at a time to a struct, and in that struct is a number that says how many bytes follow until the next struct. So there isn't a static pattern to the data that I can follow.

const int STRUCT_HEADER_SIZE = 256; //256 bytes
const int FILE_HEADER_SIZE = 1024; //1024 bytes

[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct STRUCTHEADER
{
    //Sample fields
    public ushort MagicNumber;   
    public byte SubChannelNumber; 
    public ushort NumChansToFollow;  

    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 2)]
    public ushort[] Reserved1; 

    public int NumBytesThisRecord; //This value says how many bytes there are total in the current 'packet' which includes this struct header.

    public ushort Year;
    public byte Month;
    public byte Day;
    public byte Hour;
}

FileStream stream1;
STRUCTHEADER testStruct = new STRUCTHEADER();
List<string> filePaths = new List<string>();

foreach (string filePath in filePaths) //for each binary file
{
    ReadBinaryFile(filePath); //Read the binary file
}

public void ReadBinaryFile(string filePath)
{
    try
    {
        stream1 = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.None);
    }
    catch(Exception ex)
    {
    }

    try
    {
        stream1.Position = FILE_HEADER_SIZE; //Start stream after the file header

        while (stream1.Position < stream1.Length) //While not at the end of the stream
        {
            testStruct = ReadRecFromStream<STRUCTHEADER>(stream1); //read data from binary file into STRUCTHEADER type struct

            //assigning fields here
            //ex: int year = testStruct.year;

            stream1.Position += Math.Abs(testStruct.NumBytesThisRecord - STRUCT_HEADER_SIZE); //Advance to the end of current packet
        }
        stream1.Close();
        //Adding field values to a textbox
        //ex: textbox1.Text += year;
    }
    catch
    {
        stream1.Close();
    }
}

//Used to read in data in binary file to a struct
private T ReadRecFromStream<T>(Stream stream) where T : struct
{
    byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
    stream.Read(buffer, 0, STRUCT_HEADER_SIZE);
    GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
    try
    {
        return (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
    }
    finally
    {
        handle.Free();
    }
}

Here's a link to two sample .xtf binary files (they are the same type I'm reading in): https://sourceforge.net/projects/imagejforxtf/files/?source=navbar

\$\endgroup\$
7
\$\begingroup\$

An alternative solution could be a binary reader (see also: this code project article)

Implementation: (I just replaced the 2 try catch blocks with one using):

public static void ReadBinaryReader(string filePath)
{
    using (var stream1 = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.None))
    {
        stream1.Position = FILE_HEADER_SIZE; //Start stream after the file header

        using (var reader = new BinaryReader(stream1))
        {
            while (stream1.Position < stream1.Length) //While not at the end of the stream
            {
                var testStruct = FromReader(reader); //read data from binary file into STRUCTHEADER type struct
                stream1.Position += Math.Abs(testStruct.NumBytesThisRecord - STRUCT_HEADER_SIZE); //Advance to the end of current packet
            }
        }
    }
}

private static STRUCTHEADER FromReader(BinaryReader reader)
{
    STRUCTHEADER result = new STRUCTHEADER();
    result.MagicNumber = reader.ReadUInt16();
    result.SubChannelNumber = reader.ReadByte();
    result.NumChansToFollow = reader.ReadUInt16();
    result.Reserved1 = new []
    {
        reader.ReadUInt16(),
        reader.ReadUInt16(),
    };
    result.NumBytesThisRecord = reader.ReadInt32();
    result.Year = reader.ReadUInt16();
    result.Month = reader.ReadByte();
    result.Day = reader.ReadByte();
    result.Hour = reader.ReadByte();
    return result;
}

It is a little bis faster, but not significat... However, if you have to use a binary reader for reading the other content, maybe it is a reasonable alternative to PtrToStructure.

Reading the 6 mb file 60.000 times takes:

PtrToStructure

1) 3791 ms

2) 3807 ms

3) 3794 ms

BinaryReader

1) 3650 ms

2) 3645 ms

3) 3668 ms

\$\endgroup\$
  • 3
    \$\begingroup\$ It may not be much faster, but it's much easier for your average person to read and understand, since there's a lot less magic. It's best to avoid the Marshal class unless absolutely necessary. \$\endgroup\$ – Dan Lyons Jun 16 '16 at 17:28
  • 1
    \$\begingroup\$ So I tested this against 700 files and your solution is a few seconds quicker than mine which is awesome! Thank you! \$\endgroup\$ – pfinferno Jun 16 '16 at 17:29
  • \$\begingroup\$ Using MemoryMappedFile would speed up things more \$\endgroup\$ – Konrad Mar 5 at 0:40
4
\$\begingroup\$

Your problem isn't the struct reading but the reading of small junks from disc. Consider to load the whole file into a memorystream first and then use either your aproach or use the BinaryReader like @JanDotNet suggested.


As a side note, you really should name your things better. E.g stream1 implies that there is at least another stream around.


Swallowing exceptions is the way to hell. Unless you have a very good reason, you should never ever do this.
If you have a valid good reason then make sure to place a comment stating why you are swallowing that exception.


Don't call the Close () method of the stream in the catch. Call it in the finally which makes the call in the try superflous. But much better use a using block like @JanDotNet suggested.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.