0
\$\begingroup\$

So I'm trying to read data and store it in an array as fast as possible and the fastest method I found of doing so was this.

var filePath = "data.dat";
FileStream fs = new FileStream(filePath, FileMode.Open);
bool[] buffer = new bool[fs.Length];

TimeSpan[] times = new TimeSpan[500000];
Stopwatch sw = new Stopwatch();
for (int r = 0; r < 500000; r++)
{
    sw.Start();

    int stackable = 0;
    int counter = 0;
    while ((stackable = fs.ReadByte()) != -1)
    {
        buffer[counter] = (stackable == 1);
        counter++;
    }

    sw.Stop();
    Console.WriteLine($"Elapsed: {sw.Elapsed}ms");
    times[r] = sw.Elapsed;
    sw.Reset();
}

Console.WriteLine($"Longest itteration: {times.Max()}ms");

which manages to read and process about 9000 bytes in < 3ms. The idea is to check each byte to see if it's either 1 or 0 (true or false) and store that in an array.

So my question is, is there a faster way of achieving this? What are some things to keep in mind when trying to process data fast, is it to make sure you're working with smaller data types so you don't allocate unecessary memory?

What the data looks like.

enter image description here

https://hatebin.com/dcldbvrbdm

\$\endgroup\$
8
  • 1
    \$\begingroup\$ All subsequent reading will take 0ms because you don't rewind the stream. \$\endgroup\$ Jan 19 at 8:18
  • 3
    \$\begingroup\$ Why do you read that data into a bool[]? Why don't you read it into a byte[] via File.ReadAllBytes? \$\endgroup\$ Jan 19 at 13:35
  • 2
    \$\begingroup\$ Btw, it's a strange way to store the data. Why the files saved in so inefficient data format? One byte contains 8 bits not 1. Why not save 8 per byte then? Then you'll get 8x smaller file, as result 8x faster reads. \$\endgroup\$
    – aepot
    Jan 19 at 18:19
  • \$\begingroup\$ JFYI BitArray. \$\endgroup\$
    – aepot
    Jan 19 at 18:42
  • \$\begingroup\$ If you just want to check each byte why do you need an array for it? Just keep reading till the end and do your thing along the way. \$\endgroup\$
    – t3chb0t
    Jan 19 at 20:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.