4
\$\begingroup\$

I have an application where I am receiving big byte array very fast around per 50 miliseconds.

The byte array contains some information like file name etc. The data (byte array ) may come from several sources.

Each time I receive the data, I have to find the file name and save the data to that file name.

I need some guide lines to how should I design it so that it works efficient.

Following is my code...

public class DataSaver
{
    private Dictionary<string, FileStream> _dictFileStream;

    public void SaveData(byte[] byteArray)
    {
        string fileName = GetFileNameFromArray(byteArray);
        FileStream fs = GetFileStream(fileName);
        fs.Write(byteArray, 0, byteArray.Length);
    }

    private FileStream GetFileStream(string fileName)
    {
        FileStream fs;
        bool hasStream = _dictFileStream.TryGetValue(fileName, out fs);
        if (!hasStream)
        {
            fs = new FileStream(fileName, FileMode.Append);
            _dictFileStream.Add(fileName, fs);
        }
        return fs;
    }

    public void CloseSaver()
    {
        foreach (var key in _dictFileStream.Keys)
        {
            _dictFileStream[key].Close();
        }
    }
}

How can I improve this code ? I need to create a thread maybe to do the saving.

\$\endgroup\$
1
6
\$\begingroup\$

You need to ask yourself 4 questions to start with:

  1. How "big" is "big byte array"?
  2. How fast is "very fast" ? Can we take 20 Hz as a design assumption?
  3. Should data be appended in-order?
  4. What is the expected overall size for a given file?

Just for the taste of it, appending 4MB to a file, on my 10 YO machine, takes less than 10 ms. A modern SSD and CPU will outdo this by an order of magnitude.

Anyway, knowing nothing, I would start with the most simple imaginable scheme, which is completely synchronous, and let it fail:

public static class FileSaver
{
    public static bool TrySaveData(byte[] data)
    {
        int dataOffset = 0;
        var file = GetFileName(data, out dataOffset);
        return TryAppend(file, data, dataOffset);
    }

    private static string GetFileName(byte[] data, out int offset)
    {
        string res = ...   // get the file name
        offset = ...       // depends on your format                    
        return res;
    }

    private static bool TryAppend(string file, byte[] data, int offset = 0)
    {
        try
        {
            using (var stream = new FileStream(file, FileMode.Append))
            {
                stream.Write(data, offset, data.Length - offset);
            }
            return true;
        }
        catch (Exception ex) { /* log some error */ }
        return false;
    }
}

Why is this good? Because by the time it fails, you'll be smart enough to answer 1-4, and then we would design something else based on real understanding of the problem.

\$\endgroup\$
3
\$\begingroup\$

Careful about how many files you keep open, OS handles are not free. See http://msdn.microsoft.com/en-us/library/yz8tx0w3.aspx

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.