Skip to main content
Tweeted twitter.com/StackCodeReview/status/1509999040821407745
edited tags
Link
Source Link

Reading big csv file C#

I'm trying to read quite big file with around 28 million of rows in the following way:

var jobs = new ActionBlock<string[]>((jobs) =>
    {
        //Some code
    });
    var pool = ArrayPool<string>.Shared;
    var items = pool.Rent(1000);

    using (FileStream fs = File.Open("test.csv", FileMode.Open))
    using (BufferedStream bs = new BufferedStream(fs))
    using (StreamReader sr = new StreamReader(bs, Encoding.UTF8, true, 1024))
    {
        var count = 0;
        string s;
        while ((s = sr.ReadLine()) != null)
        {
            items[count] = s;
            if (count++ == 999)
            {
                count = 0;
                jobs.Post(items);
            }
        }

        jobs.Complete();
    }

    await jobs.Completion;
    pool.Return(items);

I'm trying to read it by means of ReadLine with some pre-defined buffer. Now it takes around 17 seconds in average. Each line contains alike lines:

100.1.8.hah,2017-05-16,00:00:00,0.0,1054102.0,0001493152-17-005364,form8-k.htm,200.0,3767.0,0.0,0.0,0.0,9.0,0.0,

How can I be sure is it slow and needs enhancements or it's quite appropriate ? If someone have some clues please let me know please ...