I'm trying to read quite big file with around 28 million of rows in the following way:
var jobs = new ActionBlock<string[]>((jobs) =>
{
//Some code
});
var pool = ArrayPool<string>.Shared;
var items = pool.Rent(1000);
using (FileStream fs = File.Open("test.csv", FileMode.Open))
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs, Encoding.UTF8, true, 1024))
{
var count = 0;
string s;
while ((s = sr.ReadLine()) != null)
{
items[count] = s;
if (count++ == 999)
{
count = 0;
jobs.Post(items);
}
}
jobs.Complete();
}
await jobs.Completion;
pool.Return(items);
I'm trying to read it by means of ReadLine with some pre-defined buffer. Now it takes around 17 seconds in average. Each line contains alike lines:
100.1.8.hah,2017-05-16,00:00:00,0.0,1054102.0,0001493152-17-005364,form8-k.htm,200.0,3767.0,0.0,0.0,0.0,9.0,0.0,
How can I be sure is it slow and needs enhancements or it's quite appropriate ? If someone have some clues please let me know please ...