CSV concatenator

Question

I have the following code (running in LINQPad) which takes tens of thousands of CSV files and concatenates them to create a single file.

Each CSV file has two lines: a header line and a data line. I need the header line only once, then the rest of the lines are data.

However, it appears to be getting slower as it runs over the course of several minutes. Why is it doing this, and what can I do to make it faster overall?

Each file is only 3KB apiece. For example, the output below was from a run of about 55K files and the resulting CSV file was just under 8MB.

// Directory path - This is the directory where all of the csv's you want to combine reside
var path = @"E:\a1";        

// Destination path - Needs to be different from the directory Path we defined in the first step. This is where the combined csv file will be placed
var destination = @"E:\MasterFileA1.csv";

var dirInfo = new DirectoryInfo(path);
var enumFiles = dirInfo.EnumerateFiles("*.csv", SearchOption.TopDirectoryOnly);

int counter = 0;
string header;
string data;
DateTime start = DateTime.Now;
TimeSpan elapsed;

FileStream stream = new FileStream(destination, FileMode.Append, FileAccess.Write);
StreamWriter writer = new StreamWriter(stream);

foreach(var file in enumFiles) 
{  
    StreamReader reader = new StreamReader(file.FullName);

    header = reader.ReadLine();
    data = reader.ReadLine();

    reader.Close();

    if (counter == 0)
    {
        writer.WriteLine(header);
    }

    writer.WriteLine(data);

    counter++;

    if (counter % 1000 == 0)
    {
        //writer.Flush();
        elapsed = DateTime.Now.Subtract(start);
        Console.WriteLine("Processed: " + counter + " files.     Elapsed Time: " + String.Format("{0}:{1}", elapsed.Minutes, elapsed.Seconds) +
                          "     Time per file: " + Math.Round(elapsed.TotalMilliseconds/counter, 4) + " ms");
    }
}

writer.Flush();
writer.Close();

Output via Console:

Processed: 1000 files.     Elapsed Time: 0:2     Time per file: 2.574 ms
Processed: 2000 files.     Elapsed Time: 0:4     Time per file: 2.4671 ms
Processed: 3000 files.     Elapsed Time: 0:7     Time per file: 2.4406 ms
Processed: 4000 files.     Elapsed Time: 0:9     Time per file: 2.4353 ms
Processed: 5000 files.     Elapsed Time: 0:12     Time per file: 2.4227 ms
Processed: 6000 files.     Elapsed Time: 0:14     Time per file: 2.4123 ms
Processed: 7000 files.     Elapsed Time: 0:17     Time per file: 2.4495 ms
Processed: 8000 files.     Elapsed Time: 0:19     Time per file: 2.4368 ms
Processed: 9000 files.     Elapsed Time: 0:21     Time per file: 2.4285 ms
Processed: 10000 files.     Elapsed Time: 0:24     Time per file: 2.4231 ms
Processed: 11000 files.     Elapsed Time: 0:26     Time per file: 2.4167 ms
Processed: 12000 files.     Elapsed Time: 0:28     Time per file: 2.4118 ms
Processed: 13000 files.     Elapsed Time: 0:31     Time per file: 2.4104 ms
Processed: 14000 files.     Elapsed Time: 0:33     Time per file: 2.4094 ms
Processed: 15000 files.     Elapsed Time: 0:36     Time per file: 2.4062 ms
Processed: 16000 files.     Elapsed Time: 0:38     Time per file: 2.4031 ms
Processed: 17000 files.     Elapsed Time: 0:40     Time per file: 2.4009 ms
Processed: 18000 files.     Elapsed Time: 0:43     Time per file: 2.4004 ms
Processed: 19000 files.     Elapsed Time: 0:45     Time per file: 2.3979 ms
Processed: 20000 files.     Elapsed Time: 0:47     Time per file: 2.3955 ms
Processed: 21000 files.     Elapsed Time: 0:50     Time per file: 2.3945 ms
Processed: 22000 files.     Elapsed Time: 0:52     Time per file: 2.3936 ms
Processed: 23000 files.     Elapsed Time: 0:55     Time per file: 2.3987 ms
Processed: 24000 files.     Elapsed Time: 0:57     Time per file: 2.3981 ms
Processed: 25000 files.     Elapsed Time: 0:59     Time per file: 2.3968 ms
Processed: 26000 files.     Elapsed Time: 1:2     Time per file: 2.3966 ms
Processed: 27000 files.     Elapsed Time: 1:4     Time per file: 2.3945 ms
Processed: 28000 files.     Elapsed Time: 1:7     Time per file: 2.3938 ms
Processed: 29000 files.     Elapsed Time: 1:9     Time per file: 2.3934 ms
Processed: 30000 files.     Elapsed Time: 1:11     Time per file: 2.3927 ms
Processed: 31000 files.     Elapsed Time: 1:23     Time per file: 2.6784 ms
Processed: 32000 files.     Elapsed Time: 1:35     Time per file: 2.9779 ms
Processed: 33000 files.     Elapsed Time: 1:47     Time per file: 3.2714 ms
Processed: 34000 files.     Elapsed Time: 2:0     Time per file: 3.5462 ms
Processed: 35000 files.     Elapsed Time: 2:12     Time per file: 3.7953 ms
Processed: 36000 files.     Elapsed Time: 2:25     Time per file: 4.0352 ms
Processed: 37000 files.     Elapsed Time: 2:38     Time per file: 4.2769 ms
Processed: 38000 files.     Elapsed Time: 2:50     Time per file: 4.4893 ms
Processed: 39000 files.     Elapsed Time: 3:3     Time per file: 4.6924 ms
Processed: 40000 files.     Elapsed Time: 3:15     Time per file: 4.8848 ms
Processed: 41000 files.     Elapsed Time: 3:27     Time per file: 5.0673 ms
Processed: 42000 files.     Elapsed Time: 3:40     Time per file: 5.2461 ms
Processed: 43000 files.     Elapsed Time: 3:53     Time per file: 5.4223 ms
Processed: 44000 files.     Elapsed Time: 4:5     Time per file: 5.5781 ms
Processed: 45000 files.     Elapsed Time: 4:17     Time per file: 5.7302 ms
Processed: 46000 files.     Elapsed Time: 4:30     Time per file: 5.8799 ms
Processed: 47000 files.     Elapsed Time: 4:42     Time per file: 6.0171 ms
Processed: 48000 files.     Elapsed Time: 4:55     Time per file: 6.1547 ms
Processed: 49000 files.     Elapsed Time: 5:9     Time per file: 6.3069 ms
Processed: 50000 files.     Elapsed Time: 5:21     Time per file: 6.4373 ms
Processed: 51000 files.     Elapsed Time: 5:34     Time per file: 6.5605 ms
Processed: 52000 files.     Elapsed Time: 5:47     Time per file: 6.6796 ms
Processed: 53000 files.     Elapsed Time: 5:59     Time per file: 6.7909 ms
Processed: 54000 files.     Elapsed Time: 6:12     Time per file: 6.9046 ms
Processed: 55000 files.     Elapsed Time: 6:25     Time per file: 7.0035 ms

Update

This is an unsatisfying result, but I think the problem was that the source and destination path were both on an external HDD connected via USB 3.0. Once I copied the files over and was working only on my local hard drive, the speed of execution stayed constant.

Now working locally, I tested my original code against the suggestions provided by xDaevax - that is, I incorporated using statements for both the reader and the writer. The results:

Original code took 2.66 ms per file
New code took 5.82 ms per file

I assume this is primarily because I was opening a new StreamWriter every time, whereas in the original code I left the StreamWriter open but opened and closed the StreamReader. I also tried adding a using statement just to the StreamReader (not the Writer) and was the same speed as my original code.

What is the benefit of opening and closing the StreamWriter for each write, if not performance? (this question still hasn't been answered, so if you're reading this and want to contribute, please do so)

Update 2

Implementing Brandon's suggestions for using a StringBuilder and also reading the input CSV in one interaction, I was able to reduce the time per file by nearly 20% to 2.17 ms. I also just generally used better structure for the program as a whole, and added some formatting to the output.

As a bonus, now that I am accessing the lines of the input CSVs as array elements, I detected a file that wasn't created properly when I hit an "index outside bounds of array" error, so I added a try/catch to handle that and tell me which file it was.

Here's the "final" version (but is tinkering ever really finished?):

void Main()
{
    // Directory path
    string path = @"C:\a1";         

    // Destination path (must be different from directory path)
    string destination = @"C:\MasterFileA1.csv";

    int counter = 0;
    TimeSpan globalElapsed;
    TimeSpan intervalElapsed;

    FileStream stream = new FileStream(destination, FileMode.CreateNew, FileAccess.Write);
    StreamWriter writer = new StreamWriter(stream);
    StringBuilder sb = new StringBuilder();

    // Pull list of files into string array
    IEnumerable enumFiles = EnumerateFiles(path);

    // Start stopwatches
    Stopwatch globalTimer = Stopwatch.StartNew();
    Stopwatch intervalTimer = Stopwatch.StartNew();

    foreach(FileInfo file in enumFiles) 
    {  
        try
        {
            // Read file
            string[] fileText = File.ReadAllLines(file.FullName);

            // Append lines to StringBuilder object
            if (counter == 0)
            {
                sb.Append(fileText[0] + "\r\n");
                sb.Append(fileText[1] + "\r\n");
            }
            else
            {
                sb.Append(fileText[1] + "\r\n");
            }
        }
        catch (Exception e)
        {
            Console.WriteLine("Error occurred at file " + counter + ": " + file.FullName + ". " + e.Message);
        }

        // Log progress to console
        if (counter % 1000 == 0 && counter != 0)
        {
            globalElapsed = globalTimer.Elapsed;
            intervalElapsed = intervalTimer.Elapsed;

            Console.WriteLine("Processed: " + String.Format("{0:00000}",counter) + " files.     " +
                              "Elapsed Time: " + String.Format("{0:00}:{1:00}", globalElapsed.Minutes, globalElapsed.Seconds) + "     " +
                              "Time per file (last 1000): " + String.Format("{0:00.00}", Math.Round(intervalElapsed.TotalMilliseconds / 1000, 2)) + " ms     " +
                              "Time per file (total): " + String.Format("{0:00.00}", Math.Round(globalElapsed.TotalMilliseconds / counter, 2)) + " ms");

            intervalTimer.Reset();
            intervalTimer.Start();
        }

        counter++;
    }

    globalTimer.Stop();

    // Write concatenated string to file
    writer.Write(sb.ToString());
    writer.Flush();
    writer.Close();
}

public IEnumerable EnumerateFiles(string path)
{
    Console.WriteLine("Enumerating files...");
    Stopwatch getFilesTimer = Stopwatch.StartNew();

    var enumFiles = new DirectoryInfo(path).EnumerateFiles("*.csv", SearchOption.TopDirectoryOnly);

    getFilesTimer.Stop();
    TimeSpan getFilesElapsed = getFilesTimer.Elapsed;
    Console.WriteLine("Enumeration complete. Time taken: " + getFilesElapsed.TotalMilliseconds + " ms");

    return enumFiles;
}

The output:

Enumerating files...
Enumeration complete. Time taken: 0.2707 ms
Processed: 01000 files.     Elapsed Time: 00:02     Time per file (last 1000): 02.38 ms     Time per file (total): 02.38 ms
Processed: 02000 files.     Elapsed Time: 00:04     Time per file (last 1000): 02.23 ms     Time per file (total): 02.30 ms
Processed: 03000 files.     Elapsed Time: 00:06     Time per file (last 1000): 02.27 ms     Time per file (total): 02.29 ms
Processed: 04000 files.     Elapsed Time: 00:09     Time per file (last 1000): 02.26 ms     Time per file (total): 02.28 ms
Processed: 05000 files.     Elapsed Time: 00:11     Time per file (last 1000): 02.31 ms     Time per file (total): 02.29 ms
Processed: 06000 files.     Elapsed Time: 00:13     Time per file (last 1000): 02.07 ms     Time per file (total): 02.25 ms
Processed: 07000 files.     Elapsed Time: 00:15     Time per file (last 1000): 02.02 ms     Time per file (total): 02.22 ms
Processed: 08000 files.     Elapsed Time: 00:17     Time per file (last 1000): 02.08 ms     Time per file (total): 02.20 ms
Processed: 09000 files.     Elapsed Time: 00:19     Time per file (last 1000): 02.32 ms     Time per file (total): 02.22 ms
Processed: 10000 files.     Elapsed Time: 00:22     Time per file (last 1000): 02.32 ms     Time per file (total): 02.23 ms
Processed: 11000 files.     Elapsed Time: 00:24     Time per file (last 1000): 02.33 ms     Time per file (total): 02.24 ms
Processed: 12000 files.     Elapsed Time: 00:26     Time per file (last 1000): 02.33 ms     Time per file (total): 02.24 ms
Processed: 13000 files.     Elapsed Time: 00:29     Time per file (last 1000): 02.09 ms     Time per file (total): 02.23 ms
Processed: 14000 files.     Elapsed Time: 00:31     Time per file (last 1000): 02.18 ms     Time per file (total): 02.23 ms
Processed: 15000 files.     Elapsed Time: 00:33     Time per file (last 1000): 02.01 ms     Time per file (total): 02.21 ms
Processed: 16000 files.     Elapsed Time: 00:35     Time per file (last 1000): 02.02 ms     Time per file (total): 02.20 ms
Processed: 17000 files.     Elapsed Time: 00:37     Time per file (last 1000): 02.01 ms     Time per file (total): 02.19 ms
Processed: 18000 files.     Elapsed Time: 00:39     Time per file (last 1000): 02.01 ms     Time per file (total): 02.18 ms
Processed: 19000 files.     Elapsed Time: 00:41     Time per file (last 1000): 02.00 ms     Time per file (total): 02.17 ms
Processed: 20000 files.     Elapsed Time: 00:43     Time per file (last 1000): 02.02 ms     Time per file (total): 02.16 ms
Processed: 21000 files.     Elapsed Time: 00:45     Time per file (last 1000): 02.01 ms     Time per file (total): 02.16 ms
Processed: 22000 files.     Elapsed Time: 00:47     Time per file (last 1000): 02.37 ms     Time per file (total): 02.17 ms
Processed: 23000 files.     Elapsed Time: 00:49     Time per file (last 1000): 02.31 ms     Time per file (total): 02.17 ms
Processed: 24000 files.     Elapsed Time: 00:52     Time per file (last 1000): 02.24 ms     Time per file (total): 02.18 ms
Processed: 25000 files.     Elapsed Time: 00:54     Time per file (last 1000): 02.02 ms     Time per file (total): 02.17 ms
Processed: 26000 files.     Elapsed Time: 00:56     Time per file (last 1000): 02.00 ms     Time per file (total): 02.16 ms
Processed: 27000 files.     Elapsed Time: 00:58     Time per file (last 1000): 02.02 ms     Time per file (total): 02.16 ms
Processed: 28000 files.     Elapsed Time: 01:00     Time per file (last 1000): 02.00 ms     Time per file (total): 02.15 ms
Processed: 29000 files.     Elapsed Time: 01:02     Time per file (last 1000): 02.02 ms     Time per file (total): 02.15 ms
Processed: 30000 files.     Elapsed Time: 01:04     Time per file (last 1000): 02.01 ms     Time per file (total): 02.14 ms
Processed: 31000 files.     Elapsed Time: 01:06     Time per file (last 1000): 01.99 ms     Time per file (total): 02.14 ms
Processed: 32000 files.     Elapsed Time: 01:08     Time per file (last 1000): 02.01 ms     Time per file (total): 02.13 ms
Processed: 33000 files.     Elapsed Time: 01:10     Time per file (last 1000): 02.01 ms     Time per file (total): 02.13 ms
Processed: 34000 files.     Elapsed Time: 01:12     Time per file (last 1000): 02.14 ms     Time per file (total): 02.13 ms
Processed: 35000 files.     Elapsed Time: 01:14     Time per file (last 1000): 02.30 ms     Time per file (total): 02.14 ms
Processed: 36000 files.     Elapsed Time: 01:17     Time per file (last 1000): 02.30 ms     Time per file (total): 02.14 ms
Processed: 37000 files.     Elapsed Time: 01:19     Time per file (last 1000): 02.30 ms     Time per file (total): 02.14 ms
Processed: 38000 files.     Elapsed Time: 01:21     Time per file (last 1000): 02.33 ms     Time per file (total): 02.15 ms
Processed: 39000 files.     Elapsed Time: 01:23     Time per file (last 1000): 02.31 ms     Time per file (total): 02.15 ms
Processed: 40000 files.     Elapsed Time: 01:26     Time per file (last 1000): 02.31 ms     Time per file (total): 02.16 ms
Processed: 41000 files.     Elapsed Time: 01:28     Time per file (last 1000): 02.34 ms     Time per file (total): 02.16 ms
Processed: 42000 files.     Elapsed Time: 01:30     Time per file (last 1000): 02.33 ms     Time per file (total): 02.17 ms
Processed: 43000 files.     Elapsed Time: 01:33     Time per file (last 1000): 02.33 ms     Time per file (total): 02.17 ms
Processed: 44000 files.     Elapsed Time: 01:35     Time per file (last 1000): 02.25 ms     Time per file (total): 02.17 ms
Processed: 45000 files.     Elapsed Time: 01:37     Time per file (last 1000): 02.00 ms     Time per file (total): 02.17 ms
Error occurred at file 45257: C:\a1\W09K6240177_20160726211002_Results.csv. Index was outside the bounds of the array.
Processed: 46000 files.     Elapsed Time: 01:39     Time per file (last 1000): 02.14 ms     Time per file (total): 02.17 ms
Processed: 47000 files.     Elapsed Time: 01:41     Time per file (last 1000): 01.99 ms     Time per file (total): 02.16 ms
Processed: 48000 files.     Elapsed Time: 01:44     Time per file (last 1000): 02.38 ms     Time per file (total): 02.17 ms
Processed: 49000 files.     Elapsed Time: 01:46     Time per file (last 1000): 02.01 ms     Time per file (total): 02.16 ms
Processed: 50000 files.     Elapsed Time: 01:48     Time per file (last 1000): 02.05 ms     Time per file (total): 02.16 ms
Processed: 51000 files.     Elapsed Time: 01:50     Time per file (last 1000): 02.28 ms     Time per file (total): 02.16 ms
Processed: 52000 files.     Elapsed Time: 01:52     Time per file (last 1000): 02.32 ms     Time per file (total): 02.17 ms
Processed: 53000 files.     Elapsed Time: 01:54     Time per file (last 1000): 02.30 ms     Time per file (total): 02.17 ms
Processed: 54000 files.     Elapsed Time: 01:57     Time per file (last 1000): 02.29 ms     Time per file (total): 02.17 ms
Processed: 55000 files.     Elapsed Time: 01:59     Time per file (last 1000): 01.99 ms     Time per file (total): 02.17 ms

Not sure just yet, but for sure use the StopWatch type built into the System.Diagnostics namespace instead of using DateTime. It's much more precise and will give you better metrics. Do you have any perfcounter data to see what the memory usage is as time goes on? — xDaevax, Aug 5, 2016 at 22:20
you are not disposing of your input streams, be nice to the GC and do a using on the input stream — pm100, Aug 5, 2016 at 22:34
I am not familiar with StreamReaders, but don't then implement IDisposable and you're supposed to use them withusing? — asibahi, Aug 5, 2016 at 22:43
@pm100 If you have spotted a problem with the code, please write an answer. Comments are for seeking clarification of the question, and are subject to deletion. — 200_success, Aug 6, 2016 at 1:31

Brandon Dube · Accepted Answer · 2016-08-06 19:15:19Z

2

There are a couple of ways you can improve this I believe, though 2ms per CSV is already pretty good.

When you're writing, you use a fileStream. Streams are fast I/O, but you are still going to be hitting that stream thousands of times, and that stream interfaces with the operating system to talk about disk access. Because these are CSV files, you are better served by using a StringBuilder, replacing your FileStream and StreamWriter with

StringBuilder sb = new StringBuilder();

You will add rows to your final CSV by calling sb.Append(data).

Inside your foreach you also cause a lot of I/O chatter by calling file.ReadLine() twice in a row. It will be faster in most cases to read the entire file into memory as a string, split it at newline characters, and then keep the rows you want.

By moving all of your operations to memory, you move everything away from the millisecond domain of hard disk I/O to the picosecond domain of memory I/O, and the program should execute much faster.

When you build and run the final program, also take care to enable Optimize Code in the visual studio project file. Depending on the contents, it can speed things up 2x or more.

answered Aug 6, 2016 at 19:15

Brandon Dube

1364 bronze badges

\$\begingroup\$ I still need a StreamWriter to create the final CSV at the end, right? A place to put the sb.ToString()? \$\endgroup\$
– Dan A.
Aug 6, 2016 at 19:26
\$\begingroup\$ @DanA. you will still ultimately need to dump the string to a file, yes. Remember that IO is sloooooooooooooooow, so you only want to hit it once or twice if you can get away with it. \$\endgroup\$
– Brandon Dube
Aug 6, 2016 at 19:30
\$\begingroup\$ Thanks. One more question - I'm sure this depends on a lot of things, but is there a practical or theoretical limit to the size of the StringBuilder? \$\endgroup\$
– Dan A.
Aug 6, 2016 at 19:33
\$\begingroup\$ It's a giant string on a computer, so I'm sure there are a lot of limits. The most immediately one is probably how much memory you have. StringBuilder is just a class that is used to efficiently put strings together; if you're repeatedly changing a string, it's much higher performance than string += "txt"; over and over. In that sense, it can get as big as a string can get. \$\endgroup\$
– Brandon Dube
Aug 6, 2016 at 19:45

Add a comment |

xDaevax · Accepted Answer · 2016-08-05 22:54:10Z

I would suggest a couple of things (I can't reproduce since I don't have the files).

Here is a re-written version I did in LinqPad using Stopwatch and using statements.

void Main() {
    // Directory path - This is the directory where all of the csv's you want to combine reside
    var path = @"E:\a1";

    // Destination path - Needs to be different from the directory Path we defined in the first step. This is where the combined csv file will be placed
    var destination = @"E:\MasterFileA1.csv";

    var enumFiles = GetFilesToProcess(path);

    int counter = 0;
    string header;
    string data;
    Stopwatch globalTimer = Stopwatch.StartNew();

    foreach (var file in enumFiles) {
        Stopwatch fileReadTimer = Stopwatch.StartNew();
        using (StreamReader reader = new StreamReader(file.FullName)) {
            header = reader.ReadLine();
            data = reader.ReadLine();
        }

        fileReadTimer.Stop();
        LogTime(string.Concat("File Read: ", file.FullName), fileReadTimer);

        Stopwatch writeTimer = Stopwatch.StartNew();
        using (FileStream stream = new FileStream(destination, FileMode.Append, FileAccess.Write)) {
            using (StreamWriter writer = new StreamWriter(stream)) {
                if (counter == 0) {
                    writer.WriteLine(header);
                }

                writer.WriteLine(data);
            }
        }
        writeTimer.Stop();
        LogTime("Finished write operation.", writeTimer);

        counter++;

        if (counter % 1000 == 0) {
            //writer.Flush();
            Console.WriteLine("Processed: " + counter + " files.     Elapsed Time: " + globalTimer.Elapsed.ToString() +
                              "     Time per file: " + Math.Round((double)(globalTimer.ElapsedMilliseconds / counter), 4) + " ms");
        }
    }

    globalTimer.Stop();
}

public FileInfo[] GetFilesToProcess(string path) {
    Stopwatch timer = Stopwatch.StartNew();
    return new DirectoryInfo(path).EnumerateFiles("*.csv", SearchOption.TopDirectoryOnly).ToArray();
    timer.Stop();
    LogTime("Enumerated input CSVs", timer);
}

public void LogTime(string message, Stopwatch timer) {
    var now = DateTime.Now;
    Console.WriteLine("[{0}] - {1}: Elapsed Time: {2}", string.Concat(now.ToShortDateString(), " ", now.ToShortTimeString()), message, timer.Elapsed.ToString());
}

1. Dispose of unmanaged resources with using statements:

using (FileStream stream = new FileStream(destination, FileMode.Append, FileAccess.Write)) {}

This allows the runtime to optimize memory usage and make sure that instances get cleaned up properly.

2. Only keep the stream open for the shortest amount of time possible.

This allows the buffer to flush regularly and creates a nice atomic operation.

Let me know how this version performs.

3. Performance Profiling

When performing operations like this, the question frequently comes up as to what is the source of the performance degradation. Any number of things could cause it but to get concrete reasons to help your refactor pen, Performance Counters are your friend.

When the .NET Framework is installed, it comes with MANY handy little counters you can use to identify key metrics of your applications performance.

Here is some information on the performance counters available. https://msdn.microsoft.com/en-us/library/w8f5kw2e(v=vs.110).aspx

Using the perf counters allows you to more easily identify things like if you have a memory leak, are bound by disk I/O, are killing the CPU, or are spending all of your time in GC. Once you know that, the solution to your performance issue should be much easier to see and implement.

Thanks for the tips! I am a .NET novice so these tips help a lot. I'll give this a try and let you know how it works! — Dan A., Aug 6, 2016 at 0:35

Dmitry Nogin · Accepted Answer · 2016-08-07 05:31:54Z

I would buffer the output and read files in parallel:

class Program
{
    static void Main(string[] args)
    {
        var f = new MergedCsv(@"C:\a1");
        f.WriteTo(@"C:\MasterFileA1.csv");
    }
}

Try adjusting degreeOfParallelism and bufferSize in this helper class:

class MergedCsv : IEnumerable<string>
{
    public MergedCsv(string path, int degreeOfParallelism = 5, int bufferSize = 4096)
    {
        Path = path;
        DegreeOfParallelism = degreeOfParallelism;
        BufferSize = bufferSize;
    }

    public void WriteTo(string path) =>
        WriteTo(File.OpenWrite(path));

    public void WriteTo(Stream stream)
    {
        using (var buffer = new BufferedStream(stream, BufferSize))
        using (var writer = new StreamWriter(buffer))
            foreach (var line in this)
                writer.WriteLine(line);
    }

    public IEnumerator<string> GetEnumerator() => Header.Concat(Data).GetEnumerator();
    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();

    public IEnumerable<string> Header => Files
        .Select(f => File.ReadLines(f).Take(1))
        .First();

    public IEnumerable<string> Data => Files
        .AsParallel()
        .WithDegreeOfParallelism(DegreeOfParallelism)
        .Select(f => File.ReadAllLines(f).Skip(1))
        .AsSequential() // Should you preserve source file line order?
        .SelectMany(f => f);

    IEnumerable<string> Files => 
        Directory.GetFiles(Path, "*.csv", SearchOption.TopDirectoryOnly);

    string Path { get; }
    int DegreeOfParallelism { get; }
    int BufferSize { get; }
}

Stack Exchange Network

CSV concatenator

Update

Update 2

3 Answers 3

Your Answer

Not the answer you're looking for? Browse other questions tagged
c#
performance
csv
io
or ask your own question.

Hot Network Questions

CSV concatenator

Update

Update 2

3 Answers 3

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged c#performancecsvio or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
c#
performance
csv
io
or ask your own question.