5
\$\begingroup\$

I need to merge some large csv files but I feel that the way I am doing it is not optmized. It gets a number of csvfiles and creates one

public static void MergeFiles(IEnumerable<string> csvFileNames, string outputPath)
{
    var sb = new StringBuilder();
    bool hasHeader = false;
    foreach (string csvFileName in csvFileNames)
    {
        TextReader tr = new StreamReader(csvFileName);
        string headers = tr.ReadLine();
        if (!hasHeader)
        {
            sb.AppendLine(headers);
            hasHeader = true;
        }
        sb.AppendLine(tr.ReadToEnd());  
    }
    File.WriteAllText(outputPath, sb.ToString());
}

Any suggestion or code snippets?

\$\endgroup\$
  • \$\begingroup\$ In which way you think it's not optimized? \$\endgroup\$ – Groo Nov 28 '13 at 13:38
5
\$\begingroup\$
  1. StreamReader is IDisposable and therefor the use of it should be wrapped in a using block like this:

    using (TextReader tr = new StreamReader(csvFileName))
    {
        ...
    }
    
  2. You are reading all files into memory and then writing it out. While this probably has a fairly good performance as it's first performing a bunch of (probably) sequential reads of all the files and then one big block write you could consider writing it out as you read it:

    using (var writer = new StreamWriter(outputPath))
    {
        foreach (var csvFileName in csvFileNames)
        {
            using (var reader = new StreamReader(csvFileName))
            {
                string headers = tr.ReadLine();
    
                if (!hasHeader)
                {
                    writer.WriteLine(headers);
                    hasHeader = true;
                }
    
                writer.Write(reader.ReadToEnd());
            }
        }
    }
    

    If the files are very big you even might want to consider reading the input line by line and writing it to the output line by line. This will be less performant but also use much less memory. In the end it's a trade off between speed and memory consumption (as it often is in software development).

    The other advantage with the write as you go method is that you open the output file first - so if there is a problem with writing to the target you will know before you do any work at all.

\$\endgroup\$
  • \$\begingroup\$ Wouldn't it be faster to first read everything to the memory and then start with the I/O operations? - or did I misunderstand you? \$\endgroup\$ – Max Nov 28 '13 at 8:05
  • \$\begingroup\$ @Max: Yes, I thought that's what I said: You are reading all files into memory and then writing it out. While this probably has a fairly good performance as it's first performing a bunch of (probably) sequential reads of all the files and then one big block write \$\endgroup\$ – ChrisWue Nov 28 '13 at 8:19
  • \$\begingroup\$ o, my bad then :) \$\endgroup\$ – Max Nov 28 '13 at 8:20
  • \$\begingroup\$ @ChrisWue Thank you very much for your time and input I will do that \$\endgroup\$ – user9969 Nov 29 '13 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.