# Improve my copying of a CSV file

I need to merge some large csv files but I feel that the way I am doing it is not optmized. It gets a number of csvfiles and creates one

public static void MergeFiles(IEnumerable<string> csvFileNames, string outputPath)
{
var sb = new StringBuilder();
bool hasHeader = false;
foreach (string csvFileName in csvFileNames)
{
TextReader tr = new StreamReader(csvFileName);
string headers = tr.ReadLine();
if (!hasHeader)
{
sb.AppendLine(headers);
hasHeader = true;
}
sb.AppendLine(tr.ReadToEnd());
}
File.WriteAllText(outputPath, sb.ToString());
}


Any suggestion or code snippets?

• In which way you think it's not optimized?
– Groo
Nov 28 '13 at 13:38

## 1 Answer

1. StreamReader is IDisposable and therefor the use of it should be wrapped in a using block like this:

using (TextReader tr = new StreamReader(csvFileName))
{
...
}

2. You are reading all files into memory and then writing it out. While this probably has a fairly good performance as it's first performing a bunch of (probably) sequential reads of all the files and then one big block write you could consider writing it out as you read it:

using (var writer = new StreamWriter(outputPath))
{
foreach (var csvFileName in csvFileNames)
{
using (var reader = new StreamReader(csvFileName))
{
string headers = tr.ReadLine();

if (!hasHeader)
{
writer.WriteLine(headers);
hasHeader = true;
}

writer.Write(reader.ReadToEnd());
}
}
}


If the files are very big you even might want to consider reading the input line by line and writing it to the output line by line. This will be less performant but also use much less memory. In the end it's a trade off between speed and memory consumption (as it often is in software development).

The other advantage with the write as you go method is that you open the output file first - so if there is a problem with writing to the target you will know before you do any work at all.

• Wouldn't it be faster to first read everything to the memory and then start with the I/O operations? - or did I misunderstand you?
– Max
Nov 28 '13 at 8:05
• @Max: Yes, I thought that's what I said: You are reading all files into memory and then writing it out. While this probably has a fairly good performance as it's first performing a bunch of (probably) sequential reads of all the files and then one big block write Nov 28 '13 at 8:19
• o, my bad then :)
– Max
Nov 28 '13 at 8:20
• @ChrisWue Thank you very much for your time and input I will do that Nov 29 '13 at 6:05