# Finding Matches and writing results to file C#

I have a folder that contains around 1000 text files, their sizes varies from few KBs up to 500 MB (Total size is around 60 GB).

I have a text file that contains a 166k lines, each line contains a set of numbers (length up to 11), what I'm doing here is to go through each file in that folder and try to find a match between the 166k and those files, then store the line from the file and print that out to a file.

My approach is working but there are several issues:

• It is not fast (taking a lot of time to go through the files)
• Memory consumption, I believe that is because I'm storing everything into a list of strings and then print all of it at once

Please take a look at my code below, and let me know if I can enhance it and improve it in any possible way.

    static void Main(string[] args)
{
char splitter = '\u0001';

//166k file
string path = "Z:\\subid.txt";

//Destination file
string RxPath = "Z:\\matched.txt";

List<string> RxClaims = new List<string>();

string[] lineObject;
int count = 0;

//folder location (contains 1000 text files)
string folderPath = "Z:\\rawfiles";

foreach (string file in Directory.EnumerateFiles(folderPath))
{
Console.WriteLine("Processing " + file);
{
lineObject = line.Split(splitter);

//Check if that value is equal to any of the numbers in the 166k, if so store in the list to print out later
if(subid.Contains(lineObject[14]))
{
count++;
}

}
}

File.WriteAllLines(RxPath, RxClaims);
Console.WriteLine("Done, Number of Claims" + count);

}


Some things to speed this process up

1. subid is only used if it contains a certain string. Use a hashset or dictionary where each lookup is O(1) instead of O(n) ; which is in this case O(166K) in worst case
2. EDIT: StreamReader and File.ReadLines actually behave the same
3. Start using using for handling input/output; it makes sure that object are garbage collected one it's out of scope. Using declares a scope for these.
4. Personally I put all configuration on top
5. You could also use StreamWriter to write the file, which is just a handle (same principle as the StreamReader, but then for writing)

With only minimal adjustments, this should be significantly faster.

Improved code

public static void Main(string[] args)
{
// file paths
string path = "Z:\\subid.txt";      //166k file
string RxPath = "Z:\\matched.txt";  //Destination file
string folderPath = "Z:\\rawfiles"; //folder location (contains 1000 text files)

char splitter = '\u0001';

// subid is ONLY used to check if it contains something. Make it a hashset
List<string> RxClaims = new List<string>();

string[] lineObject;
int count = 0;

foreach (string file in Directory.EnumerateFiles(folderPath))
{
Console.WriteLine("Processing " + file);

// use a streamreader to go through files!
{
lineObject = line.Split(splitter);

//Check if that value is equal to any of the numbers in the 166k, if so store in the list to print out later
if (subid.Contains(lineObject[14]))
{
count++;
}
}
}

File.WriteAllLines(RxPath, RxClaims);
Console.WriteLine("Done, Number of Claims" + count);


• btw, File.ReadLines <- reads file line by line, so its same as StreamReader + ReadLine – tym32167 Sep 20 '16 at 9:36
• @tym32167 you are right! I saw theFile-namespace and didn't look further. I have learned something, thnx! – Ernst Sep 20 '16 at 10:10