4
\$\begingroup\$

I have a folder that contains around 1000 text files, their sizes varies from few KBs up to 500 MB (Total size is around 60 GB).

I have a text file that contains a 166k lines, each line contains a set of numbers (length up to 11), what I'm doing here is to go through each file in that folder and try to find a match between the 166k and those files, then store the line from the file and print that out to a file.

My approach is working but there are several issues:

  • It is not fast (taking a lot of time to go through the files)
  • Memory consumption, I believe that is because I'm storing everything into a list of strings and then print all of it at once

Please take a look at my code below, and let me know if I can enhance it and improve it in any possible way.

    static void Main(string[] args)
    {
        char splitter = '\u0001';

        //166k file
        string path = "Z:\\subid.txt";

        //Destination file
        string RxPath = "Z:\\matched.txt";

        List<string> subid = File.ReadAllLines(path).ToList();
        List<string> RxClaims = new List<string>();

        string[] lineObject;
        int count = 0;

        //folder location (contains 1000 text files)
        string folderPath = "Z:\\rawfiles";

        foreach (string file in Directory.EnumerateFiles(folderPath))
        {
            Console.WriteLine("Processing " + file);
            foreach (string line in File.ReadLines(file))
            {
                lineObject = line.Split(splitter);

                //Check if that value is equal to any of the numbers in the 166k, if so store in the list to print out later
                if(subid.Contains(lineObject[14]))
                {
                    count++;
                    RxClaims.Add(line);
                }

            }
        }

        File.WriteAllLines(RxPath, RxClaims);
        Console.WriteLine("Done, Number of Claims" + count);

        Console.ReadLine();
    }
\$\endgroup\$
3
\$\begingroup\$

Some things to speed this process up

  1. subid is only used if it contains a certain string. Use a hashset or dictionary where each lookup is O(1) instead of O(n) ; which is in this case O(166K) in worst case
  2. EDIT: StreamReader and File.ReadLines actually behave the same
  3. Start using using for handling input/output; it makes sure that object are garbage collected one it's out of scope. Using declares a scope for these.
  4. Personally I put all configuration on top
  5. You could also use StreamWriter to write the file, which is just a handle (same principle as the StreamReader, but then for writing)

With only minimal adjustments, this should be significantly faster.

Improved code

public static void Main(string[] args)
{
    // file paths
    string path = "Z:\\subid.txt";      //166k file
    string RxPath = "Z:\\matched.txt";  //Destination file
    string folderPath = "Z:\\rawfiles"; //folder location (contains 1000 text files)

    char splitter = '\u0001';

    // subid is ONLY used to check if it contains something. Make it a hashset     
    HashSet<string> subid = new HashSet<string>(File.ReadAllLines(path));
    List<string> RxClaims = new List<string>();

    string[] lineObject;
    int count = 0;


    foreach (string file in Directory.EnumerateFiles(folderPath))
    {
        Console.WriteLine("Processing " + file);

        // use a streamreader to go through files!
        using (StreamReader reader = new StreamReader(file))
        {
            string line = reader.ReadLine();
            lineObject = line.Split(splitter);

            //Check if that value is equal to any of the numbers in the 166k, if so store in the list to print out later
            if (subid.Contains(lineObject[14]))
            {
                count++;
                RxClaims.Add(line);
            }
        }
    }

    File.WriteAllLines(RxPath, RxClaims);
    Console.WriteLine("Done, Number of Claims" + count);

    Console.ReadLine();
}
| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ I just changed from List to HashSet and replaced File.WriteAllLines with StreamWriter. That made a difference :) Appreciate it \$\endgroup\$ – WT86 Sep 19 '16 at 2:03
  • \$\begingroup\$ Cool! Always nice to see that choosing the right datastructure can have a (massive) impact. Although at one point the I/O will always be the bottleneck.. (hint at SSDs) \$\endgroup\$ – Ernst Sep 19 '16 at 19:00
  • \$\begingroup\$ Hardware performance is another bottleneck for me, running a VM with 4 cores and 8 GB which is OK, but the IO is the real issue here. Thank you much again! \$\endgroup\$ – WT86 Sep 19 '16 at 19:34
  • \$\begingroup\$ btw, File.ReadLines <- reads file line by line, so its same as StreamReader + ReadLine \$\endgroup\$ – tym32167 Sep 20 '16 at 9:36
  • \$\begingroup\$ @tym32167 you are right! I saw theFile-namespace and didn't look further. I have learned something, thnx! \$\endgroup\$ – Ernst Sep 20 '16 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.