1
\$\begingroup\$

Finished Someone helped me see that i was framenting the output by writing to each file inside the for loop instead i needed to have one big for loop for each file so that the harddrive didnt have to move its head everytime

https://stackoverflow.com/questions/9913788/outputing-dictionary-optimally


I have 4 Dictionarys that contain 800k strings with 200 to 6000 characters. when i load it into memory it takes up about 11 gigs of memory. it is taking me 2 minutes to parse the data and 2 minutes to output the data. Is there anyway to output the data faster than what I am using below? I am only getting 20-31 MB per second disk IO and I know the hard drive can do 800ish

var hash1 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash2 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash3 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash4 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);

...

foreach (var me in mswithfilenames)
{
    filename = me.Key.ToString();
    string filenamef = filename + "index1";
    string filenameq = filename + "index2";
    string filenamefq = filename + "index3";
    string filenameqq = filename + "index4";

    StreamWriter sw = File.AppendText(filenamef);
    StreamWriter sw2 = File.AppendText(filenameq);
    StreamWriter swq = File.AppendText(filenamefq);
    StreamWriter sw2q = File.AppendText(filenameqq);

    for (i = 0; i <= totalinhash; i++)
    {
        if (hashs1[i].ContainsKey(filenamef))
        {
            sw.Write(hashs1[i][filenamef]);
        }
        if (hashs2[i].ContainsKey(filenameq))
        {
            sw2.Write(hashs2[i][filenameq]);
        }
        if (hashs3[i].ContainsKey(filenamefastaq))
        {
            swq.Write(hash4[i][filenamefastaq]);
        }

        if (hash4[i].ContainsKey(filenameqq))
        {
            sw2q.Write(hash4[i][filenameqq]);
        }
    }
    sw.Close();
    sw2.Close();
    sw3.Close();
    sw4.Close();
    swq.Close();
    sw2q.Close();
}
\$\endgroup\$
  • 2
    \$\begingroup\$ Have you profiled your code? What is the bottleneck, writing to the disk or checking the hash tables? \$\endgroup\$ – svick Mar 28 '12 at 19:23
  • \$\begingroup\$ Can you use a database here??? \$\endgroup\$ – Leonid Mar 28 '12 at 19:49
  • \$\begingroup\$ yes i have profiled my code the bottleneck seems to be poor disk io \$\endgroup\$ – user12202 Mar 28 '12 at 19:59
  • \$\begingroup\$ @Leonid what do you mean: "can you use a database here"? \$\endgroup\$ – user12202 Mar 28 '12 at 19:59
  • \$\begingroup\$ @caseyr547, you are working with a lot of data here. How are you using it? Databases are pretty good at storing, manipulating and retrieving data. Depending on what you are doing, a database could help. \$\endgroup\$ – Leonid Mar 28 '12 at 20:28
2
\$\begingroup\$

One way to speed up the lookups is to use TryGetValue() instead of ContainsKey() and then the indexer. So, for example:

if (hashs1[i].ContainsKey(filenamef))
{
    sw.Write(hashs1[i][filenamef]);
}

would become:

string value;
if (hashs1[i].TryGetValue(filenamef, out value))
{
    sw.Write(value);
}

But it's hard to tell how much would that help you.

Another thing that might help you would be combining all the hashes into one that contains object that contains all the inner hashes. Something like:

class Hashes // or struct?
{
    public Dictionary<string, string> hash1 { get; set; }
    public Dictionary<string, string> hash2 { get; set; }
    public Dictionary<string, string> hash3 { get; set; }
    public Dictionary<string, string> hash4 { get; set; }
}

…

var mainHash = new Dictionary<int, Hashes>(f.Count + 2);

Combining the two approaches lowers the number of potential lookups per iteration from 16 to 5.

Also, unrelated to performance, but you should name your variables by their contents, not implementation. I think hash1 is a very bad name.

\$\endgroup\$
  • \$\begingroup\$ yes i tried trygetvalue from a post in stackoverflow.com/questions/9913788/… but there was no change in performance. \$\endgroup\$ – user12202 Mar 28 '12 at 19:57
  • \$\begingroup\$ also i do agree hash1 is a terrible name but i had to change it to post it online :D \$\endgroup\$ – user12202 Mar 28 '12 at 20:00
1
\$\begingroup\$

Finished Someone helped me see that i was framenting the output by writing to each file inside the for loop instead i needed to have one big for loop for each file so that the harddrive didnt have to move its head everytime

https://stackoverflow.com/questions/9913788/outputing-dictionary-optimally

\$\endgroup\$
0
\$\begingroup\$

Without more context, I can't really be sure any of my advice is applicable, but, here's a shot:

First off, consider: can you output during the parsing phase? If i only increases during the parse, at least for a given filename, and as soon as you would store it into hashX[i][filenameX] you know that it will be output there: you might as well just write it. This would cut down the number of key look-ups to 0, as well as reduce the amount of memory you're allocating during the parse (assuming you don't use the hashes for anything else later). If that doesn't apply...

Secondly, consider flipping the first- and second-level keys, e.g.:

var hash1 = new Dictionary<string, Dictionary<int, string>>(f.Count + 2);
...
var hash1_for_filenamef = hash1[filenamef];
for (var i = 0; i < totalinhash; i++) {
    string value;
    if (hash1_for_filenamef.TryGetValue(i, out value))
        sw.Write(value);
}

If you can do this, you might also get a bit more out of it under the special circumstance that the TryGetValue would be returning false a large percentage of the time, by not doing any key look-ups for the second level at all, but rather sorting the list directly:

var hash1 = new Dictionary<string, Dictionary<int, string>>(f.Count + 2);
...
var sorted_hash1_for_filenamef = hash1[filenamef]
    .Where(pair => pair.Key <= totalinhash) // compatibility with your code -- maybe totalinhash is guaranteed >= i?  If so, remove this
    .OrderBy(pair => pair.Key);
foreach (var pair in sorted_hash1_for_filenamef)
    sw.Write(pair.Value);
\$\endgroup\$
  • \$\begingroup\$ thanks for the advice the problem with outputing the files durring the parsing is that the files are random so i dont know which file will be written to next making the disk out very slow as the head of the hard drive has to move back and forther constantly between any of 4 files...i'd really like to ouput it but i dont know any way to do nonblocking io \$\endgroup\$ – user12202 Mar 29 '12 at 19:04
0
\$\begingroup\$

I addition to recommended by other members I would also:

1) Replace

var hashN = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);

with

var hashN = new Dictionary<string, string>[f.Count + 2];

since it seems that all the integer keys from 0 to totalinhash are always present in dictionary.

2) Try buffering output using BufferedStream or maybe even StringBuilder to write in larger chunks.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy