Optimal way to output a dictionary

Question

---

Finished Someone helped me see that i was framenting the output by writing to each file inside the for loop instead i needed to have one big for loop for each file so that the harddrive didnt have to move its head everytime

https://stackoverflow.com/questions/9913788/outputing-dictionary-optimally

---

I have 4 Dictionarys that contain 800k strings with 200 to 6000 characters. when i load it into memory it takes up about 11 gigs of memory. it is taking me 2 minutes to parse the data and 2 minutes to output the data. Is there anyway to output the data faster than what I am using below? I am only getting 20-31 MB per second disk IO and I know the hard drive can do 800ish

var hash1 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash2 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash3 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);
var hash4 = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);

...

foreach (var me in mswithfilenames)
{
    filename = me.Key.ToString();
    string filenamef = filename + "index1";
    string filenameq = filename + "index2";
    string filenamefq = filename + "index3";
    string filenameqq = filename + "index4";

    StreamWriter sw = File.AppendText(filenamef);
    StreamWriter sw2 = File.AppendText(filenameq);
    StreamWriter swq = File.AppendText(filenamefq);
    StreamWriter sw2q = File.AppendText(filenameqq);

    for (i = 0; i <= totalinhash; i++)
    {
        if (hashs1[i].ContainsKey(filenamef))
        {
            sw.Write(hashs1[i][filenamef]);
        }
        if (hashs2[i].ContainsKey(filenameq))
        {
            sw2.Write(hashs2[i][filenameq]);
        }
        if (hashs3[i].ContainsKey(filenamefastaq))
        {
            swq.Write(hash4[i][filenamefastaq]);
        }

        if (hash4[i].ContainsKey(filenameqq))
        {
            sw2q.Write(hash4[i][filenameqq]);
        }
    }
    sw.Close();
    sw2.Close();
    sw3.Close();
    sw4.Close();
    swq.Close();
    sw2q.Close();
}

Answer 1

One way to speed up the lookups is to use TryGetValue() instead of ContainsKey() and then the indexer. So, for example:

if (hashs1[i].ContainsKey(filenamef))
{
    sw.Write(hashs1[i][filenamef]);
}

would become:

string value;
if (hashs1[i].TryGetValue(filenamef, out value))
{
    sw.Write(value);
}

But it's hard to tell how much would that help you.

Another thing that might help you would be combining all the hashes into one that contains object that contains all the inner hashes. Something like:

class Hashes // or struct?
{
    public Dictionary<string, string> hash1 { get; set; }
    public Dictionary<string, string> hash2 { get; set; }
    public Dictionary<string, string> hash3 { get; set; }
    public Dictionary<string, string> hash4 { get; set; }
}

…

var mainHash = new Dictionary<int, Hashes>(f.Count + 2);

Combining the two approaches lowers the number of potential lookups per iteration from 16 to 5.

Also, unrelated to performance, but you should name your variables by their contents, not implementation. I think hash1 is a very bad name.

Answer 2

Finished Someone helped me see that i was framenting the output by writing to each file inside the for loop instead i needed to have one big for loop for each file so that the harddrive didnt have to move its head everytime

https://stackoverflow.com/questions/9913788/outputing-dictionary-optimally

Answer 3

Without more context, I can't really be sure any of my advice is applicable, but, here's a shot:

First off, consider: can you output during the parsing phase? If i only increases during the parse, at least for a given filename, and as soon as you would store it into hashX[i][filenameX] you know that it will be output there: you might as well just write it. This would cut down the number of key look-ups to 0, as well as reduce the amount of memory you're allocating during the parse (assuming you don't use the hashes for anything else later). If that doesn't apply...

Secondly, consider flipping the first- and second-level keys, e.g.:

var hash1 = new Dictionary<string, Dictionary<int, string>>(f.Count + 2);
...
var hash1_for_filenamef = hash1[filenamef];
for (var i = 0; i < totalinhash; i++) {
    string value;
    if (hash1_for_filenamef.TryGetValue(i, out value))
        sw.Write(value);
}

If you can do this, you might also get a bit more out of it under the special circumstance that the TryGetValue would be returning false a large percentage of the time, by not doing any key look-ups for the second level at all, but rather sorting the list directly:

var hash1 = new Dictionary<string, Dictionary<int, string>>(f.Count + 2);
...
var sorted_hash1_for_filenamef = hash1[filenamef]
    .Where(pair => pair.Key <= totalinhash) // compatibility with your code -- maybe totalinhash is guaranteed >= i?  If so, remove this
    .OrderBy(pair => pair.Key);
foreach (var pair in sorted_hash1_for_filenamef)
    sw.Write(pair.Value);

Answer 4

I addition to recommended by other members I would also:

1) Replace

var hashN = new Dictionary<int, Dictionary<string, string>>(f.Count + 2);

with

var hashN = new Dictionary<string, string>[f.Count + 2];

since it seems that all the integer keys from 0 to totalinhash are always present in dictionary.

2) Try buffering output using BufferedStream or maybe even StringBuilder to write in larger chunks.