Can this be done better and more efficient way (speed/ memory)?
Your code iterates once over the input, Distinct
internally creates a set, of which you then create a dictionary (redundant work).
You then copy the keys of the dictionary to a list to iterate over them (redundant work).
You then iterate over all keys, and for each, iterate over the input again (redundant work).
Then you iterate over the dictionary again (redundant work).
This code has almost more redundant operations than lines.
This can be vastly simplified, both conceptually and concerning runtime. Unfortunately, the resulting code will be slightly longer and still contain one redundancy since C#’s dictionary implementation has a fatal error in its interface: it doesn’t allow querying and updating a value at the same time.
string[] source = { "test1", "test2", "test3", "test4", "test1", "test1", "test3" };
var frequencies = new Dictionary<string, int>();
string highestWord = null;
int highestFreq = 0;
foreach (string word in source) {
int freq;
frequencies.TryGetValue(word, out freq);
freq += 1;
if (freq > highestFreq) {
highestFreq = freq;
highestWord = word;
}
frequencies[word] = freq;
}
Console.WriteLine(highestWord);
What if file size is 10GB. How would you do it differently from my approach?
If you expect comparably few different word (say, less than 100.000), use the same approach as above, just don’t load the whole input at once, instead, do it in chunks. You can also process chunks in parallel, with each thread working on its own dictionary, and afterwards you merge those dictionaries in one final step.
If you expect that almost every word in the input is distinct (not realistic with natural language words, but if your “words” are generated strings, this could happen) then the frequency dictionary could become quite large in size and might require special treatment. But this is an extreme scenario.
cat file.txt | sort | uniq -c | sort -rn | head -n1
\$\endgroup\$