I have some code that reads a file and then does some parallel processing of the data. There are millions of lines in the file and this section of the code is the bottleneck in my program. Any information on how to improve processing times or any other suggestions to improve the code (e.g. error processing, string manipulation, or anything to increase speed) is appreciated. I'm new to parallel processing in .NET.
// Read in the source and target file and start solving the strongest path problem.
try
{
ConcurrentBag<string> strongestPaths = new ConcurrentBag<string>();
String[] allFileLines = null;
int maxSize = (int)Math.Floor((double)(Int32.MaxValue / 10000));
// Allocate memory for the source and target file array
allFileLines = new String[maxSize];
using (StreamReader sr = File.OpenText(sourceTargetArg))
{
// Find the strongest path for each of the target nodes
int x = 0;
while (!sr.EndOfStream)
{
allFileLines[x] = sr.ReadLine();
x++;
if (x == maxSize || sr.EndOfStream)
{
Parallel.For(0, allFileLines.Length, (i, loopState) =>
{
if (allFileLines[i] != null)
{
try
{
Node targetNode = getTargetNode(graph, allFileLines[i]);
// If the Target Node was not found, do not process this node, and continue
if (targetNode != null)
{
var path = calculator.GetPath(targetNode);
String targetPath = String.Empty;
for (int j = 0; j < path.Count - 1; j++)
{
targetPath += path[j].Name + " " + path[j].getEdgeWeight(path[j + 1]) + " ";
}
targetPath = sourceNode.Name + " " + targetNode.Name + " " + path[path.Count - 1].PathWeight + ": " + targetPath + targetNode.Name;
strongestPaths.Add(targetPath);
}
}
catch (Exception e)
{
// if this fails continue processing the rest of the target nodes, but warn the user
Console.WriteLine(e.Message);
}
}
else
{
// Break from the parallel loops when current loops have compeleted
// This is to stop processing the the rest of the array
loopState.Break();
}
});
// Start processing the next chuck of data
x = 0;
Array.Clear(allFileLines, 0, allFileLines.Length);
}
}
}
}
catch (OutOfMemoryException)
{
Console.WriteLine("Exception: Not enough memory to read in the source and target file.");
Console.WriteLine();
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
Console.WriteLine();
}
finally
{
// Release the memory in case
if (allFileLines != null)
{
Array.Clear(allFileLines, 0, allFileLines.Length);
allFileLines = null;
}
// Force garbage collection
GC.Collect();
}
EDIT: Added the function for getTargetNode. It is basically returning the node in my graph object. The 'calculator.GetPath' function gets the path from a target node to the source node to print out the path and weights. The paths of all the target nodes read into the source file are then written to an output file.
My graph is basically:
public class Graph
{
internal ConcurrentDictionary<string, Node> Nodes { get; private set; }
with some functions. The program is for finding the fastest path between a source node and some target nodes on the graph. The slow part of the program above is just looping through the target nodes and displaying the path.
public static Node getTargetNode(Graph graph, string targetLine)
{
Node target = null;
if (!String.IsNullOrEmpty(targetLine))
{
try
{
// Verify that the target node is a node in the graph
target = graph.getNode(targetLine);
}
catch (KeyNotFoundException e)
{
throw new KeyNotFoundException("Invalid Input: The Target Node, " + targetLine.Trim() + ", in the Source and Target file is not a node in the graph. ", e);
}
catch (Exception e)
{
throw new Exception("Invalid Input: The Target Node, " + targetLine.Trim() + ", in the Source and Target file is invalid: " + e.Message, e);
}
}
else
{
throw new Exception("Invalid Input: The Target Node in the Source and Target file is null or empty.");
}
return target;
}
getTargetNode
look like? What type iscalculator
? A plain text description of what the code is supposed to be doing would also be helpful. \$\endgroup\$ – RobH Mar 27 '16 at 19:53StringBuilder
will reduce GC burden, if you have millions of strings. But, the more important question is, how slow isgraph.getNode
, actually? What problem are you trying to solve?Parallel.For
only provides a (theoretical) constant-time improvement (i.e. 4x faster if you have 4 cores, at best). In big-O notation, this speed-up is completely irrelevant, especially if this is a programming assignment. If this is a real world app, otoh, you will typically concentrate on providing a progress bar and making the operation run in background. \$\endgroup\$ – Groo Mar 29 '16 at 8:23