# Neural Network Simulator with OpenMP

I wrote a simple neural network simulator (the biophysical kind) from scratch, and was hoping to get some feedback on how I can speed things up, or any C++ / compilation best practices that I can improve on.

The code is at this repository.

Main Problem:

OpenMP doesn't seem to be conferring speedup.

The performance-critical section of the code is in src/networks/spikingnet.cpp, but for additional context, see the rest of the code in the repository.

#pragma omp parallel for
for (size_t li=0; li < nL; ++li) {
SpikingLayer *layer = net->layers[li];
Stim *stim = rs->stimuli[li];
boolvec doSpike = stim->yield();
conn_vec pre_arr = net->pre[li];
updateLayer(layer, pre_arr, doSpike, t);
recordSpikes(results.mutable_spikes(li), layer, i);
}

// update transmission & stdp
#pragma omp parallel for
for (size_t li=0; li < nL; ++li) {
SpikingLayer *layer = net->layers[li];

// transmission
for (SpikingConnection *conn : net->post[li]) {
for (SpikingSynapse* syn : conn->synapses) {
updateTransmission(syn, layer->units[syn->s]);
}
}

// STDP
for (SpikingConnection *conn : net->pre[li]) {
SpikingLayer *source = net->layers[conn->s];
SpikingLayer *target = net->layers[conn->t];
if (conn->stdp_enabled) {
#pragma omp parallel for
for (SpikingSynapse* syn : conn->synapses) {
}
}
} // end STDP

} // end for


GProf Trace

Architectural details:

The network consists of L=9 "layers", each with 100-900 units. There are on the order of L^2 "connections" (bundles of synapses between layers), and 2000 synapses per "connection" (synapses are sparse).

During each update cycle, all the layers (neurons) are updated (conditioned on connections), and then all the connections (synapses) are updated (conditioned on layers). That is to say, layer updating is independent conditioned on connections, and connection updating is independent conditioned on layers.

Given that there are so many neurons and synapses, the program naturally spends most of its time updating layers and synapses (line 133). I thought that using OpenMP over the per-layer loop, or even per-neuron/synapse, would really speed things up, but that does not seem to be the case at all.

If it's of any interest to the reader, my machine has a 4.0Ghz CPU with 8 cores. 12K steps run in approximately 30 seconds on either single-threaded or OpenMP-enabled builds.

I'm aware that I should be using smart pointers, but memory management is fairly straightforward in this simple simulator, so I chose to live dangerously.

General C++ programming style tips / pointing out my bad practices are also welcome!

• Thank you for adding the code. I have retracted my close vote, and I hope you get some good answers soon - I know we have excellent C++ reviewers. – user34073 Apr 20 '15 at 4:24
• I would leave only this as a parallel loop: for (SpikingSynapse* syn : conn->synapses) {updateSTDP(syn, source->units[syn->s], target->units[syn->t]);} I think what happens is: the outer loop gets paralleled and there is not enough threads left in the OpneCV's opinion for the innermost loop (which as you said the most resource-hungry one) – cha Apr 20 '15 at 6:20

I would hoist as many parallel directives to the outermost loop as possible, and avoid having nested parallel directives. You may have to switch from using omp parallel for to just using omp parallel at the outermost level, then having some logic to explicitly decide which part of the data each thread works on depending on its thread number.