C++ Feed-Forward Neural Network

Question

After a few days of reading articles, watching videos and bugging my head around neural networks, I have finally managed to understand it just so I could write my own feed-forward implementation in C++.

It does have some scratch back-propagation functionality, but it needs further work (not done yet).

Here's my code, I would like you to point out any bad practices, tips, you know :)

main.cpp

#include "neural-net.hpp"

int main(int argc, char **argv)
{
    srand(time(NULL));

    /* Topology: x-y-z-...-n where a is the input layer and n is the output layer */
    /* In this case: 1 input layer with 2 neurons, 1 hidden layer with 3 neurons and an output layer with 1 neuron */
    std::vector<unsigned> vecTopology = { 2, 3, 1 };
    NeuralNet net(vecTopology, false);

    /* Set the input values and expected results for back-propagation (not finished!) */
    std::vector<double> vecInputs(vecTopology[0], 1);
    std::vector<double> vecExpected(3, 0);

    std::cout << "Inputs: ";
    for (int i = 0; i < vecInputs.size(); i++) {
        std::cout << vecInputs[i] << " ";
    }
    std::cout << "\n\n";

    net.feedForward(vecInputs);
    net.backPropagate(vecExpected);

    net.status();
}

neural-net.hpp

#ifndef NEURALNET_HPP
#define NEURALNET_HPP

#include "Neuron.hpp"

class NeuralNet
{
public:
    NeuralNet(const std::vector<unsigned> &, bool = false);
    void status();
    void setWeight(unsigned, unsigned, unsigned, double);
    void feedForward(const std::vector<double> &);
    void backPropagate(const std::vector<double> &);
    std::vector<double> getOutput();

private:
    std::vector<Layer> vecLayers;
    bool useBias;
};

#endif

neural-net.cpp

#include "neural-net.hpp"

NeuralNet::NeuralNet(const std::vector<unsigned> &vecTopology, bool useBias)
{
    this->useBias = useBias;

    /* 'Build' the network based on the topology */
    for (unsigned l = 0; l < vecTopology.size(); l++) {
        this->vecLayers.push_back(Layer());

        unsigned nAxons = (l == vecTopology.size() - 1) ? 0 : vecTopology[l + 1];
        for (unsigned n = 0; n < vecTopology[l] + (this->useBias) ? 1 : 0; n++) {
            this->vecLayers[l].push_back(Neuron(n, nAxons));
        }

        /* We do not want the bias neuron in the output layer, pop it back :) */
        if (this->useBias && l == vecTopology.size() - 1)
            this->vecLayers[l].pop_back();
    }
}

void NeuralNet::status()
{
    for (unsigned l = 0; l < this->vecLayers.size(); l++) {
        std::cout << "\nLayer " << l;

        Layer &vecLayer = this->vecLayers[l];
        for (unsigned n = 0; n < vecLayer.size(); n++) {
            std::cout << "\n    Neuron " << n << "\n";

            std::vector<Axon> vecAxons = vecLayer[n].getAxons();
            for (unsigned w = 0; w < vecAxons.size(); w++) {
                std::cout << "        Axon " << w << " weight: " << vecAxons[w].weight << "\n";
                std::cout << "        Axon " << w << " output: " << vecLayer[n].getOutput() << "\n\n";
            }

            if (l == this->vecLayers.size() - 1)
                std::cout << "        Output: " << vecLayer[n].getOutput() << "\n\n";
        }
    }
}

void NeuralNet::setWeight(unsigned layer, unsigned neuron, unsigned axon, double weight)
{
    this->vecLayers[layer][neuron].setWeight(axon, weight);
}

void NeuralNet::feedForward(const std::vector<double> &vecInputs)
{
    /* Set outputs of the input layer's neurons to the user's inputs */
    for (unsigned n = 0; n < vecInputs.size(); n++) {
        this->vecLayers[0][n].setOutput(vecInputs[n]);
    }

    /* Feed-forward! */
    for (unsigned l = 1; l < this->vecLayers.size(); l++) {
        Layer &vecLayer = this->vecLayers[l];
        for (unsigned n = 0; n < vecLayer.size(); n++) {
            vecLayer[n].feedForward(this->vecLayers[l - 1]);
        }
    }
}

void NeuralNet::backPropagate(const std::vector<double> &vecExpected)
{
    /* This needs to be finished, as I am thinking of the best way to implement back-propagation */
    Layer &vecLayer = this->vecLayers.back();
    for (unsigned n = 0; n < vecLayer.size(); n++) {
        vecLayer[n].backPropagate(this->vecLayers[this->vecLayers.size() - 2], vecExpected[n]);
    }
}

std::vector<double> NeuralNet::getOutput()
{
    /* I think this function is self-explanatory */
    std::vector<double> vecOutputs;

    Layer &vecLayer = this->vecLayers.back();
    for (unsigned n = 0; n < vecLayer.size(); n++) {
        vecOutputs.push_back(vecLayer[n].getOutput());
    }

    return vecOutputs;
}

neuron.hpp

#ifndef NEURON_HPP
#define NEURON_HPP

#include <iostream>
#include <vector>
#include <ctime>
#include <cmath>
#include <random>

class Neuron;
typedef std::vector<Neuron> Layer;

struct Axon
{
    double weight;
    double deltaWeight; 
    Axon(double weight = 0): weight(weight) {}
};

class Neuron
{
public:
    Neuron(unsigned, unsigned);
    std::vector<Axon> getAxons();
    void setOutput(double);
    double getOutput();
    void setWeight(unsigned, double);
    void feedForward(const Layer &);
    void backPropagate(Layer &, double);

private:
    unsigned index;
    std::vector<Axon> vecAxons;
    double output;
    double outputSum;

    double randomWeight();
    double sigmoid(double);
    double sigmoidDerivative(double);
};

#endif

neuron.cpp

#include "neuron.hpp"

Neuron::Neuron(unsigned index, unsigned nAxons)
{
    this->index = index;
    this->output = 0.0;

    /* Axons are just connections between neurons, each given a random weight between 0.0 and 1.0 as a starter */
    for (unsigned a = 0; a < nAxons; a++) {
        this->vecAxons.push_back(Axon(randomWeight()));
    }
}

double Neuron::randomWeight()
{
    return rand() / double(RAND_MAX);
}

std::vector<Axon> Neuron::getAxons()
{
    return this->vecAxons;
}

void Neuron::setOutput(double output)
{
    this->output = output;
}

double Neuron::getOutput()
{
    return this->output;
}

void Neuron::setWeight(unsigned axon, double weight)
{
    this->vecAxons[axon].weight = weight;
}

void Neuron::feedForward(const Layer &vecPreviousLayer)
{
    this->outputSum = 0.0;

    /* Calculate the sum of inputs * weights going to the neuron and pass it through the transfer function... */
    for (unsigned n = 0; n < vecPreviousLayer.size(); n++) {
        this->outputSum += vecPreviousLayer[n].output * vecPreviousLayer[n].vecAxons[this->index].weight;
    }

    /* ... which is sigmoid in my case */
    this->output = sigmoid(this->outputSum);
}

void Neuron::backPropagate(Layer &vecPreviousLayer, double expected)
{
    /* This is NOT considered done in any way, or should I say it's working only for output <-> outer hidden layer */
    double error = expected - this->output;
    double deltaOutputSum = sigmoidDerivative(this->outputSum) * error;

    std::cout << "Testing some back-propagation stuff, ignore next 2 lines\n";
    std::cout << "Margin of error: " << error << "\n";
    std::cout << "Delta output sum: " << deltaOutputSum << "\n\n";

    for (unsigned n = 0; n < vecPreviousLayer.size(); n++) {
        double output = vecPreviousLayer[n].output;  
        vecPreviousLayer[n].vecAxons[this->index].deltaWeight = deltaOutputSum * output;
    }
}

double Neuron::sigmoid(double t)
{
    return (1 / (1 + pow(exp(1.0), -t)));
}

double Neuron::sigmoidDerivative(double t)
{
    return (pow(exp(1.0), t) / pow(1 + pow(exp(1.0), t), 2));
}

Answer 1

So here my 2 cents.

1. I think all the this-> makes it hard to read. Also this is only necessary when there might be name conflicts, which are not there.

2. The randomWeight function should be a member of the Axon. Therefore you can initialize the axon vector simply by

   vecAxons = std::vector<Axon>(nAxon);
   

3. Even if you do not go for that, you should alwas reserve memory if you know the size of an array beforehand. Therewith you avoid reallocations.

   vecAxons.reserve(nAxons);
   

4. As you use C++ you should utilize the random library rather than rand. When you want to develop serious models rand is not your friend.

5. You should try to use range based loops, which improve readability. For example

   /* Calculate the sum of inputs * weights going to the neuron and pass it through the transfer function... */
   for (unsigned n = 0; n < vecPreviousLayer.size(); n++) {
       this->outputSum += vecPreviousLayer[n].output * vecPreviousLayer[n].vecAxons[this->index].weight;
   }
   

   can be written as

   /* Calculate the sum of inputs * weights going to the neuron and pass it through the transfer function... */
   for (const Neuron& sourceNeuron : vecPreviousLayer) {
       outputSum += sourceNeuron.output * sourceNeuron.vecAxons[index].weight;
   }
   

6. Whenever possible avoid pow() of natural numbers. I know it is tedious, but pow is just incredibly slow, especially stuff like pow(x,2) vs x*x

Answer 2

Do not use home brew stuff like axon or neuron. You are making life so hard for yourself. Instead just use standard matrix math.

In GNU octave a very simple feedforward net would look something like this:

function r = sigmoid(z)
    r = 1 ./ (1 + exp(-z));
end

x = [[0, 0, 1]; [0, 1, 1]; [1, 0, 1]; [1, 1, 1]]; 
y = [0, 1, 1, 0]';
Wih = rand(4, 3) * 2 - 1
Who = rand(1, 4) * 2 - 1
learning_rate = 0.9;

for i = 1:100000
    % forward propagate
    input = x;
    hidden = sigmoid(input * Wih');
    output = sigmoid(hidden * Who');

    % backpropagate errors
    ho_err = (y - output) .* (output .* (1 - output));
    Who += learning_rate * (ho_err' * hidden);

    ih_err = (ho_err * Who) .* (hidden .* (1 - hidden));
    Wih += learning_rate * (ih_err' * input);
end

output

You can easily convert this to a language of your choice and use corresponding matrix math library like px4 matrix (very light weight C++) or Eigen or something else. It makes it possible for you to reuse already written algorithms for weight update (where weights are just standard matrices and updates are linear algebra operations).