2
\$\begingroup\$

I got lot of suggestions for optimizing my neural network last post I made, now I wanted to post updated version of it were I got rid of most of performance eaters, now I would really appriciate neural network side help, because last post I only got c++ side stuff, program is working as excpected, but relu network dies way to quickly in network with 2 inputs, so maybe my math is incorrect somewhere in updateWeights(). (or calculateErrors())

Neural Network Class:

#pragma once
#include <vector>
#include <numeric>
#include <iostream>
#include <functional>
#include <cmath>
#include <iterator>

namespace ActivationFunctions
{
    struct relu
    {
        float operator()(float input) const noexcept { return input * (input >= 0); }
        struct deriv_t
        {
            float operator()(float input) const noexcept { return input >= 0; }
        } derivative;
    };

    struct lrelu
    {
        float operator()(float input) const noexcept { return (input < 0) * (-derivative.a * input) + (input >= 0) * input; }
        struct deriv_t
        {
            float a = 0.01f;
            float operator()(float input) const noexcept { return (input >= 0 ? 1 : a); }
        } derivative;
    };

    struct sigmoid
    {
        float operator()(float input) const noexcept { return 1.0f / (1.0f + std::expf(-input)); }
        struct deriv_t
        {
            float operator()(float input) const noexcept { float s = 1.0f / (1.0f + std::expf(-input)); return s * (1.0f - s); }
        } derivative;
    };

    struct tanh
    {
        float operator()(float input) const noexcept { return std::tanhf(input); }
        struct deriv_t
        {
            float operator()(float input) const noexcept { float t = std::tanhf(input); return 1.0f - t * t; }
        } derivative;
    };

    struct step
    {
        float operator()(float input) const noexcept { return (input <= 0 ? -1 : 1); }
        struct deriv_t
        {
            float operator()(float input) const noexcept { return 1; }
        } derivative;
    };

    struct none
    {
        float operator()(float input) const noexcept { return input; }
        struct deriv_t
        {
            float operator()(float input) const noexcept { return 1; }
        } derivative;
    };
};

template<typename Activation = ActivationFunctions::relu>
class NeuralNetwork
{
    using layer = std::vector<float>;
    using neuron_layers = std::vector<layer>;
    using batch = neuron_layers;
    using neuron_weights = std::vector<neuron_layers>;
    using size_type = size_t;
public:
    float learningRate = 0.1f;
    float weightRandRange = 0.1f;
public:
    NeuralNetwork(const std::vector<size_type>& topology)
    {
        //Resize layers to how many layers network will have
        size_type layers = topology.size() - 1;
        m_neuron_layers.resize(layers);
        m_unactivated_neuron_layers.resize(layers);
        m_neuron_layer_errors.resize(layers);

        for (size_type i = 0; i < layers; ++i)
        {
            m_neuron_layers[i].resize(topology[i + 1]);
            m_unactivated_neuron_layers[i].resize(topology[i + 1]);
            m_neuron_layer_errors[i].resize(topology[i + 1]);
        }

        srand(time(NULL));

        //Initialize weights and resize them according to last layer 
        m_neuron_weights.resize(layers);

        for (size_type i = 0; i < layers; ++i)
        {
            m_neuron_weights[i].resize(topology[i + 1]);
            for (size_type j = 0; j < topology[i + 1]; ++j)
            {
                size_type lastLayerOutputSize = topology[i];
                m_neuron_weights[i][j].resize(lastLayerOutputSize + 1);
                m_neuron_weights[i][j].back() = 0;
                for (size_type k = 0; k < lastLayerOutputSize; ++k)
                {
                    m_neuron_weights[i][j][k] = initWeightRandom();
                }
            }
        }
    }

    void forward(const float* inputs)
    {
        std::copy(inputs, inputs + m_inputs.size(), m_inputs.data());
        //Forwarding input layer according to network inputs
        //Multiplying weights over inputs for each input neuron using inner_product and adding bias (specified as last parameter of inner_product)
        for (size_type i = 0; i < m_neuron_layers[0].size(); ++i)
        {
            m_neuron_layers[0][i] = m_unactivated_neuron_layers[0][i] = std::inner_product(inputs, inputs + m_inputs.size(), m_neuron_weights[0][i].data(), m_neuron_weights[0][i].back());
        }

        //Forwarding other layer according to last layer's outputs in the same way
        for (size_type i = 1; i < m_neuron_layers.size(); ++i)
        {
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                m_neuron_layers[i][j] = m_unactivated_neuron_layers[i][j] = std::inner_product(m_neuron_layers[i - 1].begin(), m_neuron_layers[i - 1].end(), m_neuron_weights[i][j].begin(), m_neuron_weights[i][j].back());
            }
        }

        //Applying activation function to neurons
        for (size_type i = 0; i < m_neuron_layers.size() - 1; ++i)
        {
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                m_neuron_layers[i][j] = m_activation(m_neuron_layers[i][j]);
            }
        }
    }
    void backpropagate(const float* targets)
    {
        calculateErrors(targets);
        updateWeights();
    }
    void calculateErrors(const float* targets)
    {
        for (size_type i = 0; i < m_neuron_layers.back().size(); ++i)
        {
            m_neuron_layer_errors.back()[i] = targets[i] - m_neuron_layers.back()[i];
        }

        //Than using calculated output errors calculate hidden errors
        //We iterate over all the layers(except output layer) and then iterate over all the neurons in current layer
        //And then we wanna do: errors[i] = errors[i + 1] * weights[i].transpose() ("i" is layer index)
        //errors[i] and errors[i + 1] is a vector and weights[i] is a matrix
        //aligned as [inputs][neurons] (I think) so if we transpose that it will be [neurons][weights]
        //so we can translate errors[i] = errors[i + 1] * weights[i + 1].transpose(); (WARN: weights[i + 1] might be incorrect it might be weights[i])
        //to errors[i][j] += errors[i + 1][k] * weights[i + 1][k][j] ("j" is neuron index in current layer)("k" is neuron index in next layer)
        for (long i = m_neuron_layers.size() - 2; i >= 0; --i)
        {
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                m_neuron_layer_errors[i][j] = 0;
                for (size_type k = 0; k < m_neuron_layers[i + 1].size(); ++k)
                {
                    m_neuron_layer_errors[i][j] += m_neuron_layer_errors[i + 1][k] * m_neuron_weights[i + 1][k][j];
                }
            }
        }
    }
    void updateWeights()
    {
        //Input Weights
        for (size_type j = 0; j < m_neuron_layers[0].size(); ++j)
        {
            for (size_type k = 0; k < m_inputs.size(); ++k)
            {
                m_neuron_weights[0][j][k] += learningRate * m_neuron_layer_errors[0][j] * m_activation.derivative(m_unactivated_neuron_layers[0][j]) * m_inputs[k];
            }
        }
        //Weights
        for (size_type i = 1; i < m_neuron_layers.size(); ++i)
        {
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                for (size_type k = 0; k < m_neuron_layers[i - 1].size(); ++k)
                {
                    m_neuron_weights[i][j][k] += learningRate * m_neuron_layer_errors[i][j] * m_activation.derivative(m_unactivated_neuron_layers[i][j]) * m_neuron_layers[i - 1][k];
                }
            }
        }

        //Bias
        for (size_type i = 0; i < m_neuron_layers.size(); ++i)
        {
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                m_neuron_weights[i][j].back() += learningRate * m_neuron_layer_errors[i][j] * m_activation.derivative(m_unactivated_neuron_layers[i][j]);
            }
        }
    }

    void train(const batch inputs, const batch targets)
    {
        for (size_type i = 0; i < inputs.size(); ++i)
        {
            forward(inputs[i].data());
            backpropagate(targets[i].data());
        }
    }

    void train(const batch inputs, const batch targets, batch& predictions)
    {
        predictions.clear();
        predictions.resize(inputs.size());
        for (size_type i = 0; i < inputs.size(); ++i)
        {
            forward(inputs[i].data());
            backpropagate(targets[i].data());
            predictions[i] = m_neuron_layers.back();
        }
    }

public: //<UTILITIES>//
    const float* getOutput() const { return m_neuron_layers.back().data(); }
    void reset()
    {
        m_neuron_layers.clear();
        m_neuron_layer_errors.clear();
        m_unactivated_neuron_layers.clear();
        for (size_type i = 0; i < m_neuron_weights.size(); ++i)
        {
            for (size_type j = 0; j < m_neuron_weights.size(); ++j)
            {
                for (size_type k = 0; k < m_neuron_weights.size(); ++k)
                {
                    m_neuron_weights[i][j][k] = initWeightRandom();
                }
            }
        }
    }
    float mse() const
    {
        float error = 0;
        for (const auto& x : m_neuron_layer_errors.back())
            error += x * x;
        return std::sqrt(error / m_neuron_layers.back().size());
    }
private:
    float initWeightRandom() const { return (float(rand()) / RAND_MAX * 2 - 1) * float(weightRandRange); }
public: //<DEBUG>//
    friend std::ostream& operator << (std::ostream& os, const NeuralNetwork& net)
    {
        os << "Error: " << net.mse();
        return os;
    }
    void print()
    {
        std::ostream_iterator<float> carr(std::cout, " | ");
        std::cout << "Network = \n{\n";

        for (size_type i = 0; i < m_neuron_layers.size(); ++i)
        {
            std::cout << ("    Layer[" + std::to_string(i) + "] = \n    {\n");
            for (size_type j = 0; j < m_neuron_layers[i].size(); ++j)
            {
                std::cout << ("        Neuron[" + std::to_string(j) + "] = \n        {\n");
                std::cout << ("            Weights = [ | ");
                std::copy(m_neuron_weights[i][j].begin(), m_neuron_weights[i][j].end() - 1, carr);
                std::cout << "];\n";
                std::cout << ("            Bias = " + std::to_string(m_neuron_weights[i][j].back())) + ";\n";
                std::cout << ("            Unactivated_Output = " + std::to_string(m_unactivated_neuron_layers[i][j])) + ";\n";
                std::cout << ("            Output = " + std::to_string(m_neuron_layers[i][j]) + ";\n");
                std::cout << ("            Error = " + std::to_string(m_neuron_layer_errors[i][j])) + ";\n";
                std::cout << "        };\n";
            }
            std::cout << "    }\n";
        }

        std::cout << "\n    Outputs = [ | ";
        std::copy(m_neuron_layers.back().data(), m_neuron_layers.back().data() + m_neuron_layers.back().size(), carr);
        std::cout << "];\n";

        std::cout << "}" << std::endl;
    }

private:
    layer m_inputs;
    neuron_layers m_neuron_layers;
    neuron_layers m_unactivated_neuron_layers;
    neuron_layers m_neuron_layer_errors;
    neuron_weights m_neuron_weights;
    const Activation m_activation = {};
};
\$\endgroup\$
2
\$\begingroup\$

Write only one statement per line

For some reason you decided to make all the activation functions and their derivatives one-liners. This results in some very long lines, requiring sideways scrolling. I recommend you just use one statement per line.

Make constants static constexpr

The constants a (in lrelu::deriv_t), as well as learningRate and weightRandRange are never changed in the code you showed here. If they are really constant, prefer to make them static constexpr.

Use size_type consistently

You are still using long in one place as an index into m_neuron_layers. This should be size_type as well. This can be done by changing the loop from:

for (long i = m_neuron_layers.size() - 2; i >= 0; --i)

Which unfortunately depends on i going negative, to:

for (size_type i = m_neuron_layers.size() - 1; i-- > 0;)

However, I would remove size_type completely and just use std::size_t everywhere. The reason is that size_type is only used internally, and it should never be anything but std::size_t.

Naming things

Use a consistent naming scheme. Your main class name uses PascalCase, but the types you define with using inside that class are all in snake_case. Futhermore, there's size_type which has the suffix _type, but the other types don't. I recommend that you keep things as consistent as possible; either use PascalCase for all types you define, or use the _type suffix for all type aliases you define.

Input/output types

Some functions, like forward(), take raw pointers to float as input, others, like train(), take std::vectors as input, by value even! The first is rather unsafe, as it doesn't pass the size of the buffer of floats, the latter is very inefficient, since it is making unnecessary copies.

Prefer passing inputs as const references to STL container types. (If you want to be really fancy and you can use C++20, you could make those functions take std::ranges::input_ranges.) Add assert() statements to verify that the passed input has the expected size, to catch programming errors.

For getOutput(), you can return a const reference to m_neuron_layers.back().

Avoid unnecessary copies

Apart from the unnecessary copies mentioned in the above section, you more explicitly make a copy of the input given to forward(). The input values are reused by updateWeights(), but why couldn't you just pass the same pointer/reference to inputs to updateWeights()? That would allow you to remove the member variable m_inputs, and avoid having to make a copy.

Potential to use more STL algorithms

You are already using STL algorithms in std::forward(), but there are many places where you could use them. On the other hand, plain for-loops can sometimes be more readable, so I would not hold this against you.

Use C++'s random number generators

Don't use srand() and rand(), those are C functions, and they are not particularly good for random number generation. Prefer using C++11's random number generation functionality. In particular, you should be able to replace initWeightRandom() with a member variable of type std::uniform_real_distribution.

Unnecessary call to clear()

In train(), you are going to overwrite all the elements of predictions anyway, so the call to clear() is superfluous, and will unnecessarily cause the array to be filled with zeroes due to the call to resize() right afterwards. Remove the call to clear() and just keep the resize().

Printing results

It's weird to see operator<< just print the root mean square error, but having a function print() that prints the whole state of the neural network. I would expect them to do the same thing. If the user of this class just wants to print the error value, then they can just do that themselves, the difference in typing between std::cout << net and std::cout << net.mse() is minimal.

\$\endgroup\$
2
  • \$\begingroup\$ thanks, some mentions: I name all the things weirdly like that (it's just my style). long/int was necessary in that for loop. rand() works well mostly so why not, but still fixed it, learning rate is not static constexpr, user should be able to change that (so I prefer it to be just public member), others were fixed. \$\endgroup\$
    – G.Azma
    Jun 18 at 14:39
  • \$\begingroup\$ Well, if there is consistency in your weirdness it's fine I guess :) The long is avoidable, I've updated the answer to show how. \$\endgroup\$
    – G. Sliepen
    Jun 18 at 14:54
1
\$\begingroup\$

All your output statements are of the form:
std::cout << (" Layer[" + std::to_string(i) + "] = \n {\n");
that is very weird. Why are you using to_string and concatenating the strings for the output, rather than just using << normally?

That is: cout << " Layer[" << i << "] = \n {\n";

\$\endgroup\$
1
  • \$\begingroup\$ you are right, fixed it. \$\endgroup\$
    – G.Azma
    Jun 18 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.