Different neural network activation functions and gradient descent

Question

I've implemented a bunch of activation functions for neural networks, and I just want have validation that they work correctly mathematically. I implemented sigmoid, tanh, relu, arctan, step function, squash, and gaussian and I use their implicit derivative (in terms of the output) for backpropagation.

import numpy as np

def sigmoid(x, derivative=False):
    if (derivative == True):
        return x * (1 - x)
    return 1 / (1 + np.exp(-x))

def tanh(x, derivative=False):
    if (derivative == True):
        return (1 - (x ** 2))
    return np.tanh(x)

def relu(x, derivative=False):
    if (derivative == True):
        for i in range(0, len(x)):
            for k in range(len(x[i])):
                if x[i][k] > 0:
                    x[i][k] = 1
                else:
                    x[i][k] = 0
        return x
    for i in range(0, len(x)):
        for k in range(0, len(x[i])):
            if x[i][k] > 0:
                pass  # do nothing since it would be effectively replacing x with x
            else:
                x[i][k] = 0
    return x

def arctan(x, derivative=False):
    if (derivative == True):
        return (np.cos(x) ** 2)
    return np.arctan(x)

def step(x, derivative=False):
    if (derivative == True):
        for i in range(0, len(x)):
            for k in range(len(x[i])):
                if x[i][k] > 0:
                    x[i][k] = 0
        return x
    for i in range(0, len(x)):
        for k in range(0, len(x[i])):
            if x[i][k] > 0:
                x[i][k] = 1
            else:
                x[i][k] = 0
    return x

def squash(x, derivative=False):
    if (derivative == True):
        for i in range(0, len(x)):
            for k in range(0, len(x[i])):
                if x[i][k] > 0:
                    x[i][k] = (x[i][k]) / (1 + x[i][k])
                else:
                    x[i][k] = (x[i][k]) / (1 - x[i][k])
        return x
    for i in range(0, len(x)):
        for k in range(0, len(x[i])):
            x[i][k] = (x[i][k]) / (1 + abs(x[i][k]))
    return x

def gaussian(x, derivative=False):
    if (derivative == True):
        for i in range(0, len(x)):
            for k in range(0, len(x[i])):
                x[i][k] = -2* x[i][k] * np.exp(-x[i][k] ** 2)
    for i in range(0, len(x)):
        for k in range(0, len(x[i])):
            x[i][k] = np.exp(-x[i][k] ** 2)
    return x

The input x is the result of the output matrix of the previous layer multiplied by the weight matrix. The output x is the resultant of the activation function on the matrix to be used for the next layer.

Did I implement them in the way that it was intended to be used, or do some of these not give the intended results?

Here is the gradient descent algorithm I implemented:

import numpy as np
from Neural_Network import Activation_Function as af

def GD(layers, weights, proper, acti):
    deltas = []
    llayerError = layers[-1] - proper
    llayerDelta = llayerError * acti(layers[-1], True)
    deltas.append(llayerDelta)
    for i in range(-1, -len(layers), -1):
        layerError = deltas[abs((i + 1))].dot(weights[i].T)
        layerDelta = layerError * acti(layers[i-1], True)
        deltas.append(layerDelta)

    return deltas

Answer 1

Your code is correct. What can be improved is readability and speed.

First thing first, don't mix up the computation of the function and the computation of the derivative. I understand that passing True to the function to get the derivative can be convenient, but consider doing something less confusing.

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x)
    return x * (1 - x)

You can always pass both versions of the function to your gradient descent function, or create Sigmoid class with forward/backward versions. This is what PyBrain and Torch do for example. Let's look at the PyBrain ReLU since it's done in Python:

from pybrain.structure.modules.neuronlayer import NeuronLayer

class ReluLayer(NeuronLayer):
    """ Layer of rectified linear units (relu). """

    def _forwardImplementation(self, inbuf, outbuf):
        outbuf[:] = inbuf * (inbuf > 0)

    def _backwardImplementation(self, outerr, inerr, outbuf, inbuf):
        inerr[:] = outerr * (inbuf > 0)

I think looking at this is a good way to review your code. First, what you did is not different, congratulations! The differences are mostly about optimizations, because speed is really important in neural networks. Let's look at the differences:

• They did not conflate the two functions as I explained earlier
• They set up their network to avoid allocations during training: they set outbuff using inbuf inplace: once they created their neural network, they don't need to create anything, just update weights to go as fast as possible. It probably wasn't your main concern here, and that's perfectly OK if you're doing this to learn.
• They don't use loops and conditions, but stick to vectorized operations. Again, it's an optimization concern: vectorized operations get executed much faster. Maybe you did not think about the inbuf > 0 trick (which will create a matrix of zeros and ones), but it will make your code faster.
• They group all computation: the equivalent of inerr[:] = outerr * (inbuf > 0) in your code is layerDelta = layerError * relu(layers[i-1], True). I don't think this is a crucial difference.