2
\$\begingroup\$

I added learning rate and momentum to a neural network implementation from scratch I found at: https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6

However I had a few questions about my implementation:

  • Is it correct? Any suggested improvements? It appears to output adequate results generally but outside advice is very appreciated.
  • With a learning rate < 0.5 or momentum > 0.9 the network tends to gets stuck in a local optimum where loss = ~1. I assume this is because step size isn't big enough to escape this but is there a way to overcome this? Or is this inherent with the nature of the data being solved and unavoidable.

    import numpy as np
    import matplotlib.pyplot as plt
    
    
    def sigmoid(x):
        return 1 / (1 + np.exp(-x))
    
    
    def sigmoid_derivative(x):
        sig = 1 / (1 + np.exp(-x))
        return sig * (1 - sig)
    
    
    class NeuralNetwork:
        def __init__(self, x, y):
            self.input      = x
            self.weights1   = np.random.rand(self.input.shape[1], 4)
            self.weights2   = np.random.rand(4, 1)
            self.y          = y
            self.output     = np.zeros(self.y.shape)
            self.v_dw1      = 0
            self.v_dw2      = 0
            self.alpha      = 0.5
            self.beta       = 0.5
    
        def feedforward(self):
            self.layer1 = sigmoid(np.dot(self.input, self.weights1))
            self.output = sigmoid(np.dot(self.layer1, self.weights2))
    
        def backprop(self, alpha, beta):
            # application of the chain rule to find derivative of the loss function with respect to weights2 and weights1
            d_weights2 = np.dot(self.layer1.T, (2*(self.y - self.output) * sigmoid_derivative(self.output)))
            d_weights1 = np.dot(self.input.T,  (np.dot(2*(self.y - self.output) *
                                                sigmoid_derivative(self.output), self.weights2.T) *
                                                sigmoid_derivative(self.layer1)))
            # adding effect of momentum
            self.v_dw1 = (beta * self.v_dw1) + ((1 - beta) * d_weights1)
            self.v_dw2 = (beta * self.v_dw2) + ((1 - beta) * d_weights2)
            # update the weights with the derivative (slope) of the loss function
            self.weights1 = self.weights1 + (self.v_dw1 * alpha)
            self.weights2 = self.weights2 + (self.v_dw2 * alpha)
    
    
    if __name__ == "__main__":
        X = np.array([[0, 0, 1],
                      [0, 1, 1],
                      [1, 0, 1],
                      [1, 1, 1]])
        y = np.array([[0], [1], [1], [0]])
        nn = NeuralNetwork(X, y)
    
        total_loss = []
        for i in range(10000):
            nn.feedforward()
            nn.backprop(nn.alpha, nn.beta)
            total_loss.append(sum((nn.y-nn.output)**2))
    
        iteration_num = list(range(10000))
        plt.plot(iteration_num, total_loss)
        plt.show()
        print(nn.output)
    
\$\endgroup\$
  • \$\begingroup\$ "Is it correct?" Did you test it? \$\endgroup\$ – Mast Oct 24 '18 at 16:40
  • \$\begingroup\$ @Mast Yes but it gets stuck in a local optimum where loss = ~1 occasionally, and I wanted to know if I'd inputted momentum correctly. \$\endgroup\$ – Seb Squire Oct 24 '18 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.