3
\$\begingroup\$

I have the following python code which implements a simple neural network (two inputs, one hidden layer with 2 neurons, and one output) with a sigmoid activation function to learn a XOR gate. The code runs fine (ie no syntax errors or runtime errors or anything) and I have it plotting error over time to visualize its progress.

I have tested it piece by piece; I know the feedforward works correctly as I tried hardcoding in weights and feeding it through and it gave the expected results, and while I didn't have the patience to backpropagate errors for the entire network by hand, I worked out the errors on the output and one of the hidden layer nodes and got similar numbers to the code. However, when I try to train it on the 4 test cases I have cyclically, the network consistently converges to 0.5. I've tried changing various parameters (learning rate, momentum, epochs, etc.) but to no avail. I even tried replacing the sigmoid function with tanh for the activation but that didn't change the results either.

I've used numpy in some places to make the matrix/vector math cleaner. I would greatly appreciate any suggestions on how I can improve my code, as well as what could be causing my problem.

import numpy as np
from random import random
from math import tanh, exp
from copy import deepcopy
LAYERS = [2, 2, 1]
testCases = [
    [[0, 0], [0]],
    [[0, 1], [1]],
    [[1, 0], [1]],
    [[1, 1], [0]]
]

activate = np.vectorize(lambda c: 1 / (1 + exp(-c))) #activation function
_activate = np.vectorize(lambda c: activate(c) * (1 - activate(c))) #derivative of activation function




errors = [list(), list(), list(), list()]

class NeuralNetwork:
    def __init__(self, top):
        self.topology = top
        self.layers = [np.matrix([0 for i in range(size)]).T for size in top]
        self.weights = list()
        for i,j in zip(top[:-1], top[1:]):
            self.weights.append(np.matrix([[random()*2-1 for _ in range(i + 1)] for _ in range(j)]))
        self.momentum = 0.3
        self.dw = None

    def feedforward(self, inputs):
        assert len(inputs) == self.topology[0]
        self.layers[0] = np.matrix(inputs[:]).T
        for i in range(len(self.layers) - 1):
            biased = np.concatenate((self.layers[i], np.matrix([1])), axis=0)
            self.layers[i + 1] = activate(self.weights[i] @ biased)

        return self.layers[-1]


    def backpropagate(self, test):
        inputs, expected = test
        self.feedforward(inputs) # load the inputs into the network
        deltas = deepcopy(self.layers)
        errors[testCases.index(test)].append(np.linalg.norm(expected - self.layers[-1]))
        deltas[-1] = np.dot(expected - self.layers[-1], _activate(self.layers[-1]))
        for i in range(len(deltas) - 1)[::-1]:

            try:
                sc = deltas[i+1].T @ self.weights[i]
            except ValueError:
                sc = deltas[i+1][:-1,:].T @ self.weights[i]
            biased = np.concatenate((self.layers[i], [[1]])) #layer with a bias node of 1 added to it
            deltas[i] = np.multiply(_activate(biased), sc.T)
        self.deltas = deltas[1:]

    def test(self, test):
        self.backpropagate(test)
        for i in range(len(self.deltas)):
            biased = np.concatenate((self.layers[i], [[1]])).T
            if self.deltas[i].shape[0] is not self.weights[i].shape[0]:
                self.deltas[i] = self.deltas[i][:-1, :]
            assert self.deltas[i].shape[0] == self.weights[i].shape[0]
            assert biased.shape[1] == self.weights[i].shape[1]

            dw = self.deltas[1] @ biased
            if self.dw is not None:
                dw += self.momentum * self.dw
            self.weights[i] += 0.1 * dw
            self.dw = dw


nn = NeuralNetwork(LAYERS)

print(nn.weights)
nn.test(testCases[0])
print(nn.deltas)
for i in range(1000):
    nn.test(testCases[i % 4])

import matplotlib.pyplot as plt
list(map(plt.plot, errors))
plt.show()


list(map(print, nn.feedforward([0, 0])))
list(map(print, nn.feedforward([0, 1])))
list(map(print, nn.feedforward([1, 0])))
list(map(print, nn.feedforward([1, 1])))

UPDATE: I took the time and backpropagated the errors by hand for a couple of random initial configurations, and my results matched what the network calculates. Also, I ran it once on each test case and saw that the error for that test case improved marginally after the update, so I know my backprop and weight update steps are correct. Unfortunately, it still converges on bad values. Any ideas?

\$\endgroup\$
  • 1
    \$\begingroup\$ What are you saying, it works but it doesn't? \$\endgroup\$ – Mast Aug 22 '18 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.