I have the following python code which implements a simple neural network (two inputs, one hidden layer with 2 neurons, and one output) with a sigmoid activation function to learn a XOR gate. The code runs fine (ie no syntax errors or runtime errors or anything) and I have it plotting error over time to visualize its progress.
I have tested it piece by piece; I know the feedforward works correctly as I tried hardcoding in weights and feeding it through and it gave the expected results, and while I didn't have the patience to backpropagate errors for the entire network by hand, I worked out the errors on the output and one of the hidden layer nodes and got similar numbers to the code. However, when I try to train it on the 4 test cases I have cyclically, the network consistently converges to
0.5. I've tried changing various parameters (
epochs, etc.) but to no avail. I even tried replacing the
sigmoid function with
tanh for the activation but that didn't change the results either.
numpy in some places to make the matrix/vector math cleaner. I would greatly appreciate any suggestions on how I can improve my code, as well as what could be causing my problem.
import numpy as np from random import random from math import tanh, exp from copy import deepcopy LAYERS = [2, 2, 1] testCases = [ [[0, 0], ], [[0, 1], ], [[1, 0], ], [[1, 1], ] ] activate = np.vectorize(lambda c: 1 / (1 + exp(-c))) #activation function _activate = np.vectorize(lambda c: activate(c) * (1 - activate(c))) #derivative of activation function errors = [list(), list(), list(), list()] class NeuralNetwork: def __init__(self, top): self.topology = top self.layers = [np.matrix([0 for i in range(size)]).T for size in top] self.weights = list() for i,j in zip(top[:-1], top[1:]): self.weights.append(np.matrix([[random()*2-1 for _ in range(i + 1)] for _ in range(j)])) self.momentum = 0.3 self.dw = None def feedforward(self, inputs): assert len(inputs) == self.topology self.layers = np.matrix(inputs[:]).T for i in range(len(self.layers) - 1): biased = np.concatenate((self.layers[i], np.matrix()), axis=0) self.layers[i + 1] = activate(self.weights[i] @ biased) return self.layers[-1] def backpropagate(self, test): inputs, expected = test self.feedforward(inputs) # load the inputs into the network deltas = deepcopy(self.layers) errors[testCases.index(test)].append(np.linalg.norm(expected - self.layers[-1])) deltas[-1] = np.dot(expected - self.layers[-1], _activate(self.layers[-1])) for i in range(len(deltas) - 1)[::-1]: try: sc = deltas[i+1].T @ self.weights[i] except ValueError: sc = deltas[i+1][:-1,:].T @ self.weights[i] biased = np.concatenate((self.layers[i], [])) #layer with a bias node of 1 added to it deltas[i] = np.multiply(_activate(biased), sc.T) self.deltas = deltas[1:] def test(self, test): self.backpropagate(test) for i in range(len(self.deltas)): biased = np.concatenate((self.layers[i], [])).T if self.deltas[i].shape is not self.weights[i].shape: self.deltas[i] = self.deltas[i][:-1, :] assert self.deltas[i].shape == self.weights[i].shape assert biased.shape == self.weights[i].shape dw = self.deltas @ biased if self.dw is not None: dw += self.momentum * self.dw self.weights[i] += 0.1 * dw self.dw = dw nn = NeuralNetwork(LAYERS) print(nn.weights) nn.test(testCases) print(nn.deltas) for i in range(1000): nn.test(testCases[i % 4]) import matplotlib.pyplot as plt list(map(plt.plot, errors)) plt.show() list(map(print, nn.feedforward([0, 0]))) list(map(print, nn.feedforward([0, 1]))) list(map(print, nn.feedforward([1, 0]))) list(map(print, nn.feedforward([1, 1])))
UPDATE: I took the time and backpropagated the errors by hand for a couple of random initial configurations, and my results matched what the network calculates. Also, I ran it once on each test case and saw that the error for that test case improved marginally after the update, so I know my backprop and weight update steps are correct. Unfortunately, it still converges on bad values. Any ideas?