# Backpropagation in simple Neural Network

I've been working on a simple neural network implemented in python. Currently, it seems to be learning, but unfortunately it doesn't seem to be learning effectively. The graph below shows the output of my neural network when trained over about 15,000 iterations, with 1000 training examples (it's trying to learn x2). The cost of the network also seems to always bottom out just over 0.41, and I can't get it any lower.

EDIT: I tried leaving the network to learn for a few hours, over probably several million or more iterations, and the two lines on the grah did seem to come together more, however, the cost refused to budge below 0.41-ish. I suspect the issue is with my implementation of the backpropagation algorithm, since the high value for cost given by my implementation seems to correspond with the seeming inaccuracy when the network is plotted on a graph.

However, my full code is below, in case I've missed something in other parts of the implementation.

import numpy as np
class neural_network:

def __init__(self,dimensions,nonlinear_function,nonlinear_function_derivative,seed=None):
if seed:
np.random.seed(seed)
self.g=nonlinear_function
self.g_dash=nonlinear_function_derivative
self.theta_one=2*np.random.random((dimensions,dimensions))-1
self.theta_two=2*np.random.random((dimensions+1,dimensions))-1
def predict(self,x):
self.a1=x
self.z2=np.dot(x,self.theta_one)
self.a2=np.concatenate((self.g(self.z2),np.ones((np.shape(self.z2),1))),axis=1)

self.z3=np.dot(self.a2,self.theta_two)
self.a3=self.g(self.z3)
return self.a3
def backprop(self,y,alpha,reg):
self.y=y
self.delta_three=self.a3-y
self.delta_two=np.dot(self.delta_three,self.theta_two.T)[::,:-1]*self.g_dash(self.z2)

self.Delta2=np.dot(self.a2.T,self.delta_three)
self.Delta1=np.dot(self.a1.T,self.delta_two)

self.theta_one_regularisation=(0.0001/len(self.a1))*self.theta_one
self.theta_two_regularisation=(0.0001/len(self.a1))*self.theta_two

def cost(self,reg):
self.j=(1/len(self.a1))*np.sum(np.sum(-self.y*np.log(self.a3)-(1-self.y)*np.log(1-self.a3),1))
self.j+=(reg/len(self.a1))*(sum(sum(self.theta_one**2,2))+sum(sum(self.theta_two**2,2)))
return self.j
def train(self,inPut,outPut,alpha,reg):
self.predict(inPut)
self.backprop(outPut,alpha,reg)


Can anyone offer any feedback on my implementation, and whether the backprop algorithm is 'correct'?

EDIT:

Below is the testing code that I am using (just put after the neural network code in the same file). I realise that it is extremely messy, and contains many examples of bad practice, but it seems to work (however inelegantly). I have tested extensively with different hyperparameters, not just the ones used in the code below, and I have not been able to get the cost down below 0.41-ish, which does seem pretty poor. If anyone can find hyperparameters which would decrease the minimum bound on the cost significantly, that would also basically answer my question.

np.seterr(all='raise')

def plot():
plt.clf()
plt.plot([i/100 for i in range(100)],[net.predict(np.array([[1,f/100]])) for f in range(100)])
plt.plot([i/100 for i in range(100)],[((f/100)**2) for f in range(100)])
plt.pause(0.05)
import random
net=neural_network([2,400,1],lambda x:1/(1+np.exp(-x)),lambda x:(1/(1+np.exp(-x)))*(1-(1/(1+np.exp(-x)))))

count=0

import matplotlib.pyplot as plt
plt.ion()
r=[i/1000 for i in range(1,1000)]
inp=np.array([[1,f] for f in r])
outp=np.array([[f**2] for f in r])

net.predict(inp)

while True:
count+=1

net.train(inp,outp,1,0.001)
if count%100==0:

print(net.cost(0.001))
plot()

• Hey, I currently read your question as your code doesn't work as intended. Does this work as intended, but non-optimally? Or is there a problem that prevents your code from outputting the correct answer? – Peilonrayz Nov 3 '16 at 16:53
• @JoeWallis The network appears to be learning, but as can be seen in the graph, it's pretty bad at it. I'm wondering if anyone can suggest any improvements to it that would improve its learning, whether that is through pointing out an algorithmic error that I have made, or suggesting an improvement somewhere else. – penalosa Nov 3 '16 at 16:58
• The consensus in chat seems to be that "correctness" in neural networks is not binary, so asking for improvements in accuracy should be allowable. – 200_success Nov 3 '16 at 17:31
• Please actually add your full code. That includes your training set and the activation function as well as the network dimensions you chose. – Ext3h Nov 8 '16 at 13:57
• You can use gradient checking for debugging purpose(you can read more here) – AmirHossein May 1 '17 at 6:14