I've been working on a simple neural network implemented in python. Currently, it seems to be learning, but unfortunately it doesn't seem to be learning effectively. The graph below shows the output of my neural network when trained over about 15,000 iterations, with 1000 training examples (it's trying to learn x2). The cost of the network also seems to always bottom out just over 0.41, and I can't get it any lower.
EDIT: I tried leaving the network to learn for a few hours, over probably several million or more iterations, and the two lines on the grah did seem to come together more, however, the cost refused to budge below 0.41-ish.
I suspect the issue is with my implementation of the backpropagation algorithm, since the high value for cost given by my implementation seems to correspond with the seeming inaccuracy when the network is plotted on a graph.
However, my full code is below, in case I've missed something in other parts of the implementation.
import numpy as np
class neural_network:
def __init__(self,dimensions,nonlinear_function,nonlinear_function_derivative,seed=None):
if seed:
np.random.seed(seed)
self.g=nonlinear_function
self.g_dash=nonlinear_function_derivative
self.theta_one=2*np.random.random((dimensions[0],dimensions[1]))-1
self.theta_two=2*np.random.random((dimensions[1]+1,dimensions[2]))-1
def predict(self,x):
self.a1=x
self.z2=np.dot(x,self.theta_one)
self.a2=np.concatenate((self.g(self.z2),np.ones((np.shape(self.z2)[0],1))),axis=1)
self.z3=np.dot(self.a2,self.theta_two)
self.a3=self.g(self.z3)
return self.a3
def backprop(self,y,alpha,reg):
self.y=y
self.delta_three=self.a3-y
self.delta_two=np.dot(self.delta_three,self.theta_two.T)[::,:-1]*self.g_dash(self.z2)
self.Delta2=np.dot(self.a2.T,self.delta_three)
self.Delta1=np.dot(self.a1.T,self.delta_two)
self.theta_one_gradient=self.Delta1/len(self.a1)
self.theta_two_gradient=self.Delta2/len(self.a1)
self.theta_one_regularisation=(0.0001/len(self.a1))*self.theta_one
self.theta_two_regularisation=(0.0001/len(self.a1))*self.theta_two
self.theta_one_gradient[1]+=self.theta_one_regularisation[1]
self.theta_two_gradient[1]+=self.theta_two_regularisation[1]
self.theta_one-=alpha*(self.theta_one_gradient)
self.theta_two-=alpha*(self.theta_two_gradient)
def cost(self,reg):
self.j=(1/len(self.a1))*np.sum(np.sum(-self.y*np.log(self.a3)-(1-self.y)*np.log(1-self.a3),1))
self.j+=(reg/len(self.a1))*(sum(sum(self.theta_one**2,2))+sum(sum(self.theta_two**2,2)))
return self.j
def train(self,inPut,outPut,alpha,reg):
self.predict(inPut)
self.backprop(outPut,alpha,reg)
Can anyone offer any feedback on my implementation, and whether the backprop algorithm is 'correct'?
EDIT:
Below is the testing code that I am using (just put after the neural network code in the same file). I realise that it is extremely messy, and contains many examples of bad practice, but it seems to work (however inelegantly). I have tested extensively with different hyperparameters, not just the ones used in the code below, and I have not been able to get the cost down below 0.41-ish, which does seem pretty poor. If anyone can find hyperparameters which would decrease the minimum bound on the cost significantly, that would also basically answer my question.
np.seterr(all='raise')
def plot():
plt.clf()
plt.plot([i/100 for i in range(100)],[net.predict(np.array([[1,f/100]]))[0][0] for f in range(100)])
plt.plot([i/100 for i in range(100)],[((f/100)**2) for f in range(100)])
plt.pause(0.05)
import random
net=neural_network([2,400,1],lambda x:1/(1+np.exp(-x)),lambda x:(1/(1+np.exp(-x)))*(1-(1/(1+np.exp(-x)))))
count=0
import matplotlib.pyplot as plt
plt.ion()
r=[i/1000 for i in range(1,1000)]
inp=np.array([[1,f] for f in r])
outp=np.array([[f**2] for f in r])
net.predict(inp)
while True:
count+=1
net.train(inp,outp,1,0.001)
if count%100==0:
print(net.cost(0.001))
plot()
