# How to make my neural network train faster

I'm trying to train my neural network and for the most part it's going well. However, I'd like it if it could train faster and was wondering if anyone could give some advice.

I'm trying mostly to understand the concept and didn't want to use any libraries like Tensorflow or Keras to make my network with, so I expected things to be slow. I'd just like to know if there's a more efficient method I could use to train my network.

Currently the code is trying to solve for the amount of x-intercepts in a quadratic function. I used a hyperbolic tangent activation to restrict the output between 1 and -1 and then just shifted it up one unit so it ends up where all possible answers are between 2 and 0.

If there is a more efficient method of doing this (without using numpy, tensorflow or other machine learning libraries) I would like to know. Thanks in advance!

Here is my code:

def tanh(x, derivative=False):
if (derivative):
return 1-(np.tanh(x)**2)
else:
return np.tanh(x)

bias = 1
learningrate = 0.5

w43 = -0.00761961638643
w42 = -0.00277057293921
w41 = -0.00761961638643
w4bias = 0.0125605873166

w3a = 0.999999973314
w3b = 0.999999946628
w3c = 0.999999973314
w3bias = 0.999430439099

w2a = -1.76817386861
w2b = -1.87500335887
w2c = -3.53116339806
w2bias = 0.972644723607

w1a = 1.00
w1b = 0.999999946628
w1c = 0.999999946628
w1bias = 0.999430439099
f = open(y)
f.close()
f = open(y, 'w')
f.write(lines)
f.write("\n<br>"+x)
f.close()
def u1(a, b, c):
global w1a
global w1b
global w1c
global w1bias
global bias
return tanh((bias*w1bias)+(w1a*a)+(w1b*b)+(w1c*c))

def u2(a, b, c):
global w2a
global w2b
global w2c
global w2bias
global bias
return tanh((bias*w2bias)+(w2a*a)+(w2b*b)+(w2c*c))

def u3(a, b, c):

global w3a
global w3b
global w3c
global w3bias
global bias
return tanh((bias*w3bias)+(w3a*a)+(w3b*b)+(w3c*c))

def u4(a, b, c):
global w43
global w42
global w41
global w4bias
global bias
return tanh((bias*w4bias)+(w43*u3(a,b,c))+(w42*u2(a,b,c))+(w41*u1(a,b,c)))

def feedforward(a, b, c):
return u4(a,b,c)+1

def error(a,b,c,d):
return ((d-feedforward(a,b,c))**2)/100

def backpropagation(a,b,c,d):
global learningrate
global w43
global w42
global w41
global w4bias
global w3a
global w3b
global w3c
global w3bias
global w2a
global w2b
global w2c
global w2bias
global w1a
global w1b
global w1c
global w1bias

result = feedforward(a,b,c)
delta0 = (d-result)*result*((1-result)**2)

delta3 = (delta0*w43)*u3(a,b,c)*(1-u3(a,b,c))
delta2 = (delta0*w42)*u2(a,b,c)*(1-u4(a,b,c))
delta1 = (delta0*w41)*u1(a,b,c)*(1-u1(a,b,c))

def train(data):
testing = [[2,3,4,0], [1,3,5,0], [1,5,3,2], [2,12,4,2], [4,8,4,0], [6,12,6,0]]
passed = False
count = 0
while (not passed):
for _ in range(500):
avgerr = 0
for i in data:
backpropagation(i[0], i[1], i[2], i[3])
avgerr += error(i[0], i[1], i[2], i[3])
avgerr = avgerr/len(data)
if (_%100 == 0):
print avgerr
c = 0
for i in testing:
if int(round(feedforward(i[0], i[1], i[2]))) == i[3]:
c+=1
if c == 6:
break
f = open('weights-temp.txt', 'w')
f.write("w43 " + str(w43))
f.write("\nw42 " + str(w42))
f.write("\nw41 " + str(w41))
f.write("\nw4bias " + str(w4bias))
f.write("\nw3a " + str(w3a))
f.write("\nw3b " + str(w3b))
f.write("\nw3c " + str(w3c))
f.write("\nw3bias " + str(w3bias))
f.write("\nw2a " + str(w2a))
f.write("\nw2b " + str(w2b))
f.write("\nw2c " + str(w2c))
f.write("\nw2bias " + str(w2bias))
f.write("\nw1a " + str(w1a))
f.write("\nw1b " + str(w1b))
f.write("\nw1c " + str(w1c))
f.write("\nw1bias " + str(w1bias))
f.close()
train(dataset)
a = ""
while a != "quit":
a = raw_input(">> ")
if a == "test":
alpha = int(raw_input("alpha: "))
beta = int(raw_input("beta: "))
gamma = int(raw_input("gamma: "))
print feedforward(alpha, beta, gamma)
if a == "train":
train(dataset)
if a == "weights":
print w43
print w42
print w41
print w4bias
print
print w3c
print w3b
print w3a
print w3bias
print
print w2c
print w2b
print w2a
print w2bias
print
print w1c
print w1b
print w1a
print w1bias

f = open('weights.txt', 'w')
f.write("w43 " + str(w43))
f.write("w42 " + str(w42))
f.write("w41 " + str(w41))
f.write("w4bias " + str(w4bias))
f.write("w3c " + str(w3c))
f.write("w3b " + str(w3b))
f.write("w3a " + str(w3a))
f.write("w3bias " + str(w3bias))
f.write("w2c " + str(w2c))
f.write("w2b " + str(w2b))
f.write("w2a " + str(w2a))
f.write("w2bias " + str(w2bias))
f.write("w1c " + str(w1c))
f.write("w1b " + str(w1b))
f.write("w1a " + str(w1a))
f.write("w1bias " + str(w1bias))
f.close()


I don't know much about machine learning, so I'll be reviewing your style. Also, this is my first review, I hope I do it right.

## Use Python 3

Judging by your print statements, you are currently using Python 2.x. Python 2 support will be dropped soon, and there is no reason to use it today (except if maintaining legacy code, which is not your case). More info on the subject

For your case, this involves mainly changing lines like print "something" to print("something"), as print became a function with Python 3.

## Don't repeat yourself.

It makes the code harder to read, harder to maintain, error-prone... Reading the code for review is needlessly long. Instead, make use of classes to group variables that belong together and arrays and loops instead of repeating the same line with different variables.

## Naming

Your naming convention needs work. Between single-letter variables and misleading function names, it's hard to figure out what code does. For example, the name w3c doesn't carry any meaning by itself.

Worse is a function like add(x,y), which is a very misleading name. I'd expect it to add two numbers, whereas it actually appends a string to a file. A name like append(string, file) would be much more meaningful.

Your code doesn't include any comment or docstring. Sure, you can tell how it works for now, but will you remember when you'll reuse it later? At best, it's going to be hard, and it's made worse by your naming conventions.

## Single responsability principle

Your function tanh is really 2 function put in a single method: tanh and its derivative. Separating them into 2 methods makes more sense.

## Include a fully working code

The code you provided won't run. It doesn't import NumPy before using it, and you don't provide the dataset or detail its format to make use of it. As such, I can't test your code or the modifications I suggest.

First, let's take care of that last point:

import numpy as np


That was easy. You can also add some constant parameters at the beginning of your file:

BIAS = 1
LEARNING_RATE =  0.5


Alternatively, you could put your whole code in some methods a use a __main__ guard, to make your code importable. More on that later.

Then, single responsability principle:

def tanh_derivative(x):
return 1-(np.tanh(x)**2)


And use np.tanh where needed. Also, the derivative doesn't looks like it's used. I suppose it will be used at some point later on, so I included it. If it's not used, it should generally be removed from the code, though.

Make a class for neurons, holding all the weights and methods together:

class neuron:
"""Docstring: describe here what the class is used for"""

def __init__(self, weight_a, weight_b, weight_c, bias)
self.weight_a = weight_a
self.weight_b = weight_b
self.weight_c = weight_c
self.bias = bias

def u(self, a, b, c):
"""Describe the purpose of that method.
There are probably better names for it and its arguments, too"""

return tanh(BIAS * self.bias
+ self.weight_a * a
+ self.weight_b * b
+ self.weight_c * c)

def print_weigths_bias(self):
"""prints all the weights and bias of the neuron."""
print('weight_a = {}\n'.format(self.weight_a))
print('weight_b = {}\n'.format(self.weight_b))
print('weight_c = {}\n'.format(self.weight_c))
print('bias = {}\n'.format(self.bias)


Then you can put neurons in an array and turn this:

w43 = -0.00761961638643
w42 = -0.00277057293921
w41 = -0.00761961638643
w4bias = 0.0125605873166

w3a = 0.999999973314
w3b = 0.999999946628
w3c = 0.999999973314
w3bias = 0.999430439099

w2a = -1.76817386861
w2b = -1.87500335887
w2c = -3.53116339806
w2bias = 0.972644723607

w1a = 1.00
w1b = 0.999999946628
w1c = 0.999999946628
w1bias = 0.999430439099


into this:

neurons = [neuron(1.0,
0.999999946628,
0.999999946628,
0.999430439099),
neuron(-1.76817386861,
-1.87500335887,
-3.53116339806,
0.972644723607),
neuron(0.999999973314,
0.999999946628,
0.999999973314,
0.999430439099)
neuron(-0.00761961638643,
-0.00277057293921,
-0.00761961638643,
0.0125605873166)]


Which then allows things like this:

for n in neurons:
n.print_weights_bias()


instead of 16 lines of print statements.

You could also consider writing a method for writing the values to a file (or appending them).

Finally, writing a neural_network class with an array of neurons as a property, alongside the global bias and learning rate, and methods for feedforward, backpropagation, would probably be nice and bonus for making your code importable as a script.

## The __main__ guard

import numpy as np

def tanh_derivative(x):
return 1-(np.tanh(x)**2)

class neural_neuron:
"""Don't forget to document yur code"""

def __init__(self, weights, bias)
# some code here

def __repr__(self):
"""Useful for getting a string representation of the neuron"""
# some code here

def u(self, ...):
# some code here

# more methods as needed

class neural_network:
"""Don't forget to document yur code"""

def __init__(self, bias, learning_rate)
# some code here

# some code here

def feedforward(self, ...):
# some code here

def backpropagation(self, ...):
# some code here

def train(self, dataset):
# some code here

def process(self, dataset):
# some code here

# more methods as needed


This would make the script importable, so you can reuse your neural net for various applications. In order to make it also executable, add a if __name__ == __main__ condition. This will be True if you execute your script, and False if you import it. You can use this to test or run your script while working on it.

After the classes and function definitions, add something like this:

if __name__ == 'main':
nn = neural_network(1.0, 0.5)

1. Use numpy. It isn't "cheating" if you want to learn about neural networks, but it will make every computation much faster and help you understand how to use matrices in Python. Numpy is much faster than vanilla Python when it comes to these kind of operations, it's also much easier to write clean code.
2. Don't declare everything as global. That's just... bad practice. If you want everything to be in one place, put it in one method (Not that I'd recommend this either), but you need to learn to separate your code without having to globalize everything.