3
\$\begingroup\$

I'm trying to train my neural network and for the most part it's going well. However, I'd like it if it could train faster and was wondering if anyone could give some advice.

I'm trying mostly to understand the concept and didn't want to use any libraries like Tensorflow or Keras to make my network with, so I expected things to be slow. I'd just like to know if there's a more efficient method I could use to train my network.

Currently the code is trying to solve for the amount of x-intercepts in a quadratic function. I used a hyperbolic tangent activation to restrict the output between 1 and -1 and then just shifted it up one unit so it ends up where all possible answers are between 2 and 0.

If there is a more efficient method of doing this (without using numpy, tensorflow or other machine learning libraries) I would like to know. Thanks in advance!

Here is my code:

def tanh(x, derivative=False):
    if (derivative):
        return 1-(np.tanh(x)**2)
    else:
        return np.tanh(x)

bias = 1
learningrate = 0.5

w43 = -0.00761961638643
w42 = -0.00277057293921
w41 = -0.00761961638643
w4bias = 0.0125605873166

w3a = 0.999999973314
w3b = 0.999999946628
w3c = 0.999999973314
w3bias = 0.999430439099

w2a = -1.76817386861
w2b = -1.87500335887
w2c = -3.53116339806
w2bias = 0.972644723607

w1a = 1.00
w1b = 0.999999946628
w1c = 0.999999946628
w1bias = 0.999430439099
def add(x,y):
    f = open(y)
    lines = f.read()
    f.close()
    f = open(y, 'w')
    f.write(lines)
    f.write("\n<br>"+x)
    f.close()
def u1(a, b, c):
    global w1a
    global w1b
    global w1c
    global w1bias
    global bias
    return tanh((bias*w1bias)+(w1a*a)+(w1b*b)+(w1c*c))

def u2(a, b, c):
    global w2a
    global w2b
    global w2c
    global w2bias
    global bias
    return tanh((bias*w2bias)+(w2a*a)+(w2b*b)+(w2c*c))

def u3(a, b, c):

    global w3a
    global w3b
    global w3c
    global w3bias
    global bias
    return tanh((bias*w3bias)+(w3a*a)+(w3b*b)+(w3c*c))

def u4(a, b, c):
    global w43
    global w42
    global w41
    global w4bias
    global bias
    return tanh((bias*w4bias)+(w43*u3(a,b,c))+(w42*u2(a,b,c))+(w41*u1(a,b,c)))


def feedforward(a, b, c):
    return u4(a,b,c)+1

def error(a,b,c,d):
    return ((d-feedforward(a,b,c))**2)/100

def backpropagation(a,b,c,d):
    global learningrate
    global w43
    global w42
    global w41
    global w4bias
    global w3a
    global w3b
    global w3c
    global w3bias
    global w2a
    global w2b
    global w2c
    global w2bias
    global w1a
    global w1b
    global w1c
    global w1bias

    result = feedforward(a,b,c)
    delta0 = (d-result)*result*((1-result)**2)

    delta3 = (delta0*w43)*u3(a,b,c)*(1-u3(a,b,c))
    delta2 = (delta0*w42)*u2(a,b,c)*(1-u4(a,b,c))
    delta1 = (delta0*w41)*u1(a,b,c)*(1-u1(a,b,c))

    #adjustments
    outputAdjustment = (learningrate*delta0)
    w43 += outputAdjustment*u3(a,b,c)
    w42 += outputAdjustment*u2(a,b,c)
    w41 += outputAdjustment*u1(a,b,c)
    w4bias += outputAdjustment

    u3Adjustment = learningrate*delta3
    w3c += u3Adjustment*c
    w3b += u3Adjustment*b
    w3a += u3Adjustment*a
    w3bias += u3Adjustment

    u2Adjustment = learningrate*delta2
    w2c += u2Adjustment*c
    w2b += u2Adjustment*b
    w2a += u2Adjustment*a
    w2bias += u2Adjustment

    u1Adjustment = learningrate*delta1
    w1c += u1Adjustment*c
    w1b += u1Adjustment*b
    w1c += u1Adjustment*a
    w1bias += u1Adjustment


def train(data):
    testing = [[2,3,4,0], [1,3,5,0], [1,5,3,2], [2,12,4,2], [4,8,4,0], [6,12,6,0]]
    passed = False
    count = 0
    while (not passed):
        for _ in range(500):
            avgerr = 0
            for i in data:
                backpropagation(i[0], i[1], i[2], i[3])
                avgerr += error(i[0], i[1], i[2], i[3])
            avgerr = avgerr/len(data)
            if (_%100 == 0):
                print avgerr
                add(str(avgerr), "index.html")
        c = 0
        for i in testing:
            if int(round(feedforward(i[0], i[1], i[2]))) == i[3]:
                c+=1
        if c == 6:
            break
        f = open('weights-temp.txt', 'w')
        f.write("w43 " + str(w43))
        f.write("\nw42 " + str(w42))
        f.write("\nw41 " + str(w41))
        f.write("\nw4bias " + str(w4bias))
        f.write("\nw3a " + str(w3a))
        f.write("\nw3b " + str(w3b))
        f.write("\nw3c " + str(w3c))      
        f.write("\nw3bias " + str(w3bias))
        f.write("\nw2a " + str(w2a))
        f.write("\nw2b " + str(w2b))
        f.write("\nw2c " + str(w2c))
        f.write("\nw2bias " + str(w2bias))
        f.write("\nw1a " + str(w1a))
        f.write("\nw1b " + str(w1b))
        f.write("\nw1c " + str(w1c))
        f.write("\nw1bias " + str(w1bias))
        f.close()
train(dataset)
a = ""
while a != "quit":
    a = raw_input(">> ")
    if a == "test":
        alpha = int(raw_input("alpha: "))
        beta = int(raw_input("beta: "))
        gamma = int(raw_input("gamma: "))
        print feedforward(alpha, beta, gamma)
    if a == "train":
        train(dataset)
    if a == "weights":
        print w43
        print w42
        print w41
        print w4bias
        print
        print w3c
        print w3b
        print w3a
        print w3bias
        print
        print w2c
        print w2b
        print w2a
        print w2bias
        print
        print w1c
        print w1b
        print w1a
        print w1bias

        f = open('weights.txt', 'w')
        f.write("w43 " + str(w43))
        f.write("w42 " + str(w42))
        f.write("w41 " + str(w41))
        f.write("w4bias " + str(w4bias))
        f.write("w3c " + str(w3c))
        f.write("w3b " + str(w3b))
        f.write("w3a " + str(w3a))
        f.write("w3bias " + str(w3bias))
        f.write("w2c " + str(w2c))
        f.write("w2b " + str(w2b))
        f.write("w2a " + str(w2a))
        f.write("w2bias " + str(w2bias))
        f.write("w1c " + str(w1c))
        f.write("w1b " + str(w1b))
        f.write("w1a " + str(w1a))
        f.write("w1bias " + str(w1bias))
        f.close()
\$\endgroup\$
7
\$\begingroup\$

I don't know much about machine learning, so I'll be reviewing your style. Also, this is my first review, I hope I do it right.

Use Python 3

Judging by your print statements, you are currently using Python 2.x. Python 2 support will be dropped soon, and there is no reason to use it today (except if maintaining legacy code, which is not your case). More info on the subject

For your case, this involves mainly changing lines like print "something" to print("something"), as print became a function with Python 3.

Don't repeat yourself.

It makes the code harder to read, harder to maintain, error-prone... Reading the code for review is needlessly long. Instead, make use of classes to group variables that belong together and arrays and loops instead of repeating the same line with different variables.

Naming

Your naming convention needs work. Between single-letter variables and misleading function names, it's hard to figure out what code does. For example, the name w3c doesn't carry any meaning by itself.

Worse is a function like add(x,y), which is a very misleading name. I'd expect it to add two numbers, whereas it actually appends a string to a file. A name like append(string, file) would be much more meaningful.

Document your code

Your code doesn't include any comment or docstring. Sure, you can tell how it works for now, but will you remember when you'll reuse it later? At best, it's going to be hard, and it's made worse by your naming conventions.

Single responsability principle

Your function tanh is really 2 function put in a single method: tanh and its derivative. Separating them into 2 methods makes more sense.

Include a fully working code

The code you provided won't run. It doesn't import NumPy before using it, and you don't provide the dataset or detail its format to make use of it. As such, I can't test your code or the modifications I suggest.

Refactoring your code

First, let's take care of that last point:

import numpy as np

That was easy. You can also add some constant parameters at the beginning of your file:

BIAS = 1
LEARNING_RATE =  0.5

Alternatively, you could put your whole code in some methods a use a __main__ guard, to make your code importable. More on that later.

Then, single responsability principle:

def tanh_derivative(x):
    return 1-(np.tanh(x)**2)

And use np.tanh where needed. Also, the derivative doesn't looks like it's used. I suppose it will be used at some point later on, so I included it. If it's not used, it should generally be removed from the code, though.

Make a class for neurons, holding all the weights and methods together:

class neuron:
    """Docstring: describe here what the class is used for"""

    def __init__(self, weight_a, weight_b, weight_c, bias)
        self.weight_a = weight_a
        self.weight_b = weight_b
        self.weight_c = weight_c
        self.bias = bias

    def u(self, a, b, c):
        """Describe the purpose of that method.
        There are probably better names for it and its arguments, too"""

        return tanh(BIAS * self.bias
                    + self.weight_a * a
                    + self.weight_b * b
                    + self.weight_c * c)

    def print_weigths_bias(self):
        """prints all the weights and bias of the neuron."""
        print('weight_a = {}\n'.format(self.weight_a))
        print('weight_b = {}\n'.format(self.weight_b))
        print('weight_c = {}\n'.format(self.weight_c))
        print('bias = {}\n'.format(self.bias)

Then you can put neurons in an array and turn this:

w43 = -0.00761961638643
w42 = -0.00277057293921
w41 = -0.00761961638643
w4bias = 0.0125605873166

w3a = 0.999999973314
w3b = 0.999999946628
w3c = 0.999999973314
w3bias = 0.999430439099

w2a = -1.76817386861
w2b = -1.87500335887
w2c = -3.53116339806
w2bias = 0.972644723607

w1a = 1.00
w1b = 0.999999946628
w1c = 0.999999946628
w1bias = 0.999430439099

into this:

neurons = [neuron(1.0, 
                  0.999999946628, 
                  0.999999946628,
                  0.999430439099),
           neuron(-1.76817386861,
                  -1.87500335887,
                  -3.53116339806,
                  0.972644723607),
           neuron(0.999999973314,
                  0.999999946628,
                  0.999999973314,
                  0.999430439099)
           neuron(-0.00761961638643, 
                  -0.00277057293921, 
                  -0.00761961638643, 
                  0.0125605873166)]

Which then allows things like this:

for n in neurons:
    n.print_weights_bias()

instead of 16 lines of print statements.

You could also consider writing a method for writing the values to a file (or appending them).

Finally, writing a neural_network class with an array of neurons as a property, alongside the global bias and learning rate, and methods for feedforward, backpropagation, would probably be nice and bonus for making your code importable as a script.

The __main__ guard

If you follow my advice, your code would look something like this:

import numpy as np


def tanh_derivative(x):
    return 1-(np.tanh(x)**2)


class neural_neuron:
    """Don't forget to document yur code"""

    def __init__(self, weights, bias)
        # some code here

    def __repr__(self):
        """Useful for getting a string representation of the neuron"""
        # some code here

    def u(self, ...):
        # some code here

    # more methods as needed


class neural_network:
    """Don't forget to document yur code"""

    def __init__(self, bias, learning_rate)
        # some code here

    def add_neuron(self, ...):
        # some code here

    def feedforward(self, ...):
        # some code here

    def backpropagation(self, ...):
        # some code here

    def train(self, dataset):
        # some code here

    def process(self, dataset):
        # some code here

    # more methods as needed

This would make the script importable, so you can reuse your neural net for various applications. In order to make it also executable, add a if __name__ == __main__ condition. This will be True if you execute your script, and False if you import it. You can use this to test or run your script while working on it.

After the classes and function definitions, add something like this:

if __name__ == 'main':
    nn = neural_network(1.0, 0.5)
    nn.add_neuron(neuron(1.0, 1.0, 1.0, 1.0))
    nn.train("C:/taining data.csv")
    # your code goes here

If you follow this advice, your code will not only be much easier to read and review the actual algorithms, but also much easier to work with, including maintenance (bug fixing, adding features...) and reuse (as an importable script - you would be able to import and reuse your neural net in other projects just as easily as you import and use NumPy).

\$\endgroup\$
0
\$\begingroup\$

This will be a short review because the neural network seems okay.

  1. Use numpy. It isn't "cheating" if you want to learn about neural networks, but it will make every computation much faster and help you understand how to use matrices in Python. Numpy is much faster than vanilla Python when it comes to these kind of operations, it's also much easier to write clean code.

  2. Don't declare everything as global. That's just... bad practice. If you want everything to be in one place, put it in one method (Not that I'd recommend this either), but you need to learn to separate your code without having to globalize everything.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.