6
\$\begingroup\$

I am trying to learn the basics about neural networks by coding from scratch the perceptron model.

Since I am not a programmer and would like to improve my coding skills I would like to get your comments about this code. What do you think about it (I tried to respect the different steps: forward and backward for backpropagation)? I suppose it is quite horrific for a Python programmer and would be pleased to gain some insight about some of my evidently bad practices ;)

Here is my work (inspired by Rashka book):

class Perceptron(object):
    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs       

    def fit(self, X, Y):
        self.w_ = np.zeros(X.shape[1])
        self.b_=0
        self.cost_ = []   
        for _ in range(self.epochs):
            errors=0
            for x_i,y_i in zip(X,Y):
                y_hat=self.forward(x_i)
                gradL_W,gradL_b=self.backward(y_i,y_hat,x_i)
                self.w_,self.b_=self.update(gradL_W,gradL_b)  
                errors+=(y_hat!=y_i)*1
            self.cost_.append(errors)   
        return self

    def net_input(self, x):
        """Calculate net input"""
        return x.dot(self.w_) + self.b_            

    def activation(self,z):
        heaviside= lambda x: (1, -1)[x<0]
        return heaviside(z)

    def forward(self,x):
        s= self.net_input(x)
        y_hat=self.activation(s)
        return y_hat    

    def backward(self,y,y_hat,x):
        error=-(y-y_hat)
        gradL_b=error*1
        gradL_W=error*x
        return gradL_W,gradL_b        

    def update(self,gradL_W,gradL_b):
        self.w_=self.w_-self.eta*gradL_W
        self.b_=self.b_-self.eta*gradL_b
        return self.w_,self.b_

    def predict(self,X):
        s=X.dot(self.w_)+self.b_
        y_hat=list(map(lambda x: (1, -1)[x<0],s))
        return y_hat
\$\endgroup\$
1
\$\begingroup\$

Style

Python comes with an official Style Guide often just called PEP8, and especially as a beginnger it's a good starting point to get going. Of course coding style is often a matter of choice, however, there are aspects you should definitely follow.

Whitespace
As you know, Python's code structure is build upon indentation, so some parts are already language-defined. The style guide also has recommendations on how to use whitespace within statements and expressions. For example, , should always be followed be a single space character, while = is preceded and followed by a single space when used in assignments (there should be no whitespace around = if used as keyword arguments to functions such as foo(batz='bar')). So you would go from

gradL_W,gradL_b=self.backward(y_i,y_hat,x_i)
self.w_,self.b_=self.update(gradL_W,gradL_b)

to

gradL_W, gradL_b = self.backward(y_i, y_hat, x_i)
self.w_, self.b_ = self.update(gradL_W, gradL_b)

which does look way cleaner IMHO.

Variable names
It's always good practice to use descriptive variable names. Simply writing self.weights instead of self.w_ wont hurt that much. Apart from that, the Style Guide also recommends to use a single leading underscore for non-public variable names. Note, this is only a convention, since Python has no "real" private member variables as you may know from other languages.

Documentation
There is a single function in your perceptron class that has a) documentation and b) also follows the docstring-style from the Style Guide. You should apply that to the other functions and maybe even to the class as well. Your current project might be simple, but if projects become more complex, good documentation will save you a lot of headache. Following the official docstring style also has the nice benefit that Python's help(...) function as well as most IDEs will easily find it.

The code

Apart from the stylistic issues mentioned above, there are some more direct code-focused aspects you should work on.

Code duplication
There is some duplicated code in your implementation of the perceptron model. E.g., you implement the scalar product at the input of the perceptron twice! The first time at net_input, which is likely where it's supposed to go, and the second time in predict. predict also reimplements pretty much the whole point of activation. It's pretty straightforward to implement predict using net_input and activation:

def predict(self, X):
    s = self.net_input(X)
    y_hat = list(map(self.activation, s))
    return y_hat

If you use Python for a while you will also learn to use list comprehensions more often than the list(map(...)) construct, simply because it's more flexible and readable. Rewriting the calculation of y_hat as list comprehension would look like y_hat = [self.activation(i) for i in s]. After you have arrived at

def predict(self, X):
    s = self.net_input(X)
    y_hat = [self.activation(i) for i in s]
    return y_hat

you may also find the similarity between predict and forward quite striking. forward can now be boiled down to simply

def forward(self, x):
    return self.predict(x)[0]

Getting the "first" result of the return value using [0] is mainly to ensure that the new version just returns a single number as it was before. Otherwise it would be a list with a single element.

Explicit is better than implicit

(Taken from the Zen of Python)

Using code like

heaviside = lambda x: (1, -1)[x < 0]
return heaviside(z)

might be clever, but clever does not equal good or readable most of the time. The same thing can also be expressed as return -1 if z < 0 else 1, which is almost a verbatim translation of the mathematical definition of the step function as presented by Rashka. If you don't like the inline-if, you may also write it as "normal" if condition:

if z < 0:
    return -1
else:
    return 1

gradL_b = error * 1 is another instance of cleverness. This time, I am not even sure what you are actually trying to accomplish here. Depending on the type of your labels y, error will either be of type int or maybe also float, and multiplying it with 1 will not change that.

errors += (y_hat != y_i) * 1 is more obvious, but could also be expressed explicitly as errors += int(y_hat != y_i). Python should even be able to do errors += y_hat != y_i without changing the end result.

Return values
I'm not 100% sure what you are trying to accomplish with some of your function return values. For example, I cannot see the point in fit returning self. updateis another instance of that topic. I can see you're using it as self.w_, self.b_ = self.update(gradL_W, gradL_b), but there is not a real reason to do this, since update already modifies the weights internally.

You will likely have to think about that topic as you further progress in your journey into the depths of neural networks (pun intended). Often, the multilayer perceptron is one of the next logical steps in this process. At this stage, you will need the gradients of previous layers to be able to propagate errors through the network. But time will tell1.

Until then: Happy coding!


1You will also probably soon find out that modelling single neurons is not sufficient (and computationally efficient) for what there is to come. Depending on your learning materials, matrix operations will pop up more and more often. This is the time where the NumPy library will help you shine.

\$\endgroup\$
  • \$\begingroup\$ Thanks for your help. Concerning the "self" for the fit method, i mimic what S. Raschka do in his train method sebastianraschka.com/Articles/…; i thought i was perhaps missing something ... if someone know a good reason to add return self it interest me ... \$\endgroup\$ – laurent pincemaille May 16 at 6:36
  • \$\begingroup\$ Returning self would allow you to do some kind of "chaining" like nn.fit(X, Y).fit(other_X, other_Y), but in your implementation there is no point in doing this since the network parameters are reset each time. Apart from that you can always express it as nn.fit(X, Y), followed by nn.fit(other_X, other_Y). \$\endgroup\$ – AlexV May 16 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.