-1
\$\begingroup\$

I recently learned about backpropagation online and tried to implement it. I am not sure I have it correct yet. I am confused and would love a second pair of eyes on this code. Please help me understand how I can improve this and if this is even correct.

Implementation:

"""
Artificial Neural Network
"""

import numpy as np


def sigmoid(x, derivative=False):
    output = 1 / (1 + np.exp(-x))
    if derivative:
        return output * (1 - output)
    return output


def tanh(x, derivative=False):
    output = 2 * sigmoid(2 * x) - 1
    if derivative:
        return 1 - output ** 2
    return output


class Layer:
    def __init__(self, num_input, num_output, activation_fn=tanh):
        # num_rows = num_input, num_cols = num_output
        self.weights = activation_fn(np.random.rand(num_input, num_output))
        self.bias = activation_fn(np.random.rand(1, num_output))
        self.activation_fn = activation_fn
        self.raw = None
        self.activated = None
        self.input_data = None

    def __str__(self):
        return "Weights:\n" + str(self.weights) + "\nBias:\n" + str(self.bias)

    def output(self, x):
        self.input_data = x
        self.raw = np.dot(x, self.weights) + self.bias
        self.activated = self.activation_fn(self.raw)
        return self.activated

    def output_der(self):
        return self.activation_fn(self.activated, derivative=True)


class NeuralNetwork:
    def __init__(self, layers, eta=0.1):
        if len(layers) < 2:
            raise Exception("Layers needs to have input and output")
        self.layers = self.init_layers(layers)
        self.eta = eta

    def __str__(self):
        s = ""
        for i, l in enumerate(self.layers):
            s += "Layer {}.\n{}\n\n".format(i, str(l))
        s += "--------------"
        return s

    def init_layers(self, layers):
        ann = []
        i = 0
        while i < len(layers) - 1:
            ann.append(Layer(layers[i], layers[i + 1]))
            i += 1
        return ann

    def train(self, dataset, times=45):
        # item[0] = input
        # item[1] = label
        # backpropagation is an online algorithm (one example at a time)
        for i in range(times):
            error = 0
            output = 0
            for item in dataset:
                output = self.forward(np.array(item[0]))
                # loss_at_output_layer = -(label - output)
                self.backward(output - item[1])
                error += (item[1] - output) ** 2
            print(error)

    def hot_encode(self, output):
        top = max(output)
        for i in range(len(output)):
            output[i] = 0 if output[i] != top else 1
        return output

    def test(self, dataset):
        accuracy = 0
        for item in dataset:
            output = self.forward(np.array(item[0]))
            if np.array_equal(self.hot_encode(output[0]), item[1]):
                accuracy += 1
        print("Accuracy: {}\n".format(accuracy / len(dataset)))

    def forward(self, item):
        x = item
        for l in self.layers:
            x = l.output(x)
        return x

    def backward(self, loss):
        # delta_i = error * Δoutput_i
        deltas = []
        for l in reversed(self.layers):
            deltas.append(np.multiply(loss, l.output_der()))
            loss = np.dot(deltas[-1], l.weights.T)
            l.weights -= self.eta * np.multiply(deltas[-1], l.weights)
            l.bias -= self.eta * deltas[-1]

#---------------------------------------------------------------------#

"""
Driver Code
"""

def normalize(data):
    return (2 * (data - min(data)) / (max(data) - min(data))) - 1


def get_dataset(filename):
    with open(filename, "r") as f:
        raw = {}
        for line in f:
            line_data = line.split(",")
            feature = normalize(np.array(list(map(float, line_data[:-1]))))
            if line_data[-1] in raw:
                raw[line_data[-1]].append(feature)
            else:
                raw[line_data[-1]] = [feature]
        hot_encoding = {}
        for i, label in enumerate(raw):
            hot_encoding[label] = np.zeros(len(raw))
            hot_encoding[label][i] = 1
        dataset = []
        for label in raw:
            for feature in raw[label]:
                dataset.append([feature, hot_encoding[label]])
        return dataset


if __name__ == "__main__":
    classifier = NeuralNetwork([4, 3], eta=0.01)
    print("{}\n".format(classifier))
    dataset = get_dataset("./iris.data")
    np.random.shuffle(dataset)
    print(dataset)
    split_ratio = int(2 * len(dataset) / 3)
    classifier.train(dataset[:split_ratio], times=50)
    classifier.test(dataset[split_ratio:])
    print("{}\n".format(classifier))

I am trying to test it on the Iris dataset but it doesn't converge. I have tried the following architectures:

  • 4 Input, 3 * 3 Hidden, 3 Output (Hot encoded)
  • 4 '' , 3 '' , 3 ''
  • 4 '' , 5 '' , 3 ''

Please help me identify how to improve the algorithm, representation, and code quality. Please let me know how to improve this question to help you better answer this as well! Thank you so much!

| improve this question | | | | |
\$\endgroup\$
  • 5
    \$\begingroup\$ Please help me understand how I can improve this and if this is even correct. This site is for reviewing working code. The author should understand the code. I am not sure this is the case here. Visit our help center for more info: codereview.stackexchange.com/help/on-topic. \$\endgroup\$ – dfhwze Sep 8 '19 at 19:28
1
\$\begingroup\$

Coding Style

First, I'm not a code reviewer. Your code seems to be OK though. There are some basic coding conventions in writing Python scripts, such as variable naming, commenting, docstring, and such, which I don't go through it, since I'm learning myself, and you can find it here.

Implementation

There are a few things that hold a basic ANN not to properly converge such as:

  • IRIS dataset is a pretty small dataset, to start with; which normally one uses some 70% of a dataset for training, if supervised (which is the case here), and the rest for validation.

  • It is difficult for me to go through your mathematical debugging, but it might be a reason that the math might have some problems and the network doesn't converge. To make sure, you can test it step by step (Neuron by Neuron maybe, if you will) with a very simple training and testing dataset, much simpler than IRIS, to see if there might be some bugs.

  • If there is no bug, the architecture of ANN is another thing that would impact the convergence. I guess you might not need three layers of hiddens, and one hidden layer with maybe 10 to 30 neurons might be just OK for IRIS. Sometimes, adding too many neurons would trap the network into mathematical local minima dilemma. You might want to make sure that the Input and Output layers have the exact correct number of neurons according to the dataset.

  • There might be some related tutorials to implement ANNs from scratch, wouldn't be such a bad idea to look them up. Maybe, something with a Neuron class.

Integration

You can also apply some already built-in modules to do that, such with KNN in this case, which I'm pretty sure you know:

from sklearn import neighbors, datasets, preprocessing
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

iris = datasets.load_iris() 
X, y = iris.data[:, :], iris.target

Xtrain, Xtest, y_train, y_test = train_test_split(X, y)
scaler = preprocessing.StandardScaler().fit(Xtrain)
Xtrain = scaler.transform(Xtrain)
Xtest = scaler.transform(Xtest)

knn = neighbors.KNeighborsClassifier(n_neighbors=4)
knn.fit(Xtrain, y_train)
y_pred = knn.predict(Xtest)

print(accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))

Output

0.8947368421052632
              precision    recall  f1-score   support

           0       1.00      1.00      1.00        13
           1       0.79      0.92      0.85        12
           2       0.91      0.77      0.83        13

    accuracy                           0.89        38
   macro avg       0.90      0.90      0.89        38
weighted avg       0.90      0.89      0.89        38

[[13  0  0]
 [ 0 11  1]
 [ 0  3 10]]

Overall

I think it is great that you are trying to implement a ANN from scratch.

| improve this answer | | | | |
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.