# Python ANN Implementation [closed]

I recently learned about backpropagation online and tried to implement it. I am not sure I have it correct yet. I am confused and would love a second pair of eyes on this code. Please help me understand how I can improve this and if this is even correct.

Implementation:

"""
Artificial Neural Network
"""

import numpy as np

def sigmoid(x, derivative=False):
output = 1 / (1 + np.exp(-x))
if derivative:
return output * (1 - output)
return output

def tanh(x, derivative=False):
output = 2 * sigmoid(2 * x) - 1
if derivative:
return 1 - output ** 2
return output

class Layer:
def __init__(self, num_input, num_output, activation_fn=tanh):
# num_rows = num_input, num_cols = num_output
self.weights = activation_fn(np.random.rand(num_input, num_output))
self.bias = activation_fn(np.random.rand(1, num_output))
self.activation_fn = activation_fn
self.raw = None
self.activated = None
self.input_data = None

def __str__(self):
return "Weights:\n" + str(self.weights) + "\nBias:\n" + str(self.bias)

def output(self, x):
self.input_data = x
self.raw = np.dot(x, self.weights) + self.bias
self.activated = self.activation_fn(self.raw)
return self.activated

def output_der(self):
return self.activation_fn(self.activated, derivative=True)

class NeuralNetwork:
def __init__(self, layers, eta=0.1):
if len(layers) < 2:
raise Exception("Layers needs to have input and output")
self.layers = self.init_layers(layers)
self.eta = eta

def __str__(self):
s = ""
for i, l in enumerate(self.layers):
s += "Layer {}.\n{}\n\n".format(i, str(l))
s += "--------------"
return s

def init_layers(self, layers):
ann = []
i = 0
while i < len(layers) - 1:
ann.append(Layer(layers[i], layers[i + 1]))
i += 1
return ann

def train(self, dataset, times=45):
# item[0] = input
# item[1] = label
# backpropagation is an online algorithm (one example at a time)
for i in range(times):
error = 0
output = 0
for item in dataset:
output = self.forward(np.array(item[0]))
# loss_at_output_layer = -(label - output)
self.backward(output - item[1])
error += (item[1] - output) ** 2
print(error)

def hot_encode(self, output):
top = max(output)
for i in range(len(output)):
output[i] = 0 if output[i] != top else 1
return output

def test(self, dataset):
accuracy = 0
for item in dataset:
output = self.forward(np.array(item[0]))
if np.array_equal(self.hot_encode(output[0]), item[1]):
accuracy += 1
print("Accuracy: {}\n".format(accuracy / len(dataset)))

def forward(self, item):
x = item
for l in self.layers:
x = l.output(x)
return x

def backward(self, loss):
# delta_i = error * Δoutput_i
deltas = []
for l in reversed(self.layers):
deltas.append(np.multiply(loss, l.output_der()))
loss = np.dot(deltas[-1], l.weights.T)
l.weights -= self.eta * np.multiply(deltas[-1], l.weights)
l.bias -= self.eta * deltas[-1]

#---------------------------------------------------------------------#

"""
Driver Code
"""

def normalize(data):
return (2 * (data - min(data)) / (max(data) - min(data))) - 1

def get_dataset(filename):
with open(filename, "r") as f:
raw = {}
for line in f:
line_data = line.split(",")
feature = normalize(np.array(list(map(float, line_data[:-1]))))
if line_data[-1] in raw:
raw[line_data[-1]].append(feature)
else:
raw[line_data[-1]] = [feature]
hot_encoding = {}
for i, label in enumerate(raw):
hot_encoding[label] = np.zeros(len(raw))
hot_encoding[label][i] = 1
dataset = []
for label in raw:
for feature in raw[label]:
dataset.append([feature, hot_encoding[label]])
return dataset

if __name__ == "__main__":
classifier = NeuralNetwork([4, 3], eta=0.01)
print("{}\n".format(classifier))
dataset = get_dataset("./iris.data")
np.random.shuffle(dataset)
print(dataset)
split_ratio = int(2 * len(dataset) / 3)
classifier.train(dataset[:split_ratio], times=50)
classifier.test(dataset[split_ratio:])
print("{}\n".format(classifier))


I am trying to test it on the Iris dataset but it doesn't converge. I have tried the following architectures:

• 4 Input, 3 * 3 Hidden, 3 Output (Hot encoded)
• 4 '' , 3 '' , 3 ''
• 4 '' , 5 '' , 3 ''

• Please help me understand how I can improve this and if this is even correct. This site is for reviewing working code. The author should understand the code. I am not sure this is the case here. Visit our help center for more info: codereview.stackexchange.com/help/on-topic. – dfhwze Sep 8 '19 at 19:28

### Coding Style

First, I'm not a code reviewer. Your code seems to be OK though. There are some basic coding conventions in writing Python scripts, such as variable naming, commenting, docstring, and such, which I don't go through it, since I'm learning myself, and you can find it here.

### Implementation

There are a few things that hold a basic ANN not to properly converge such as:

• IRIS dataset is a pretty small dataset, to start with; which normally one uses some 70% of a dataset for training, if supervised (which is the case here), and the rest for validation.

• It is difficult for me to go through your mathematical debugging, but it might be a reason that the math might have some problems and the network doesn't converge. To make sure, you can test it step by step (Neuron by Neuron maybe, if you will) with a very simple training and testing dataset, much simpler than IRIS, to see if there might be some bugs.

• If there is no bug, the architecture of ANN is another thing that would impact the convergence. I guess you might not need three layers of hiddens, and one hidden layer with maybe 10 to 30 neurons might be just OK for IRIS. Sometimes, adding too many neurons would trap the network into mathematical local minima dilemma. You might want to make sure that the Input and Output layers have the exact correct number of neurons according to the dataset.

• There might be some related tutorials to implement ANNs from scratch, wouldn't be such a bad idea to look them up. Maybe, something with a Neuron class.

### Integration

You can also apply some already built-in modules to do that, such with KNN in this case, which I'm pretty sure you know:

from sklearn import neighbors, datasets, preprocessing
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

X, y = iris.data[:, :], iris.target

Xtrain, Xtest, y_train, y_test = train_test_split(X, y)
scaler = preprocessing.StandardScaler().fit(Xtrain)
Xtrain = scaler.transform(Xtrain)
Xtest = scaler.transform(Xtest)

knn = neighbors.KNeighborsClassifier(n_neighbors=4)
knn.fit(Xtrain, y_train)
y_pred = knn.predict(Xtest)

print(accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


### Output

0.8947368421052632
precision    recall  f1-score   support

0       1.00      1.00      1.00        13
1       0.79      0.92      0.85        12
2       0.91      0.77      0.83        13

accuracy                           0.89        38
macro avg       0.90      0.90      0.89        38
weighted avg       0.90      0.89      0.89        38

[[13  0  0]
[ 0 11  1]
[ 0  3 10]]


### Overall

I think it is great that you are trying to implement a ANN from scratch.