Deep Neural Net implementation in Python3

Question

I created my first implementation of an arbitrary feed forward neural network with a simple back-propagation training implementation.

class NeuralNet(object):
    learning_rate = .1

    # Error function
    @staticmethod
    def J(guess, target):
        return 0.5 * np.linalg.norm(guess - target, 2) ** 2

    def __init__(self, structure=tuple(), activation_functions=tuple(), activation_derivatives=tuple()):
        self.w = []
        self.b = []
        self.f = list(activation_functions)
        self.df = list(activation_derivatives)

        li = len(structure) - 1
        while li > 0:
            self.w.insert(0, np.random.uniform(low=-1.0, high=1.0, size=(structure[li - 1], structure[li])))
            self.b.insert(0, np.random.uniform(low=-1.0, high=1.0, size=structure[li]))

            li -= 1

    def forward(self, x):
        z = [0] * len(self.w)
        a = [x]

        for i, (wi, bi, fi) in enumerate(zip(self.w, self.b, self.f)):
            z[i] = a[i] @ wi + bi
            a.append(fi(z[i]))

        return a[-1]

    def train(self, training_data, labels):
        # Get a permutation vector for shuffling the inputs and labels in each epoch:
        permutation = np.random.permutation(len(inputs))

        # Keeping track of all MSE values:
        errors = []

        # Training loop:
        for epoch in range(10000):

            # Shuffling the inputs and labels for each epoch:
            X = training_data[permutation]
            Y = labels[permutation]

            #                                          n
            # Keeping track of the error: MSE = 1/n * SUM ||Activation - Target|| ** 2
            #                                         i=1
            error = 0.0
            for xi, yi in zip(X, Y):
                # Forward pass:
                z = [0] * len(self.w)
                a = [xi]

                for i, (wi, bi, fi) in enumerate(zip(self.w, self.b, self.f)):
                    z[i] = a[i] @ wi + bi
                    a.append(fi(z[i]))

                # Calculate error for (xi, yi) according to 0.5 * ||xi - yi|| ** 2
                error += self.J(a[-1], yi)

                # Backwards pass:
                #  - Calculate deltas:
                layer_delta = []
                iter_zi = iter(zip(reversed(range(len(z))), reversed(z)))
                layer_delta.insert(0, (a[-1] - yi) * self.df[1](next(iter_zi)[1]))
                for i, zi in iter_zi:
                    delta_i = np.multiply(layer_delta[0] @ self.w[i+1].T, self.df[i](z[i]))
                    layer_delta.insert(0, delta_i)

                #  - Calculate weight deltas:
                weight_delta = []
                for i, di in zip(reversed(range(len(layer_delta))), reversed(layer_delta)):
                    delta_wi = a[i].reshape(-1, 1) * layer_delta[i]
                    weight_delta.insert(0, delta_wi)

                # w[i](new) := w[i](old) - LR * dJ/dw[i]
                # b[i](new) := b[i](old) - LR * dJ/db[i]
                for i in range(len(self.w)):
                    self.w[i] = self.w[i] - self.learning_rate * weight_delta[i]
                    self.b[i] = self.b[i] - self.learning_rate * layer_delta[i]

            errors.append(error / len(X))

            # Convergence testing, according to the last N errors:
            error_delta = sum(reversed(errors[-5:]))
            if error_delta < 1.0e-6:
                print("Error delta reached, ", epoch, " exiting.")
                break

        return errors

This seem to work just fine, at least it successfully learnt the XOR problem:

# Inputs and their respective labels:
inputs = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
labels = np.array([[0], [1], [1], [0]])

nn = NeuralNet(structure=(2, 2, 1),
               activation_functions=(lambda x: np.tanh(x),
                                     lambda x: .25 * x if x < 0 else x),
               activation_derivatives=(lambda x: 1.0 - np.tanh(x) ** 2,
                                       lambda x: .25 if x < 0 else 1))
nn.learning_rate = 0.1
print("Before training:")
print(nn.forward(np.array([0, 0])))
print(nn.forward(np.array([0, 1])))
print(nn.forward(np.array([1, 0])))
print(nn.forward(np.array([1, 1])))

errors = nn.train(inputs, labels)

print("After training:")
print(nn.forward(np.array([0, 0])))
print(nn.forward(np.array([0, 1])))
print(nn.forward(np.array([1, 0])))
print(nn.forward(np.array([1, 1])))

plt.plot(errors)
plt.show()

The calculations for layer_deltas seem particularly complicated, but whenever I try to simplify it, I break something. Any tips I could simplify that or any tips for the above code in general?

Answer 1

The not so arbitrary Network

Your original claim was that your network is "arbitrary". From what I see, I would tend to say that it's not so arbitrary as one might expect.

Arbitrary:

• number of neurons per layer (you call it "structure")
• activation function
• their derivatives (which means they are not totally arbitrary)

Not arbitrary:

• learning rate (0.1)
• network weight initialization
• error function (Somehow set to half of the squared Euclidean distance? Maybe you wanted the Mean Squared Error here?)
• number of epochs in training (10000)
• training mode (sample vs. batch learning)
• training termination condition (error_delta < 1.0e-6)
• optimizer
• ...

Those are all (more or less) important aspects one might like to tune without "monkey-patching" them into the original class (if possible). Some of these parameters could easily be set in the constructor or when calling train (for number of epochs etc.). The error function could also easily be set at initialization, especially since it's already a @staticmethod which does not rely on class internals.

I assume a flexible optimizer would be out of scope for what you want to do (and there are a plethora of frameworks that can do this).

The code itself

The initialization of the network weights and biases is using a while loop and is unnecessarily complicated. You don't need a while loop here since you know exactly how many iterations have to be done. So use a for loop instead, which would lead you to:

for i in range(len(structure)-1):
    self.w.append(np.random.uniform(low=-1.0, high=1.0, size=(structure[i], structure[i+1])))
    self.b.append(np.random.uniform(low=-1.0, high=1.0, size=(structure[i+1])))

Apart from not needing to keep tracking of the indexing variable, you also don't need indexing tricks with negatives indices and you are able to use .append(...) instead of inserting at the front.

Oh and while we are at initialization: there is no need to convert the functions/derivative tuples to lists. You can iterate over/index tuples the same way as over lists.

---

Speaking of iterations, this

layer_delta = []
iter_zi = iter(zip(reversed(range(len(z))), reversed(z)))
layer_delta.insert(0, (a[-1] - yi) * self.df[1](next(iter_zi)[1]))
for i, zi in iter_zi:
    delta_i = np.multiply(layer_delta[0] @ self.w[i+1].T, self.df[i](z[i]))
    layer_delta.insert(0, delta_i)

is madness™ IMHO (and there is also likely a bug with self.df[1]). See this SO post on how to reverse enumerate a Python list, if you really think you need to. I don't think you need to because you use the actual list value only once and the index in all other cases. That would bring that monster down to something like

layer_delta = [(a[-1] - yi) * self.df[-1](z[-1])]
for i in reversed(range(len(z)-1)):
    delta_i = np.multiply(layer_delta[0] @ self.w[i+1].T, self.df[i](z[i]))
    layer_delta.insert(0, delta_i)

You can and should adapt your other appearances of the monster above as well. Work carefully while refactoring these parts of your code, I cannot guarantee that I didn't make a mistake while deciphering them.

---

The implementation of the forward pass is also a little bit to complicated IMHO. Since you're only going through the network once, there is no need to store the activation and output of all layers. You only need the last one.

You also implement the forward pass twice, in it's own forward function and in train. If you stick to your original implementation, think about if you would like to return the other values as well. That would leave you with only one piece of code to fix if something is wrong. If you're concerned about usability since the application phase now sees all the internal values, you could implement an internal function , e.g. _forward which does the heavy-lifting and let forward just return the final output.

---

The comment and variable names at

# Convergence testing, according to the last N errors:
error_delta = sum(reversed(errors[-5:]))
if error_delta < 1.0e-6:
    print("Error delta reached, ", epoch, " exiting.")
    break

are also a little off. The termination criterion does seem to check if the sum of the last N (where N is hard coded to 5) is below your (arbitrarily chosen) threshold. Also, there is no need to reverse the list here since sum does not care about the order.

Since you are working in Python 3, you can also use the new f-string syntax for output formatting:

print(f"Error delta reached in {epoch} exiting.")

---

Speaking of comments, the methods of your class all lack user visible documentation. For that, Python programmers usually use so called """doc strings""" on their methods/classes/functions. As an example:

def train(self, training_data, labels):
    """Train the neural network with all the available data

    The training continues until the maximum number of epochs is reached or
    the termination criterion is hit.
    """
    # your code here ...

Documentation written in this kind of format will be picked up by all major Python IDEs as well as by Python's built-in help(...) function.

---

I know there are common naming conventions in the neural network community, and when you implement it, you should stick to them as closely as possible which you mostly do. But bear in mind not to sacrifice the clarity and readability of your code. E.g. z could become activation and a could also be named layer_output (plurals might apply where they are used as list).