Feed forward neural network

Question

I have made a basic neural network in python. The idea is the neural network can have any structure you want, not just the standard layers where every neuron is connected to every neuron in the next layer.

from numpy import exp

class Feed_forward_network:
    """
    Feed_forward_network
    inputs: the number of inputs, int
    outputs: the number of outputs, int
    neuron_data: the neuron data, list of tuples|None

    the first inputs of neuron_data needs to be None
    each item in neuron_data is data about the neuron
    tuple[0]: the activation function of the neuron, function(float) -> float
    tuple[1]: the bias of the neuron, float
    tuple[2]: the connections of the neuron, list of tuples

    each item in connections is data about the connection
    tuple[0]: the neuron the connection is to
    tuple[1]: the weight of the connection
    """
    def __init__(self, inputs: int, outputs: int, neuron_data):
        if outputs > len(neuron_data):
            raise RuntimeError("outputs < len(neuron_data)")
        self.inputs = inputs
        self.outputs = outputs
        self.neuron_data = neuron_data
        self.neuron_values = [None]*(len(neuron_data))

    def activate(self, inputs):
        if self.inputs != len(inputs):
            raise RuntimeError("self.inputs != len(inputs)")
        
        self.neuron_values = [None]*len(self.neuron_values)
        
        for i in range(len(inputs)):
            self.neuron_values[i] = inputs[i]
        
        return tuple([self.calculate_neuron(i+self.inputs) for i in range(self.outputs)])
    
    def calculate_neuron(self, neuron):
        if neuron < self.inputs:
            return self.neuron_values[neuron]
        
        neuron_value = self.neuron_values[neuron]

        if neuron_value == None:
            neuron_data = self.neuron_data[neuron]

            self.neuron_values[neuron] = 0 # avoid RecursionError

            value = neuron_data[0](sum([self.calculate_neuron(conn)*weight for conn, weight in neuron_data[2]]) * neuron_data[1])
            
            self.neuron_values[neuron] = value
            return value

        return neuron_value
        

def sigmoid(x: float) -> float:
    return 1 / (1 + exp(-(x)))

if __name__ == "__main__":
    ffn = Feed_forward_network(2, 1, [None, None, (sigmoid, 1.0, [(0, 1.0), (1, 1.0), (3, 1.0)]), (sigmoid, 1.0, [(0, 1.0), (1, 1.0), (3, 1.0)])])
    print(ffn.activate([1, 1]))
    print(ffn.neuron_values)

I am mainly looking for performance improvements so here is the output when profiled (I made a for loop that repeats the if __name__ == "__main__" bit 100 times because they were just all 0.000 otherwise):

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(__new__)
    100    0.001    0.000    0.001    0.000 <string>:20(__init__)
    100    0.001    0.000    0.009    0.000 <string>:28(activate)
        1    0.000    0.000    0.000    0.000 <string>:3(Feed_forward_network)
    100    0.000    0.000    0.008    0.000 <string>:37(<listcomp>)
700/100    0.002    0.000    0.008    0.000 <string>:39(calculate_neuron)
200/100    0.001    0.000    0.005    0.000 <string>:50(<listcomp>)
    200    0.005    0.000    0.005    0.000 <string>:58(sigmoid)

Answer 1

    the first inputs of neuron_data needs to be None
    each item in neuron_data is data about the neuron
    tuple[0]: the activation function of the neuron, function(float) -> float
    tuple[1]: the bias of the neuron, float
    tuple[2]: the connections of the neuron, list of tuples

Thank you for the """docstring""", it is helpful, I appreciate it.

But rather than a class docstring, please move it down one line so it is a method docstring.

I found the in / out parameters slightly surprising. Maybe name them num_inputs / num_outputs ?

Giving details about the three tuple elements is very helpful and I thank you, your heart is in the right place, you want folks to be clear on what each element means. But we could do better by passing in a namedtuple or a dataclass, letting us reason about names rather than integer indexes.

Pep-8 asks that you spell it class FeedForwardNetwork:

---

Kudos on the error checking. I really like that we raise if caller messed up the contract.

---

            neuron_data = self.neuron_data[neuron]
            ...            
            self.neuron_values[neuron] = value

That 1st line is not terrific, as we're using the same spelling of an identifier to represent two very different things. I guess I could ask that you assign to neuron_datum, singular? But that still doesn't seem very satisfying. The trouble is that "data" is always super vague, and it's not helping us to pin things down right here. Maybe your namedtuple will be a Neuron and then the naming will be self explanatory?

In contrast, the 2nd assignment uses plural and singular properly, making the meaning clear to the Gentle Reader.

---

Thank you for the profiler output -- you're on the right path! But it looks like you want to iterate more than a hundred times. Or use a bigger neural net, closer to the size you care about in production.

About half the time is spent on the activation function, evaluating exp() and reciprocal. Consider benching a table-driven approximation.

Consider benching ReLU or other alternative activation functions. There's a whole literature devoted just to that.

If we spend half the cycles in sigmoid(), I have to believe that there's a lot of function call overhead. The code is admirably simple. You say you care about speed. Consider writing a vector of input values into a numpy ndarray, and then a single function call evaluates N activation functions all at once. And I apologize, as such batching will make the code less simple.

def sigmoid(x: float) -> float:

    return 1 / (1 + exp(-(x)))

I initially thought we were calling math.exp on a single value.

Please rephrase it explicitly as np.exp(-x),

And, I guess we're passing in just a single scalar? Doing vector operations on N values would give you just 1/N-th the function call overhead.

---

This class achieves many of its design objectives.

I would be willing to delegate or accept maintenance tasks on this codebase.

Answer 2

Nit:

In calculate_neuron(), there's a block controlled by if neuron_value == None: (use is/not None (PEP 8)). Fine, but it is followed by an unconditional return <value>:
Use the negated condition for an early out and save one indentation level - "mentally", too.

    if neuron_value is not None:
        return neuron_value

(revised version:
if neuron_value is not None or neuron < self.num_inputs:)

Answer 3

Thanks @J_H. Here is a revised version of my code:

from numpy import exp

class FeedForwardNetwork:
    """
    FeedForwardNetwork
    ==================
    - num_inputs: the number of inputs, int
    - num_outputs: the number of outputs, int
    - neuron_data: the neuron data, list of tuples|None

    The first num_outputs of neuron_data are ouput neurons
    ------------------------------------------------------
    Each item in neuron_data is data about the neuron
    -------------------------------------------------
    - tuple[0]: the activation function of the neuron, function(float) -> float
    - tuple[1]: the bias of the neuron, float
    - tuple[2]: the connections of the neuron, list of tuples

    Each item in connections is data about the connection
    -----------------------------------------------------
    - tuple[0]: the neuron the connection is to
    - tuple[1]: the weight of the connection
    """
    def __init__(self, num_inputs: int, num_outputs: int, neuron_data: tuple):
        if num_outputs > len(neuron_data):
            raise RuntimeError("outputs < len(neuron_data)")
        self.num_inputs = num_inputs
        self.num_outputs = num_outputs
        self.neuron_data = neuron_data
        self.neuron_values = [None]*(len(neuron_data))
        self.inputs = [None]*num_inputs

    def activate(self, inputs):
        if self.num_inputs != len(inputs):
            raise RuntimeError("self.num_inputs != len(inputs)")
        
        self.neuron_values = [None]*len(self.neuron_values)
        
        for i in range(len(inputs)):
            self.inputs[i] = inputs[i]
        
        return tuple([self.calculate_neuron(i) for i in range(self.num_outputs)])
    
    def calculate_neuron(self, neuron):
        if neuron < 0:
            return self.inputs[neuron]

        neuron_value = self.neuron_values[neuron]

        if neuron_value is not None:
            return neuron_value

        current_neuron_data = self.neuron_data[neuron]

        self.neuron_values[neuron] = 0 # avoid RecursionError

        value = current_neuron_data[0](sum([self.calculate_neuron(conn)*weight for conn, weight in current_neuron_data[2]]) * current_neuron_data[1])
        
        self.neuron_values[neuron] = value
        return value
        

def sigmoid(x: float) -> float:
    return 1 / (1 + exp(-(x)))

if __name__ == "__main__":
    ffn = FeedForwardNetwork(2, 1, [(sigmoid, 1.0, [(-1, 1.0), (-2, 1.0), (1, 1.0)]), (sigmoid, 1.0, [(-1, 1.0), (-2, 1.0), (1, 1.0)])])
    print(ffn.activate([1, 1]))
    print(ffn.neuron_values)

• reorganised the calculate_neuron() functions if logic again
• renamed inputs and outputs to num_inputs and num_outputs
• renamed neuron_data to current_neuron_data
• reorganized how the neuron_data and neuron_values are used
• changed docstring