Simple Java Neural Network

Question

I've written a toy neural network in Java. I ran it several million times with the same outputs with only the randomized weights changing from run to run. The average of all of the outputs is not 0.5, as I would have expected. The code is in this Github Repository.

The Layer class:

public class Layer extends AbstractLayer {

    private double[][] weights;

    public Layer(int neurons, int prevLayerNeurons) {
        super(neurons);
        weights = new double[neurons][prevLayerNeurons + 1];
        randomize(weights);
    }

    protected void randomize(double[][] x) {
        for (int j = 0; j < x.length; j++) {
            for (int i = 0; i < x[j].length; i++) {
                weights[j][i] = Math.random();
            }
        }
    }

    @Override
    public double[] compute(double inputs[]) {
        Preconditions.checkArgument(inputs.length == weights[0].length - 1, "incorrect number of inputs");
        double[] ret = new double[neurons];
        for (int i = 0; i < neurons; i++) {
            double acc = 0;
            for (int j = 0; j < inputs.length; j++) {
                acc += inputs[j] * weights[i][j];
            }
            acc -= weights[i][weights[i].length - 1];
            acc = sigmoid(acc);
            ret[i] = acc;
        }
        return ret;
    }

    private double sigmoid(double x) {
        return 1 / (1 + Math.exp(-x));
    }
}

And the Network class:

public class Network {

    private List<AbstractLayer> layers;

    public Network(int[] neuronCounts) {
        layers = new LinkedList<AbstractLayer>();
        layers.add(new InputLayer(neuronCounts[0]));
        for (int i = 1; i < neuronCounts.length; i++) {
            layers.add(new Layer(neuronCounts[i], neuronCounts[i - 1]));
        }
    }

    public boolean[] run(double[] inputs) {
        double[] tmps = inputs;
        for (int i = 0; i < layers.size(); i++) {
            tmps = layers.get(i).compute(tmps);
        }
        boolean[] rets = new boolean[tmps.length];
        for (int i = 0; i < rets.length; i++) {
            rets[i] = tmps[i] > 0.5;
        }
        return rets;
    }

    public AbstractLayer getLayer(int i) {
        return layers.get(i);
    }

}

These are the most relevant classes, but there are several others in the repository.

As it stands now, the class Bootstrap will run a 3-layer network with a fixed arrangement and set of inputs.

I'd appreciate any comments on the structure of the program, and on the correctness (or incorrectness) of the output.

Answer 1

Finding the average output

Let's figure out what your average output should be. Your neural network has 3 inputs in the first layer, 2 nodes in the second layer, and one output. Each weight is randomized to a value from 0..1, so call it 0.5 on average.

The inputs you use in the program are: 1, 0, 1. On each layer, you also have a "bias" input of -1. So with average weights of 0.5, the the input layer will do the following:

inputs = 1, 0, 1, -1
output = sigmoid(1*0.5 + 0*0.5 + 1*0.5 - 1*0.5) = sigmoid(0.5) = 0.62

The second layer has two nodes, each with average input 0.62. It will do:

inputs = 0.62, 0.62, -1
output = sigmoid(0.62*0.5 + 0.62*0.5 - 1*0.5) = sigmoid(0.12) = 0.53

So your average output should be 0.53. I modified your program to sum the outputs and found that this was close (actual was 0.528). Now as for the percentage of time the output is above 0.5, that depends on the distribution of the output, and I don't know that it is easy to compute by hand. But your program shows through experimentation that the answer is roughly 60% of the time.

Random weights

I think that your choice of random weights from the range 0..1 is the source of your confusion. If you were to choose random weights in the range -1..1 like this:

weights[j][i] = Math.random()*2 - 1.0;

then your output would be 0.5 on average and the percentage of outputs greater than 0.5 would be 50% (I modified your program to verify this). Perhaps that is what you were expecting.

As far as your neural net code goes, it appears to be correct as far as I can tell.

Answer 2

1. The randomize method is an overridable method and you calling it from the constructor, which is quite dangerous in Java. Methods which you call from the constructor shouldn't be overridable, i.e.: private or final or static. Check this out.

2. The precondition in the compute method would throw an exception if the input is null, so you can add an additional guard for that:

   Preconditions.checkNotNull(inputs);
   

3. At #Layer(neurons,prevLayerNeurons), you're creating arrays based on the passed arguments. Are sure that the passed arguments are valid as an array length? Why don't you protect it?

   Preconditions.checkArgument(neurons > 0);
   Preconditions.checkArgument(prevLayerNeurons > 0);