a prototype of finding the (almost) best learning rate and initial weights so that a perceptron converges with the minimal iteration

Question

First of all, I chose the nearest data points/training examples

import numpy as np
import copy

nearest_setosa = np.array([[1.9, 0.4],[1.7, 0.5]])
nearest_versicolour = np.array([[3. , 1.1]])

and then I labeled negative examples as -1, and kept the label for the positive example.

x_train = np.concatenate((nearest_setosa, nearest_versicolour), axis=0)
y_train = [-1, -1, 1]

This is a simplified version of sign function.

def predict(x):
    if np.dot(model_w, x) + model_b >= 0:
        return 1
    else:
        return -1

I decided to update the weights once the model makes a wrong prediction.

def update_weights(idx, verbose=False):
    global model_w, model_b, eta
    model_w += eta * y_train[idx] * x_train[idx]
    model_b += eta * y_train[idx]
    if verbose:
        print(model_b)
        print(model_w)

The following code tests a bunch of learning rates(eta) and initial weights to find one which have the model converge with the minimal iteration.

eta_weights = []
for w in np.arange(-1.0, 1.0, .1):
    for eta in np.arange(.1, 2.0, .1):
        model_w = np.asarray([w, w])
        model_b = 0.0
        init_w = copy.deepcopy(w)
        for j in range(99):
            indicator = 0
            for i in range(3):
                if y_train[i] != predict(x_train[i]):
                    update_weights(i)
                else:
                    indicator+=1
            if indicator>=3:
                break
        eta_weights.append([j, eta, init_w, model_w, model_b])

I'm not sure if some classic search algorithms, e.g. binary search, are applicable to this particular case.

Is it common to loop so many layers? Is there a better way to do the job?

Answer 1

There are certainly things you could improve

This use of global is quite confusing: you are using eta, model_w and model_b as a local variables to the for eta in np.arange(.1, 2.0, .1), not as a global state. A cleaner way to do that would be to pass them as parameters to update_weights.

---

The way j is used outside of the loop is not very clear (even if it should work as intended). What I suggest is to use another variable for that (why not calling it n_iter?). It would look like:

    n_iter = 99
    for j in range(n_iter):
        indicator = 0
        ...
        if indicator>=3:
            n_iter = j
            break
    eta_weights.append([n_iter, eta, init_w, model_w, model_b])

As a bonus now you can tell if your loop broke because of the last point (n_iter=98) or if it ran until the end (n_iter=99)

---

Why are you using a copy, let even a deepcopy, here? w is only a number (non-mutable), so you can simply do: init_w = w

---

Layers of loop are not an issue as long as the code stays readable. Here I'd say it is ok. If you still want to shorten it you can use itertools.product:

import itertools
for w, eta in itertools.product(np.arange(-1.0, 1.0, .1), np.arange(.1, 2.0, .1)):
    print(f"w={w}, eta={eta}")

---

I am not sure where you want to use search algorithms. Yes, in the worst case you have to loop 300 times but there is no easy fix here.