Improve Performance of Comparing two Numpy Arrays

Question

I had a code challenge for a class I'm taking that built a NN algorithm. I got it to work but I used really basic methods for solving it. There are two 1D NP Arrays that have values 0-2 in them, both equal length. They represent two different trains and test data The output is a confusion matrix that shows which received the right predictions and which received the wrong (doesn't matter ;).

This code is correct - I just feel I took the lazy way out working with lists and then turning those lists into a ndarray. I would love to see if people have some tips on maybe utilizing Numpy for this? Anything Clever?

import numpy as np

x = [0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
     2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 2, 0, 0, 0, 0, 0, 1, 0]
y = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
     2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

testy = np.array(x)
testy_fit = np.array(y)

row_no = [0,0,0]
row_dh = [0,0,0]
row_sl = [0,0,0]

# Code for the first row - NO
for i in range(len(testy)):
    if testy.item(i) == 0 and testy_fit.item(i) == 0:
        row_no[0] += 1
    elif testy.item(i) == 0 and testy_fit.item(i) == 1:
        row_no[1] += 1
    elif testy.item(i) == 0 and testy_fit.item(i) == 2:
        row_no[2] += 1

# Code for the second row - DH
for i in range(len(testy)):
    if testy.item(i) == 1 and testy_fit.item(i) == 0:
        row_dh[0] += 1
    elif testy.item(i) == 1 and testy_fit.item(i) == 1:
        row_dh[1] += 1
    elif testy.item(i) == 1 and testy_fit.item(i) == 2:
        row_dh[2] += 1

# Code for the third row - SL
for i in range(len(testy)):
    if testy.item(i) == 2 and testy_fit.item(i) == 0:
        row_sl[0] += 1
    elif testy.item(i) == 2 and testy_fit.item(i) == 1:
        row_sl[1] += 1
    elif testy.item(i) == 2 and testy_fit.item(i) == 2:
        row_sl[2] += 1

confusion = np.array([row_no,row_dh,row_sl])

print(confusion)

the result of the print is correct as follow:

[[16 10  0]
 [ 2 10  0]
 [ 2  0 22]]

Answer 1

This can be implemented concisely by using numpy.add.at:

In [2]: c = np.zeros((3, 3), dtype=int)                                         

In [3]: np.add.at(c, (x, y), 1)                                                 

In [4]: c                                                                       
Out[4]: 
array([[16, 10,  0],
       [ 2, 10,  0],
       [ 2,  0, 22]])

Answer 2

For now disregarding that there is a (way) better numpy solution to this, as explained in the answer by @WarrenWeckesser, here is a short code review of your actual code.

• testy.item(i) is a very unusual way to say testy[i]. It is probably also slower as it involves an attribute lookup.

• Don't repeat yourself. You test e.g. if testy.item(i) == 0 three times, each time with a different second condition. Just nest them in an if block:

  for i in range(len(testy)):
      if testy[i] == 0:
          if testy_fit[i] == 0:
              row_no[0] += 1
          elif testy_fit[i] == 1:
              row_no[1] += 1
          elif testy_fit[i] == 2:
              row_no[2] += 1
  

• Loop like a native. Don't iterate over the indices of iterables, iterate over the iterable(s)! You can also use the fact that the value encodes the position you want to increment:

  for test, fit in zip(testy, testy_fit):
      if test == 0 and fit in {0, 1, 2}:
          row_no[fit] += 1
  

• You can even use the fact that the first value encodes the list you want to use and iterate only once. Or even better, make it a list of lists right away:

  n = 3
  confusion_matrix = [[0] * n for _ in range(n)]
  for test, fit in zip(testy, testy_fit):
      confusion_matrix[test][fit] += 1
  
  print(np.array(confusion_matrix))
  

• Don't put everything into the global space, to be run whenever you interact with the script at all. Put your code into functions, document them with a docstring, and execute them under a if __name__ == "__main__": guard, which allows you to import from this script from another script without your code running:

  def confusion_matrix(x, y):
      """Return the confusion matrix for two vectors `x` and `y`.
      x and y must only have values from 0 to n and 0 to m, respectively.
      """
      n, m = np.max(x) + 1, np.max(y) + 1
      matrix = [[0] * m for _ in range(n)]
      for a, b in zip(x, y):
          matrix[a][b] += 1
      return matrix
  
  if __name__ == "__main__":
      x = ...
      y = ...
      print(np.array(confusion_matrix(x, y)))
  

Once you have come this far, you can just swap the implementation of this function to the faster numpy one without changing anything (except that it then directly returns a numpy.array instead of a list of lists).