I need to randomly erase 33% of the non-zero elements in each line of a matrix.

Side note: The end goal is to train a denoising autoencoder to remove this noise.

My solution:

import numpy as np

matrix = np.random.rand(1000,3000)

for i in range(matrix.shape[0]):
    clean = matrix[i, :] # original matrix line
    # find non zero elements
    msk = np.nonzero(clean)
    assert sum(msk[0]) != 0
    # keep 66% of them
    idx = np.random.randint(0, len(msk[0]), size=max(1, len(msk[0])//3)) #erase at least 1
    msk = np.delete(msk, idx)

    dirty = clean
    dirty = [j if i in msk else 0 for i,j in enumerate(dirty)]
    assert sum(clean-dirty) != 0
    #save clean and dirty

My guess is that the bottleneck is drawing random numbers at each iteration.

Is there an alternative way to do this?


This seems to run quickly.

The idea is to use nonzero() grab the indices of non-zero elements. Use choice() to select 1/3 of them. and set those indices to zero. Repeat for each row in the matrix.

rng = np.random.default_rng()

matrix = np.random.rand(1000,3000)

for k in range(matrix.shape[0]):
    r = matrix[k,:].nonzero()[0]
    n = round(len(r) * 0.33)
    i = rng.choice(r, n, replace=False)
    matrix[k,i] = 0
  • \$\begingroup\$ Why switch from their len(r) // 3 to round(len(r) * 0.33)? More code and smaller values... \$\endgroup\$ – superb rain Jul 30 '20 at 19:22
  • \$\begingroup\$ Also, you removed their "erase at least 1" aspect. \$\endgroup\$ – superb rain Jul 30 '20 at 19:29
  • \$\begingroup\$ @superb rain, I missed the "erase at least one", because the comment was off the screen. Using floor divide (//) means that at most 33% will be zeroed. By using round(), I thought the average over all the lines would be closer to 33%. Preferring * over / is just a very old habit. \$\endgroup\$ – RootTwo Jul 30 '20 at 22:26
  • \$\begingroup\$ 3000 // 3 is 1000. That's more than 33%. And more than your 990. \$\endgroup\$ – superb rain Jul 30 '20 at 22:32
  • \$\begingroup\$ Ah, I see. In the text, the OP said "33%". Meh, I suspect they really meant a third, as suggested by their code. \$\endgroup\$ – superb rain Jul 30 '20 at 22:41

My guess is that the bottleneck is drawing random numbers at each iteration.

Why guess when you can measure? I changed the height to 100 so the whole thing takes me about 8 seconds, reasonable to play around with. Removing the assert line on the bottom makes it a bit faster. Also removing the dirty = [j if...] line makes it a lot faster. As you'd expect from that inefficient line.

Not sure why you're making it so complicated. I think clean[idx] = 0 does the deletions you want, and it's fast.

  • \$\begingroup\$ I'm not sure. If I do dirty[idx] = 0 then dirty is unchanged and still equal to clean (assertion error). But I agree, that line sucks. \$\endgroup\$ – shamalaia Jul 29 '20 at 10:53
  • \$\begingroup\$ What makes you think dirty is unchanged? It's not. About them being equal after the change: Yeah, of course they are. With dirty = clean you make them the same object. So if you change that object through one of those variables, of course you then also see the change through the other variable. If you want dirty to be a copy (which you can change independently), then make it a copy. Although I'd maybe instead simply store the clean version before making the change. \$\endgroup\$ – superb rain Jul 29 '20 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.