# Faster generation of list with multiple sublists [closed]

I'm trying to improve the performance of some rather large code. One of the bottlenecks is the line below. It doesn't look like much but it is processed millions of times.

Is there any way to improve its performance at the code level (ie: no parallelization)? I'm perfectly fine with the final outcome being a numpy array instead of a list.

import numpy as np

def func(m1, m2):
return -2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))

# Generate data.
N1, N2 = 300, 200
aa = np.random.random((5, N1))
idxs = np.random.choice(N1, N2, replace=False)
bb = np.random.random((10, N2))

# I need to improve the performance of this line.
cc = [func(m[idxs], bb[i]) for (i, m) in enumerate(aa)]


As Mibac suggested, the issue here might actually be the successive calls to func(), not the generation of the cc list itself.

If this is the case, then I'd need to improve the performance of the line:

-2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))


or equivalently:

-2.5 * np.log10(c ** m1 + c ** m2)


where c=10**(-0.4).

• It would be more helpful if we could look at the context in which this line is called million times rather than an extremelly simplified hypothetical case. – 409_Conflict Sep 8 '17 at 9:13
• The context is this code github.com/asteca/ASteCA (see this module). The particular bit of code posted here is used to turn simple star systems into binaries. – Gabriel Sep 8 '17 at 14:00

aa = np.random.random((5, N1))
idxs = np.random.choice(N1, N2, replace=False)
bb = np.random.random((10, N2))

# I need to improve the performance of this line.
cc = [func(m[idxs], bb[i]) for (i, m) in enumerate(aa)]


Unless I'm misunderstanding something, half of bb is never used. Shouldn't the 5 in the instantiation of aa and the 10 in the instantiation of bb be the same value?

def func(m1, m2):
return -2.5 * np.log10(10 ** (-0.4 * m1) + 10 ** (-0.4 * m2))


Algebraically, $$\log_{10}(10^x + 10^y) = \log_{10}(10^x (1 + 10^{y - x})) = x + \log_{10}(1 + 10^{y - x})$$ Since in this scenario $-1 \le y - x \le 1$ there doesn't seem to be any numerical-analytic reason not to simplify like this.

Depending on your error requirements, it may be possible to speed up the logarithm further by approximating it with a Taylor expansion $$\log_{10}(1 + 10^z) = \frac{\ln 2}{\ln 10} + \frac{z}{2} + \frac{\ln 10}{8} z^2 - \frac{\ln^3 10}{192} z^4 + \frac{\ln^5 10}{2880}z^6 - \ldots$$ or with a Padé approximant.

• You are understanding correctly, bb has that shape because I'm respecting the overall structure of the larger code where this block was taken from. That range of x,y is only for this example, my actual data will not necessarily have that range. I actually found a similar expansion to the one you mention, here stackoverflow.com/a/3975283/1391441 It improved substantially the execution time. Maybe you could add it to your answer? Thank you! – Gabriel Sep 6 '17 at 12:13
• It's not similar: it's the same. These are properties of logarithms so basic that some people use them as the definition of a logarithm. – Peter Taylor Sep 6 '17 at 12:33

I think the bottleneck of the line you commented is mostly the func function because as far as I know these kind of math operations are kind of expensive. Calculating 10 ** -0.4 once before and then storing it somewhere should improve performance. So later you only do storedMagicNumber ** m1 and storedMagicNumber ** m2 (you probably should choose a better name).

Disclaimer: I have no experience in Python

• Thank you Mibac. Although it sounds reasonable, there is little to no improvement with the method you proposed. I agree that func might be the issue here, though. – Gabriel Sep 5 '17 at 21:21
• How are you measuring improvement? Trying the magic # method improves my time from 138 µs ± 1.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) to 12.6 ns ± 0.0534 ns per loop (mean ± std. dev. of 7 runs, 100000000 loops each), which is much faster. Depending on the size of your data you could also expand your m1,m2 into full arrays then just call the function on the entire array once instead of calling the function n times in your list comprehension. – mochi Sep 6 '17 at 0:14
• I did @mochi, I saw almost no improvement with this approach. – Gabriel Sep 6 '17 at 12:36