I am using LMFIT to fit the transfer function of a large impedance network to measured data, this involves some ~20 parameters and ~40 lines of (naively implemented) computations.
there are a lot of computations that go into computing this transfer function, and many of those do not need to be repeated if the parameters that they use have not changed.
I have noticed that LMFIT does not vary all parameters all the time, it only varies a few of the ~20 odd parameters at a time, so I would like to rewrite my model to detect which parameters have changed, and then only do the computations that are needed.
Ideally I would have liked to not have to "detect" what parameters have changed and to implement this manually, but to instead have LMFIT handle it (know what computations to re-do depending on what parameters have been changed), but I have not been able to find a way to do this with the features already implemented in LMFIT.
It currently takes ~1.5 hours to do the fit, and my model will only get more complicated with time, so I really need a solution with absolute minimal overhead!. Keep in mind that is is an attempt to greatly reduce computation time, not increase it.
At first I thought that I would have implemented this in an afternoon, because I didn't really think that if would be that complicated/hard to do, now I am here hoping that someone (perhaps with experience in doing a similar thing) can help me with suggestions as to how to go about this, as it turned out to be a lot harder than I had expected.
All my computations are currently hard-coded, I did that out of fear of overhead.
z_x = 1234.56 def model(s, r1, l1, r2, l2, c2, r3, l3, r4, l4, c4, r5, l5, c5, r6, l6, r7, l7, c7, r8, l8, r9, l9, c9, r10, l10, c10, r11, l11, r12, l12, c12, v_source): z1 = r1 + l1 * s z2 = r2 + l2 * s + 1 / (c2 * s) z3 = r3 + l3 * s z4 = r4 + l4 * s + 1 / (c4 * s) z5 = r5 + l5 * s + 1 / (c5 * s) z12 = r12 + l12 * s + 1 / (c12 * s) z_a = 1/(1/(z1 + 1 / (1/z2 + 1 / (z3 + 1 / (1/z4 + 1/z5)))) + 1/z12) z6 = r6 + l6 * s z7 = r7 + l7 * s + 1 / (c7 * s) z8 = r8 + l8 * s z9 = r9 + l9 * s + 1 / (c9 * s) z10 = r10 + l10 * s + 1 / (c10 * s) z11 = 1 / (1/z7 + 1/(z8 + 1 / (1/z9 + 1/z10))) ratio = z11 / (z6 + z11) z_b = z6 * ratio + r11 + l11 * s v_b = v_source * ratio z_c = z_a + z_b return 20*np.log10(np.abs(v_source * z_x / (z_a + z_x))),\ 20*np.log10(np.abs(v_b * z_x / (z_b + z_x))),\ 20*np.log10(np.abs(v_b * z_x / (z_c + z_x)))
What I have "tried" or "considered";
- Somehow putting all input parameters in a np.array, compare it to the previous np.array of parameters to get an array of True/False for changed or not-changed, and then use this array as a "mask" whenever doing any computations on data in the array.
I have been working on implementing this approach, the main problem is that it makes the code so unreadable that before I make it to the end I am unable to read my own code and so fail to get it working.
- Wrapping all the math in custom class objects called "Expr", "Add", "Sub" etc. (like sympy and mpmath does) and then have each resulting expression object be evaluated in the last moment, and have the expression objects contain their previous value and return that if nothing has changed.
This is a solution, but not one that I am satisfied with, both because I don't want to have to write and to maintain my own library of expression wrappers etc. and because I fear the impact of the overhead that this could cause if I am not careful.
- Completely hard coded solution (If a != a_previous: ..., if b != b_previous: ...), as you may understand, this is not a solution that I am satisfied with either.
So the question is; Given the code example above, what is the most efficient method that you can come up with for only doing each of the computations whenever a value that is used in that computation has changed.
Or alternatively; If you have experience with a similar situation when using LMFIT, how did you solve it?