1
\$\begingroup\$

Massive data problem, 7/8 variables are repeated 1000 times in a for loop, sending 8 input vectors * 50,000 values per loop for each function call. Only 1 input vector changes in each loop iteration. The function being called is vectorized accepting list inputs and can accept any length vectors. The function returns its values into the variable FEUPnL.

Is there a way to vectorize the entire for loop without making copies of the input data that stays constant? Or just not make copies of the static data. Memory issue otherwise, and very slow.

(Pandas DataFrame: FEU (input data), NumPy array S (changing values each loop))

import pandas as pd
import numpy as np

FEU = pd.read(FEU_inputs.csv)

#function outputs tuple format
FEUPnL = np.zeros(sims+1*len(FEU),dtype=tuple)
FEUPnL.resize(sims+1,len(FEU))

#function takes list inputs (FEU.x from a Pandas DataFrame)
FEU_strike = FEU.STRIKE.reset_index().values[:,1].tolist()
FEU_texpiry = FEU.TIMETOEXPIRYYEARS.reset_index().values[:,1].tolist()
FEU_vol = FEU.VOLATILITY.reset_index().values[:,1].tolist()
FEU_callput = FEU.CALL_FLAG.reset_index().values[:,1].tolist()
FEU_IR = FEU.IR.reset_index().values[:,1].tolist()
FEU_premium = np.zeros(len(FEU)).tolist()

for i in range(0,1001):
    FEUPnL[i] = optionpricer('p', FEU_callput, S[FEU.RFmap.values,i].tolist(), FEU_vol, FEU_strike, FEU_texpiry, FEU_IR, FEU_premium)
\$\endgroup\$
  • 1
    \$\begingroup\$ Welcome to Code Review! If I understand your post correctly, I believe it is off-topic. Is the code you have posted sample code, or is it real code? I only question this because you say "A code sample is below". \$\endgroup\$ – SirPython Mar 11 '16 at 23:43
  • 1
    \$\begingroup\$ Could you tell us more about what the code is actually trying to accomplish and why, rather than the mechanics of what the code is doing? See How to Ask. \$\endgroup\$ – 200_success Mar 12 '16 at 0:19
  • \$\begingroup\$ The code is calling a COM object C++ DLL repeatedly over the loop (optionpricer). It is proprietary so I can't post the DLL. This is more of a memory management question than anything else, i.e., how can I take a for loop that has 7 constant variables and 1 dynamic into a lower memory form (hopefully increasing the speed in the process). \$\endgroup\$ – Matt Mar 12 '16 at 22:48
  • \$\begingroup\$ All the tolist calls look fishy given that you're already using NumPy. Take a look at the C-API instead of converting the data frame. For the subsetting it's really hard to tell. If the only changing thing is i again to the subsetting vs. FEU.RFmap first and the i on the C side. \$\endgroup\$ – ferada Mar 13 '16 at 14:34
  • \$\begingroup\$ Yes the tolist() routines are a definite waste here in the code due to the way the COM interface was setup. I pointed him to the NumPy C-API (thanks) and he found something easier here: pythonhosted.org/comtypes although these interfaces look easy as well in boost: github.com/ndarray/Boost.NumPy/commit/… and are super memory efficient since the same memory is used to create the C++ or NumPy array (no copying). Further down this road for optimization is placing time consuming functions in the C++ side as you suggested. \$\endgroup\$ – Matt Mar 14 '16 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.