I like to compute dummy variables based on the existing values in a dataframe and a dictionary with certain "cut off" points. I wonder whether I need to get rid of the for loop in the example below. And if so, how?
The second answer in this Stackoverflow post compellingly urges to avoid looping through dataframes. In my real dataset (about 20.000 rows and 50 columns/variables) the "apply lambda" for the rows indeed performs much better than a for loop (which after a few hours still wasn't finished). Yet, the for loop for the columns doesn't really give a problem with time and it is handy to automatically name the variables.
Here is an example dataset
import pandas as pd
# create df
dict={'file':['a', 'b', 'c'],'t_0':[1.5, 2.5, 3.5], 't_1':[0.5, 1.5, 0]}
df=pd.DataFrame(dict,index=['0', '1', '3'])
print(df)
# create dictionary with cut off points for dummy variable
d = dict={'t_0_cut_off':1, 't_1_cut_off':2}
print(d)
And here the for loop and apply lambda function which I use to compute the dummy variables.
for column in df.columns[-2:]:
df[f'{column}_result']=df[f'{column}'].apply(lambda x: 1 if x> d[f'{column}_cut_off'] else 0)
df
The results look like this (and appear as I expect)
file t_0 t_1 t_0_result t_1_result
0 a 1.5 0.5 1 0
1 b 2.5 1.5 1 0
3 c 3.5 0.0 1 0