I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the operations twice. Can you help me to make it efficient and elegant?

I have a dataframe df in which I have to apply certain rules (see at the bottom PEEP_con and Fio2_con). PEEP_con has to be applied only to records with item_name == PEEP (df_P) and FiO2_con to item_name == Fio2 (df_F)

Before I apply the final conditions, I do certain preprocessing tasks which is 100 pc same for both the conditions but has to be done once for df_P and df_F . Hope this helps

Please find my code below (credit to SO users for help)

df = pd.read_csv('sample_data.csv')
df['charttime'] = pd.to_datetime(df['charttime'])
df = df.loc[df.value.mod(1).eq(0)]  # removes records with values like 5.4,6.1 etc

# I am creating two dataframes here. Is it possible to do this on the fly without creating two dataframes
df_P = df[df['item_name'] == 'PEEP']
df_P['value'] = np.where(df_P['value'] < 5, 5, df_P['value'])
df_F = df[df['item_name'] == 'Fi02/100']

# for the first time, I manually keep it to df_P and later I change it to df_F manually
dfa = df_F

# below operations is same for both the dataframes
m = dfa.assign(date=dfa.charttime.dt.date).groupby(['date',dfa.value.ne(dfa.value.shift()).cumsum()]).first().reset_index(drop=True)
c = pd.to_timedelta(24,unit='h')-(m.charttime-m.charttime.dt.normalize())
m['total_hr'] = m.groupby(['subject_id'])['charttime'].diff().shift(-1).fillna(c).dt.total_seconds()/3600
final_df = m.assign(date = m.charttime.dt.date)
gone_hr = final_df[final_df['total_hr'] > 1]
lone_hr = final_df[final_df['total_hr'] < 1]
min_gone_hr_data = gone_hr.groupby(['subject_id','date'])['value'].min().reset_index()
min_lone_hr_data = lone_hr.groupby(['subject_id','date'])['value'].min().reset_index()
op1 = gone_hr.merge(min_gone_hr_data, on = ['subject_id','date','value'],how = 'inner')
op2 = lone_hr.merge(min_lone_hr_data, on = ['subject_id','date','value'],how = 'inner')
op = (op1.set_index(['subject_id','date']).combine_first(op2.set_index(['subject_id','date'])).reset_index())

# This condition is only for `df_P`
PEEP_con = lambda x: (x.shift(2).ge(x.shift(1))) & (x.ge(x.shift(2).add(3))) & (x.shift(-1).ge(x.shift(2).add(3)))

# This condition is only for `df_F`
Fi02_con = lambda x: (x.shift(2).ge(x.shift(1))) & (x.ge(x.shift(2).add(20))) & (x.shift(-1).ge(x.shift(2).add(20)))

# this field should only be applied for df_P
op['fake_VAC'] = op.groupby('subject_id')['value'].transform(PEEP_con).map({True:'fake VAC',False:''})

# this field should only be applied for df_F
op['Fi02_flag'] = op.groupby('subject_id')['value'].transform(Fi02_con).map({True:'VAC',False:''})

I request your inputs in making this more elegant and efficient. Though suggestions w.r.t to logic are welcome, I am mostly looking for ways to optimize the code structure

Currently I generate two csv file output (one for PEEP and other for Fi02). Instead I would like to have both the columns Fi02_flag and fake_VAC in the same csv file (one final dataframe)


1 Answer 1


You're indeed doing a lot of stuff twice. The first step is recognizing that you have a problem :D

The following collection of functions could be used to decrease the repetition:

def load_df(item_name):
   return df[df['item_name'] == item_name]

def get_hr_op(predicate):
   """ Predicate is a lambda """
   hr = final_df[predicate(final_df['total_hr'])]
   min_hr = hr.groupby(['subject_id','date'])['value'].min().reset_index()
   op = hr.merge(min_hr, on = ['subject_id','date','value'],how = 'inner')
   return op.set_index(['subject_id','date'])

def make_con(add_qty):
   def con(x):
      return (
      ) & (
      ) & (
   return con

def apply_field(name, title, add_qty):
   con = make_con(add_qty)
   op[name] = op.groupby('subject_id')['value'].transform(con).map({True:title,False:''})

You get the idea. Basically, functions are your friend. There may be some fancier way of cleaning this up using Pandas-specific stuff, but this is the Python way.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.