Elegant way to do the dataframe operations twice

Question

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the operations twice. Can you help me to make it efficient and elegant?

I have a dataframe df in which I have to apply certain rules (see at the bottom PEEP_con and Fio2_con). PEEP_con has to be applied only to records with item_name == PEEP (df_P) and FiO2_con to item_name == Fio2 (df_F)

Before I apply the final conditions, I do certain preprocessing tasks which is 100 pc same for both the conditions but has to be done once for df_P and df_F . Hope this helps

Please find my code below (credit to SO users for help)

df = pd.read_csv('sample_data.csv')
df['charttime'] = pd.to_datetime(df['charttime'])
df = df.loc[df.value.mod(1).eq(0)]  # removes records with values like 5.4,6.1 etc

# I am creating two dataframes here. Is it possible to do this on the fly without creating two dataframes
df_P = df[df['item_name'] == 'PEEP']
df_P['value'] = np.where(df_P['value'] < 5, 5, df_P['value'])
df_F = df[df['item_name'] == 'Fi02/100']

# for the first time, I manually keep it to df_P and later I change it to df_F manually
dfa = df_F

# below operations is same for both the dataframes
m = dfa.assign(date=dfa.charttime.dt.date).groupby(['date',dfa.value.ne(dfa.value.shift()).cumsum()]).first().reset_index(drop=True)
c = pd.to_timedelta(24,unit='h')-(m.charttime-m.charttime.dt.normalize())
m['total_hr'] = m.groupby(['subject_id'])['charttime'].diff().shift(-1).fillna(c).dt.total_seconds()/3600
final_df = m.assign(date = m.charttime.dt.date)
gone_hr = final_df[final_df['total_hr'] > 1]
lone_hr = final_df[final_df['total_hr'] < 1]
min_gone_hr_data = gone_hr.groupby(['subject_id','date'])['value'].min().reset_index()
min_lone_hr_data = lone_hr.groupby(['subject_id','date'])['value'].min().reset_index()
op1 = gone_hr.merge(min_gone_hr_data, on = ['subject_id','date','value'],how = 'inner')
op2 = lone_hr.merge(min_lone_hr_data, on = ['subject_id','date','value'],how = 'inner')
op = (op1.set_index(['subject_id','date']).combine_first(op2.set_index(['subject_id','date'])).reset_index())

# This condition is only for `df_P`
PEEP_con = lambda x: (x.shift(2).ge(x.shift(1))) & (x.ge(x.shift(2).add(3))) & (x.shift(-1).ge(x.shift(2).add(3)))

# This condition is only for `df_F`
Fi02_con = lambda x: (x.shift(2).ge(x.shift(1))) & (x.ge(x.shift(2).add(20))) & (x.shift(-1).ge(x.shift(2).add(20)))

# this field should only be applied for df_P
op['fake_VAC'] = op.groupby('subject_id')['value'].transform(PEEP_con).map({True:'fake VAC',False:''})

# this field should only be applied for df_F
op['Fi02_flag'] = op.groupby('subject_id')['value'].transform(Fi02_con).map({True:'VAC',False:''})

I request your inputs in making this more elegant and efficient. Though suggestions w.r.t to logic are welcome, I am mostly looking for ways to optimize the code structure

Currently I generate two csv file output (one for PEEP and other for Fi02). Instead I would like to have both the columns Fi02_flag and fake_VAC in the same csv file (one final dataframe)

Answer 1

You're indeed doing a lot of stuff twice. The first step is recognizing that you have a problem :D

The following collection of functions could be used to decrease the repetition:

def load_df(item_name):
   return df[df['item_name'] == item_name]

def get_hr_op(predicate):
   """ Predicate is a lambda """
   hr = final_df[predicate(final_df['total_hr'])]
   min_hr = hr.groupby(['subject_id','date'])['value'].min().reset_index()
   op = hr.merge(min_hr, on = ['subject_id','date','value'],how = 'inner')
   return op.set_index(['subject_id','date'])

def make_con(add_qty):
   def con(x):
      return (
         x.shift(2).ge(x.shift(1))
      ) & (
         x.ge(x.shift(2).add(3))
      ) & (
         x.shift(-1).ge(
            x.shift(2).add(add_qty)
         )
      )
   return con

def apply_field(name, title, add_qty):
   con = make_con(add_qty)
   op[name] = op.groupby('subject_id')['value'].transform(con).map({True:title,False:''})

You get the idea. Basically, functions are your friend. There may be some fancier way of cleaning this up using Pandas-specific stuff, but this is the Python way.