I'm new to python and pandas.
I would like to use pandas groupby()
to flag values in a df that are outliers. I think I've got it working, but as I'm new to python, wanted to ask if there is a more obvious / pythonic approach.
Given input data with two groups, two variables X and Y:
n=10000
df= pd.DataFrame({'key': ['a']*n+['b']*n
,"x" : np.hstack((
np.random.normal(10, 1.0, size=n)
,np.random.normal(100, 1.0, size=n)
))
,"y" : np.hstack((
np.random.normal(20, 1.0, size=n)
,np.random.normal(200, 1.0, size=n)
))
})
To identify outliers I need to calculate the quartiles and inter-quartile range for each group to calculate the limits. Seemed reasonable to create a function:
def get_outlier(x,tukeymultiplier=2):
Q1=x.quantile(.25)
Q3=x.quantile(.75)
IQR=Q3-Q1
lowerlimit = Q1 - tukeymultiplier*IQR
upperlimit = Q3 + tukeymultiplier*IQR
return (x<lowerlimit) | (x>upperlimit)
And then use groupby()
and call the function via transform, e.g.:
g=df.groupby('key')[['x','y']]
df['x_outlierflag']=g.x.transform(get_outlier)
df['y_outlierflag']=g.y.transform(get_outlier)
df.loc[df.x_outlierflag==True]
df.loc[df.y_outlierflag==True]
I'm not worried about performance at this point, because the data are small. But not sure if there is a more natural way to do this? For example, it's not clear to me how apply() differs from transform(). Is there an apply()
approach that would be better?
Is this approach reasonably pythonic / in line with best practices? I would like to stick with pandas. I realize there are SQL approaches etc.