New answers tagged pandas
0
Lets try another approach involving np.frompyfunc:
#Creating a df where the values for Id and Anticipation are sorted in ascending & descending
m=df.sort_values(['Id','Anticipation'],ascending=[True,False])
Anticipation Id Size
3 10 bar 10
4 9 bar 9
5 8 bar 10
6 10 baz 7
0 ...
1
Normally, I really despise the fact that itertools.groupby only groups adjacent elements with the same key... in your case, though, this seems ideal. Forgive me for just rewriting the code rather than critique'ing what you have, but using this grouping function completely changes how the overall task is best approached.
Let's use itertools.groupby to ...
3
Since you're already calling .apply, I'd stick with that approach to iteration rather than mix that with a list comprehension.
It's generally better to avoid making data modifications in-place within a function unless explicitly asked to (via an argument, like inplace=False that you'll see in many Pandas methods) or if it's made clear by the functions name ...
1
You repeat a lot of work in each loop. A simple one is extracting the tumor_sizes from the lists. Each row you do sizes = np.array(list(compare_df['tumor_size'])). If you do tumor_sizes = df["tumor_size"].apply(pd.Series) at the beginning of the calculation, you have a series with all tumorsizes, indexed the same as your df.
You can save your results in a ...
5
There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed.
Dates. Format the dates in your dataframe before the first for loop with something like
df['Date'] = pd.to_datetime(df['Date']).dt.date
Notice I’m converting the datetime ...
1
The most suspicious thing to me is this:
except ValueError:
pass
It's on a very broad try block, which leads me to believe that it was thrown in to attempt a pseudo fail-safe loop. Fail-safe loops are not a bad thing, but this isn't a great way to go about it. Contrary to most circumstances, it's actually a good idea here to broaden your caught ...
2
This seems mostly to be an API question? Assuming all that data for
each of the companies is required and there's no combined API endpoint
that can be fed with, say, 10s or 100s of symbols, this would seem to
be the best way. If so, there'd still be possible concurrent execution
options to make multiple API requests at the same time and possible get
more ...
2
One minor thing that you can do is make your function reduce-friendly:
import numpy as np
from scipy.stats import chi2_contingency
from functools import reduce
from itertools import chain, repeat
def abtest(df, args):
tot, convert = args
df = df.copy()
df['nonconvert'] = df[tot] - df[convert]
grp = np.split(df.index.values,df.index.size//2)
...
2
You're indeed doing a lot of stuff twice. The first step is recognizing that you have a problem :D
The following collection of functions could be used to decrease the repetition:
def load_df(item_name):
return df[df['item_name'] == item_name]
def get_hr_op(predicate):
""" Predicate is a lambda """
hr = final_df[predicate(final_df['total_hr'])]
...
4
leave the original data intact
Since df is the original data, adding columns to it can have a strange side effect in another part of the calculation. Best is not to touch the original data, and do all the calculations etc in separate series or copied DataFrames
pd.grouper
You can use pd.Grouper to group per day and hour
You can group the date per subject ...
2
When you have a finite number of members in a group A and B. Instead of split into groups, hstack the DataFrame like this:
pd.concat(
[
df[df.index % 2 == 0].add_prefix('a_').reset_index(),
df[df.index % 2 == 1].add_prefix('b_').reset_index()
], axis=1
)
a_grp a_tot a_lgn a_read index b_grp b_tot b_lgn b_read
0 ...
1
Finally I got an amazing improvement thanks to stack overflow, regarding two things:
https://stackoverflow.com/questions/10715965/add-one-row-to-pandas-dataframe
https://stackoverflow.com/questions/37757844/pandas-df-locz-x-y-how-to-improve-speed
Also, as hpaulj pointed, changing to json.loads slightly increases the performance.
It went from 16 hours to ...
2
The other thing that might make the code run slower is I am creating an arrow for each line, but I want the arrow to point correctly relative to the direction of travel so I need to manually calculate the direction of the arrow, which I believe is also making the code run slower.
That would be it. trace_creator() alone takes 10s on my machine, which is ...
Top 50 recent answers are included
