New answers tagged

0

Lets try another approach involving np.frompyfunc: #Creating a df where the values for Id and Anticipation are sorted in ascending & descending m=df.sort_values(['Id','Anticipation'],ascending=[True,False]) Anticipation Id Size 3 10 bar 10 4 9 bar 9 5 8 bar 10 6 10 baz 7 0 ...


1

Normally, I really despise the fact that itertools.groupby only groups adjacent elements with the same key... in your case, though, this seems ideal. Forgive me for just rewriting the code rather than critique'ing what you have, but using this grouping function completely changes how the overall task is best approached. Let's use itertools.groupby to ...


3

Since you're already calling .apply, I'd stick with that approach to iteration rather than mix that with a list comprehension. It's generally better to avoid making data modifications in-place within a function unless explicitly asked to (via an argument, like inplace=False that you'll see in many Pandas methods) or if it's made clear by the functions name ...


1

You repeat a lot of work in each loop. A simple one is extracting the tumor_sizes from the lists. Each row you do sizes = np.array(list(compare_df['tumor_size'])). If you do tumor_sizes = df["tumor_size"].apply(pd.Series) at the beginning of the calculation, you have a series with all tumorsizes, indexed the same as your df. You can save your results in a ...


5

There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed. Dates. Format the dates in your dataframe before the first for loop with something like df['Date'] = pd.to_datetime(df['Date']).dt.date Notice I’m converting the datetime ...


1

The most suspicious thing to me is this: except ValueError: pass It's on a very broad try block, which leads me to believe that it was thrown in to attempt a pseudo fail-safe loop. Fail-safe loops are not a bad thing, but this isn't a great way to go about it. Contrary to most circumstances, it's actually a good idea here to broaden your caught ...


2

This seems mostly to be an API question? Assuming all that data for each of the companies is required and there's no combined API endpoint that can be fed with, say, 10s or 100s of symbols, this would seem to be the best way. If so, there'd still be possible concurrent execution options to make multiple API requests at the same time and possible get more ...


2

One minor thing that you can do is make your function reduce-friendly: import numpy as np from scipy.stats import chi2_contingency from functools import reduce from itertools import chain, repeat def abtest(df, args): tot, convert = args df = df.copy() df['nonconvert'] = df[tot] - df[convert] grp = np.split(df.index.values,df.index.size//2) ...


2

You're indeed doing a lot of stuff twice. The first step is recognizing that you have a problem :D The following collection of functions could be used to decrease the repetition: def load_df(item_name): return df[df['item_name'] == item_name] def get_hr_op(predicate): """ Predicate is a lambda """ hr = final_df[predicate(final_df['total_hr'])] ...


4

leave the original data intact Since df is the original data, adding columns to it can have a strange side effect in another part of the calculation. Best is not to touch the original data, and do all the calculations etc in separate series or copied DataFrames pd.grouper You can use pd.Grouper to group per day and hour You can group the date per subject ...


2

When you have a finite number of members in a group A and B. Instead of split into groups, hstack the DataFrame like this: pd.concat( [ df[df.index % 2 == 0].add_prefix('a_').reset_index(), df[df.index % 2 == 1].add_prefix('b_').reset_index() ], axis=1 ) a_grp a_tot a_lgn a_read index b_grp b_tot b_lgn b_read 0 ...


1

Finally I got an amazing improvement thanks to stack overflow, regarding two things: https://stackoverflow.com/questions/10715965/add-one-row-to-pandas-dataframe https://stackoverflow.com/questions/37757844/pandas-df-locz-x-y-how-to-improve-speed Also, as hpaulj pointed, changing to json.loads slightly increases the performance. It went from 16 hours to ...


2

The other thing that might make the code run slower is I am creating an arrow for each line, but I want the arrow to point correctly relative to the direction of travel so I need to manually calculate the direction of the arrow, which I believe is also making the code run slower. That would be it. trace_creator() alone takes 10s on my machine, which is ...


Top 50 recent answers are included