15
votes
Accepted
Parse complex text files using Python
There are a few performance tricks we can apply here:
add __slots__ to the class definition should help with memory and performance as well:
...
- 17.2k
13
votes
Accepted
Dropping rows from a PANDAS dataframe where some of the columns have value 0
I think you need create boolean DataFrame by compare all filtered columns values by scalar for not equality and then check all Trues per rows by ...
- 246
13
votes
Accepted
Basic function to convert Country name to ISO code using Pycountry
I always get my code to be as clean as possible before starting work on performance. Here you have a bare except, which can hide errors. Change it to something ...
- 42.7k
12
votes
Accepted
Analyze frequency and content of political fundraising E-mails
Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... ...
- 28.6k
11
votes
Accepted
Table of Tribonacci sequence using NumPy and PANDAS
You’re using the wrong tool for the job. Basically, you do all the computation in Python, use numpy for intermediate storage and ...
- 28.6k
11
votes
Lookup closest value in Pandas DataFrame
Not sure if this will help, but I'm using this to find nearest in a sorted column: (time series stuff)
...
- 111
11
votes
Accepted
William Fractal technical indicator implementation
Update: There was a post (now deleted) about parameterizing the number of shift periods, so I've added a period param to both the ...
- 701
10
votes
Chi Square Independence Test for Two Pandas DF columns
Using pandas.crosstab, this can be done in a single step:
pandas.crosstab(index=test_df['var1'],columns=test_df['var2'])
It ...
- 101
10
votes
Modifying Titration Data analysis results
I've never used numpy or matplotlib, so I can only speak to issues of style.
You're allowing for far too much nesting here. Your code consists of a giant, dense, deeply nested chunk. As a result, the ...
- 16.2k
9
votes
Accepted
Groupby and moving average function in pandas works but is slow
Iterating in Python is slow, iterating in C is fast. So, it's best to keep as much as possible within Pandas to take advantage of its C implementation and avoid Python.
You can use apply on groupby ...
- 1,124
9
votes
Calculating time deltas between rows in a Pandas dataframe
Use the diff().
x['time_delta'] = x.timestamp.diff().fillna(x['time_delta'])
This works as below, in a simpler example.
You could use the ...
- 91
8
votes
Accepted
Extract unique terms from a PANDAS series
It doesn't look like you really need regular expressions. This construct just using basic string operations is about 10x faster than the construct with the regular expressions:
...
- 4,191
8
votes
Modifying Titration Data analysis results
I'm going to put this code into an editor, "proofread" it from top to bottom, give notes as I go, and then paste the final result to show the effect of the edits.
Code editors don't usually wrap ...
- 3,990
7
votes
Accepted
Reading from a .txt file to a pandas dataframe
When opening very large files, first concern would be memory availability on your system to avoid swap on slower devices (i.e. disk).
Pandas is shipped with built-in reader methods. For example the <...
- 238
7
votes
Accepted
Interpret YYYYMMDD as the nth day of the year
As you already use pandas, you are right that there is no overhead on using Timestamps (the kind of objects returned by ...
- 28.6k
7
votes
Accepted
Code for creating combinations taking a long time to finish
You take the combinations in a way too convoluted way. For starter, I would simplify the retrieval of "same memberid" questions:
...
- 28.6k
7
votes
Basic function to convert Country name to ISO code using Pycountry
Apart from speeding up your code, there are some other improvements possible that I don't see mentioned yet.
The speedup
As other users have already picked up, you only need to transform around 200 ...
- 418
6
votes
Analyze frequency and content of political fundraising E-mails
Imagine how much of a pain it would be to add another politician to this list. In addition to the Politician class suggested in another answer, I would suggest ...
- 862
6
votes
Analyze frequency and content of political fundraising E-mails
Setting aside PEP 8 (official style guide) issues I would make the following change:
Rather than keeping your politician bins, names and email addresses in separate data structures (the only way they ...
- 161
6
votes
Finding the states with the three most populous counties
nlargest could help, it finds the maximum n values in pandas series.
In this line of code, groupby groups the frame according to state name, then apply finds the 3 largest values in column ...
- 191
6
votes
Finding the states with the three most populous counties
SUMLEV is explained here
Definitely want to use nlargest
The advantage of nlargest is that ...
- 226
6
votes
Accepted
Split latitude/longitude by degree to make file names and folder directory names
Bug and accidental quadratic runtime
...
- 1,694
6
votes
Accepted
Preprocessing steps to follow while cleaning and extracting text data from tweets
Copying my answer from SO:
You can use pandas vectorized string methods to do your processing and it also removes the for loop ...
- 176
6
votes
Comparing the size of tumors over time using PANDAS
There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed.
Dates. ...
- 61
6
votes
Table of Tribonacci sequence using NumPy and PANDAS
Some general tips
Put code in functions
That way you can test each part individually. Here the generation of the sequence, calculation of the ratio and exporting to pandas are clear divisions in the ...
- 9,100
6
votes
Accepted
Python pandas: Take column of counts and create DataFrame with a row per count
You are correct in thinking that iterrows is a very bad sign for Pandas code. Even worse is building up a DataFrame one row at a time like this with ...
- 211
6
votes
Accepted
My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report
The description of the problem in the post says that the case to be detected is redeeming "over 3 time" but what the code actually detects is 3 times or more. Which is right?
The description of the ...
- 49.6k
6
votes
Accepted
Making a dataframe of parent IDs
You should decide if your functions modify the object they receive or if the return a modified object. Doing both is just asking for disaster. After your code has finished, ...
- 40.8k
6
votes
Accepted
Python to write multiple dataframes and highlight rows inside an excel file
First, starting from your code, you should realize that you are repeating yourself, three times. This goes against the principle Don't repeat Yourself (DRY).
The only real difference between ...
- 40.8k
Only top scored, non community-wiki answers of a minimum length are eligible