15 votes
Accepted

Parse complex text files using Python

There are a few performance tricks we can apply here: add __slots__ to the class definition should help with memory and performance as well: ...
alecxe's user avatar
  • 17.3k
13 votes
Accepted

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I think you need create boolean DataFrame by compare all filtered columns values by scalar for not equality and then check all Trues per rows by ...
jezrael's user avatar
  • 246
13 votes
Accepted

Basic function to convert Country name to ISO code using Pycountry

I always get my code to be as clean as possible before starting work on performance. Here you have a bare except, which can hide errors. Change it to something ...
Peilonrayz's user avatar
  • 43.3k
12 votes

Lookup closest value in Pandas DataFrame

Not sure if this will help, but I'm using this to find nearest in a sorted column: (time series stuff) ...
Doug Griggs's user avatar
12 votes
Accepted

William Fractal technical indicator implementation

Update: There was a post (now deleted) about parameterizing the number of shift periods, so I've added a period param to both the ...
tdy's user avatar
  • 875
11 votes
Accepted

Table of Tribonacci sequence using NumPy and PANDAS

You’re using the wrong tool for the job. Basically, you do all the computation in Python, use numpy for intermediate storage and ...
301_Moved_Permanently's user avatar
10 votes

Chi Square Independence Test for Two Pandas DF columns

Using pandas.crosstab, this can be done in a single step: pandas.crosstab(index=test_df['var1'],columns=test_df['var2']) It ...
Thameem Ansari's user avatar
10 votes

Modifying Titration Data analysis results

I've never used numpy or matplotlib, so I can only speak to issues of style. You're allowing for far too much nesting here. Your code consists of a giant, dense, deeply nested chunk. As a result, the ...
Carcigenicate's user avatar
9 votes
Accepted

Groupby and moving average function in pandas works but is slow

Iterating in Python is slow, iterating in C is fast. So, it's best to keep as much as possible within Pandas to take advantage of its C implementation and avoid Python. You can use apply on groupby ...
mochi's user avatar
  • 1,124
9 votes

Calculating time deltas between rows in a Pandas dataframe

Use the diff(). x['time_delta'] = x.timestamp.diff().fillna(x['time_delta']) This works as below, in a simpler example. You could use the ...
PhasorLaser's user avatar
8 votes
Accepted

Extract unique terms from a PANDAS series

It doesn't look like you really need regular expressions. This construct just using basic string operations is about 10x faster than the construct with the regular expressions: ...
Stephen Rauch's user avatar
8 votes

Modifying Titration Data analysis results

I'm going to put this code into an editor, "proofread" it from top to bottom, give notes as I go, and then paste the final result to show the effect of the edits. Code editors don't usually wrap ...
Samwise's user avatar
  • 4,000
7 votes
Accepted

Reading from a .txt file to a pandas dataframe

When opening very large files, first concern would be memory availability on your system to avoid swap on slower devices (i.e. disk). Pandas is shipped with built-in reader methods. For example the <...
rdbisme's user avatar
  • 238
7 votes
Accepted

Interpret YYYYMMDD as the nth day of the year

As you already use pandas, you are right that there is no overhead on using Timestamps (the kind of objects returned by ...
301_Moved_Permanently's user avatar
7 votes
Accepted

Code for creating combinations taking a long time to finish

You take the combinations in a way too convoluted way. For starter, I would simplify the retrieval of "same memberid" questions: ...
301_Moved_Permanently's user avatar
7 votes

Basic function to convert Country name to ISO code using Pycountry

Apart from speeding up your code, there are some other improvements possible that I don't see mentioned yet. The speedup As other users have already picked up, you only need to transform around 200 ...
Ivo Merchiers's user avatar
6 votes

Finding the states with the three most populous counties

nlargest could help, it finds the maximum n values in pandas series. In this line of code, groupby groups the frame according to state name, then apply finds the 3 largest values in column ...
sgDysregulation's user avatar
6 votes

Finding the states with the three most populous counties

SUMLEV is explained here Definitely want to use nlargest The advantage of nlargest is that ...
piRSquared's user avatar
6 votes
Accepted

Split latitude/longitude by degree to make file names and folder directory names

Bug and accidental quadratic runtime ...
mkrieger1's user avatar
  • 1,694
6 votes
Accepted

Preprocessing steps to follow while cleaning and extracting text data from tweets

Copying my answer from SO: You can use pandas vectorized string methods to do your processing and it also removes the for loop ...
umutto's user avatar
  • 176
6 votes

Comparing the size of tumors over time using PANDAS

There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed. Dates. ...
pkaercher's user avatar
6 votes

Table of Tribonacci sequence using NumPy and PANDAS

Some general tips Put code in functions That way you can test each part individually. Here the generation of the sequence, calculation of the ratio and exporting to pandas are clear divisions in the ...
Maarten Fabré's user avatar
6 votes
Accepted

Python pandas: Take column of counts and create DataFrame with a row per count

You are correct in thinking that iterrows is a very bad sign for Pandas code. Even worse is building up a DataFrame one row at a time like this with ...
miradulo's user avatar
  • 211
6 votes
Accepted

My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report

The description of the problem in the post says that the case to be detected is redeeming "over 3 time" but what the code actually detects is 3 times or more. Which is right? The description of the ...
Gareth Rees's user avatar
  • 49.6k
6 votes
Accepted

Making a dataframe of parent IDs

You should decide if your functions modify the object they receive or if the return a modified object. Doing both is just asking for disaster. After your code has finished, ...
Graipher's user avatar
  • 41k
6 votes
Accepted

Python to write multiple dataframes and highlight rows inside an excel file

First, starting from your code, you should realize that you are repeating yourself, three times. This goes against the principle Don't repeat Yourself (DRY). The only real difference between ...
Graipher's user avatar
  • 41k
6 votes
Accepted

Groupby count, then sum and get the percentage

I think your code is already nearly optimal and Pythonic. But there is some little things to improve: cluster_count.sum() returns you a Series object so if you are ...
vurmux's user avatar
  • 1,355
6 votes
Accepted

Parsing Addresses with GeoPanda's GeoDataFrame

Nomenclature Function names should be snake_case per PEP8, i.e. address_parsing. Type hinting For function parameters and return values mainly, type hinting will ...
Reinderien's user avatar
6 votes
Accepted

Pandas add calculated row for every row in a dataframe

iterrows Of all the ways to iterate over a pandas DataFrame, iterrows is the worst. This creates a new series for each row. this series also has a single dtype, so ...
Maarten Fabré's user avatar
6 votes
Accepted

Finding the most frequent words in Pandas dataframe

Note: The data you're going through is 370k+ lines. Because I tend to run different versions of code a lot during review, I've limited my version to 1000 lines. Your code goes all over the place. ...
Mast's user avatar
  • 13.4k

Only top scored, non community-wiki answers of a minimum length are eligible