Hot answers tagged

15 votes
Accepted

Parse complex text files using Python

There are a few performance tricks we can apply here: add __slots__ to the class definition should help with memory and performance as well: ...
user avatar
  • 17.1k
13 votes

Chi Square Independence Test for Two Pandas DF columns

I would try to use existing pandas features where possible to keep this code minimal - this aids readability and reduces the possibility of bugs being introduced in complicated loop structures. ...
user avatar
  • 131
13 votes
Accepted

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I think you need create boolean DataFrame by compare all filtered columns values by scalar for not equality and then check all Trues per rows by ...
user avatar
  • 246
12 votes
Accepted

Analyze frequency and content of political fundraising E-mails

Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... ...
user avatar
12 votes
Accepted

Basic function to convert Country name to ISO code using Pycountry

I always get my code to be as clean as possible before starting work on performance. Here you have a bare except, which can hide errors. Change it to something ...
user avatar
  • 41.7k
11 votes
Accepted

Table of Tribonacci sequence using NumPy and PANDAS

You’re using the wrong tool for the job. Basically, you do all the computation in Python, use numpy for intermediate storage and ...
user avatar
11 votes
Accepted

William Fractal technical indicator implementation

Update: There was a post (now deleted) about parameterizing the number of shift periods, so I've added a period param to both the ...
user avatar
  • 671
10 votes

Chi Square Independence Test for Two Pandas DF columns

Using pandas.crosstab, this can be done in a single step: pandas.crosstab(index=test_df['var1'],columns=test_df['var2']) It ...
user avatar
10 votes

Modifying Titration Data analysis results

I've never used numpy or matplotlib, so I can only speak to issues of style. You're allowing for far too much nesting here. Your code consists of a giant, dense, deeply nested chunk. As a result, the ...
user avatar
9 votes

Lookup closest value in Pandas DataFrame

Not sure if this will help, but I'm using this to find nearest in a sorted column: (time series stuff) ...
user avatar
9 votes

Calculating time deltas between rows in a Pandas dataframe

Use the diff(). x['time_delta'] = x.timestamp.diff().fillna(x['time_delta']) This works as below, in a simpler example. You could use the ...
user avatar
8 votes
Accepted

Chi Square Independence Test for Two Pandas DF columns

AFAIK it does what you want (but on this site you should generally be sure that your code does what you want beforehand). Beauty, eye of the Beholder, ...; that said, this code can be rewritten in a ...
user avatar
  • 11k
8 votes
Accepted

Extract unique terms from a PANDAS series

It doesn't look like you really need regular expressions. This construct just using basic string operations is about 10x faster than the construct with the regular expressions: ...
user avatar
8 votes
Accepted

Groupby and moving average function in pandas works but is slow

Iterating in Python is slow, iterating in C is fast. So, it's best to keep as much as possible within Pandas to take advantage of its C implementation and avoid Python. You can use apply on groupby ...
user avatar
  • 1,114
8 votes

Modifying Titration Data analysis results

I'm going to put this code into an editor, "proofread" it from top to bottom, give notes as I go, and then paste the final result to show the effect of the edits. Code editors don't usually wrap ...
user avatar
  • 3,984
7 votes

Efficient Pandas to MySQL "UPDATE... WHERE"

You have eight conditions to match for every UPDATE. A typical solution would store timestamps using a DATETIME or ...
user avatar
7 votes
Accepted

Reading from a .txt file to a pandas dataframe

When opening very large files, first concern would be memory availability on your system to avoid swap on slower devices (i.e. disk). Pandas is shipped with built-in reader methods. For example the <...
user avatar
  • 238
7 votes
Accepted

Interpret YYYYMMDD as the nth day of the year

As you already use pandas, you are right that there is no overhead on using Timestamps (the kind of objects returned by ...
user avatar
7 votes
Accepted

Code for creating combinations taking a long time to finish

You take the combinations in a way too convoluted way. For starter, I would simplify the retrieval of "same memberid" questions: ...
user avatar
6 votes

Monte Carlo estimation of the Hypergeometric Function

I am the Adams of Adams-Skopek. I want to point out a couple of things: You get some truncation error when you sum or multiply a series of values in different orders. You need to estimate this and ...
user avatar
6 votes
Accepted

A big "Game of Life"

DISCLAIMER: I don't know cython and have never used it, so if any of my advice doesn't apply because of cython limitations, feel free to disregard it. Counting neighbors in a game board is very easy ...
user avatar
  • 1,596
6 votes

Analyze frequency and content of political fundraising E-mails

Setting aside PEP 8 (official style guide) issues I would make the following change: Rather than keeping your politician bins, names and email addresses in separate data structures (the only way they ...
user avatar
  • 161
6 votes

Analyze frequency and content of political fundraising E-mails

Imagine how much of a pain it would be to add another politician to this list. In addition to the Politician class suggested in another answer, I would suggest ...
user avatar
6 votes

Finding the states with the three most populous counties

nlargest could help, it finds the maximum n values in pandas series. In this line of code, groupby groups the frame according to state name, then apply finds the 3 largest values in column ...
user avatar
6 votes

Finding the states with the three most populous counties

SUMLEV is explained here Definitely want to use nlargest The advantage of nlargest is that ...
user avatar
6 votes
Accepted

Split latitude/longitude by degree to make file names and folder directory names

Bug and accidental quadratic runtime ...
user avatar
  • 1,684
6 votes
Accepted

Preprocessing steps to follow while cleaning and extracting text data from tweets

Copying my answer from SO: You can use pandas vectorized string methods to do your processing and it also removes the for loop ...
user avatar
  • 176
6 votes

Comparing the size of tumors over time using PANDAS

There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed. Dates. ...
user avatar
6 votes

Table of Tribonacci sequence using NumPy and PANDAS

Some general tips Put code in functions That way you can test each part individually. Here the generation of the sequence, calculation of the ratio and exporting to pandas are clear divisions in the ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible