Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
16
votes
1answer
372 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
14
votes
3answers
748 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...
13
votes
5answers
27k views

Chi Square Independence Test for Two Pandas DF columns

I want to calculate the scipy.stats.chi2_contingency() for two columns of a pandas DataFrame. The data is categorical, like this: ...
13
votes
1answer
30k views

Parse complex text files using Python

I'm looking for a simple way of parsing complex text files into a pandas DataFrame. Below is a sample file, what I want the result to look like after parsing, and my current method. Is there any way ...
12
votes
1answer
572 views

A big “Game of Life”

Our quest: Create a big simulation for Conway's Game of Life, and record the entire simulation history. Current Approach: Cython is used for an iterate method. The ...
11
votes
1answer
2k views

Simplifying Python Pandas code for selecting co-occurrences in a window of time

I am a beginner at programming. I was able to build the thing below, which achieves what I want with a small dataset. With larger datasets, my RAM gets swamped bringing the computer to a halt (2014 ...
10
votes
3answers
2k views

Table of Tribonacci sequence using NumPy and PANDAS

What I am trying to accomplish is calculate the Tribonacci numbers and the ratio. I can do this in Excel easy like this. So I tried do the same using Python 3 now. It works and gives such output. ...
10
votes
1answer
8k views

Tkinter GUI for making very simple edits to pandas DataFrames

It is part of a separate application that allows users to interact very loosely with different databases and check for possible errors and make corrections. ...
9
votes
2answers
52k views

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I am dropping rows from a PANDAS dataframe when some of its columns have 0 value. I got the output by using the below code, but I hope we can do the same with less code — perhaps in a single line. df:...
9
votes
2answers
264 views

Curried function

From the question and since I'm currently learning functional programming I was inspired to write the following (curried) function: ...
8
votes
2answers
750 views

Extract unique terms from a PANDAS series

Background I have process tons of DataFrames with shapes of ~230 columns x ~2000-50000+ rows. Here is an extremely simplified example; ...
8
votes
2answers
730 views

Project Euler #19: Counting Sundays in the 20th century using Pandas

Project Euler #19 asks: How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)? I'm hoping I wasn't too off course from the spirit of the exercise ...
8
votes
1answer
193 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
8
votes
2answers
142 views

Developing an investment strategy based on stock movements

Below is my code ...
8
votes
1answer
98 views

Querying houses similar to a given house

I was given this task as an interview coding challenge and was wondering If the code is well structured and follows Python guidelines. I chose to sort the houses based on a similarity metric and then ...
8
votes
1answer
96 views

Calculating distance and time between waypoints for large files

This is very similar to other code I've posted, however this is designed for very large CSV files, for example 35GB. The files typically look like this: ...
7
votes
3answers
757 views

Rating tennis players in a database, taking days to run

I have this project in data analysis for creating a ranking of tennis players. Currently, it takes more than 6 days to run on my computer. Can you review the code and see where's the problem? ...
7
votes
2answers
327 views

Monte Carlo estimation of the Hypergeometric Function

I am trying to implement the algorithm described in the paper Statistical Test for the Comparison of Samples from Mutational Spectra (Adams & Skopek, 1986) DOI: 10.1016/0022-2836(87)90669-3: $$p =...
7
votes
1answer
9k views

Efficient Pandas to MySQL “UPDATE… WHERE”

I have a pandas DataFrame and a (MySQL) database with the same columns. The database is not managed by me. I want to update the values in the database in an "UPDATE... WHERE" style, updating only ...
7
votes
2answers
104 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
7
votes
1answer
293 views

Average across direction-specific average speeds and probabilities

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-...
7
votes
2answers
105 views

Speed up script that calculates distribution of every character from user input

I have a data set with close to 6 million rows of user input. Specifically, users were supposed to type in their email addresses, but because there was not pattern validation put in place we have a ...
7
votes
1answer
7k views

Remove duplicates from csv based on conditions

The task is basically this: I am given the following csv file with lots of duplicate email addresses ...
7
votes
3answers
219 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
7
votes
1answer
184 views

Coalesce consecutive failures in a DataFrame of hourly sensor readings

I have a PANDAS DataFrame that contains sensor data that is recorded every hour (sample included below). It is important to note that every hour is not necessarily in the dataframe, as sometimes the ...
7
votes
1answer
97 views

Outputting scatter plots [closed]

I have written a python function that outputs scatter plots using Matplotlib after processing the data a little. It works but it's painfully slow. I was wondering if anybody had any suggestions as to ...
7
votes
2answers
163 views

Apply function to every subset combination and return square matrix

I do not know how to do this without four nested for loops. I'd like to apply a function to every possible combination of subsets for ...
7
votes
1answer
158 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
7
votes
1answer
279 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
7
votes
1answer
629 views

Similarity research : K-Nearest Neighbour(KNN) using a linear regression to determine the weights

I have a set of houses with categorical and numerical data. Later I will have a new house and my goal will be to find the 20 closest houses. The code is working fine, and the result are not so bad but ...
7
votes
2answers
357 views

Appointment Schedule

In the below code there are two data frames given, free_schedule and appointments which represent a pre-defined available ...
7
votes
1answer
659 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
7
votes
1answer
437 views

PANDAS DataFrame operations to analyze top Server Fault tags

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
6
votes
2answers
1k views

Code for creating combinations taking a long time to finish

I have the following code that I'm using to create combinations of elements in my dataset: ...
6
votes
1answer
291 views

Python API for Crimson Hexagon

I'd like some feedback on the readability, style, and potential problems or issues. In particular I'm not too happy with how I handle ratelimits. ...
6
votes
1answer
23k views

Apriori algorithm using Pandas

I want to optimize my Apriori algorithm for speed: ...
6
votes
2answers
2k views

Python Pandas Dataframe code takes too long to finish

I have a Pandas DataFrame with users subscription dates in the following format: UserId, StartingDate, EndingDate And I try to calculate the Churn Rate metric for ...
6
votes
1answer
2k views

String split for street addresses

I have 856k addresses of pattern: 660 1st Ave New York, NY 10016 (with 3 spaces between Ave and New York) I need to parse and store parts as: ...
6
votes
2answers
7k views

Reading and processing a file using Pandas

I am trying to read a file using pandas and then process it. For opening the file I use the following function: ...
6
votes
2answers
994 views

Python code to split csv into smaller csvs, not splitting IDs

I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like ...
6
votes
3answers
726 views

Matching values from html table for updating values in pandas dataframe

This is more of an exercise for me to get use to Pandas and its dataframes. For those who didn't hear of it: Pandas is a Python package providing fast, flexible, and expressive data structures ...
6
votes
1answer
100 views

Slicing time spans into calendar months

I have apparently correct code that still runs for weeks on my data (tens of millions of rows). I show the entire code for reference (and maybe other gains to be made), but the key operation is in the ...
6
votes
1answer
9k views

A custom Pandas dataframe to_string method

Oftentimes I find myself converting pandas.DataFrame objects to lists of formatted row strings, so I can print the rows into, e.g. a ...
6
votes
2answers
129 views

Listing page visits for each visitor

I have some data in the following format: ...
6
votes
1answer
396 views

Expanding url from shortened url obtained from tweet

I have a twitter data set. I have extracted all the expanded urls from the json and now am trying to resolve the shortened ones. Also, I need to check which urls are still working and only keep those. ...
6
votes
1answer
67 views

Undoing corrections to a big dataframe

I have 2 dataframes. The first one (900 lines) contains corrections that have been applied to a deal. The second dataframe (140,000 lines) contains the list of deals with corrected values. What I am ...
6
votes
2answers
29k views

Select the n most frequent items from a pandas groupby dataframe

I´m working on trying to get the n most frequent items from a pandas dataframe similar to ...
6
votes
1answer
810 views

Reading an Excel file and comparing the amino acid sequence of each data pair

Since I am fairly new to Python I was wondering whether anyone can help me by making the code more efficient. I know the output stinks; I will be using Pandas to make this a little nicer. ...
6
votes
1answer
52 views

Get nearest major city of an image (using EXIF GPS data)

I am writing a program that will take images in a folder, get the GPS Coordinates (from the EXIF data), then return the closest "major city". (I define a major city as one with an airport, so ...
6
votes
1answer
769 views

Bank saving account class in Python (pandas)

I am setting up a kids bank account class to teach them how to save, and myself how to write a class in Python. The original idea was to create a pandas DataFrame that will show how the money gains ...