Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
16
votes
1answer
407 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
14
votes
3answers
752 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...
14
votes
1answer
35k views

Parse complex text files using Python

I'm looking for a simple way of parsing complex text files into a pandas DataFrame. Below is a sample file, what I want the result to look like after parsing, and my current method. Is there any way ...
13
votes
5answers
32k views

Chi Square Independence Test for Two Pandas DF columns

I want to calculate the scipy.stats.chi2_contingency() for two columns of a pandas DataFrame. The data is categorical, like this: ...
12
votes
1answer
610 views

A big “Game of Life”

Our quest: Create a big simulation for Conway's Game of Life, and record the entire simulation history. Current Approach: Cython is used for an iterate method. The ...
11
votes
1answer
2k views

Simplifying Python Pandas code for selecting co-occurrences in a window of time

I am a beginner at programming. I was able to build the thing below, which achieves what I want with a small dataset. With larger datasets, my RAM gets swamped bringing the computer to a halt (2014 ...
10
votes
3answers
2k views

Table of Tribonacci sequence using NumPy and PANDAS

What I am trying to accomplish is calculate the Tribonacci numbers and the ratio. I can do this in Excel easy like this. So I tried do the same using Python 3 now. It works and gives such output. ...
10
votes
1answer
9k views

Tkinter GUI for making very simple edits to pandas DataFrames

It is part of a separate application that allows users to interact very loosely with different databases and check for possible errors and make corrections. ...
9
votes
2answers
66k views

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I am dropping rows from a PANDAS dataframe when some of its columns have 0 value. I got the output by using the below code, but I hope we can do the same with less code — perhaps in a single line. df:...
9
votes
2answers
282 views

Curried function

From the question and since I'm currently learning functional programming I was inspired to write the following (curried) function: ...
8
votes
3answers
623 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
8
votes
2answers
780 views

Extract unique terms from a PANDAS series

Background I have process tons of DataFrames with shapes of ~230 columns x ~2000-50000+ rows. Here is an extremely simplified example; ...
8
votes
2answers
785 views

Project Euler #19: Counting Sundays in the 20th century using Pandas

Project Euler #19 asks: How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)? I'm hoping I wasn't too off course from the spirit of the exercise ...
8
votes
1answer
2k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
8
votes
1answer
313 views

Average across direction-specific average speeds and probabilities

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-...
8
votes
1answer
208 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
8
votes
2answers
144 views

Developing an investment strategy based on stock movements

Below is my code ...
8
votes
1answer
101 views

Querying houses similar to a given house

I was given this task as an interview coding challenge and was wondering If the code is well structured and follows Python guidelines. I chose to sort the houses based on a similarity metric and then ...
8
votes
1answer
103 views

Calculating distance and time between waypoints for large files

This is very similar to other code I've posted, however this is designed for very large CSV files, for example 35GB. The files typically look like this: ...
7
votes
4answers
24k views

Extracting contents of dictionary contained in Pandas dataframe to make new dataframe columns

I created a Pandas dataframe from a MongoDB query. c = db.runs.find().limit(limit) df = pd.DataFrame(list(c)) Right now one column of the dataframe corresponds ...
7
votes
3answers
801 views

Rating tennis players in a database, taking days to run

I have this project in data analysis for creating a ranking of tennis players. Currently, it takes more than 6 days to run on my computer. Can you review the code and see where's the problem? ...
7
votes
2answers
331 views

Monte Carlo estimation of the Hypergeometric Function

I am trying to implement the algorithm described in the paper Statistical Test for the Comparison of Samples from Mutational Spectra (Adams & Skopek, 1986) DOI: 10.1016/0022-2836(87)90669-3: $$p =...
7
votes
1answer
10k views

Efficient Pandas to MySQL “UPDATE… WHERE”

I have a pandas DataFrame and a (MySQL) database with the same columns. The database is not managed by me. I want to update the values in the database in an "UPDATE... WHERE" style, updating only ...
7
votes
2answers
107 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
7
votes
2answers
105 views

Speed up script that calculates distribution of every character from user input

I have a data set with close to 6 million rows of user input. Specifically, users were supposed to type in their email addresses, but because there was not pattern validation put in place we have a ...
7
votes
1answer
9k views

Remove duplicates from csv based on conditions

The task is basically this: I am given the following csv file with lots of duplicate email addresses ...
7
votes
3answers
223 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
7
votes
1answer
199 views

Coalesce consecutive failures in a DataFrame of hourly sensor readings

I have a PANDAS DataFrame that contains sensor data that is recorded every hour (sample included below). It is important to note that every hour is not necessarily in the dataframe, as sometimes the ...
7
votes
1answer
97 views

Outputting scatter plots [closed]

I have written a python function that outputs scatter plots using Matplotlib after processing the data a little. It works but it's painfully slow. I was wondering if anybody had any suggestions as to ...
7
votes
2answers
169 views

Apply function to every subset combination and return square matrix

I do not know how to do this without four nested for loops. I'd like to apply a function to every possible combination of subsets for ...
7
votes
1answer
234 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
7
votes
1answer
300 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
7
votes
1answer
650 views

Similarity research : K-Nearest Neighbour(KNN) using a linear regression to determine the weights

I have a set of houses with categorical and numerical data. Later I will have a new house and my goal will be to find the 20 closest houses. The code is working fine, and the result are not so bad but ...
7
votes
2answers
521 views

Appointment Schedule

In the below code there are two data frames given, free_schedule and appointments which represent a pre-defined available ...
7
votes
1answer
795 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
7
votes
0answers
142 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
7
votes
1answer
448 views

PANDAS DataFrame operations to analyze top Server Fault tags [closed]

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
6
votes
2answers
1k views

Code for creating combinations taking a long time to finish

I have the following code that I'm using to create combinations of elements in my dataset: ...
6
votes
2answers
93 views

Comparing the size of tumors over time using PANDAS

I have a dataframe in pandas like this ...
6
votes
2answers
25k views

Excel's SUMIFS implemented using PANDAS, the Python Data Analysis Library

I've implemented Excel's SUMIFS function in Pandas using the following code. Is there a better — more Pythonic — implementation? ...
6
votes
1answer
3k views

String split for street addresses

I have 856k addresses of pattern: 660 1st Ave New York, NY 10016 (with 3 spaces between Ave and New York) I need to parse and store parts as: ...
6
votes
1answer
320 views

Python API for Crimson Hexagon

I'd like some feedback on the readability, style, and potential problems or issues. In particular I'm not too happy with how I handle ratelimits. ...
6
votes
1answer
24k views

Apriori algorithm using Pandas

I want to optimize my Apriori algorithm for speed: ...
6
votes
1answer
135 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
6
votes
2answers
3k views

Python Pandas Dataframe code takes too long to finish

I have a Pandas DataFrame with users subscription dates in the following format: UserId, StartingDate, EndingDate And I try to calculate the Churn Rate metric for ...
6
votes
2answers
8k views

Reading and processing a file using Pandas

I am trying to read a file using pandas and then process it. For opening the file I use the following function: ...
6
votes
2answers
1k views

Python code to split csv into smaller csvs, not splitting IDs

I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like ...
6
votes
3answers
822 views

Matching values from html table for updating values in pandas dataframe

This is more of an exercise for me to get use to Pandas and its dataframes. For those who didn't hear of it: Pandas is a Python package providing fast, flexible, and expressive data structures ...
6
votes
1answer
104 views

Slicing time spans into calendar months

I have apparently correct code that still runs for weeks on my data (tens of millions of rows). I show the entire code for reference (and maybe other gains to be made), but the key operation is in the ...
6
votes
1answer
10k views

A custom Pandas dataframe to_string method

Oftentimes I find myself converting pandas.DataFrame objects to lists of formatted row strings, so I can print the rows into, e.g. a ...

1 2 3 4 5 8