Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
15
votes
1answer
453 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
14
votes
5answers
41k views

Chi Square Independence Test for Two Pandas DF columns

I want to calculate the scipy.stats.chi2_contingency() for two columns of a pandas DataFrame. The data is categorical, like this: ...
14
votes
3answers
761 views

Analyze frequency and content of political fundraising E-mails

Since I'm a big politics nerd, I wanted to write a little script that would analyze the frequency and content of political fundraising emails. I signed up for the e-mails of 6 campaigns, donated a ...
13
votes
1answer
40k views

Parse complex text files using Python

I'm looking for a simple way of parsing complex text files into a pandas DataFrame. Below is a sample file, what I want the result to look like after parsing, and my current method. Is there any way ...
13
votes
1answer
761 views

A big “Game of Life”

Our quest: Create a big simulation for Conway's Game of Life, and record the entire simulation history. Current Approach: Cython is used for an iterate method. The ...
11
votes
1answer
2k views

Simplifying Python Pandas code for selecting co-occurrences in a window of time

I am a beginner at programming. I was able to build the thing below, which achieves what I want with a small dataset. With larger datasets, my RAM gets swamped bringing the computer to a halt (2014 ...
10
votes
3answers
2k views

Table of Tribonacci sequence using NumPy and PANDAS

What I am trying to accomplish is calculate the Tribonacci numbers and the ratio. I can do this in Excel easy like this. So I tried do the same using Python 3 now. It works and gives such output. ...
10
votes
1answer
12k views

Tkinter GUI for making very simple edits to pandas DataFrames

It is part of a separate application that allows users to interact very loosely with different databases and check for possible errors and make corrections. ...
9
votes
2answers
96k views

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I am dropping rows from a PANDAS dataframe when some of its columns have 0 value. I got the output by using the below code, but I hope we can do the same with less code — perhaps in a single line. df:...
9
votes
4answers
36k views

Extracting contents of dictionary contained in Pandas dataframe to make new dataframe columns

I created a Pandas dataframe from a MongoDB query. c = db.runs.find().limit(limit) df = pd.DataFrame(list(c)) Right now one column of the dataframe corresponds ...
9
votes
2answers
5k views

Calculating time deltas between rows in a Pandas dataframe

I am trying to compute the difference in timestamps and make a delta time column in a Pandas dataframe. This is the code I am currently using: ...
9
votes
2answers
819 views

Extract unique terms from a PANDAS series

Background I have process tons of DataFrames with shapes of ~230 columns x ~2000-50000+ rows. Here is an extremely simplified example; ...
9
votes
2answers
307 views

Curried function

From the question and since I'm currently learning functional programming I was inspired to write the following (curried) function: ...
9
votes
1answer
230 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
8
votes
4answers
3k views

Basic function to convert Country name to ISO code using Pycountry

I am working on a dataset which contains a column with common country names. Task: To convert country name into standard ISO names I have written a basic function which converts country names into ...
8
votes
3answers
643 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
8
votes
3answers
477 views

Priority based categorization using pandas/python

I have invoice and code data in the below Dataframes Invoices ...
8
votes
2answers
870 views

Project Euler #19: Counting Sundays in the 20th century using Pandas

Project Euler #19 asks: How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)? I'm hoping I wasn't too off course from the spirit of the exercise ...
8
votes
1answer
4k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
8
votes
1answer
345 views

Average across direction-specific average speeds and probabilities

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-...
8
votes
1answer
11k views

A custom Pandas dataframe to_string method

Oftentimes I find myself converting pandas.DataFrame objects to lists of formatted row strings, so I can print the rows into, e.g. a ...
8
votes
2answers
148 views

Developing an investment strategy based on stock movements

Below is my code ...
8
votes
1answer
103 views

Querying houses similar to a given house

I was given this task as an interview coding challenge and was wondering If the code is well structured and follows Python guidelines. I chose to sort the houses based on a similarity metric and then ...
8
votes
1answer
108 views

Calculating distance and time between waypoints for large files

This is very similar to other code I've posted, however this is designed for very large CSV files, for example 35GB. The files typically look like this: ...
7
votes
2answers
151k views

Reading from a .txt file to a pandas dataframe

Having a text file './inputs/dist.txt' as: ...
7
votes
3answers
868 views

Rating tennis players in a database, taking days to run

I have this project in data analysis for creating a ranking of tennis players. Currently, it takes more than 6 days to run on my computer. Can you review the code and see where's the problem? ...
7
votes
2answers
188 views

Parsing Addresses with GeoPanda's GeoDataFrame

Long time reader of Code Review with my first question. I'm self-taught and suspect this code can be improved. The project is really important to me and the team and I could really use the help. I'm ...
7
votes
2answers
343 views

Monte Carlo estimation of the Hypergeometric Function

I am trying to implement the algorithm described in the paper Statistical Test for the Comparison of Samples from Mutational Spectra (Adams & Skopek, 1986) DOI: 10.1016/0022-2836(87)90669-3: $$p =...
7
votes
1answer
15k views

Efficient Pandas to MySQL “UPDATE… WHERE”

I have a pandas DataFrame and a (MySQL) database with the same columns. The database is not managed by me. I want to update the values in the database in an "UPDATE... WHERE" style, updating only ...
7
votes
1answer
7k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
7
votes
2answers
112 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
7
votes
1answer
243 views

Extracting time duration in the session from 30 million rows

I am looking for making my code faster. I am working on yoochoose recsys 2015 dataset.. and trying to perform some transformations.. [recsys2015], it has got 30 million plus rows of data. The goal of ...
7
votes
2answers
106 views

Speed up script that calculates distribution of every character from user input

I have a data set with close to 6 million rows of user input. Specifically, users were supposed to type in their email addresses, but because there was not pattern validation put in place we have a ...
7
votes
1answer
10k views

Remove duplicates from csv based on conditions

The task is basically this: I am given the following csv file with lots of duplicate email addresses ...
7
votes
3answers
225 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
7
votes
1answer
213 views

Coalesce consecutive failures in a DataFrame of hourly sensor readings

I have a PANDAS DataFrame that contains sensor data that is recorded every hour (sample included below). It is important to note that every hour is not necessarily in the dataframe, as sometimes the ...
7
votes
1answer
99 views

Outputting scatter plots [closed]

I have written a python function that outputs scatter plots using Matplotlib after processing the data a little. It works but it's painfully slow. I was wondering if anybody had any suggestions as to ...
7
votes
1answer
2k views

Parsing URLs in Pandas DataFrame

My client needs their Google AdWords destination URL query parsed and the values spell checked to eliminate any typos ("use" instead of "us", etc). I'm pulling the data using the AdWords API and ...
7
votes
2answers
174 views

Apply function to every subset combination and return square matrix

I do not know how to do this without four nested for loops. I'd like to apply a function to every possible combination of subsets for ...
7
votes
1answer
1k views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
7
votes
1answer
331 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
7
votes
1answer
656 views

Similarity research : K-Nearest Neighbour(KNN) using a linear regression to determine the weights

I have a set of houses with categorical and numerical data. Later I will have a new house and my goal will be to find the 20 closest houses. The code is working fine, and the result are not so bad but ...
7
votes
1answer
1k views

Calculating a unique count within a rolling time window

I have a Pandas DataFrame that contains a row per member per day, expressing member interaction with a website. Members interact only on some days, each member is identified with an ID. Here is a ...
7
votes
2answers
1k views

Appointment Schedule

In the below code there are two data frames given, free_schedule and appointments which represent a pre-defined available ...
7
votes
1answer
974 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
7
votes
0answers
81 views

Using get_dummies to create a Simple Recommender System - Cold Start

Question: was using get_dummies a good choice for converting categorical strings? I used get_dummies to convert categorical ...
7
votes
0answers
149 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
7
votes
1answer
451 views

PANDAS DataFrame operations to analyze top Server Fault tags [closed]

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
6
votes
3answers
22k views

Finding the states with the three most populous counties

I just started to use Python and Pandas. My current solution to a problem looks ugly and inefficient. I would like to know how to improve it. Data file is Census 2010 can be viewed here Question: ...
6
votes
2answers
2k views

Code for creating combinations taking a long time to finish

I have the following code that I'm using to create combinations of elements in my dataset: ...

1
2 3 4 5
11