Questions tagged [pandas]

Pandas is a Python data analysis library.

84 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
1answer
437 views

PANDAS DataFrame operations to analyze top Server Fault tags

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
6
votes
0answers
2k views

Calculating T-Test within Large Pandas Dataframes

The below code runs a t-statistic within a large dataframe (rnadf) based on masked values from another dataframe (cnvdf_maked). ...
5
votes
0answers
65 views

Comparing the size of tumors over time using PANDAS

I have a dataframe in pandas like this ...
5
votes
0answers
165 views

Code for training machine learning linear regression and SVM

Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used ...
5
votes
0answers
3k views

Calculate relationship between 2 categorical variables in a pandas Dataset with chi square test

I'm working on a Machine Learning project and I'm in Data Exploration step, and my dataset has both categorical and continuous attributes. I decided to compute a chi square test between 2 categorical ...
4
votes
1answer
69 views

Optimising iteration and substitution over large dataset

I have the following code: ...
4
votes
0answers
28 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
4
votes
0answers
39 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
4
votes
0answers
1k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
4
votes
0answers
1k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
4
votes
0answers
931 views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
4
votes
0answers
395 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
4
votes
0answers
376 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
4
votes
0answers
966 views

Python Script to query OSRM (drive-times)

The below script takes two arguments Path to the OSRM Map Path to a .csv containing the columns ...
3
votes
1answer
24 views

Parse table names and attributes from SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
3
votes
0answers
51 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
3
votes
0answers
19 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
3
votes
0answers
26 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
3
votes
1answer
90 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
3
votes
0answers
891 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
159 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
101 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
93 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
206 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
1answer
69 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
0answers
338 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
3
votes
0answers
49 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
3
votes
0answers
396 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
3
votes
0answers
420 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
3
votes
0answers
75 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
3
votes
0answers
37 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
3
votes
0answers
679 views

KNN pipeline w/ cross_validation_scores

Using the wine quality dataset, I'm attempting to perform a simple KNN classification (w/ a scaler, and the classifier in a pipeline). It works, but I've never used ...
3
votes
0answers
3k views

Rolling OLS algorithm in a dataframe

I want to be able to find a solution to run the following code in a much faster fashion (ideally something like dataframe.apply(func) which has the fastest speed, ...
3
votes
0answers
575 views

Speed up Pandas DataFrame expansion to include time-lagged information about events

Using pandas and Python 3, information about a simple timeseries data set is being processed. Within the span of .5 seconds, 3 names are being said. We record the onset of each utterance, the length ...
2
votes
0answers
33 views

Faster way to replace strings in pandas series using dictionary in Python

I'm working with a dictionary with 11830 keys/values, that have to loop over 13905 strings, that can have up to 50 words each. My series looks like this: ...
2
votes
0answers
37 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
2
votes
0answers
151 views

Python, Selenium, Pandas project; for populating web form page on loop

I have inherited this nightmare Python, Selenium, and Pandas project. I am learning Python as I go; and have been cleaning up and refactoring the code where I see necessary - It was worst if you can ...
2
votes
0answers
48 views

Efficiently/pythonically calculating new Dataframe rows

I'm new to Pandas, and slightly new to Python as well (but not development in general). I've got a chunk of code that works, but it feels like I'm missing out on the Pandas/Python way of doing ...
2
votes
0answers
67 views

Sum, validate, and plot monthly inventory levels

The idea is that I have a subset of the movements of material of a factory (over 700k rows), and I want to separate these movements by the reference of each product to plot the stock of each product ...
2
votes
0answers
32 views

Function that plots data frames, supporting several variants

I defined a function that generates tables and creates countplots and barplots based on the arguments that it receives. But the snippets of code inside the function are repeated several times, which ...
2
votes
0answers
248 views

Finding peaks in a DataFrame

I would like to compare the min/max values of a time-series with a test time-series. Additionally, I would like to compare the time of the "peaks". However, I'm having trouble extracting these ...
2
votes
0answers
91 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
2
votes
1answer
80 views

Importing text into PANDAS and counting certain words

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
2
votes
0answers
60 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
2
votes
1answer
387 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
0answers
33 views

Determine file provider based on column headers, replace columns and process data

Each day I receive files from different customers relaying the same information, but the customers use proprietary formats. For example, ...
2
votes
0answers
41 views

Average coverage between each sphere of each spheres groups

Let me start with a litle bit of context. I have Spheres, a large pandas DataFrame with positions and radius of multiple spheres during time. (in real life 10 000 ...
2
votes
0answers
155 views

Network traffic flow detection using pandas dataframe

I have a pandas dataframe containing network traffic of multiple hosts: ...