Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
3
votes
1answer
4k views

Inventory simulation using Pandas DataFrame

I've been learning Python like for a year and started learning a little about pandas DataFrame. I made this little program to practice all the concepts and would ...
3
votes
2answers
69 views

Checking whether customers in a dataframe have social media accounts

I am using an API to check if customers have social media profiles. Depending on whether they have phone or email or both there's a different search type. There's a lot of conditions in this ...
3
votes
1answer
41 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
3
votes
1answer
76 views

Tracking stock prices using Python and Google Sheets

The purpose of this program is to keep track of some stocks that we bought in a game and every week we must list the closing prices of each stock in a Google Sheet. I thought since I had a lot of ...
3
votes
1answer
69 views

Evaluating association strengths of items

I have a list of animals with their counts: ...
3
votes
1answer
620 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
160 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
3
votes
1answer
52 views

Pivot and annotate Pandas DataFrame

Given the CSV data: ,fan1,fan2,foil1,foil2 0,0.0,0.0,0.0,0.125 1,0.0625,0.0,0.0625,0.125 2,0.0625,0.0,0.0,0.3125 Which I want to turn into a kind of annotated ...
3
votes
1answer
1k views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
3
votes
1answer
565 views

Subclassing pandas DataFrame for an ETL

I'm writing a small ETL, which loads data from CSVs, cleans each table a bit, and then loads everything into a PostgreSQL database. I was planning to use pandas for its built-in capabilities, but am ...
3
votes
1answer
101 views

Gap-fills and trims sensor data to match trial period (pre-processing)

This is a script/tool I've created for work to help us prepare sensor data that comes off of field trials that had data loggers installed in commercially grown agriculture crops. This is the first ...
3
votes
1answer
8k views

Calculating sum of multiple columns in pandas

I have data from one data provider in very thin demographic units: Adults_18_21,Adults_22_24,Adults_25_27, etc. Then in the original data frame that is built like this: ...
3
votes
1answer
295 views

Categorization algorithm for discrete variables

I am trying to categorize some data. For that I check the distribution of the data. Then I split based on the number of appearance of each value. The algorithm I have is working so far but really ...
3
votes
1answer
1k views

Optimize calculation for the amount of time between when a value changes in Pandas

I have some sensor data that contains timestamps from when a machine is turned on and an indicator variable showing whether or not the machine is actively running. The data is mostly recorded every ...
3
votes
1answer
2k views

Calculate working minutes between two timestamps

I've created a function to calculate the working minutes between two timestamps. I class working minutes as those between 9am - 5pm and not on weekends or national holidays. The holidays are those ...
3
votes
2answers
444 views

Optimize a simple and quick python script for transposing a .csv file

I need to transpose the following file output1.csv, which is is a result from a quantum chemistry calculation into a single colum efficiently: ...
3
votes
1answer
11k views

Split excel file with multiple sheets, manipulate the data and create final out file

I have an excel file with 20+ separate sheets containing tables of data. My script iterates through each sheet, manipulates the data into the format I want it and then saves it to a final output file. ...
3
votes
1answer
518 views

SQL GROUPING SETS in Python using Pandas

The code below is intended to provide SQL's GROUPING SETS functionality in Python with the aid of Pandas. Background on SQL GROUPING SETS There are at least two advantages to doing this in Python: ...
3
votes
1answer
58 views

SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
3
votes
1answer
31 views

Appending values in Pandas column

I have a dictionary: x = {'[a]':'(1234)', '[b]':'(2345)', '[c]':'(xyzad)'} and a dataframe ...
3
votes
1answer
57 views

turn a dict that may contain a pandas dataframe to several dicts

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows). I want to create a new dict for each row in the DataFrame's, ...
3
votes
1answer
137 views

One minute cron Python script reads from MySQL and updates Redis based on calculations

The following code works and will be put into cron on a AWS server (not sure what specifications, cores, kernels etc) at a every 1 minute frequency. However, it's my first time putting a script into ...
3
votes
1answer
206 views

Complete missing data in timeseries pandas

I collected data using a smartphone. The phone could run out of battery or the collection software could fail, leading to missing data. For this example let's say I already compute the number of ...
3
votes
1answer
334 views

Monte Carlo Simulation of P-Value

I'm testing Python 3 code to perform a Monte Carlo simulation based on the result of an statistical test. I currently have the result of the statistical test in a pandas dataframe, like this. ...
3
votes
1answer
2k views

Python - Faster random business date generation

I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
3
votes
2answers
493 views

Generating a PANDAS DataFrame of simulated coin tosses

It's taking my machine quite a long time to execute 1 billion (1st loop x 10, 2nd loop x 1000, 3rd loop x 100,000) instructions. Suggestions for performance enhancements? Sources of potential concern: ...
3
votes
1answer
397 views

Creating a time-course dependent, correlation-based directed graph with Networkx

I have a correlation matrix containing 4 time points, each with multiple samples. Each sample is identified with a time point with its name. What I am trying to accomplish here is to create a directed ...
3
votes
0answers
33 views

Analyzing car exchange transactions

I'm analyzing some transaction data for a colleague's car exchange platform. I have made this program to do some operations on the data but I'm wondering if I can improve the performance (time and ...
3
votes
2answers
43 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
3
votes
0answers
62 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
3
votes
0answers
21 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
3
votes
0answers
27 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
3
votes
1answer
113 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
3
votes
0answers
949 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
164 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
102 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
93 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
210 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
1answer
205 views

Python/Pandas code creating 12 month product balance forecast

I have an input DataFrame with one row per product with balances, rates, etc. I want to be able to take that starting balance, and other inputs from the row (characteristics of the product), and ...
3
votes
1answer
69 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
0answers
354 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
3
votes
0answers
49 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
3
votes
0answers
409 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
3
votes
0answers
427 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
3
votes
0answers
77 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
3
votes
0answers
37 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
3
votes
0answers
3k views

Rolling OLS algorithm in a dataframe

I want to be able to find a solution to run the following code in a much faster fashion (ideally something like dataframe.apply(func) which has the fastest speed, ...
3
votes
0answers
575 views

Speed up Pandas DataFrame expansion to include time-lagged information about events

Using pandas and Python 3, information about a simple timeseries data set is being processed. Within the span of .5 seconds, 3 names are being said. We record the onset of each utterance, the length ...
2
votes
2answers
59 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...