Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
3
votes
1answer
70 views

Compute leaderboard of competitive riders throughout a season

I have a table in my DB that contains Riders as rows and their points throughout the season as columns. Here are the columns: ...
3
votes
1answer
712 views

Pivot pandas DataFrame while removing index duplicates

The goal of my code is to pivot a pandas DataFrame which is shown below. The idea is to use the Individual column as the column IDs and the ...
3
votes
1answer
88 views

Fetch and append player statistics to an Excel spreadsheet

I am a new programmer in python, and I need a bit of help to structure my code. ...
3
votes
1answer
55 views

Function that fills a time series row-by-row by using the values in the row before

Summary: performance issue with for-loop, vectorization possible? I am processing a time series of securities price data, stored in a pandas dataframe df_buy_sell. ...
3
votes
1answer
63 views

Filling in gym membership prices by joining PANDAS dataframes

I have two dataframes, df which contains a list of members in addition to the type of contract that they purchased on a given date, ...
3
votes
1answer
108 views

Excel Laboratory Data Entry from Python 2.7

I've written a script to automate the entry of laboratory instrument data into an Excel spreadsheet using pandas and win32com. I've got the script functioning correctly, but it is painfully slow. In ...
3
votes
1answer
6k views

Working with pandas dataframes for stock backtesting exercise

I'm attempting to apply a long set of conditions and operations onto a pandas dataframe (see the dataframe below with VTI, upper, lower, etc). I attempted to use apply, but I was having a lot of ...
3
votes
1answer
4k views

Inventory simulation using Pandas DataFrame

I've been learning Python like for a year and started learning a little about pandas DataFrame. I made this little program to practice all the concepts and would ...
3
votes
1answer
85 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
2answers
81 views

Checking whether customers in a dataframe have social media accounts

I am using an API to check if customers have social media profiles. Depending on whether they have phone or email or both there's a different search type. There's a lot of conditions in this ...
3
votes
1answer
42 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
3
votes
1answer
99 views

Tracking stock prices using Python and Google Sheets

The purpose of this program is to keep track of some stocks that we bought in a game and every week we must list the closing prices of each stock in a Google Sheet. I thought since I had a lot of ...
3
votes
1answer
69 views

Evaluating association strengths of items

I have a list of animals with their counts: ...
3
votes
1answer
693 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
169 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
3
votes
1answer
56 views

Pivot and annotate Pandas DataFrame

Given the CSV data: ,fan1,fan2,foil1,foil2 0,0.0,0.0,0.0,0.125 1,0.0625,0.0,0.0625,0.125 2,0.0625,0.0,0.0,0.3125 Which I want to turn into a kind of annotated ...
3
votes
1answer
1k views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
3
votes
1answer
601 views

Subclassing pandas DataFrame for an ETL

I'm writing a small ETL, which loads data from CSVs, cleans each table a bit, and then loads everything into a PostgreSQL database. I was planning to use pandas for its built-in capabilities, but am ...
3
votes
1answer
105 views

Gap-fills and trims sensor data to match trial period (pre-processing)

This is a script/tool I've created for work to help us prepare sensor data that comes off of field trials that had data loggers installed in commercially grown agriculture crops. This is the first ...
3
votes
1answer
8k views

Calculating sum of multiple columns in pandas

I have data from one data provider in very thin demographic units: Adults_18_21,Adults_22_24,Adults_25_27, etc. Then in the original data frame that is built like this: ...
3
votes
1answer
318 views

Categorization algorithm for discrete variables

I am trying to categorize some data. For that I check the distribution of the data. Then I split based on the number of appearance of each value. The algorithm I have is working so far but really ...
3
votes
1answer
1k views

Optimize calculation for the amount of time between when a value changes in Pandas

I have some sensor data that contains timestamps from when a machine is turned on and an indicator variable showing whether or not the machine is actively running. The data is mostly recorded every ...
3
votes
1answer
2k views

Calculate working minutes between two timestamps

I've created a function to calculate the working minutes between two timestamps. I class working minutes as those between 9am - 5pm and not on weekends or national holidays. The holidays are those ...
3
votes
2answers
460 views

Optimize a simple and quick python script for transposing a .csv file

I need to transpose the following file output1.csv, which is is a result from a quantum chemistry calculation into a single colum efficiently: ...
3
votes
1answer
11k views

Split excel file with multiple sheets, manipulate the data and create final out file

I have an excel file with 20+ separate sheets containing tables of data. My script iterates through each sheet, manipulates the data into the format I want it and then saves it to a final output file. ...
3
votes
1answer
540 views

SQL GROUPING SETS in Python using Pandas

The code below is intended to provide SQL's GROUPING SETS functionality in Python with the aid of Pandas. Background on SQL GROUPING SETS There are at least two advantages to doing this in Python: ...
3
votes
1answer
44 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing a code to automate some weekly reports. I had help over on overflow, My original question. I have a code that works for the most part, however there are few things that I just cant ...
3
votes
1answer
69 views

SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
3
votes
1answer
31 views

Appending values in Pandas column

I have a dictionary: x = {'[a]':'(1234)', '[b]':'(2345)', '[c]':'(xyzad)'} and a dataframe ...
3
votes
1answer
59 views

turn a dict that may contain a pandas dataframe to several dicts

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows). I want to create a new dict for each row in the DataFrame's, ...
3
votes
1answer
158 views

One minute cron Python script reads from MySQL and updates Redis based on calculations

The following code works and will be put into cron on a AWS server (not sure what specifications, cores, kernels etc) at a every 1 minute frequency. However, it's my first time putting a script into ...
3
votes
1answer
216 views

Complete missing data in timeseries pandas

I collected data using a smartphone. The phone could run out of battery or the collection software could fail, leading to missing data. For this example let's say I already compute the number of ...
3
votes
1answer
365 views

Monte Carlo Simulation of P-Value

I'm testing Python 3 code to perform a Monte Carlo simulation based on the result of an statistical test. I currently have the result of the statistical test in a pandas dataframe, like this. ...
3
votes
1answer
2k views

Python - Faster random business date generation

I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
3
votes
2answers
510 views

Generating a PANDAS DataFrame of simulated coin tosses

It's taking my machine quite a long time to execute 1 billion (1st loop x 10, 2nd loop x 1000, 3rd loop x 100,000) instructions. Suggestions for performance enhancements? Sources of potential concern: ...
3
votes
1answer
401 views

Creating a time-course dependent, correlation-based directed graph with Networkx

I have a correlation matrix containing 4 time points, each with multiple samples. Each sample is identified with a time point with its name. What I am trying to accomplish here is to create a directed ...
3
votes
0answers
30 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
3
votes
2answers
88 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
3
votes
0answers
79 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
3
votes
0answers
26 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
3
votes
0answers
28 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
3
votes
0answers
40 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
3
votes
1answer
135 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
3
votes
0answers
1k views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
178 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
102 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
93 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
216 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
1answer
225 views

Python/Pandas code creating 12 month product balance forecast

I have an input DataFrame with one row per product with balances, rates, etc. I want to be able to take that starting balance, and other inputs from the row (characteristics of the product), and ...
3
votes
1answer
73 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...