Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
4
votes
0answers
70 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
4
votes
0answers
37 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
4
votes
0answers
63 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
4
votes
1answer
61 views

Set of one-hot encoders in Python

In the absence of feature-complete and easy-to-use one-hot encoders in the Python ecosystem I've made a set of my own. This is intended to be a small library, so I want to make sure it's as clear and ...
4
votes
0answers
1k views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
4
votes
0answers
408 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
4
votes
0answers
516 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
4
votes
0answers
1k views

Python Script to query OSRM (drive-times)

The below script takes two arguments Path to the OSRM Map Path to a .csv containing the columns ...
3
votes
2answers
529 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...
3
votes
2answers
1k views

Slow code pandas 8 million rows

I have the following code which takes 10-15 minutes to execute. That is way too slow considering that my database is growing daily. Is there any chance to make it faster? ...
3
votes
3answers
5k views

Plotting from a Pandas dataframe

I want to improve my code. Is it possible to get the plot without repeating the same instructions multiple lines? The data comes from a Pandas' dataframe, but I am only plotting the last column (...
3
votes
1answer
238 views

Calculate percentage based on dummy variable in Pandas

Today I had a simple task: I have products, quantities and a dummy, and I needed to know which percentage of the total quantities of that product the dummy represented. My DataFrame looked like this: ...
3
votes
1answer
2k views

Compute conditional median of PANDAS dataframe

I am new to Python/Pandas. Consider the following code: ...
3
votes
1answer
56 views

Map price values to bins where bins are of a larger price range

I have two different pandas data frames where in the first data frame (price), I have two columns. The first column named value has some values in it and the second column amount has the available ...
3
votes
1answer
118 views

Python: Combining Two Rows with Pandas read_excel

I am reading an Excel file using Pandas and I feel like there has to be a better way to handle the way I create column names. This is something like the Excel file I'm reading: ...
3
votes
1answer
73 views

Checking whether fund prices changed over multiple CSV files

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes. 1st download - 2 Fund ...
3
votes
1answer
76 views

Django module to transform data from database to be displayed in Slickgrid tables

This is a Django tag with 2 classes, but can be viewed as a module: What does the module do? it loads a database, performs some transformations on the data, returns the output as a dictionary of json ...
3
votes
1answer
47 views

Getting the average score for hotel scores in different countries using pandas

I am diving into Data Analysis with pandas, and I have just written this Python script to calculate the average of hotel review scores of each country. The dataset contains an individual average score ...
3
votes
1answer
176 views

Code runs Speed Test CLI using the python wrapper and then stores values within CSV

I wanted to make use of the speedtest-cli while also incorporating pandas into the project. My initial goal, which I am posting here, was to take the values of the speed test and incorporate them into ...
3
votes
1answer
71 views

Compute leaderboard of competitive riders throughout a season

I have a table in my DB that contains Riders as rows and their points throughout the season as columns. Here are the columns: ...
3
votes
1answer
876 views

Pivot pandas DataFrame while removing index duplicates

The goal of my code is to pivot a pandas DataFrame which is shown below. The idea is to use the Individual column as the column IDs and the ...
3
votes
1answer
88 views

Fetch and append player statistics to an Excel spreadsheet

I am a new programmer in python, and I need a bit of help to structure my code. ...
3
votes
1answer
55 views

Function that fills a time series row-by-row by using the values in the row before

Summary: performance issue with for-loop, vectorization possible? I am processing a time series of securities price data, stored in a pandas dataframe df_buy_sell. ...
3
votes
1answer
65 views

Filling in gym membership prices by joining PANDAS dataframes

I have two dataframes, df which contains a list of members in addition to the type of contract that they purchased on a given date, ...
3
votes
1answer
112 views

Excel Laboratory Data Entry from Python 2.7

I've written a script to automate the entry of laboratory instrument data into an Excel spreadsheet using pandas and win32com. I've got the script functioning correctly, but it is painfully slow. In ...
3
votes
1answer
6k views

Working with pandas dataframes for stock backtesting exercise

I'm attempting to apply a long set of conditions and operations onto a pandas dataframe (see the dataframe below with VTI, upper, lower, etc). I attempted to use apply, but I was having a lot of ...
3
votes
1answer
5k views

Inventory simulation using Pandas DataFrame

I've been learning Python like for a year and started learning a little about pandas DataFrame. I made this little program to practice all the concepts and would ...
3
votes
2answers
52 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
3
votes
1answer
96 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
2answers
82 views

Checking whether customers in a dataframe have social media accounts

I am using an API to check if customers have social media profiles. Depending on whether they have phone or email or both there's a different search type. There's a lot of conditions in this ...
3
votes
1answer
43 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
3
votes
1answer
141 views

Tracking stock prices using Python and Google Sheets

The purpose of this program is to keep track of some stocks that we bought in a game and every week we must list the closing prices of each stock in a Google Sheet. I thought since I had a lot of ...
3
votes
1answer
70 views

Evaluating association strengths of items

I have a list of animals with their counts: ...
3
votes
1answer
821 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
183 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
3
votes
1answer
1k views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
3
votes
1answer
107 views

Gap-fills and trims sensor data to match trial period (pre-processing)

This is a script/tool I've created for work to help us prepare sensor data that comes off of field trials that had data loggers installed in commercially grown agriculture crops. This is the first ...
3
votes
1answer
8k views

Calculating sum of multiple columns in pandas

I have data from one data provider in very thin demographic units: Adults_18_21,Adults_22_24,Adults_25_27, etc. Then in the original data frame that is built like this: ...
3
votes
1answer
336 views

Categorization algorithm for discrete variables

I am trying to categorize some data. For that I check the distribution of the data. Then I split based on the number of appearance of each value. The algorithm I have is working so far but really ...
3
votes
1answer
1k views

Optimize calculation for the amount of time between when a value changes in Pandas

I have some sensor data that contains timestamps from when a machine is turned on and an indicator variable showing whether or not the machine is actively running. The data is mostly recorded every ...
3
votes
1answer
2k views

Calculate working minutes between two timestamps

I've created a function to calculate the working minutes between two timestamps. I class working minutes as those between 9am - 5pm and not on weekends or national holidays. The holidays are those ...
3
votes
2answers
497 views

Optimize a simple and quick python script for transposing a .csv file

I need to transpose the following file output1.csv, which is is a result from a quantum chemistry calculation into a single colum efficiently: ...
3
votes
1answer
12k views

Split excel file with multiple sheets, manipulate the data and create final out file

I have an excel file with 20+ separate sheets containing tables of data. My script iterates through each sheet, manipulates the data into the format I want it and then saves it to a final output file. ...
3
votes
1answer
594 views

SQL GROUPING SETS in Python using Pandas

The code below is intended to provide SQL's GROUPING SETS functionality in Python with the aid of Pandas. Background on SQL GROUPING SETS There are at least two advantages to doing this in Python: ...
3
votes
1answer
93 views

SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
3
votes
1answer
34 views

Appending values in Pandas column

I have a dictionary: x = {'[a]':'(1234)', '[b]':'(2345)', '[c]':'(xyzad)'} and a dataframe ...
3
votes
1answer
65 views

turn a dict that may contain a pandas dataframe to several dicts

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows). I want to create a new dict for each row in the DataFrame's, ...
3
votes
1answer
476 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
3
votes
1answer
191 views

One minute cron Python script reads from MySQL and updates Redis based on calculations

The following code works and will be put into cron on a AWS server (not sure what specifications, cores, kernels etc) at a every 1 minute frequency. However, it's my first time putting a script into ...

1 2 3 4 5 8