Questions tagged [pandas]

Pandas is a Python data analysis library.

110 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
63 views

Using get_dummies to create a Simple Recommender System - Cold Start

Question: was using get_dummies a good choice for converting categorical strings? I used get_dummies to convert categorical ...
7
votes
0answers
148 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
6
votes
0answers
71 views

Code optimisation: Converting dataframe to numpy's ndarray

I am working with a dataframe of over 21M rows. ...
6
votes
0answers
43 views

Subtracting elements of datasets of an HDF5 file

I am trying to solve the following problem: Input: Input is two arrays (Nx4, sorted in column-2) stored in datasets-1 and 2 in HDF5 file (input.h5). N is huge (...
6
votes
0answers
181 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
6
votes
0answers
2k views

Calculating T-Test within Large Pandas Dataframes

The below code runs a t-statistic within a large dataframe (rnadf) based on masked values from another dataframe (cnvdf_maked). ...
5
votes
2answers
62 views

Get exactly `n` unique randomly sampled rows per category in a Dataframe

I want to get exactly n unique randomly sampled rows per category in a Dataframe. This proved to be involve more steps than the description would lead you to ...
5
votes
1answer
86 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
5
votes
0answers
184 views

Code for training machine learning linear regression and SVM

Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used ...
4
votes
1answer
52 views

Python : adding a columns with count of missing values by row

I have a big python data-frame and I am trying to add a column to it with average number of missing values by row. I have inherited some code that is working but I'd like to reduce memory usage by ...
4
votes
0answers
44 views

Analyze Stock Market Bottom Timing without Looping over Data Frame Rows

I am replicating this blog post in Python. The blog post shows that even if you know future stock market bottoms, only buying stocks when the stock market is below future stock market bottoms provides ...
4
votes
0answers
37 views

Get Asset Returns for multiple Tickers and Holding Periods

I've written some code that will take multiple stock tickers and calculate their returns over yearly, monthly, bi-weekly, and weekly holding periods. The results are written to an xlsx file. I'm ...
4
votes
0answers
25 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
4
votes
0answers
36 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
4
votes
0answers
82 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
4
votes
0answers
38 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
4
votes
0answers
87 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
4
votes
0answers
1k views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
4
votes
0answers
418 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
4
votes
0answers
4k views

Calculate relationship between 2 categorical variables in a pandas Dataset with chi square test

I'm working on a Machine Learning project and I'm in Data Exploration step, and my dataset has both categorical and continuous attributes. I decided to compute a chi square test between 2 categorical ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
4
votes
0answers
2k views

Python Script to query OSRM (drive-times)

The below script takes two arguments Path to the OSRM Map Path to a .csv containing the columns ...
3
votes
0answers
74 views

Finding the most frequent words in Pandas dataframe

I am new in python coding. I think the code could be written in a better and more compact form. It compiles quite slowly due to the method of removing stop-words. I wanted to find the top 10 most ...
3
votes
0answers
48 views

Reorder the Columns in a CSV File in Descending Order

I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like ...
3
votes
0answers
174 views

Match lists and get value of one column based on values of other columns from dataframe optimization

I have an input dataframe like below, where 'ID' is unique identifier, 'Signals_in_Group' is a derived field containing list of all unique 'Signal' column values present in a 'Group'. And '...
3
votes
0answers
87 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
0answers
42 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
3
votes
0answers
36 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
3
votes
0answers
50 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
3
votes
0answers
2k views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
324 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
103 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
103 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
243 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
0answers
474 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
3
votes
0answers
51 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
3
votes
0answers
545 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
3
votes
0answers
466 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
3
votes
0answers
78 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
3
votes
0answers
39 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
0answers
701 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
3
votes
0answers
3k views

Rolling OLS algorithm in a dataframe

I want to be able to find a solution to run the following code in a much faster fashion (ideally something like dataframe.apply(func) which has the fastest speed, ...
3
votes
0answers
586 views

Speed up Pandas DataFrame expansion to include time-lagged information about events

Using pandas and Python 3, information about a simple timeseries data set is being processed. Within the span of .5 seconds, 3 names are being said. We record the onset of each utterance, the length ...
3
votes
1answer
406 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
2
votes
0answers
28 views

Populating 2 columns in a dataframe with frequencies from a dictionary of dataframes

I have a pandas dataframe, my_calculation, that is a Cartesian product of 4 different categories (Name, Freqset, Formula, and Location) and two additional ...
2
votes
0answers
53 views

How can I shorten the runtime of my simulation?

Before you read the code below, note the following explanation: I have three classes: Driver, Vehicle, and ...
2
votes
1answer
70 views

Creating a weekly sheet for employees

I am looking for any feedback for this first script I have written. It is a mix of openpyxl and pandas, where it takes employee times from different Excel workbooks, then creates a weekly sheet for ...
2
votes
0answers
34 views

How can the following pandas script be made more efficient?

I have an excel sheet G0.xlsx which has 3 sheets in it Winter, Summer, ...
2
votes
0answers
39 views

Stock price history SQL database with Python, Pandas and yfinance

I have a program that lists and downloads a year of history of two stock tickers (AMZN and GOOG). I download these to a SQL database, C:\Databases\hist_prices.db, ...