Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
3
votes
0answers
34 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
3
votes
0answers
49 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
3
votes
0answers
2k views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
311 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
103 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
102 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
243 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
1answer
439 views

Python/Pandas code creating 12 month product balance forecast

I have an input DataFrame with one row per product with balances, rates, etc. I want to be able to take that starting balance, and other inputs from the row (characteristics of the product), and ...
3
votes
0answers
470 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
3
votes
0answers
51 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
3
votes
0answers
536 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
3
votes
0answers
465 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
3
votes
0answers
78 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
3
votes
0answers
39 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
0answers
686 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
3
votes
0answers
3k views

Rolling OLS algorithm in a dataframe

I want to be able to find a solution to run the following code in a much faster fashion (ideally something like dataframe.apply(func) which has the fastest speed, ...
3
votes
0answers
585 views

Speed up Pandas DataFrame expansion to include time-lagged information about events

Using pandas and Python 3, information about a simple timeseries data set is being processed. Within the span of .5 seconds, 3 names are being said. We record the onset of each utterance, the length ...
3
votes
1answer
395 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
2
votes
2answers
79 views

Reduce repetition in Pandas DataFrame column assignments [closed]

Could the aggregate functions after using grouping functions like groupby() and resample() be looped for cleaner code ____DateTime.____ | Steps | day_segment 2017-10-25 00:00:00| 4 | night 2017-...
2
votes
2answers
96 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...
2
votes
1answer
80 views

Pythonic way of managing data for plotting 2D variance plots

I have collected large dataset of trajectories in 3D space. Below is the information about dataset- Each trajectory (shape: n x 4) is saved into CSV file with the following header: ...
2
votes
1answer
54 views

Check value existence in two lists and gather the result in a csv

I have two lists of protein sequences, I have to check every entry's existence in the two lists, say like ...
2
votes
1answer
24 views

Merge multiple pandas dataframe along with np.where to process values

I have a pandas dataframe, df1: id level val 1 l1 0.2 1 l2 0.1 and another dataframe which contains type for <...
2
votes
1answer
124 views

Building optimal portfolios for all combinations of stocks

Hi I recently wrote some code in python that does the following: 1.) Pulls stock closing data from yahoo finance for x number of stock 2.) finds all possible combinations of x stocks in groups of y ...
2
votes
1answer
62 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
2
votes
2answers
61 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
2
votes
1answer
54 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
2
votes
1answer
120 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...
2
votes
1answer
41 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
1answer
113 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
2
votes
1answer
2k views

Conditional joining Pandas dataframes

I'm looking for an optimum way to join two Dataframes where the same record exists in both. Examples would be: ...
2
votes
1answer
1k views

Finding recessions in US GDP data using pandas

For an assignment, I am identifying the first quarter of the 2008 recession in the United States. The Excel data I'm using can be downloaded here: gdplev.xls. How can I improve this pandas code to ...
2
votes
2answers
45 views

Compare an array with a file and form groups from elements of an array

I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. ...
2
votes
1answer
3k views

Loading .txt file content and filename into pandas dataframe

Is there a better way to read in the name of a textfile and the content of a text file into a dataframe? (Or is my implementation even okay?) Can I avoid storing the data in lists? ...
2
votes
1answer
623 views

Operating multiple columns of one pandas DataFrame using data from another

I have a DataFrame of data from a survey that was repeated over several years, asking people about their income and how much money they had in savings. For simplicity, let's pretend it looks like this:...
2
votes
1answer
76 views

Stock data REST API extractor with GCP storage options

I'm currently working on an AI-driven trading system, the code below aims to extract ticker data from polygon REST API, this is a paid service so, in order to test the code you will need to subscribe /...
2
votes
1answer
38 views

Construct a numpy array by repeating a 1 dimensional array sliced at different indices

I have a pandas dataframe (called base_mortality) with 1 column and n rows, which is of the following form: ...
2
votes
1answer
162 views

What is an efficient ways to parse a bar separated usr file in Python

I am trying to create a robust, generic way to parse a bar delimited usr files, now I can read the file in and separate it by | ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
2
votes
1answer
43 views

Optimal way to filter DataFrame by many arguments, without usage of multiple ifs and for loops

I am learning about survival analysis and I made some implementation in Python using lifelines and plotly. However depending on ...
2
votes
1answer
34 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
2
votes
1answer
671 views

Generating scatter plot from a CSV file

Assume you have the following data in the form of a csv-file. The content looks something like this: ...
2
votes
1answer
78 views

Check values in datatables, with validation rules that vary according to the data source

Background I receive dataframes from several sources (below I call these sources s1, s2 and ...
2
votes
1answer
161 views

Function for setting a streak of strings in a Pandas

I have a df with artist names in one column, I want to find how many times the artists repeats through the column. Example: ...
2
votes
1answer
408 views

Pivot DataFrame with DateTimeIndex

I have some data, which I've faked using: ...
2
votes
1answer
1k views

Processing large time-series data

I am currently processing some large time-series data with pandas, and I have a function that is intolerably slow, and I'm sure it could be done faster. The problem is: I'm studying a factory that ...
2
votes
2answers
16k views

Using Pandas to merge .csv files

Hope you can help me out with this one because it is really slow. Is there a way to do this without loading the whole .csv into memory? The thing is... I have files containing timeseries data with 10 ...
2
votes
1answer
4k views

Create all lags of given columns

I'm creating a pandas.DataFrame out of lags from an existing pandas.DataFrame, using ...

1
4 5
6
7 8
10