Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
3
votes
0answers
46 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
3
votes
1answer
193 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
3
votes
0answers
1k views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
3
votes
0answers
191 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
0answers
102 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
0answers
96 views

Filtering OTC market data

It's been one month I do programming in python. Hence, my experience is a bit limited. Here is what I've done so far. The program is only filtering OTC market data according to certain features. Could ...
3
votes
0answers
220 views

Iterative pandas row additions and row dropping

I have written a program that uses pandas dataframes to quickly access and filter data (DNA strings). Come to find out, pandas ...
3
votes
1answer
251 views

Python/Pandas code creating 12 month product balance forecast

I have an input DataFrame with one row per product with balances, rates, etc. I want to be able to take that starting balance, and other inputs from the row (characteristics of the product), and ...
3
votes
1answer
74 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
0answers
390 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
3
votes
0answers
51 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
3
votes
0answers
457 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
3
votes
0answers
441 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
3
votes
0answers
77 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
3
votes
0answers
37 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
3
votes
0answers
2k views

Pandas calculation speed of stock beta on many dataframes

I have many (4000+) CSVs of stock data (Date, Open, High, Low, Close) which I import into individual Pandas dataframes to perform analysis. I am new to Python and want to calculate a rolling 12month ...
3
votes
0answers
3k views

Rolling OLS algorithm in a dataframe

I want to be able to find a solution to run the following code in a much faster fashion (ideally something like dataframe.apply(func) which has the fastest speed, ...
3
votes
0answers
579 views

Speed up Pandas DataFrame expansion to include time-lagged information about events

Using pandas and Python 3, information about a simple timeseries data set is being processed. Within the span of .5 seconds, 3 names are being said. We record the onset of each utterance, the length ...
2
votes
2answers
66 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...
2
votes
1answer
64 views

Pythonic way of managing data for plotting 2D variance plots

I have collected large dataset of trajectories in 3D space. Below is the information about dataset- Each trajectory (shape: n x 4) is saved into CSV file with the following header: ...
2
votes
1answer
51 views

Check value existence in two lists and gather the result in a csv

I have two lists of protein sequences, I have to check every entry's existence in the two lists, say like ...
2
votes
1answer
58 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
2
votes
2answers
51 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
2
votes
1answer
46 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
2
votes
1answer
49 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...
2
votes
1answer
33 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
1answer
77 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
2
votes
1answer
660 views

Conditional joining Pandas dataframes

I'm looking for an optimum way to join two Dataframes where the same record exists in both. Examples would be: ...
2
votes
1answer
523 views

Finding recessions in US GDP data using pandas

For an assignment, I am identifying the first quarter of the 2008 recession in the United States. The Excel data I'm using can be downloaded here: gdplev.xls. How can I improve this pandas code to ...
2
votes
2answers
44 views

Compare an array with a file and form groups from elements of an array

I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. ...
2
votes
1answer
618 views

Operating multiple columns of one pandas DataFrame using data from another

I have a DataFrame of data from a survey that was repeated over several years, asking people about their income and how much money they had in savings. For simplicity, let's pretend it looks like this:...
2
votes
1answer
50 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
2
votes
1answer
32 views

Optimal way to filter DataFrame by many arguments, without usage of multiple ifs and for loops

I am learning about survival analysis and I made some implementation in Python using lifelines and plotly. However depending on ...
2
votes
1answer
99 views

Parsing data from JSON API in Pandas

I made a little program to parse data from an API. I do not have experience with Pandas. It is working but I would like to know how to do that better and more efficiently. ...
2
votes
1answer
33 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
2
votes
1answer
418 views

Generating scatter plot from a CSV file

Assume you have the following data in the form of a csv-file. The content looks something like this: ...
2
votes
1answer
71 views

Check values in datatables, with validation rules that vary according to the data source

Background I receive dataframes from several sources (below I call these sources s1, s2 and ...
2
votes
1answer
108 views

Function for setting a streak of strings in a Pandas

I have a df with artist names in one column, I want to find how many times the artists repeats through the column. Example: ...
2
votes
1answer
122 views

Pivot DataFrame with DateTimeIndex

I have some data, which I've faked using: ...
2
votes
1answer
2k views

Loading .txt file content and filename into pandas dataframe

Is there a better way to read in the name of a textfile and the content of a text file into a dataframe? (Or is my implementation even okay?) Can I avoid storing the data in lists? ...
2
votes
1answer
875 views

Processing large time-series data

I am currently processing some large time-series data with pandas, and I have a function that is intolerably slow, and I'm sure it could be done faster. The problem is: I'm studying a factory that ...
2
votes
2answers
15k views

Using Pandas to merge .csv files

Hope you can help me out with this one because it is really slow. Is there a way to do this without loading the whole .csv into memory? The thing is... I have files containing timeseries data with 10 ...
2
votes
1answer
50 views

Calculate Twitter Follower Changes Over Time

I have a code that grabs quickly the twitter followers for 12 different users. After appending to a Pandas DataFrame, the data is compared to the same file pulled ...
2
votes
1answer
82 views

Counting matches won by teams

This script counts how many matches were won by the team with a first half, second half, and full game time of possession differential advantage. The data is stored in a pandas data frame: ...
2
votes
1answer
40 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
2
votes
1answer
64 views

Loop to extract json from dataframe and storing in a new dataframe

I have a dataframe (obtained from a csv saved from mysql) with several columns, and one of them consist of a string which is the representation of a json. The data looks like: ...
2
votes
1answer
86 views

Pandas updating/adding columns to rows incrementally using dictionary key values

I want to populate columns of a dataframe (df) by iteratively looping over a list (A_list) generating a dictionary where the keys are the names of the desired columns of df (in the example below the ...
2
votes
1answer
172 views

Faster way to 3 nested for loop python

I am calculating parameters for a time series and the data size is very big. How can I make my function faster? The following function, param, calculates parameters ...
2
votes
1answer
646 views

Pandas nested loop which checks if values from one series are in another series

I have two different series in pandas that I have created a nested for loop which checks if the values of the first series are in the other series. But this is time consuming in pandas and I cannot ...