Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Pandas is a Python data analysis library.

-2
votes
0answers
20 views

Loop on dataframe takes a lot of time [on hold]

The dataframe subset feature is being used in a for loop across the dataframe rows. The result seems accurate however, the time taken to complete the loop on 2000 odd rows is more than 4 minutes. Any ...
3
votes
0answers
35 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
6
votes
1answer
50 views

Get nearest major city of an image (using EXIF GPS data)

I am writing a program that will take images in a folder, get the GPS Coordinates (from the EXIF data), then return the closest "major city". (I define a major city as one with an airport, so ...
3
votes
1answer
37 views

Groupby count, then sum and get the percentage

I wrote this code. It works, but I think there is a more elegant and Pythonic way to this task. Groupby and count the different occurences Get the sum of all the occurences Divide each occurrence ...
3
votes
1answer
31 views

Calculate percentage based on dummy variable in Pandas

Today I had a simple task: I have products, quantities and a dummy, and I needed to know which percentage of the total quantities of that product the dummy represented. My DataFrame looked like this: ...
1
vote
2answers
43 views

Function to count words in Python

I am new to developing in Python and would like to see comments about my word count solution. Please indicate my mistakes or best solutions. ...
3
votes
0answers
18 views

Grouping a Pandas Dataframe by two parameters

I have a .csv file of 8k+ rows which looks like this: ...
4
votes
0answers
26 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
2
votes
0answers
28 views

Faster way to replace strings in pandas series using dictionary in Python

I'm working with a dictionary with 11830 keys/values, that have to loop over 13905 strings, that can have up to 50 words each. My series looks like this: ...
2
votes
1answer
32 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
3
votes
1answer
39 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
1
vote
1answer
96 views

Comparing columns in pandas different data frames and fill in a new column

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists ...
1
vote
0answers
28 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
3
votes
0answers
15 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
1
vote
1answer
46 views

Efficient way to add a colon before and after every word in strings inside a Series

I was working with twitter data, and extracted every emoji from the data. But, when I passed that data through CountVectorizer, colons where substracted from the ...
2
votes
1answer
64 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
4
votes
1answer
34 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
1
vote
0answers
18 views

Requests / Facebook API - Dealing with Errors

I work in marketing and I am currently trying to aggregate data for 50 facebook pages on the same Business Page. I have a loop ...
2
votes
1answer
163 views

Faster way to 3 nested for loop python

I am calculating parameters for a time series and the data size is very big. How can I make my function faster? The following function, param, calculates parameters ...
3
votes
1answer
31 views

Python: Combining Two Rows with Pandas read_excel

I am reading an Excel file using Pandas and I feel like there has to be a better way to handle the way I create column names. This is something like the Excel file I'm reading: ...
2
votes
2answers
55 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...
4
votes
0answers
37 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
3
votes
0answers
26 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
2
votes
1answer
114 views

Pandas nested loop which checks if values from one series are in another series

I have two different series in pandas that I have created a nested for loop which checks if the values of the first series are in the other series. But this is time consuming in pandas and I cannot ...
1
vote
1answer
21 views

Applying a dataframe function to a pandas groupby object

I am trying to apply a function to each group in a pandas dataframe where the function requires access to the entire group (as opposed to just one row). For this I am iterating over each group in the ...
2
votes
0answers
37 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
1
vote
1answer
38 views

speed up/optimize pandas code

I am making a data-driven bond screening and I have as input a big dataset of 1526 columns and 2412 rows. For 10 columns it takes 2 minutes processing time at the moment, which is too much. The ...
7
votes
2answers
163 views

Apply function to every subset combination and return square matrix

I do not know how to do this without four nested for loops. I'd like to apply a function to every possible combination of subsets for ...
2
votes
1answer
58 views

Conditional joining Pandas dataframes

I'm looking for an optimum way to join two Dataframes where the same record exists in both. Examples would be: ...
2
votes
0answers
113 views

Python, Selenium, Pandas project; for populating web form page on loop

I have inherited this nightmare Python, Selenium, and Pandas project. I am learning Python as I go; and have been cleaning up and refactoring the code where I see necessary - It was worst if you can ...
1
vote
2answers
41 views

Groupby, apply custom function to data, return results in new columns

I want to group by id, apply a custom function to the data, and create a new column with the results. It seems there must be a faster/more efficient way to do this ...
7
votes
0answers
115 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
-1
votes
1answer
41 views

Write/read efficiently dataframe objects into memory or disk [closed]

i'm running a for loop that loops over all the rows of a pandas dataframe, then it calculates the euclidean distance from one point at a time to all the other points in the dataframe, then it pass the ...
3
votes
1answer
69 views

Checking whether fund prices changed over multiple CSV files

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes. 1st download - 2 Fund ...
3
votes
1answer
59 views

Tracking stock prices using Python and Google Sheets

The purpose of this program is to keep track of some stocks that we bought in a game and every week we must list the closing prices of each stock in a Google Sheet. I thought since I had a lot of ...
2
votes
0answers
48 views

Efficiently/pythonically calculating new Dataframe rows

I'm new to Pandas, and slightly new to Python as well (but not development in general). I've got a chunk of code that works, but it feels like I'm missing out on the Pandas/Python way of doing ...
0
votes
0answers
74 views

Calculating Relative Strength Index for Multiple Pandas column

I want calculate RSI indicator value for multiple column in Pandas DataFrame. I am looking for a method to avoid loop, here is the code I am using: ...
4
votes
1answer
106 views

Cumulative counts of items in a Pandas dataframe

i'm using a dataframe full of headers/field names to develop a schema and I need unique values for each header - but each header should be represented. I want to loop though the dataframe list of ...
3
votes
1answer
68 views

Evaluating association strengths of items

I have a list of animals with their counts: ...
3
votes
1answer
73 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
4
votes
2answers
291 views

Python to write multiple dataframes and highlight rows inside an excel file

I am trying to write multiple dataframes into an excel file one after another with the same logic. Nothing changes for any of the data frame, except the number of columns or number of records. The ...
3
votes
1answer
30 views

Appending values in Pandas column

I have a dictionary: x = {'[a]':'(1234)', '[b]':'(2345)', '[c]':'(xyzad)'} and a dataframe ...
2
votes
1answer
235 views

Finding recessions in US GDP data using pandas

For an assignment, I am identifying the first quarter of the 2008 recession in the United States. The Excel data I'm using can be downloaded here: gdplev.xls. How can I improve this pandas code to ...
1
vote
0answers
118 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
4
votes
1answer
137 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
2
votes
1answer
312 views

Unit test for a function that transforms a PANDAS dataframe

I am currently working on a project that uses many dataframe wrangling. The intention is to write test to check that the output functions are correct. I have written a test for a dataframe function ...
1
vote
1answer
253 views

Customer segmentation using RFM analysis [closed]

Currently, my code works perfectly well but i would like to make it cleaner for other users by removing the duplicate and similar lines of code into functions or for loops. Because I am still new ...
0
votes
0answers
15 views

Create an interpolant based on data in a csv

I have a csv file with many columns (e.g. A,B,C,D,...
1
vote
1answer
39 views

Pandas: updating cells with new value plus old value

I have a folder with zip files that contain CSVs for each day. Each CSV contains rows with IPs and a company ID. I want to calculate a distribution of the number of firms accessed. Meaning that I want ...
1
vote
1answer
48 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...