Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
2answers
62 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...
4
votes
0answers
43 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
3
votes
0answers
27 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
2
votes
1answer
333 views

Pandas nested loop which checks if values from one series are in another series

I have two different series in pandas that I have created a nested for loop which checks if the values of the first series are in the other series. But this is time consuming in pandas and I cannot ...
1
vote
1answer
36 views

Applying a dataframe function to a pandas groupby object

I am trying to apply a function to each group in a pandas dataframe where the function requires access to the entire group (as opposed to just one row). For this I am iterating over each group in the ...
2
votes
0answers
38 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...
1
vote
1answer
54 views

speed up/optimize pandas code

I am making a data-driven bond screening and I have as input a big dataset of 1526 columns and 2412 rows. For 10 columns it takes 2 minutes processing time at the moment, which is too much. The ...
7
votes
2answers
165 views

Apply function to every subset combination and return square matrix

I do not know how to do this without four nested for loops. I'd like to apply a function to every possible combination of subsets for ...
2
votes
1answer
259 views

Conditional joining Pandas dataframes

I'm looking for an optimum way to join two Dataframes where the same record exists in both. Examples would be: ...
2
votes
0answers
234 views

Python, Selenium, Pandas project; for populating web form page on loop

I have inherited this nightmare Python, Selenium, and Pandas project. I am learning Python as I go; and have been cleaning up and refactoring the code where I see necessary - It was worst if you can ...
1
vote
2answers
49 views

Groupby, apply custom function to data, return results in new columns

I want to group by id, apply a custom function to the data, and create a new column with the results. It seems there must be a faster/more efficient way to do this ...
7
votes
1answer
180 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
-1
votes
1answer
44 views

Write/read efficiently dataframe objects into memory or disk [closed]

i'm running a for loop that loops over all the rows of a pandas dataframe, then it calculates the euclidean distance from one point at a time to all the other points in the dataframe, then it pass the ...
3
votes
1answer
70 views

Checking whether fund prices changed over multiple CSV files

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes. 1st download - 2 Fund ...
3
votes
1answer
84 views

Tracking stock prices using Python and Google Sheets

The purpose of this program is to keep track of some stocks that we bought in a game and every week we must list the closing prices of each stock in a Google Sheet. I thought since I had a lot of ...
2
votes
0answers
50 views

Efficiently/pythonically calculating new Dataframe rows

I'm new to Pandas, and slightly new to Python as well (but not development in general). I've got a chunk of code that works, but it feels like I'm missing out on the Pandas/Python way of doing ...
0
votes
0answers
128 views

Calculating Relative Strength Index for Multiple Pandas column

I want calculate RSI indicator value for multiple column in Pandas DataFrame. I am looking for a method to avoid loop, here is the code I am using: ...
4
votes
1answer
258 views

Cumulative counts of items in a Pandas dataframe

i'm using a dataframe full of headers/field names to develop a schema and I need unique values for each header - but each header should be represented. I want to loop though the dataframe list of ...
3
votes
1answer
69 views

Evaluating association strengths of items

I have a list of animals with their counts: ...
3
votes
1answer
121 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
4
votes
2answers
388 views

Python to write multiple dataframes and highlight rows inside an excel file

I am trying to write multiple dataframes into an excel file one after another with the same logic. Nothing changes for any of the data frame, except the number of columns or number of records. The ...
3
votes
1answer
31 views

Appending values in Pandas column

I have a dictionary: x = {'[a]':'(1234)', '[b]':'(2345)', '[c]':'(xyzad)'} and a dataframe ...
2
votes
1answer
395 views

Finding recessions in US GDP data using pandas

For an assignment, I am identifying the first quarter of the 2008 recession in the United States. The Excel data I'm using can be downloaded here: gdplev.xls. How can I improve this pandas code to ...
1
vote
0answers
129 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
4
votes
1answer
360 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
2
votes
1answer
753 views

Unit test for a function that transforms a PANDAS dataframe

I am currently working on a project that uses many dataframe wrangling. The intention is to write test to check that the output functions are correct. I have written a test for a dataframe function ...
1
vote
1answer
432 views

Customer segmentation using RFM analysis [closed]

Currently, my code works perfectly well but i would like to make it cleaner for other users by removing the duplicate and similar lines of code into functions or for loops. Because I am still new ...
0
votes
0answers
17 views

Create an interpolant based on data in a csv

I have a csv file with many columns (e.g. A,B,C,D,...
1
vote
1answer
40 views

Pandas: updating cells with new value plus old value

I have a folder with zip files that contain CSVs for each day. Each CSV contains rows with IPs and a company ID. I want to calculate a distribution of the number of firms accessed. Meaning that I want ...
1
vote
1answer
51 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
2
votes
2answers
135 views

Pandas calculations per columns and per rows for very big datasets

I am using a panda's dataframe and I am doing filtering and some calculations per column and per row. The dataframe looks this: ...
0
votes
1answer
62 views

Linear Regression on Pandas

I'm working on a simple statistics problem with Pandas and sklearn. I'm aware that my code is ugly, but how can I improve it? <...
0
votes
1answer
447 views

Get all node descendants in a tree

I have a CSV holding a "flat" table of a tree's edges (NOT binary, but a node cannot have two parents), ~1M edges: ...
4
votes
1answer
70 views

Google Analytic Webcrawler for every country code

At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
4
votes
1answer
52 views

Set of one-hot encoders in Python

In the absence of feature-complete and easy-to-use one-hot encoders in the Python ecosystem I've made a set of my own. This is intended to be a small library, so I want to make sure it's as clear and ...
2
votes
2answers
42 views

Compare an array with a file and form groups from elements of an array

I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. ...
2
votes
1answer
293 views

Generating scatter plot from a CSV file

Assume you have the following data in the form of a csv-file. The content looks something like this: ...
5
votes
1answer
237 views

Calculating time deltas between rows in a Pandas dataframe

I am trying to compute the difference in timestamps and make a delta time column in a Pandas dataframe. This is the code I am currently using: ...
2
votes
0answers
69 views

Sum, validate, and plot monthly inventory levels

The idea is that I have a subset of the movements of material of a factory (over 700k rows), and I want to separate these movements by the reference of each product to plot the stock of each product ...
4
votes
1answer
64 views

Python script for to evaluate business data

The following script is part of a further education I'm currently enrolled into. Not all of the code is written by myself. Function signatures for example. Therefore I have put the sections, written ...
3
votes
1answer
53 views

Django module to transform data from database to be displayed in Slickgrid tables

This is a Django tag with 2 classes, but can be viewed as a module: What does the module do? it loads a database, performs some transformations on the data, returns the output as a dictionary of json ...
0
votes
1answer
61 views

How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done. ...
1
vote
0answers
23 views

Groupby Count on user defined time periods pandas

I have a data frame like: ...
0
votes
1answer
32 views

Assigning zero to random cells in a pandas.DataFrame

I am trying to assign 0 to random cells in a one dimensional pandas.DataFrame. The below code is the best way I could think of ...
5
votes
1answer
211 views

Making a dataframe of parent IDs

I am quite new to scripting, and here is my first code I wrote. The goal of the code is to extract a parent ID based on a child ID. I would appreciate some constructive criticism and aggressive ...
1
vote
1answer
77 views

My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report

I have a requirement to write a Python 3.6 script which should be able to detect following case within a time period and be able to create a text file report that lists the condition violated and ...
2
votes
1answer
57 views

twitter data mining script in python

I have written a simple script that searches twitter for keywords and saves them to a csv file if they contain those words. It can be found on my github here. How can I improve this code to generally ...
1
vote
0answers
57 views

Batch gradient descent and stochastic gradient descent

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use ...
1
vote
1answer
37 views

Display the difference between DataFrames' dtypes?

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having ...
0
votes
2answers
52 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...