Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1
vote
2answers
435 views

Permute and count between nested dictionaries

Goal: Permute inner values between two dictionaries with matching outer key, Save permuatations in counter, Move to next outer key, update counter with new permutations. Problem: Many lookups on ...
1
vote
1answer
103 views

Comparing columns in pandas different data frames and fill in a new column

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists ...
1
vote
1answer
45 views

Applying a dataframe function to a pandas groupby object

I am trying to apply a function to each group in a pandas dataframe where the function requires access to the entire group (as opposed to just one row). For this I am iterating over each group in the ...
1
vote
1answer
60 views

speed up/optimize pandas code

I am making a data-driven bond screening and I have as input a big dataset of 1526 columns and 2412 rows. For 10 columns it takes 2 minutes processing time at the moment, which is too much. The ...
1
vote
1answer
62 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
1
vote
1answer
38 views

Display the difference between DataFrames' dtypes?

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having ...
1
vote
1answer
599 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
1
vote
1answer
5k views

Multiply two columns of Census data and groupby

I have census data that looks like this ...
1
vote
1answer
271 views

Pandas dataframe custom forward fillna optimisation

I have a function that synchronizes according to some specifications columns of a dataframe. The functions works, nevertheless I was wondering how to: Improve ...
1
vote
1answer
9k views

Extracting specific words from PANDAS dataframe

I'm new to data analysis and doing some online training. I have a task to extract specific words from specific column in data frame, then count those words and then just to do some min/max/mean ant ...
1
vote
1answer
526 views

Write millions of lines to a file - Python, Dataframes and Redis [closed]

I have the following code snippet that reads a CSV into a dataframe, and writes out key-values pairs to a file in a Redis protocol-compliant fashion, i.e. ...
1
vote
1answer
3k views

Convert sum from monthly to quarterly values

I'm attempting to aggregate multiple columns of monthly data into quarterly chunks. Currently, I'm applying a rolling sum to the columns and then selecting only every third row to 'delete' or hide the ...
1
vote
1answer
46 views

Applying function to entire dataframe [closed]

I'm wondering if there is a faster way to do the following code, which works fine. It needs to be applied to every row of the DataFrame df. ...
1
vote
1answer
127 views

Faster function iteration on Pandas df with axis=1

I have the following function that allows me to calculate the marketing spend on my advertising campaigns. The problem is that it's now taking too long (over 15 minutes) to apply the function to ...
1
vote
1answer
80 views

SKlearn automate data pre treatment

I want to make a simple wrapper for sklearn models. The idea is that the wrapper automatically takes care of factors (columns of type "object") replacing them with ...
1
vote
1answer
70 views

Multi-linear regression with trivial model refinement algorithms

I wrote this code for an assignment. It was originally meant to be written in Maple, but got very frustrated with some of Maple's idiosyncrasies that I decided to play around with Pandas instead. This ...
1
vote
0answers
36 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...
1
vote
0answers
39 views

is there way to make faster to write to a file?

...
1
vote
0answers
29 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
1
vote
1answer
23 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
1
vote
0answers
34 views

Sortings stocks into quantiles based on their signal

This function sorts stocks into quantiles. To do so I used this function that accepts sig_df (dataframe with the timeseries of stocks signal) and number of ...
1
vote
0answers
30 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
1
vote
0answers
18 views

Requests / Facebook API - Dealing with Errors

I work in marketing and I am currently trying to aggregate data for 50 facebook pages on the same Business Page. I have a loop ...
1
vote
0answers
130 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
1
vote
0answers
24 views

Groupby Count on user defined time periods pandas

I have a data frame like: ...
1
vote
0answers
66 views

Batch gradient descent and stochastic gradient descent

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use ...
1
vote
0answers
29 views

Calculating how much $ goes into each defaulting bucket

I have the following code which I use to determine how much of each repayment goes into each late payments bucket (i.e. 0-7 days past due, 7-15 days past due... etc) For example, the repayment amount ...
1
vote
1answer
102 views

Importing text into PANDAS and counting certain words [closed]

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
1
vote
0answers
36 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
1
vote
0answers
59 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
1
vote
0answers
28 views

Writing excelsheet from python dataframe

I am reading dataframe and writing exccel, I cannot use direct df.to_excel method as I am writing data in a very particular way. Looping on dataframe is taking much time, could anything in below ...
1
vote
0answers
72 views

Dependency resolving functions

Written in Python 3.6 and I must admit, this might be too far... Tell me what you think of this code. There must be a better way... It's (another decorator...) that allows you to compose functions ...
1
vote
0answers
49 views

bullet-proof way to generate a list of object attributes (if object has them)

Background I wrote a a couple functions that make it possible to generate a list of attributes for any given object that has them. I use the list that is ...
1
vote
0answers
3k views

Using pandas and sklearn for forecasting stock market return

I have been using R for stock analysis and machine learning purpose but read somewhere that python is lot faster than R, so I am trying to learn Python for that. I am using Yhat's rodeo IDE (Python ...
1
vote
0answers
589 views

ChiSquare test Code in Python

I was trying to write a code from scratch for Chi-square test.This is the code that I had written in python using pandas.I had a doubt whether the code can produce the desired output or can be written ...
1
vote
0answers
125 views

Memory efficient alternative passing mostly static variables w/ 1 dynamic variable to an external function (COM) instead of a for loop in Python

Massive data problem, 7/8 variables are repeated 1000 times in a for loop, sending 8 input vectors * 50,000 values per loop for each function call. Only 1 input ...
1
vote
0answers
97 views

Generate features for future ML analysis of asset returns

I have built the following code to download stock data from Yahoo Finance. The plan is to then use the built-in pandas functions to calculate metrics from this data ...
0
votes
2answers
14k views

Interpret YYYYMMDD as the nth day of the year [closed]

I am provided a bunch of dates in the format (YYYYMMDD) such as: date='20170503' My goal is to convert that date into the ...
0
votes
1answer
6k views

Searching for keywords in dataframe

Given a dataframe with three columns of text blobs to search through, which can be found in this Gist. And three keywords that I want to identify in this text: ...
0
votes
1answer
37 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
0
votes
1answer
70 views

Linear Regression on Pandas

I'm working on a simple statistics problem with Pandas and sklearn. I'm aware that my code is ugly, but how can I improve it? <...
0
votes
1answer
63 views

How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done. ...
0
votes
1answer
49 views

Assigning zero to random cells in a pandas.DataFrame

I am trying to assign 0 to random cells in a one dimensional pandas.DataFrame. The below code is the best way I could think of ...
0
votes
1answer
45 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
0
votes
1answer
680 views

Simple k-means implemention using Python3 and Pandas

Is there anything I can improve? The distance function is Pearson correlation. ...
0
votes
1answer
669 views

Get all node descendants in a tree

I have a CSV holding a "flat" table of a tree's edges (NOT binary, but a node cannot have two parents), ~1M edges: ...
0
votes
2answers
58 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...
0
votes
1answer
8k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
0
votes
1answer
5k views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
0
votes
1answer
40 views

Filter if any value in group is null

The following code works, but the filter operation is bottlenecking my analysis. Do you have an idea how I can make the filter operation faster? Do you have other comments about the codestyle? ...