Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
0answers
813 views

Speed up projection of a bipartitie network for a big file using NetworkX and Pandas

I have a pretty big file (3 million lines) with each line being a person-to-event relationship. Ultimate, I want to project this bipartite network onto a single-mode, weighted, network, and write it ...
1
vote
1answer
6k views

Groupby and moving average function in pandas works but is slow

I have some time series data collected for a lot of people (over 50,000) over a two year period on 1 day intervals. I want to applying a exponential weighted moving average function for each person ...
1
vote
1answer
78 views

My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report

I have a requirement to write a Python 3.6 script which should be able to detect following case within a time period and be able to create a text file report that lists the condition violated and ...
1
vote
1answer
2k views

Parse Bloomberg Excel/CSV with Pandas DataFrame

I retrieved Bloomberg data using the Excel API. In the typical fashion, the first row contains tickers in every fourth column, and the second row has the labels Date, PX_LAST, [Empty Column], Date, ...
1
vote
1answer
44 views

Elegant way to do the dataframe operations twice

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the ...
1
vote
2answers
400 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
1
vote
1answer
48 views

Efficient way to add a colon before and after every word in strings inside a Series

I was working with twitter data, and extracted every emoji from the data. But, when I passed that data through CountVectorizer, colons where substracted from the ...
1
vote
1answer
50 views

Calls many stored dataframes, change them and combine them

I call many dataframes (using pickle), then I extract the interesting values of each one and finally combine them. Is there a better way to do it? For example with a for loop call them automatically ...
1
vote
1answer
31 views

tablename alias in attribute name

I have dataframe like this: ...
1
vote
1answer
58 views

Efficient way to get value from a dataframe and append new dataframe

I have a dataframe that have about 200 million rows. the example of dataframe is like this: ...
1
vote
2answers
54 views

Function to count words in Python

I am new to developing in Python and would like to see comments about my word count solution. Please indicate my mistakes or best solutions. ...
1
vote
2answers
60 views

Groupby, apply custom function to data, return results in new columns

I want to group by id, apply a custom function to the data, and create a new column with the results. It seems there must be a faster/more efficient way to do this ...
1
vote
1answer
600 views

Customer segmentation using RFM analysis [closed]

Currently, my code works perfectly well but i would like to make it cleaner for other users by removing the duplicate and similar lines of code into functions or for loops. Because I am still new ...
1
vote
1answer
40 views

Pandas: updating cells with new value plus old value

I have a folder with zip files that contain CSVs for each day. Each CSV contains rows with IPs and a company ID. I want to calculate a distribution of the number of firms accessed. Meaning that I want ...
1
vote
1answer
3k views

Create all lags of given columns

I'm creating a pandas.DataFrame out of lags from an existing pandas.DataFrame, using ...
1
vote
2answers
455 views

Permute and count between nested dictionaries

Goal: Permute inner values between two dictionaries with matching outer key, Save permuatations in counter, Move to next outer key, update counter with new permutations. Problem: Many lookups on ...
1
vote
1answer
103 views

Comparing columns in pandas different data frames and fill in a new column

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists ...
1
vote
1answer
55 views

Applying a dataframe function to a pandas groupby object

I am trying to apply a function to each group in a pandas dataframe where the function requires access to the entire group (as opposed to just one row). For this I am iterating over each group in the ...
1
vote
1answer
65 views

speed up/optimize pandas code

I am making a data-driven bond screening and I have as input a big dataset of 1526 columns and 2412 rows. For 10 columns it takes 2 minutes processing time at the moment, which is too much. The ...
1
vote
1answer
71 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
1
vote
1answer
38 views

Display the difference between DataFrames' dtypes?

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having ...
1
vote
1answer
671 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
1
vote
1answer
5k views

Multiply two columns of Census data and groupby

I have census data that looks like this ...
1
vote
1answer
293 views

Pandas dataframe custom forward fillna optimisation

I have a function that synchronizes according to some specifications columns of a dataframe. The functions works, nevertheless I was wondering how to: Improve ...
1
vote
1answer
9k views

Extracting specific words from PANDAS dataframe

I'm new to data analysis and doing some online training. I have a task to extract specific words from specific column in data frame, then count those words and then just to do some min/max/mean ant ...
1
vote
1answer
553 views

Write millions of lines to a file - Python, Dataframes and Redis [closed]

I have the following code snippet that reads a CSV into a dataframe, and writes out key-values pairs to a file in a Redis protocol-compliant fashion, i.e. ...
1
vote
1answer
3k views

Convert sum from monthly to quarterly values

I'm attempting to aggregate multiple columns of monthly data into quarterly chunks. Currently, I'm applying a rolling sum to the columns and then selecting only every third row to 'delete' or hide the ...
1
vote
1answer
46 views

Applying function to entire dataframe [closed]

I'm wondering if there is a faster way to do the following code, which works fine. It needs to be applied to every row of the DataFrame df. ...
1
vote
1answer
127 views

Faster function iteration on Pandas df with axis=1

I have the following function that allows me to calculate the marketing spend on my advertising campaigns. The problem is that it's now taking too long (over 15 minutes) to apply the function to ...
1
vote
1answer
80 views

SKlearn automate data pre treatment

I want to make a simple wrapper for sklearn models. The idea is that the wrapper automatically takes care of factors (columns of type "object") replacing them with ...
1
vote
1answer
70 views

Multi-linear regression with trivial model refinement algorithms

I wrote this code for an assignment. It was originally meant to be written in Maple, but got very frustrated with some of Maple's idiosyncrasies that I decided to play around with Pandas instead. This ...
1
vote
0answers
18 views

DataSet Scaler with pandas

This is a python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
1
vote
0answers
64 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
1
vote
0answers
57 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...
1
vote
0answers
39 views

is there way to make faster to write to a file?

...
1
vote
0answers
31 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
1
vote
0answers
34 views

Sortings stocks into quantiles based on their signal

This function sorts stocks into quantiles. To do so I used this function that accepts sig_df (dataframe with the timeseries of stocks signal) and number of ...
1
vote
0answers
31 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
1
vote
0answers
18 views

Requests / Facebook API - Dealing with Errors

I work in marketing and I am currently trying to aggregate data for 50 facebook pages on the same Business Page. I have a loop ...
1
vote
0answers
132 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
1
vote
0answers
24 views

Groupby Count on user defined time periods pandas

I have a data frame like: ...
1
vote
0answers
72 views

Batch gradient descent and stochastic gradient descent

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use ...
1
vote
0answers
29 views

Calculating how much $ goes into each defaulting bucket

I have the following code which I use to determine how much of each repayment goes into each late payments bucket (i.e. 0-7 days past due, 7-15 days past due... etc) For example, the repayment amount ...
1
vote
1answer
103 views

Importing text into PANDAS and counting certain words [closed]

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
1
vote
0answers
37 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
1
vote
0answers
60 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
1
vote
0answers
28 views

Writing excelsheet from python dataframe

I am reading dataframe and writing exccel, I cannot use direct df.to_excel method as I am writing data in a very particular way. Looping on dataframe is taking much time, could anything in below ...
1
vote
0answers
73 views

Dependency resolving functions

Written in Python 3.6 and I must admit, this might be too far... Tell me what you think of this code. There must be a better way... It's (another decorator...) that allows you to compose functions ...
1
vote
0answers
49 views

bullet-proof way to generate a list of object attributes (if object has them)

Background I wrote a a couple functions that make it possible to generate a list of attributes for any given object that has them. I use the list that is ...
1
vote
0answers
3k views

Using pandas and sklearn for forecasting stock market return

I have been using R for stock analysis and machine learning purpose but read somewhere that python is lot faster than R, so I am trying to learn Python for that. I am using Yhat's rodeo IDE (Python ...