Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

ML Pipeline with OneHotEncoder and model

can you please provide some advice on my codes below? There is 7 categorical features and 1 categorical target. This is a classification problem and I'm using RandomForest to train the data. I have ...
1
vote
0answers
22 views

DataSet Scaler with pandas

This is a Python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
1
vote
0answers
87 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
1
vote
0answers
99 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...
1
vote
0answers
39 views

is there way to make faster to write to a file?

...
1
vote
0answers
32 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
1
vote
0answers
34 views

Sortings stocks into quantiles based on their signal

This function sorts stocks into quantiles. To do so I used this function that accepts sig_df (dataframe with the timeseries of stocks signal) and number of ...
1
vote
0answers
31 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
1
vote
0answers
18 views

Requests / Facebook API - Dealing with Errors

I work in marketing and I am currently trying to aggregate data for 50 facebook pages on the same Business Page. I have a loop ...
1
vote
0answers
135 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
1
vote
0answers
25 views

Groupby Count on user defined time periods pandas

I have a data frame like: ...
1
vote
0answers
75 views

Batch gradient descent and stochastic gradient descent

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use ...
1
vote
0answers
29 views

Calculating how much $ goes into each defaulting bucket

I have the following code which I use to determine how much of each repayment goes into each late payments bucket (i.e. 0-7 days past due, 7-15 days past due... etc) For example, the repayment amount ...
1
vote
1answer
103 views

Importing text into PANDAS and counting certain words [closed]

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
1
vote
0answers
44 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
1
vote
0answers
61 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
1
vote
0answers
28 views

Writing excelsheet from python dataframe

I am reading dataframe and writing exccel, I cannot use direct df.to_excel method as I am writing data in a very particular way. Looping on dataframe is taking much time, could anything in below ...
1
vote
0answers
74 views

Dependency resolving functions

Written in Python 3.6 and I must admit, this might be too far... Tell me what you think of this code. There must be a better way... It's (another decorator...) that allows you to compose functions ...
1
vote
0answers
49 views

bullet-proof way to generate a list of object attributes (if object has them)

Background I wrote a a couple functions that make it possible to generate a list of attributes for any given object that has them. I use the list that is ...
1
vote
0answers
3k views

Using pandas and sklearn for forecasting stock market return

I have been using R for stock analysis and machine learning purpose but read somewhere that python is lot faster than R, so I am trying to learn Python for that. I am using Yhat's rodeo IDE (Python ...
1
vote
0answers
667 views

ChiSquare test Code in Python

I was trying to write a code from scratch for Chi-square test.This is the code that I had written in python using pandas.I had a doubt whether the code can produce the desired output or can be written ...
1
vote
0answers
125 views

Memory efficient alternative passing mostly static variables w/ 1 dynamic variable to an external function (COM) instead of a for loop in Python

Massive data problem, 7/8 variables are repeated 1000 times in a for loop, sending 8 input vectors * 50,000 values per loop for each function call. Only 1 input ...
1
vote
0answers
97 views

Generate features for future ML analysis of asset returns

I have built the following code to download stock data from Yahoo Finance. The plan is to then use the built-in pandas functions to calculate metrics from this data ...
0
votes
2answers
16k views

Interpret YYYYMMDD as the nth day of the year [closed]

I am provided a bunch of dates in the format (YYYYMMDD) such as: date='20170503' My goal is to convert that date into the ...
0
votes
1answer
7k views

Searching for keywords in dataframe

Given a dataframe with three columns of text blobs to search through, which can be found in this Gist. And three keywords that I want to identify in this text: ...
0
votes
1answer
84 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
0
votes
1answer
41 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
0
votes
1answer
94 views

Linear Regression on Pandas

I'm working on a simple statistics problem with Pandas and sklearn. I'm aware that my code is ugly, but how can I improve it? <...
0
votes
1answer
68 views

How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done. ...
0
votes
1answer
115 views

Assigning zero to random cells in a pandas.DataFrame

I am trying to assign 0 to random cells in a one dimensional pandas.DataFrame. The below code is the best way I could think of ...
0
votes
1answer
54 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
0
votes
1answer
687 views

Simple k-means implemention using Python3 and Pandas

Is there anything I can improve? The distance function is Pearson correlation. ...
0
votes
1answer
40 views

Calculate the statistical mean of different files and save it into a CSV file

In this code I calculate the statistical functions of values from different files and save them into CSV files (one filer for each statistical function). ...
0
votes
1answer
954 views

Get all node descendants in a tree

I have a CSV holding a "flat" table of a tree's edges (NOT binary, but a node cannot have two parents), ~1M edges: ...
0
votes
2answers
66 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...
0
votes
2answers
11k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
0
votes
1answer
8k views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
0
votes
1answer
41 views

Filter if any value in group is null

The following code works, but the filter operation is bottlenecking my analysis. Do you have an idea how I can make the filter operation faster? Do you have other comments about the codestyle? ...
0
votes
1answer
94 views

Finding the difference in time between two events for a specific ID

Details about code: Dataframe (x) shape: (20494463, 31) In this Dataframe, each id shows up multiple times with different events and the event dates. What I'm looking for is the time between the ...
0
votes
1answer
194 views

Calls one column of a dataframe, turns it into an array and plots it

My code calls 1 column of a dataframe, turns it into an array and plots it. I want to able to do this for all the columns without having to repeat the code many times. How can I do this? ...
0
votes
0answers
2 views

Improving loop through xml files ad add to DF

With this code, i loop through similar xml files and add them to a DataFrame. How can i make this code better so it is more pythonic? ...
0
votes
0answers
20 views

Grouping people (that live close together) in a group using pandas dataframe [closed]

Note: the following code is kind of working, but it's not really doing what I want it to do. I hope this is okay and doesn't violate this community guidelines!!! If so, please apologies! I hope you ...
0
votes
0answers
32 views

Script for matching regex matches between two lists

gist of this project is that I have two datasets, 1. dfCore, which contains station_code and station_name 2. dfMetrix, which contains station_name the ...
0
votes
0answers
79 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
0
votes
1answer
35 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
0
votes
0answers
30 views

Search a better way to work with pandas dataframe when i use lamba and apply

I have this code to take prices from a database into pandas, and what i'm doing here is group my prices based on itemnumber and price group, and then i want to push it to a dataframe where i use apply....
0
votes
0answers
21 views

Grouping by on Pandas df with focus on performance

I am doing a group by on a subset of a dataframe. I am working with 9m+ rows atm, subject to sharp increase. I tried different approaches already and wanted to know if i could squeeze a bit more ...
0
votes
0answers
286 views

Calculating Relative Strength Index for Multiple Pandas column

I want calculate RSI indicator value for multiple column in Pandas DataFrame. I am looking for a method to avoid loop, here is the code I am using: ...
0
votes
1answer
5k views

Convert Excel files to CSV

I wrote the following script to run through several excel files and format them before saving as a CSV for upload to a Quickbase app I'm creating. Each excel file is roughly 100k lines and the code ...
-1
votes
1answer
47 views

Write/read efficiently dataframe objects into memory or disk [closed]

i'm running a for loop that loops over all the rows of a pandas dataframe, then it calculates the euclidean distance from one point at a time to all the other points in the dataframe, then it pass the ...

1
4 5 6 7
8