Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1
vote
0answers
622 views

ChiSquare test Code in Python

I was trying to write a code from scratch for Chi-square test.This is the code that I had written in python using pandas.I had a doubt whether the code can produce the desired output or can be written ...
1
vote
0answers
125 views

Memory efficient alternative passing mostly static variables w/ 1 dynamic variable to an external function (COM) instead of a for loop in Python

Massive data problem, 7/8 variables are repeated 1000 times in a for loop, sending 8 input vectors * 50,000 values per loop for each function call. Only 1 input ...
1
vote
0answers
97 views

Generate features for future ML analysis of asset returns

I have built the following code to download stock data from Yahoo Finance. The plan is to then use the built-in pandas functions to calculate metrics from this data ...
0
votes
2answers
15k views

Interpret YYYYMMDD as the nth day of the year [closed]

I am provided a bunch of dates in the format (YYYYMMDD) such as: date='20170503' My goal is to convert that date into the ...
0
votes
1answer
6k views

Searching for keywords in dataframe

Given a dataframe with three columns of text blobs to search through, which can be found in this Gist. And three keywords that I want to identify in this text: ...
0
votes
1answer
80 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
0
votes
1answer
38 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
0
votes
1answer
80 views

Linear Regression on Pandas

I'm working on a simple statistics problem with Pandas and sklearn. I'm aware that my code is ugly, but how can I improve it? <...
0
votes
1answer
66 views

How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done. ...
0
votes
1answer
73 views

Assigning zero to random cells in a pandas.DataFrame

I am trying to assign 0 to random cells in a one dimensional pandas.DataFrame. The below code is the best way I could think of ...
0
votes
1answer
51 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
0
votes
1answer
686 views

Simple k-means implemention using Python3 and Pandas

Is there anything I can improve? The distance function is Pearson correlation. ...
0
votes
1answer
814 views

Get all node descendants in a tree

I have a CSV holding a "flat" table of a tree's edges (NOT binary, but a node cannot have two parents), ~1M edges: ...
0
votes
2answers
58 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...
0
votes
2answers
9k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
0
votes
1answer
6k views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
0
votes
1answer
40 views

Filter if any value in group is null

The following code works, but the filter operation is bottlenecking my analysis. Do you have an idea how I can make the filter operation faster? Do you have other comments about the codestyle? ...
0
votes
1answer
93 views

Finding the difference in time between two events for a specific ID

Details about code: Dataframe (x) shape: (20494463, 31) In this Dataframe, each id shows up multiple times with different events and the event dates. What I'm looking for is the time between the ...
0
votes
1answer
172 views

Calls one column of a dataframe, turns it into an array and plots it

My code calls 1 column of a dataframe, turns it into an array and plots it. I want to able to do this for all the columns without having to repeat the code many times. How can I do this? ...
0
votes
0answers
66 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
0
votes
1answer
35 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
0
votes
0answers
30 views

Search a better way to work with pandas dataframe when i use lamba and apply

I have this code to take prices from a database into pandas, and what i'm doing here is group my prices based on itemnumber and price group, and then i want to push it to a dataframe where i use apply....
0
votes
0answers
20 views

Grouping by on Pandas df with focus on performance

I am doing a group by on a subset of a dataframe. I am working with 9m+ rows atm, subject to sharp increase. I tried different approaches already and wanted to know if i could squeeze a bit more ...
0
votes
0answers
226 views

Calculating Relative Strength Index for Multiple Pandas column

I want calculate RSI indicator value for multiple column in Pandas DataFrame. I am looking for a method to avoid loop, here is the code I am using: ...
0
votes
0answers
17 views

Create an interpolant based on data in a csv

I have a csv file with many columns (e.g. A,B,C,D,...
0
votes
1answer
5k views

Convert Excel files to CSV

I wrote the following script to run through several excel files and format them before saving as a CSV for upload to a Quickbase app I'm creating. Each excel file is roughly 100k lines and the code ...
-1
votes
1answer
46 views

Write/read efficiently dataframe objects into memory or disk [closed]

i'm running a for loop that loops over all the rows of a pandas dataframe, then it calculates the euclidean distance from one point at a time to all the other points in the dataframe, then it pass the ...
-3
votes
0answers
13 views

Parsing text from multiple downloaded htmls [closed]

I want to extract from a directory (where i have stored my downloaded htmls), all the "executives". In this directory their are app 1.000 htmls ( (example: https://www.dropbox.com/s/uka24w7o5006ole/...
-3
votes
1answer
27 views

Need help getting the output of a for loop into Excel/CSV using OS.WALK [closed]

How can I print the results list into a CSV/Excel file? ...