Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Pandas: process data on dataframe subsets

I have written a piece of code which select each ID whose value equals something, then select a subset for each ID and then work with the subdata. My dataset has over a million entries, thus the code ...
0
votes
1answer
47 views

Extract and write fasta files using Biopython based on input from another file

I have the below code that takes a sequence file and another file with a list of contigs and extracts the sequences and writes them to a file, specifically based on the file with the contig list. The ...
0
votes
0answers
20 views

Optimizing code for speed for use of large data sets when iterating and performing operations on multiple pandas data frames contained within a list

I have four data frames namely: U,B,V,R They each contain two columns: wavelength and fraction, filled with floats/integers. I am applying an operation to all these data frames and iterating over them....
2
votes
1answer
66 views

How do you efficiently calculate YTD values in a pandas dataframe?

I implemented the following code to calculate the YTD sum in Pandas: ...
1
vote
0answers
28 views

Pandas rolling evaluation

I want to perform some calculations over rolling periods of a pandas DataFrame and to abstract all this in a dedicated class. The job of the class below is simply, given backward and forward looking ...
1
vote
1answer
72 views

Slicing multi index DataFrame into JSON object

I have a MultiIndex pd.DataFrame that I generated from a .txt that is forecast model data. ...
1
vote
1answer
70 views

Better way to create a contingency table with pandas for film genres from a Film DataFrame

From a public dataset available on film rating I created a contingency table as follow. Honestly I don't like all these "for-loops" I think the quality of the code can be definitely improved ...
1
vote
1answer
60 views

Multi-step aggregation of cryptocurrency data with high precision

My goal is to aggregate a DataFrame with several million rows including Decimal('...') columns without any precision loss. My ...
1
vote
0answers
22 views

simulation of updating a dataset from real time data

In this code (reproducible example) I want to generate a dataframe composed by random dates in the first col, and by random numbers (in reality they will be generate from a socket) in the second ...
0
votes
1answer
44 views

Binning multicolumn Pandas Dataframe

I'm trying to bin multicolumn Pandas Dataframe, and use the upper limit of the interval for further analysis. This is my approach: # Creating a dummy dataframe: ...
1
vote
1answer
96 views

Calculate a time weighted average of a feature

I'm trying to calculate a time weighted average of a feature (feat) based on previous rows by date for a given ...
1
vote
3answers
62 views

count number of rows given column names and values - pandas

I am currently switching for R to Python so please be patient with me. Is the following a good way to count the number of rows given column names and values? ...
3
votes
1answer
62 views

Transposing an Excel sheet with pandas adding additional information for a different readability - Using Fake Pizzeria Data

The goal is to transpose the excel file that comes in the following form into this The code I created works properly, but I think it can be improved. I would also like to avoid using ...
0
votes
1answer
91 views

How to speed up the search for matching values in a second data frame in a line-by-line iteration over a first data frame?

I have a survey response dataframe for which each survey has a code: df ...
2
votes
1answer
132 views

Compute the mean of every X rows of a pandas DataFrame

I coded a little loop that reduces a dataset by computing the mean of X rows every X rows. I'm working with a dataset of a certain size (8 Millions rows for 10 columns) and my code takes too long to ...
0
votes
1answer
97 views

Cleanup dataset for visualization

This code cleans up a big dataset into a very clean and a flat file that can be further used downstream like visualization. I am expecting to improve the code to essentially run it faster and cleanup ...
3
votes
1answer
116 views

Web scraping international trade data

I'm trying to scrape arrival data from this website. My script takes extremely long time to scrape the data. Is there any way I can speed up the scraping process? Here's my script: ...
2
votes
1answer
57 views

Get data from binance api and save to ClickHouse DB

I did the following Created DB and table in prepate_table function Extracted data from binance in GetHistoricalData function Saved the data to DB My code works fine but I want to optimize my code ...
1
vote
1answer
70 views

using pandas and numpy to parse dirty .csv

I'm relatively new to Python have been writing for a few months now. I've started a Pandas Numpy project that starts with parsing large and somewhat sloppy formatted textfile, its not exactly csv but ...
1
vote
1answer
238 views

How to speed up the search by patterns matching and modifying a DataFrame

I need to classify URLs from a DataFrame and modify it by exact match and contains conditions: ...
0
votes
2answers
81 views

Combining iterrows of pandas with os.walk given I have a lot of directories

I have the following code snippet: ...
7
votes
1answer
109 views

Output number densities in model atmosphere of the Sun

I have completed my Python project which calculates and outputs the number densities of dozens of different species (over 11 chemical elements total) in a model atmosphere of the Sun. The code reads ...
2
votes
1answer
56 views

Read CSV, with date filtering and resampling

I have written the following function to read multiple CSV files into pandas dataframes. Depending on the use case, the user can pass optional resampling (frequency and method) and/or a date range (...
2
votes
1answer
48 views

Finding the difference in one column in a Pandas GroupBy

I am trying to calculate the time difference between samples that a satellite takes at the same location. I have data currently in a Pandas DataFrame that has latitude, longitude, and time of the ...
1
vote
1answer
80 views

Scrape from Craigslist

I'm new to Python and just started to learn about scraping and pandas library. Here is a little scraper I wrote. I'd like to know what's a professional code for this would look like. I have a sense my ...
0
votes
1answer
99 views

add column with count by constraint

Could someone please help me out, I'm trying to remove the need to iterate through the dataframe and know it is likely very easy for someone with the knowledge. Dataframe: ...
1
vote
1answer
71 views

Python program to accept user input for Pandas methods

I know this is a weird program but I am trying to learn how to write Python programs. This program will accept user input, starting by having the user specify a file path then continuously looping ...
3
votes
2answers
66 views

Pandas split column into multiple: performance improvement

Aim of the Code: The aim of the function I'm working on is to take a particular column from a DataFrame that contains multiple <tag><content> pairs in ...
1
vote
1answer
110 views

Filtering data on a dataframe, Pandas-Jupyter

I have a dataframe which contains information of programmers like: country, programming languages. etc: ...
5
votes
1answer
67 views

GUI that reads data and generates/ saves charts

I have a program that uses pandas to read csv files and then generates and saves graphical charts. I have been trying to follow the SOLID principles so I have tried to seperate responsibilities. So ...
3
votes
0answers
60 views

pandas.DaraFrame resample OHLC based on a non-time value

I have a csv file that looks like this: ...
1
vote
0answers
34 views

quant. code. to calculate trend, momentum, correlation and volatility in Pandas(Python)

I am new to quant. finance and trying to calculate trend, momentum, correlation and ...
1
vote
0answers
57 views

Optimising this web-scraping code

A member of SO who I immensely respect just told me that the code below makes him uncomfortable. ...
1
vote
0answers
112 views

Python, create a network with a given node and edges from pandas dataframe

I'm working on a code snippet that generates a network graph. ...
3
votes
1answer
64 views
0
votes
1answer
38 views

Multiple CSV Imports with Pandas

I have a class that is importing a CSV file into a Pandas DataFrame. However, this CSV has a multiline header that changes length seemingly at random. Sometimes it is 27 lines, sometimes 31, sometimes ...
2
votes
0answers
74 views

Compute damages and savings for motor insurance claims

I am working on some claims data, where I have features such as The descriptions of the columns are: (A) the year of claim, (B) who's the insuring company (C) whether the vehicle is driven, (D) ...
1
vote
0answers
66 views

How to Better Parse CSV file with Pandas

I am working on a CSV data Sheet and want to parse and filter the data out it, While working on a code I found a similar code someone has asked on SO POST there and ...
2
votes
1answer
56 views

Find the most similar pair of rows from a table

I have a list of n cities and hourly weather prediction for each one of them for h hours. I'm writing a script to find a pair of ...
0
votes
0answers
30 views

How to most efficiently make repeated calls to API with Pandas/Python

Looking for some help on how to speed up the code below as I'm still relatively new to using Pandas. The playoff_stats function makes thousands of calls to the PlayerCareerStats API - and while it ...
0
votes
1answer
153 views

create weekly data for stocks faster based on daily data

daily data: ...
4
votes
2answers
69 views

For each of the visitors, find if they were present within a time period: code is running slow

I'm working with a pandas dataframe which describes the start and end time of visits. A toy example of the data could be as follows: ...
3
votes
0answers
42 views

Extracting Korean text for sentiment analysis

It is quite long so I'm not sure I can ask you to review my code. But I have no choice indeed. What I'm doing is analyzing the Korean language. To do this, first I collect data and loaded the data via ...
0
votes
0answers
42 views

How do I make my html object-to-table array script run faster? (Pandas)

I have a data frame that has several columns, one of which contains html objects (containing tables). I want a column of table arrays. My problem is that this piece of code takes a long time to run. ...
0
votes
1answer
38 views

Get a "train or test" slicing elegantly after splitting

I am learning numpy, pandas, and sklearn for machine learning now. My teacher gives me code like below, but I guess it is ugly. This code is for splitting the data into training set and testing set, ...
1
vote
1answer
48 views

Check if any of the coordinates are within a certain distance using apply in Python pandas with two data frames

I wrote this code that takes the vessel coordinates and checks them against every possible port coordinates. If the distance between the vessel and the port coordinates is within x meters (in this ...
0
votes
0answers
33 views

Plotting 2 types of data from DataFrame with different colors

This is the task: ...
1
vote
1answer
58 views

Pandas merge a "%Y%M%D" date column with a "%H:%M:%S" time column

I have a dataframe with a column composed by date object and a column composed by time ...
0
votes
0answers
35 views

Keras: Using Autoencoders for prediction

I have a dataset which I divided into two sections horizontally. Column A of first section is the input variable and Column A of second section is the target variable. I am trying to build a Denoising ...
3
votes
1answer
2k views

William Fractal technical indicator implementation

I'm really new to both Python, data analysis and Panda and for gathering a bit of familiarity with this components and trying to replicate a trading strategy I'm creating a function that give me the ...

1
2 3 4 5
11