Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
0 votes
1 answer
15 views

Treat a list by generating a dataframe and sending the data to function via multiprocessing

To collect the list with the data from an API, I need to do these steps: ...
user avatar
-1 votes
1 answer
54 views

Forecast experimental results based on temperature [closed]

I would really welcome any hints on making this code more concise please. In this example we have some experiment results. For each planned experiment We have predicted results based on 4 temperatures ...
user avatar
  • 3
0 votes
2 answers
45 views

Speeding up row insertion to Pandas DataFrames [closed]

I have the below code that seems to be taking a while to run over time. ...
user avatar
  • 27
3 votes
1 answer
47 views

Crawler/scraper for soccer match results

I wrote this code some time ago as part of the web-scraping learning. Every now and then I find mistakes in it, as well as I have doubts. Feedback please, is this code compliant with the common best ...
user avatar
1 vote
1 answer
49 views

Generate new column based on two columns of dataframe

I need to generate column a_b based on column a and column b of ...
user avatar
  • 135
0 votes
0 answers
42 views

Deep Reinforcement Learning training on gym data

Related question: Extract the last row in a dataframe with a timestamp before some given time This is part of the code for Deep Reinforcement Learning training on a standards-based gym env. What I ...
user avatar
  • 135
1 vote
1 answer
49 views

Design Pattern: Builder - BeautifulSoup directory navigation and scraping

I wrote a class on top of BeautifulSoup using the builder design pattern that allows for the navigation of the necp data directory. There are a couple navigation ...
user avatar
1 vote
1 answer
21 views

dataframe replace (numeric) categorical values by their frequency of label = 1

Here is my dataframe: data = [['a1','b1',0], ['a2','b3',0], ['a1','b2',1], ['a1','b1',1], ['a2','b3',0]] df = pd.DataFrame(data=data, columns = ['A','B','label']) ...
user avatar
2 votes
1 answer
86 views

Extract the last row in a dataframe with a timestamp before some given time

Related questions: I need to take specific row from different df to generate new df. How can I make my code faster? I want to take the last data before the specified time from different time intervals ...
user avatar
  • 135
3 votes
1 answer
97 views

web scraping with pandas and apscheduler

Below is a script to scrape data from the National Severe Storms Laboratory Probsevere dataset. Their data is available at a 2 minute interval and is only available for 24 hours before moving to the ...
user avatar
2 votes
2 answers
122 views

Generate date_range for string intervals like 09:00-10:00/11:00-14:00/15:00-18:00 and calculate the working minutes

I'm a bit new in python and I'm trying to achieve what I can't with VBA, get things faster. I have large datasets like this in the source(just a sample): This is done also by python and it's a CSV ...
user avatar
  • 178
2 votes
2 answers
96 views

Create new columns based on existing columns

The code: ...
user avatar
1 vote
1 answer
45 views

Cleaning up a PANDAS dataframe of Brazilian football statistics

I am trying to cleanup a dataframe (and the code) in Pandas. I used to use the modin library before but I have since switched from Win 10 to Win 11 and none of <...
user avatar
2 votes
2 answers
81 views

Regex and pandas to read forecast sky condition string

DataFrame methods to parse the sky condition from a terminal aerodrome forecast. A line in a taf can report zero-eight cloud layers. Cloud layers are required in predominate lines, and optional in ...
user avatar
3 votes
0 answers
88 views

Simple python quiz

I'm learning python and i'm trying to practice what I learned recently. I wanted to make a quiz game using pandas. What do you guys think? Is there anything I could do better? ...
user avatar
  • 31
2 votes
1 answer
54 views

Regex pattern matching to generate pandas multi index

Mostly just looking for a review of my regex and implementation of capture groups. its something I've been working to improve. The indexes have somewhat of a pattern to them of being.... ...
user avatar
4 votes
2 answers
78 views

Read a file format with mixed CSV and non-CSV content

I have a file format that I need to read which contains blocks of CSV data interspersed by non-CSV (or at least, inconsistent CSV) lines. Other than the very first line of the file (which has comma-...
user avatar
  • 263
0 votes
1 answer
59 views

Stacking Z axis on multiple [1440x720] DataFrames (X, Y)

Each datum represents a point in a 1440x720 image of the globe, The values are 0-9. Each new layer is of a higher elevation. I need to structure a DataFrame in a ...
user avatar
4 votes
1 answer
91 views

Parsing an IP routing report with half a million lines into a PANDAS dataframe

I have a file that has around 440K lines of data. I need to read these data and find the actual "table" in the text file. Part of the text file looks like this. ...
user avatar
0 votes
0 answers
50 views

mypy type hinting with an extended pandas Series

I'm learning to use mypy and the pyproject.toml config pyproject.toml ...
user avatar
3 votes
1 answer
96 views

Fetch CSV file into PANDAS and send report of Spotify statistics

I'm relatively new and self taught in Python. I've written this code which gets a CSV file from a website, plays around with the data then creates an email with information gleaned. I know this is too ...
user avatar
  • 33
3 votes
0 answers
46 views

Scrape classifications from Wikipedia with bs4

I wrote my first bs4 script that scrapes animal classifications from Wikipedia. It works as long as the classifications are stored within these tags ...
user avatar
  • 131
2 votes
1 answer
68 views

type hinting/documenting/extension of a Cython lib

I've updated some of the type hinting/documentation in a lib called pygrib. The source documentation can be found here. The goal is to extend the ...
user avatar
1 vote
1 answer
47 views

change object column to numeric above cardinality threshold pandas

Is the following an acceptable way to change an object column to numeric if cardinality threshold (cardinality_threshold) is being breached? I think it would be good to check if column values are ...
user avatar
  • 631
-1 votes
1 answer
41 views

Extraction of subsets by date and intervals from a dataframe with a specific date format

Given the following dataframe: ...
user avatar
  • 121
1 vote
1 answer
135 views

Optimise a function with numerous conditions that depends on the previous row in a Python dataframe

I have the following dataframe: country_ID ID direction date ESP_1 0 IN 2021-02-28 ENG 0 IN 2021-03-03 ENG 0 OUT 2021-03-04 ESP_2 0 IN 2021-03-05 FRA 1 OUT 2021-03-07 ENG 1 OUT 2021-03-09 ENG ...
user avatar
  • 121
2 votes
1 answer
68 views

dataframe mean outliers to NaN to derive a higher quality mean for area,speed,azimuth

I have some data that represents the area, speed(meters per second), and azimuth(rads) of a polygon. The objective is to determine the mean for each set of parameters. With that mean value, apply a ...
user avatar
3 votes
0 answers
66 views

Scraping housing dataset row by row from Estate Agent's website

I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
user avatar
2 votes
0 answers
42 views

Python data entry into excel based on equality check with dataframe

My objective is to enter/type in values into Excel cells if the column names and indices match between Excel and dataframe. So, my code does the below Gets the ID values from the dataframe for each ...
user avatar
  • 235
0 votes
2 answers
109 views

Python script for Web scraping

I wrote a Python script for Web scraping of a Website. Please review my code and suggest me any changes or make me aware of my blunders/mistakes?. I wrote the almost same script for other websites ...
user avatar
3 votes
1 answer
100 views

pandas dataframe of temps and numpy.linalg.lstsq

The problem is this. Forecast model data indicates at 0700Z the 1000 milibar temps will be -4C°. The forecaster knows that is incorrect, so they choose to make a correction. This correction should ...
user avatar
0 votes
1 answer
71 views

Apply a funtion to each element of a dataframe according to other elements values in the corresponding row in pandas and python

I am a little new in the Pandas and Python framework. I would like to understand if my code is more or less fine or it too related to my old program framework way of thinking. I have this is the input ...
user avatar
  • 101
0 votes
1 answer
86 views

Calculating the value of a stock-portfolio over time using Python Pandas

I am quite at the beginning of learning Python and decided to start my own little coding project to improve my skills. The goal of my project is to simulate the past growth of a stock portfolio based ...
user avatar
4 votes
1 answer
107 views

pandas conditions across rows and columns

Is there a better way to accomplish this? The function determines the Category for each row based on some conditions. Another set of conditions determines the <...
user avatar
0 votes
1 answer
38 views

find maximum corresponding column for maximum value after group by

For my haves: I want to find the key values for the maximum value whilst grouping by g1 and g2 to get these wants: This is my current possibly (?) clumsy code: ...
user avatar
  • 631
1 vote
0 answers
39 views

Pandas Numpy Nvector to generate a storm track algorithm

I made previous post here, relating to this topic. The class uses now uses some additional 3rd party libs nvector & pygc. ...
user avatar
1 vote
1 answer
60 views

Multiple GeoJSON Feature Collections to a Pandas MultiIndex DataFrame

Edit: The end objective is to monitor the movement of severe weather, determine its vector and make a prediction. I'm working with a GeoJSON ...
user avatar
0 votes
0 answers
449 views

Zigzag indicator for stock prices

I have read on SO and replicated an indicator for stock prices that works as intended. It's called ZigZag and projects peaks and valleys on historical prices. I pass a pandas dataframe with OHLC ...
user avatar
6 votes
2 answers
431 views

Construct Pandas DataFrame whose (i,j)th entry is the probability that a person aged i+100 will still be alive after j years

The following function takes the probability of a person aged 100+i dying in the next year (conditional on them being alive at the start of the year) and returns the probability that they will be ...
user avatar
1 vote
1 answer
56 views

Get last n rows idxmin for each group by Pandas

Goal I want to get the index of the min of last 3 rows based on low column. ...
user avatar
  • 161
2 votes
2 answers
168 views

Pandas MultiIndex DataFrame groupby apply to generate List[Dict[str,str|Dict[...]]]

I have a MultiIndex DataFrame ...
user avatar
0 votes
0 answers
27 views

Pandas: process data on dataframe subsets

I have written a piece of code which select each ID whose value equals something, then select a subset for each ID and then work with the subdata. My dataset has over a million entries, thus the code ...
user avatar
  • 101
0 votes
1 answer
81 views

Extract and write fasta files using Biopython based on input from another file

I have the below code that takes a sequence file and another file with a list of contigs and extracts the sequences and writes them to a file, specifically based on the file with the contig list. The ...
user avatar
0 votes
0 answers
25 views

Optimizing code for speed for use of large data sets when iterating and performing operations on multiple pandas data frames contained within a list

I have four data frames namely: U,B,V,R They each contain two columns: wavelength and fraction, filled with floats/integers. I am applying an operation to all these data frames and iterating over them....
user avatar
  • 1
2 votes
1 answer
208 views

How do you efficiently calculate YTD values in a pandas dataframe?

I implemented the following code to calculate the YTD sum in Pandas: ...
user avatar
  • 123
1 vote
0 answers
32 views

Pandas rolling evaluation

I want to perform some calculations over rolling periods of a pandas DataFrame and to abstract all this in a dedicated class. The job of the class below is simply, given backward and forward looking ...
user avatar
  • 983
1 vote
1 answer
149 views

Slicing multi index DataFrame into JSON object

I have a MultiIndex pd.DataFrame that I generated from a .txt that is forecast model data. ...
user avatar
1 vote
1 answer
105 views

Better way to create a contingency table with pandas for film genres from a Film DataFrame

From a public dataset available on film rating I created a contingency table as follow. Honestly I don't like all these "for-loops" I think the quality of the code can be definitely improved ...
user avatar
1 vote
1 answer
64 views

Multi-step aggregation of cryptocurrency data with high precision

My goal is to aggregate a DataFrame with several million rows including Decimal('...') columns without any precision loss. My ...
user avatar
1 vote
0 answers
27 views

simulation of updating a dataset from real time data

In this code (reproducible example) I want to generate a dataframe composed by random dates in the first col, and by random numbers (in reality they will be generate from a socket) in the second ...
user avatar
  • 111

1
2 3 4 5
12