Questions tagged [pandas]
Pandas is a Python data analysis library.
608
questions
2
votes
0
answers
144
views
Can the k-index in either of the two individual locations be used to predict the estimated kp-index?
The k-index measures the condition of the magnetosphere. It is usually averaged over three hour, so each day has 8 measurements.
The planetary k-index (kp-index) is an average of the measures taken ...
- 201
1
vote
0
answers
46
views
Combined or separate data-cleaning routine
I am a junior data engineer that have 3 years of experience with Python. I write a lot of Python code for my job and I came up with this question I can't solve by my own. I don't have the chance to ...
- 11
2
votes
1
answer
62
views
Use row data from a database to find rows in dataframes that match and use data to generate a separate dataframe
I have a DataFrame (database_df) that contains the general record with the IDs that are the same team in each of the lines, containing these values I need to find ...
- 205
1
vote
2
answers
52
views
Imrove performance when updating DataFrame rows based on complex criteria
My question got rejected the last time so I am trying a better approach to getting a solution:
...
- 21
2
votes
1
answer
127
views
Is this the right implementation for Linear Programming (puLP) on python?
I have created a LP function to help maximize a set of features. My first time playing with this library and also conducting LP.
Variables:
Number of features => X
Number of Categories => Y
...
- 33
1
vote
0
answers
26
views
Pandas to combine data files & add new calculated columns to result
I currently have the following python code that adds a few calculated columns to my consol file. Essentially it combines all the sales files into one combined DF and then adds 4 new sales columns with ...
- 11
1
vote
1
answer
53
views
Using Pandas to group data based on name and see if column value is greater than or equal to values based on group names
As you'll see from the below code, I'm creating separate data frames of a much larger data frame, then updating a column for each one. What I'm doing is looking at the second column and checking to ...
- 173
1
vote
2
answers
41
views
Split Pandas dataset column based on values (suffixes: string operation)
In Python using Pandas, I am splitting a dataset column into 4 lists based on the suffix of the values. For the 3 suffixes I am using a list comprehension then for the 4th one, a set operation that ...
- 111
2
votes
1
answer
64
views
Generating Test Data with Python
Background: I'm a BI developer building a new dashboard for a client. They want to track performance for the week/month/year to date against the prior period. Unfortunately, I don't have direct access ...
- 121
1
vote
0
answers
42
views
Recurrent Neural Network loss is NAN
I am training a neural network to use approximately 600 features (4103rd to last column of a df) to predict approximately 4000 values (7th to 4102nd column of the same df). I have standardized the ...
- 11
2
votes
1
answer
64
views
Flag tukey outliers using python pandas groupby
I'm new to python and pandas.
I would like to use pandas groupby() to flag values in a df that are outliers. I think I've got it working, but as I'm new to python, ...
- 123
1
vote
1
answer
58
views
Applying cointegration function from statsmodels on a large dataframe
I need to apply the coint function from the statsmodels library to 207 times series with 1397 points each, two by two.
Currently, it takes between 35-40 minutes on my computer with an Intel 24 Cores ...
- 113
1
vote
1
answer
53
views
Finding highly correlated variables in a dataframe by evaluating its correlation matrix's values
I read data from Excel into a Pandas DataFrame, so that every column represents a different variable, and every row represents a different sample. I made the function below to identify potential ...
- 25
5
votes
2
answers
192
views
1
vote
1
answer
111
views
Protecting functions from empty DataFrames
Pandas likes to throw cryptic errors when you feed its functions with empty DataFrames saying nothing that would help you to identify the root cause. In order to ...
- 44.1k
1
vote
1
answer
485
views
Mapping pandas' Series to dataclasses
I've got something really simple this time where I'm mapping pandas' Series to dataclasses with a oneliner helper function (as ...
- 44.1k
1
vote
0
answers
134
views
Generic[type[Enum], Protocol[DataFrame]] Dataset with mapped to enum types
Below is my solution for managing multiple DataFrames, in an abstract enough way that it may apply to objects outside of a pandas.DataFrame hence the ...
- 628
1
vote
2
answers
140
views
Replace personal names and addresses with company ones
The problem:
I am given a data frame. Somewhere in that dataframe there is 3*N
number of columns that I need to modify based on a condition. The
columns of interest look like this:
names_1
address_1
...
- 129
4
votes
0
answers
164
views
constraint solving graduation using HTML Parsing, pandas, and z3
not sure if this project fits on code review, but my code is getting extremely messy, and would love some tips to clean it up!
Overview
The project is designed to take in an HTML file (a degree audit),...
- 169
1
vote
1
answer
39
views
simulated samples for central limit theorem
I am trying to help students visualize the central limit theorem and wanted to do this with simulated data.
I created a population dataset with three variables:
...
0
votes
0
answers
248
views
Collect data from the Betfair API using the betfairlightweight module and create a DataFrame including existing values in each Series
My code uses a Series like the one below to create a final DataFrame adding other values that will be collected after access Betfair API.
Example for row_df:
...
- 205
1
vote
1
answer
152
views
Pandas Upsampling Time Series Splitting Equally the values through the weeks starting on monday
I build my code studying this question: "Divide total sum equally to higher sampled time periods when upsampling with pandas".
I am wondering if can be improved the code and if it is right.
...
- 579
0
votes
0
answers
77
views
How to efficiently store and compute forecasting curves?
Let PD be a Plane/Date couple. For each PD, I would like to predict the forecasted booking curve, i.e. the cumuled amount of bookings registered each day between X days before departure and departure. ...
- 53
1
vote
1
answer
66
views
Write a Python script to generate a random DataFrame based on specific inputs
I found myself many times in the past trying to generate fake DataFrames in pandas. I decided just for fun, to write a script that I can specify some inputs and ...
- 157
4
votes
2
answers
158
views
Unstructured to Structured TOC
The following code tries to convert an unstructured TOC with bounding box layout data given by the output of pdftotext -bbox-layout -f 11 -l 13 new_book.pdf toc.html...
- 417
3
votes
0
answers
560
views
Python BeautifulSoup - preparing HTML rows and td tags for Pandas
I'm using BeautifulSoup to parse a bunch of combined tables' rows, row by row, column by column to prepare it for import into Pandas. I can't use to_html() because ...
- 31
0
votes
1
answer
134
views
Efficient way to read files python - 10 folders with 100k txt files in each one
i am looking for an efficient way to read and append texts of .txt files to a dataframe. I currently have 10 folders with 100k documents each.
What i specifically need to do is:
getting the names of ...
1
vote
1
answer
60
views
Make unique id based on text data column with similarity scoring
I have the following dataframe:
...
- 111
0
votes
0
answers
40
views
Find profitable bets from historic results
Each of the lines in my CSV is a possibility of investment that I register on historic, but I would only make the investment if in the existing history (previous lines) the sum of the results is above ...
- 205
1
vote
1
answer
38
views
Create new columns in a DataFrame using functions and reposition the new columns
I would like a review regarding the method I use to create the new columns and then reposition them in the correct place where they should be.
The new column called ...
- 205
-4
votes
1
answer
32
views
Find characters from same homeworld as Chewbacca [closed]
The problem is
Find the names of all characters which are from the same homeworld as Chewbacca
My code is
...
- 1
5
votes
1
answer
149
views
Web scraper for data sources from Statistics Canada
I've written a parser to scrape data from Canadian Statistics Bureau.
...
- 153
2
votes
1
answer
111
views
Efficient List comprehension with multiple conditions using shift? [closed]
I am new to python.
I am trying to get the total number of failures by checking first how did the transition of the column Failure Sensor. Then creating the Start column from devicetimestamp if the ...
- 21
3
votes
1
answer
48
views
Cleaning Float Column of Longitude
I am cleaning a dataset where columns lat and long are presenting some values multiplied by 10. Not only 10, but changing 10^n. I wrote the code below. I am not sure if it is the best way, but is ...
- 77
1
vote
0
answers
40
views
BoundingBox dataclass implementation with cupy, cudf, and nvector
The dataset I'm working with is rather large so I've been experimenting with cudf and cupy. Here you can find instructions for ...
- 628
2
votes
1
answer
202
views
python: requests large.zip -> unzip -> fix -> filter ->gunzip
I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive.
The zip files appear to be malformed and results in a BadZipFileError hence ...
- 628
3
votes
2
answers
178
views
Intercolumn statistics between columns in a dataframe
I have a df and need to count how many adjacent columns have the same sign as other columns based on the sign of the first column, and multiply by the sign of the ...
- 177
1
vote
1
answer
74
views
Create charts after querying database
I'm at the end of the IBM Data Analyst course, and I wanted to ask for a rating of a piece of code I wrote as a solution to its exercises from the final chapter. I know I could write it on the forum ...
- 13
2
votes
2
answers
262
views
Pivoting and then Padding a Pandas DataFrame with NaN between specific columns - Case Study
This question is about pivoting and padding columns, two very frequent activities in Pandas.
I have a raw dataframe. I need to manipulate from long to ...
- 579
1
vote
1
answer
60
views
Plot time windows based on interested value
I use the following code to identify some interested values into a dataframe and them plot a time window before and after that value appeared. It works very well, but I would like to know if there is ...
- 77
2
votes
2
answers
243
views
Convert a mapping of arbitrary pairs into a one-to-many map
The accepted solution is much cleaner and outperforms the algorithm below for small maps, it doesn't do as well with larger ones: my code takes twice as long as the accepted code for maps of 10 pairs (...
- 133
1
vote
1
answer
66
views
Grouping and summing only n variables of m with m>n using a column as key in pandas
I have the following df
...
- 579
2
votes
0
answers
146
views
Do Mann-Kendall and Pettitt tests on each CSV file
Here is a function that will take each text file in a directory, do the Mann-Kendall and Pettitt tests, and then write the output to a text file. Would you please suggest me improve the code to make ...
- 21
1
vote
1
answer
278
views
Iterate and assign weights based on two columns (python)
FI_name
ISN
Sector
Industry
REC
INE02
PS
FS
HDB
INE03
PR
FS
ABC
INE04
PR
FS
RHC
INE05
PR
CO
ZHE
INE06
PR
FS
HSE
INE07
PR
FS
ZAK
INE08
PS
MT
HGB
INE09
PR
FS
YUJ
INE10
PR
MT
WSD
INE11
PS
FS
...
- 13
1
vote
1
answer
60
views
Testing string membership using (in) keyword in python is very slow
I have the following text dataset:
4 million paragraphs of length between (10-60 words each).
...
- 111
2
votes
1
answer
183
views
Slow processing of a python dataframe when aggregating across rows and columns
I would do this in SQL using string_agg but the server is SQL Server 2012 and beyond my control. So I'm trying a python approach.
I have a dataframe of shape [20225 rows x 7 columns], and there a bit ...
- 123
2
votes
1
answer
40
views
Obtaining error code information that occurs before, during, and after a fix/repair using date data
I have completed a project I was working on using the methods I know how, but it is very inefficient. I am a beginner trying to figure out how I can improve my work by using software solutions.
I have ...
- 23
2
votes
1
answer
92
views
Remove duplicates from a Pandas dataframe taking into account lowercase letters and accents
I have the following DataFrame in pandas:
code
town
district
suburb
02
Benalmádena
Málaga
Arroyo de la Miel
03
Alicante
Jacarilla
Jacarilla, Correntias Bajas (Jacarilla)
04
Cabrera d'Anoia
...
- 143
0
votes
1
answer
78
views
Performance issue on updating a Pandas DataFrame with Series based on DateRange
I have two Pandas data frames: one with Daily data and one with Weekly data.
I want to add the weekly data to each row of the daily data for each group of column A.
For example, for each row on the ...
- 113
1
vote
0
answers
36
views
Iterate tables from table id from href links until no table with specific table id is found
I am doing web scraping to the next web page (which is my root URL to start scraping tables): https://www.iso.org/standards-catalogue/browse-by-ics.html
What I am trying to achieve is to parse the ...
- 11