Skip to main content

Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
0 votes
1 answer
62 views

Replace iterrow loops in pandas matrices with something else to shorten the running time

This post is modified from this one: https://codereview.stackexchange.com/posts/292885/edit (Alternatives to iterrow loops in python pandas dataframes). I have a piece of code to calculate price ...
Laura's user avatar
  • 61
5 votes
2 answers
632 views

Alternatives to iterrow loops in python pandas dataframes

I have a piece of code to calculate price sensitivity based on the product and its rating. Below is the original data set with product type, reported year, customer’s rating, price per unit, and ...
Laura's user avatar
  • 61
2 votes
1 answer
37 views

Maintain a log containing values if certain conditions are met

I'm trying to capture profits and set a stop loss in my trading strategy. I want the stop loss to be set daily based on the past data and if the current price, i.e., price for the date falls below the ...
driver's user avatar
  • 232
2 votes
1 answer
222 views

Python using generators with Excelwriter - Performance

I'm looking to understand if my code has an obvious blockage or performance pain point that will cause it to operate slower or use more memory than it should. The current Excelfile i am processing ...
sayth's user avatar
  • 131
3 votes
1 answer
268 views

Transferring dataframe columns into dataframe rows

I have the following data: ...
mahmoud988's user avatar
1 vote
1 answer
108 views

Custom neural network implementation in TensorFlow to compare normalisation vs. no normalisation on data

I am performing a sports prediction multi-class classification problem, and wanted to compare the differences in model performance between normalised and non-normalised data. You can see the 2 ...
pastybake2002's user avatar
3 votes
1 answer
210 views

Machine learning training, hyperparameter tuning and testing with 3 different models

I am trying to solve a multi-class classification involving prediction the outcome of a football match (target variable = Win, Lose or Draw). With a dataset of 2280 rows, which is 6 seasons of ...
pastybake2002's user avatar
3 votes
1 answer
74 views

Calculating premium splits for policies

Looking for a better approach to write below transformation using Python. Is it possible to avoid loop and still achieve the desired output? It is too slow for 10 million rows. ...
user278818's user avatar
5 votes
2 answers
98 views

Creating csvs using Pandas on large dataset for document retrieval

I am trying to build a useable NLP corpus but getting bottlenecked by how long the program takes (200 hours). With so much data I know that optimizing my code even a little bit will net me huge time ...
evader110's user avatar
  • 143
1 vote
1 answer
60 views

Extending die roll simulations for complex data science tasks

I've developed a Python script that simulates die rolls and analyses the results. I'm now looking to extend and modify this code for more complex data science tasks and simulations. Is this code ...
Attila Vajda's user avatar
3 votes
3 answers
152 views

Syntactic sugar for derived variables from Pandas DataFrame columns

Update: Okay, after trying to use this for a while, I think it's probably a bad idea. Please use (lambda x: x["a"] + x["b"])(df) if really ...
user1537366's user avatar
0 votes
2 answers
120 views

Optimize a Python code which indicates duplicated values in an excel file [closed]

I wrote this code to indicate duplicated values. It actually works but I hope to know if there's another possible solution to optimize this process. Thanks. ...
peternish's user avatar
1 vote
0 answers
63 views

Combined or separate data-cleaning routine

I am a junior data engineer that have 3 years of experience with Python. I write a lot of Python code for my job and I came up with this question I can't solve by my own. I don't have the chance to ...
Izem's user avatar
  • 11
2 votes
1 answer
67 views

Use row data from a database to find rows in dataframes that match and use data to generate a separate dataframe

I have a DataFrame (database_df) that contains the general record with the IDs that are the same team in each of the lines, containing these values I need to find ...
Digital Farmer's user avatar
1 vote
2 answers
53 views

Imrove performance when updating DataFrame rows based on complex criteria

My question got rejected the last time so I am trying a better approach to getting a solution: ...
PyNoob's user avatar
  • 21
2 votes
1 answer
179 views

Is this the right implementation for Linear Programming (puLP) on python?

I have created a LP function to help maximize a set of features. My first time playing with this library and also conducting LP. Variables: Number of features => X Number of Categories => Y ...
Kale 's user avatar
  • 23
1 vote
0 answers
35 views

Pandas to combine data files & add new calculated columns to result

I currently have the following python code that adds a few calculated columns to my consol file. Essentially it combines all the sales files into one combined DF and then adds 4 new sales columns with ...
Neo's user avatar
  • 11
1 vote
1 answer
113 views

Using Pandas to group data based on name and see if column value is greater than or equal to values based on group names

As you'll see from the below code, I'm creating separate data frames of a much larger data frame, then updating a column for each one. What I'm doing is looking at the second column and checking to ...
jp207's user avatar
  • 173
1 vote
2 answers
67 views

Split Pandas dataset column based on values (suffixes: string operation)

In Python using Pandas, I am splitting a dataset column into 4 lists based on the suffix of the values. For the 3 suffixes I am using a list comprehension then for the 4th one, a set operation that ...
evilmandarine's user avatar
2 votes
1 answer
73 views

Generating Test Data with Python

Background: I'm a BI developer building a new dashboard for a client. They want to track performance for the week/month/year to date against the prior period. Unfortunately, I don't have direct access ...
DixieFlatline's user avatar
1 vote
0 answers
48 views

Recurrent Neural Network loss is NAN

I am training a neural network to use approximately 600 features (4103rd to last column of a df) to predict approximately 4000 values (7th to 4102nd column of the same df). I have standardized the ...
Manas Garg's user avatar
2 votes
1 answer
159 views

Flag tukey outliers using python pandas groupby

I'm new to python and pandas. I would like to use pandas groupby() to flag values in a df that are outliers. I think I've got it working, but as I'm new to python, ...
Quentin's user avatar
  • 123
2 votes
1 answer
115 views

Applying cointegration function from statsmodels on a large dataframe

I need to apply the coint function from the statsmodels library to 207 times series with 1397 points each, two by two. Currently, it takes between 35-40 minutes on my computer with an Intel 24 Cores ...
Begoodpy's user avatar
  • 125
1 vote
1 answer
341 views

Finding highly correlated variables in a dataframe by evaluating its correlation matrix's values

I read data from Excel into a Pandas DataFrame, so that every column represents a different variable, and every row represents a different sample. I made the function below to identify potential ...
Pimsel's user avatar
  • 25
6 votes
2 answers
583 views

groupby in pandas and plot

I have a csv file that looks like this: ...
Fang's user avatar
  • 545
1 vote
1 answer
385 views

Protecting functions from empty DataFrames

Pandas likes to throw cryptic errors when you feed its functions with empty DataFrames saying nothing that would help you to identify the root cause. In order to ...
t3chb0t's user avatar
  • 44.3k
2 votes
2 answers
2k views

Mapping pandas' Series to dataclasses

I've got something really simple this time where I'm mapping pandas' Series to dataclasses with a oneliner helper function (as ...
t3chb0t's user avatar
  • 44.3k
1 vote
0 answers
225 views

Generic[type[Enum], Protocol[DataFrame]] Dataset with mapped to enum types

Below is my solution for managing multiple DataFrames, in an abstract enough way that it may apply to objects outside of a pandas.DataFrame hence the ...
Jason Leaver's user avatar
1 vote
2 answers
149 views

Replace personal names and addresses with company ones

The problem: I am given a data frame. Somewhere in that dataframe there is 3*N number of columns that I need to modify based on a condition. The columns of interest look like this: names_1 address_1 ...
Glue's user avatar
  • 129
4 votes
0 answers
174 views

constraint solving graduation using HTML Parsing, pandas, and z3

not sure if this project fits on code review, but my code is getting extremely messy, and would love some tips to clean it up! Overview The project is designed to take in an HTML file (a degree audit),...
retep's user avatar
  • 179
1 vote
1 answer
65 views

simulated samples for central limit theorem

I am trying to help students visualize the central limit theorem and wanted to do this with simulated data. I created a population dataset with three variables: ...
Damon C. Roberts's user avatar
1 vote
1 answer
359 views

Pandas Upsampling Time Series Splitting Equally the values through the weeks starting on monday

I build my code studying this question: "Divide total sum equally to higher sampled time periods when upsampling with pandas". I am wondering if can be improved the code and if it is right. ...
Andrea Ciufo's user avatar
1 vote
1 answer
91 views

Write a Python script to generate a random DataFrame based on specific inputs

I found myself many times in the past trying to generate fake DataFrames in pandas. I decided just for fun, to write a script that I can specify some inputs and ...
Tasos's user avatar
  • 159
4 votes
2 answers
183 views

Unstructured to Structured TOC

The following code tries to convert an unstructured TOC with bounding box layout data given by the output of pdftotext -bbox-layout -f 11 -l 13 new_book.pdf toc.html...
Sati's user avatar
  • 417
3 votes
0 answers
917 views

Python BeautifulSoup - preparing HTML rows and td tags for Pandas

I'm using BeautifulSoup to parse a bunch of combined tables' rows, row by row, column by column to prepare it for import into Pandas. I can't use to_html() because ...
Meghan M.'s user avatar
0 votes
1 answer
182 views

Efficient way to read files python - 10 folders with 100k txt files in each one

i am looking for an efficient way to read and append texts of .txt files to a dataframe. I currently have 10 folders with 100k documents each. What i specifically need to do is: getting the names of ...
Piergiorgio Di Pasquale's user avatar
1 vote
1 answer
78 views

Make unique id based on text data column with similarity scoring

I have the following dataframe: ...
illuminato's user avatar
0 votes
0 answers
49 views

Find profitable bets from historic results

Each of the lines in my CSV is a possibility of investment that I register on historic, but I would only make the investment if in the existing history (previous lines) the sum of the results is above ...
Digital Farmer's user avatar
1 vote
1 answer
63 views

Create new columns in a DataFrame using functions and reposition the new columns

I would like a review regarding the method I use to create the new columns and then reposition them in the correct place where they should be. The new column called ...
Digital Farmer's user avatar
-4 votes
1 answer
33 views

Find characters from same homeworld as Chewbacca [closed]

The problem is Find the names of all characters which are from the same homeworld as Chewbacca My code is ...
su sahin's user avatar
5 votes
1 answer
166 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
alphamu's user avatar
  • 153
2 votes
1 answer
159 views

Efficient List comprehension with multiple conditions using shift? [closed]

I am new to python. I am trying to get the total number of failures by checking first how did the transition of the column Failure Sensor. Then creating the Start column from devicetimestamp if the ...
Noobie1997's user avatar
3 votes
1 answer
61 views

Cleaning Float Column of Longitude

I am cleaning a dataset where columns lat and long are presenting some values multiplied by 10. Not only 10, but changing 10^n. I wrote the code below. I am not sure if it is the best way, but is ...
GregOliveira's user avatar
1 vote
0 answers
46 views

BoundingBox dataclass implementation with cupy, cudf, and nvector

The dataset I'm working with is rather large so I've been experimenting with cudf and cupy. Here you can find instructions for ...
Jason Leaver's user avatar
2 votes
1 answer
316 views

python: requests large.zip -> unzip -> fix -> filter ->gunzip

I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive. The zip files appear to be malformed and results in a BadZipFileError hence ...
Jason Leaver's user avatar
3 votes
2 answers
195 views

Intercolumn statistics between columns in a dataframe

I have a df and need to count how many adjacent columns have the same sign as other columns based on the sign of the first column, and multiply by the sign of the ...
jaried's user avatar
  • 177
1 vote
1 answer
96 views

Create charts after querying database

I'm at the end of the IBM Data Analyst course, and I wanted to ask for a rating of a piece of code I wrote as a solution to its exercises from the final chapter. I know I could write it on the forum ...
kkkkrkrkkk's user avatar
2 votes
2 answers
421 views

Pivoting and then Padding a Pandas DataFrame with NaN between specific columns - Case Study

This question is about pivoting and padding columns, two very frequent activities in Pandas. I have a raw dataframe. I need to manipulate from long to ...
Andrea Ciufo's user avatar
1 vote
1 answer
94 views

Plot time windows based on interested value

I use the following code to identify some interested values into a dataframe and them plot a time window before and after that value appeared. It works very well, but I would like to know if there is ...
GregOliveira's user avatar
2 votes
2 answers
363 views

Convert a mapping of arbitrary pairs into a one-to-many map

The accepted solution is much cleaner and outperforms the algorithm below for small maps, it doesn't do as well with larger ones: my code takes twice as long as the accepted code for maps of 10 pairs (...
shortorian's user avatar

1
2 3 4 5
13