Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1 vote
1 answer
105 views

Custom neural network implementation in TensorFlow to compare normalisation vs. no normalisation on data

I am performing a sports prediction multi-class classification problem, and wanted to compare the differences in model performance between normalised and non-normalised data. You can see the 2 ...
pastybake2002's user avatar
3 votes
1 answer
206 views

Machine learning training, hyperparameter tuning and testing with 3 different models

I am trying to solve a multi-class classification involving prediction the outcome of a football match (target variable = Win, Lose or Draw). With a dataset of 2280 rows, which is 6 seasons of ...
pastybake2002's user avatar
3 votes
1 answer
72 views

Calculating premium splits for policies

Looking for a better approach to write below transformation using Python. Is it possible to avoid loop and still achieve the desired output? It is too slow for 10 million rows. ...
user278818's user avatar
5 votes
2 answers
97 views

Creating csvs using Pandas on large dataset for document retrieval

I am trying to build a useable NLP corpus but getting bottlenecked by how long the program takes (200 hours). With so much data I know that optimizing my code even a little bit will net me huge time ...
evader110's user avatar
  • 143
1 vote
1 answer
59 views

Extending die roll simulations for complex data science tasks

I've developed a Python script that simulates die rolls and analyses the results. I'm now looking to extend and modify this code for more complex data science tasks and simulations. Is this code ...
Attila Vajda's user avatar
3 votes
3 answers
144 views

Syntactic sugar for derived variables from Pandas DataFrame columns

Update: Okay, after trying to use this for a while, I think it's probably a bad idea. Please use (lambda x: x["a"] + x["b"])(df) if really ...
user1537366's user avatar
0 votes
2 answers
117 views

Optimize a Python code which indicates duplicated values in an excel file [closed]

I wrote this code to indicate duplicated values. It actually works but I hope to know if there's another possible solution to optimize this process. Thanks. ...
peternish's user avatar
1 vote
0 answers
61 views

Combined or separate data-cleaning routine

I am a junior data engineer that have 3 years of experience with Python. I write a lot of Python code for my job and I came up with this question I can't solve by my own. I don't have the chance to ...
Izem's user avatar
  • 11
2 votes
1 answer
66 views

Use row data from a database to find rows in dataframes that match and use data to generate a separate dataframe

I have a DataFrame (database_df) that contains the general record with the IDs that are the same team in each of the lines, containing these values I need to find ...
Digital Farmer's user avatar
1 vote
2 answers
53 views

Imrove performance when updating DataFrame rows based on complex criteria

My question got rejected the last time so I am trying a better approach to getting a solution: ...
PyNoob's user avatar
  • 21
2 votes
1 answer
165 views

Is this the right implementation for Linear Programming (puLP) on python?

I have created a LP function to help maximize a set of features. My first time playing with this library and also conducting LP. Variables: Number of features => X Number of Categories => Y ...
Kale 's user avatar
  • 33
1 vote
0 answers
33 views

Pandas to combine data files & add new calculated columns to result

I currently have the following python code that adds a few calculated columns to my consol file. Essentially it combines all the sales files into one combined DF and then adds 4 new sales columns with ...
Neo's user avatar
  • 11
1 vote
1 answer
100 views

Using Pandas to group data based on name and see if column value is greater than or equal to values based on group names

As you'll see from the below code, I'm creating separate data frames of a much larger data frame, then updating a column for each one. What I'm doing is looking at the second column and checking to ...
jp207's user avatar
  • 173
1 vote
2 answers
61 views

Split Pandas dataset column based on values (suffixes: string operation)

In Python using Pandas, I am splitting a dataset column into 4 lists based on the suffix of the values. For the 3 suffixes I am using a list comprehension then for the 4th one, a set operation that ...
evilmandarine's user avatar
2 votes
1 answer
71 views

Generating Test Data with Python

Background: I'm a BI developer building a new dashboard for a client. They want to track performance for the week/month/year to date against the prior period. Unfortunately, I don't have direct access ...
DixieFlatline's user avatar
1 vote
0 answers
47 views

Recurrent Neural Network loss is NAN

I am training a neural network to use approximately 600 features (4103rd to last column of a df) to predict approximately 4000 values (7th to 4102nd column of the same df). I have standardized the ...
Manas Garg's user avatar
2 votes
1 answer
134 views

Flag tukey outliers using python pandas groupby

I'm new to python and pandas. I would like to use pandas groupby() to flag values in a df that are outliers. I think I've got it working, but as I'm new to python, ...
Quentin's user avatar
  • 123
2 votes
1 answer
103 views

Applying cointegration function from statsmodels on a large dataframe

I need to apply the coint function from the statsmodels library to 207 times series with 1397 points each, two by two. Currently, it takes between 35-40 minutes on my computer with an Intel 24 Cores ...
Begoodpy's user avatar
  • 125
1 vote
1 answer
264 views

Finding highly correlated variables in a dataframe by evaluating its correlation matrix's values

I read data from Excel into a Pandas DataFrame, so that every column represents a different variable, and every row represents a different sample. I made the function below to identify potential ...
Pimsel's user avatar
  • 25
5 votes
2 answers
376 views

groupby in pandas and plot

I have a csv file that looks like this: ...
Fang's user avatar
  • 535
1 vote
1 answer
308 views

Protecting functions from empty DataFrames

Pandas likes to throw cryptic errors when you feed its functions with empty DataFrames saying nothing that would help you to identify the root cause. In order to ...
t3chb0t's user avatar
  • 44.3k
1 vote
1 answer
1k views

Mapping pandas' Series to dataclasses

I've got something really simple this time where I'm mapping pandas' Series to dataclasses with a oneliner helper function (as ...
t3chb0t's user avatar
  • 44.3k
1 vote
0 answers
208 views

Generic[type[Enum], Protocol[DataFrame]] Dataset with mapped to enum types

Below is my solution for managing multiple DataFrames, in an abstract enough way that it may apply to objects outside of a pandas.DataFrame hence the ...
Jason Leaver's user avatar
1 vote
2 answers
144 views

Replace personal names and addresses with company ones

The problem: I am given a data frame. Somewhere in that dataframe there is 3*N number of columns that I need to modify based on a condition. The columns of interest look like this: names_1 address_1 ...
Glue's user avatar
  • 129
4 votes
0 answers
170 views

constraint solving graduation using HTML Parsing, pandas, and z3

not sure if this project fits on code review, but my code is getting extremely messy, and would love some tips to clean it up! Overview The project is designed to take in an HTML file (a degree audit),...
retep's user avatar
  • 179
1 vote
1 answer
63 views

simulated samples for central limit theorem

I am trying to help students visualize the central limit theorem and wanted to do this with simulated data. I created a population dataset with three variables: ...
Damon C. Roberts's user avatar
1 vote
1 answer
318 views

Pandas Upsampling Time Series Splitting Equally the values through the weeks starting on monday

I build my code studying this question: "Divide total sum equally to higher sampled time periods when upsampling with pandas". I am wondering if can be improved the code and if it is right. ...
Andrea Ciufo's user avatar
1 vote
1 answer
87 views

Write a Python script to generate a random DataFrame based on specific inputs

I found myself many times in the past trying to generate fake DataFrames in pandas. I decided just for fun, to write a script that I can specify some inputs and ...
Tasos's user avatar
  • 159
4 votes
2 answers
177 views

Unstructured to Structured TOC

The following code tries to convert an unstructured TOC with bounding box layout data given by the output of pdftotext -bbox-layout -f 11 -l 13 new_book.pdf toc.html...
Sati's user avatar
  • 417
3 votes
0 answers
866 views

Python BeautifulSoup - preparing HTML rows and td tags for Pandas

I'm using BeautifulSoup to parse a bunch of combined tables' rows, row by row, column by column to prepare it for import into Pandas. I can't use to_html() because ...
Meghan M.'s user avatar
0 votes
1 answer
174 views

Efficient way to read files python - 10 folders with 100k txt files in each one

i am looking for an efficient way to read and append texts of .txt files to a dataframe. I currently have 10 folders with 100k documents each. What i specifically need to do is: getting the names of ...
Piergiorgio Di Pasquale's user avatar
1 vote
1 answer
67 views

Make unique id based on text data column with similarity scoring

I have the following dataframe: ...
illuminato's user avatar
0 votes
0 answers
41 views

Find profitable bets from historic results

Each of the lines in my CSV is a possibility of investment that I register on historic, but I would only make the investment if in the existing history (previous lines) the sum of the results is above ...
Digital Farmer's user avatar
1 vote
1 answer
58 views

Create new columns in a DataFrame using functions and reposition the new columns

I would like a review regarding the method I use to create the new columns and then reposition them in the correct place where they should be. The new column called ...
Digital Farmer's user avatar
-4 votes
1 answer
32 views

Find characters from same homeworld as Chewbacca [closed]

The problem is Find the names of all characters which are from the same homeworld as Chewbacca My code is ...
su sahin's user avatar
5 votes
1 answer
164 views

Web scraper for data sources from Statistics Canada

I've written a parser to scrape data from Canadian Statistics Bureau. ...
alphamu's user avatar
  • 153
2 votes
1 answer
140 views

Efficient List comprehension with multiple conditions using shift? [closed]

I am new to python. I am trying to get the total number of failures by checking first how did the transition of the column Failure Sensor. Then creating the Start column from devicetimestamp if the ...
Noobie1997's user avatar
3 votes
1 answer
58 views

Cleaning Float Column of Longitude

I am cleaning a dataset where columns lat and long are presenting some values multiplied by 10. Not only 10, but changing 10^n. I wrote the code below. I am not sure if it is the best way, but is ...
GregOliveira's user avatar
1 vote
0 answers
44 views

BoundingBox dataclass implementation with cupy, cudf, and nvector

The dataset I'm working with is rather large so I've been experimenting with cudf and cupy. Here you can find instructions for ...
Jason Leaver's user avatar
2 votes
1 answer
294 views

python: requests large.zip -> unzip -> fix -> filter ->gunzip

I wrote a function to download a large zipfile 5-7gb from Iowa State MRMS data archive. The zip files appear to be malformed and results in a BadZipFileError hence ...
Jason Leaver's user avatar
3 votes
2 answers
191 views

Intercolumn statistics between columns in a dataframe

I have a df and need to count how many adjacent columns have the same sign as other columns based on the sign of the first column, and multiply by the sign of the ...
jaried's user avatar
  • 177
1 vote
1 answer
93 views

Create charts after querying database

I'm at the end of the IBM Data Analyst course, and I wanted to ask for a rating of a piece of code I wrote as a solution to its exercises from the final chapter. I know I could write it on the forum ...
kkkkrkrkkk's user avatar
2 votes
2 answers
397 views

Pivoting and then Padding a Pandas DataFrame with NaN between specific columns - Case Study

This question is about pivoting and padding columns, two very frequent activities in Pandas. I have a raw dataframe. I need to manipulate from long to ...
Andrea Ciufo's user avatar
1 vote
1 answer
86 views

Plot time windows based on interested value

I use the following code to identify some interested values into a dataframe and them plot a time window before and after that value appeared. It works very well, but I would like to know if there is ...
GregOliveira's user avatar
2 votes
2 answers
335 views

Convert a mapping of arbitrary pairs into a one-to-many map

The accepted solution is much cleaner and outperforms the algorithm below for small maps, it doesn't do as well with larger ones: my code takes twice as long as the accepted code for maps of 10 pairs (...
shortorian's user avatar
1 vote
1 answer
66 views

Grouping and summing only n variables of m with m>n using a column as key in pandas

I have the following df ...
Andrea Ciufo's user avatar
2 votes
0 answers
219 views

Do Mann-Kendall and Pettitt tests on each CSV file

Here is a function that will take each text file in a directory, do the Mann-Kendall and Pettitt tests, and then write the output to a text file. Would you please suggest me improve the code to make ...
Geoge Cittal's user avatar
1 vote
1 answer
399 views

Iterate and assign weights based on two columns (python)

FI_name ISN Sector Industry REC INE02 PS FS HDB INE03 PR FS ABC INE04 PR FS RHC INE05 PR CO ZHE INE06 PR FS HSE INE07 PR FS ZAK INE08 PS MT HGB INE09 PR FS YUJ INE10 PR MT WSD INE11 PS FS ...
Raj's user avatar
  • 13
1 vote
1 answer
62 views

Testing string membership using (in) keyword in python is very slow

I have the following text dataset: 4 million paragraphs of length between (10-60 words each). ...
Akbar Hussein's user avatar
2 votes
1 answer
278 views

Slow processing of a python dataframe when aggregating across rows and columns

I would do this in SQL using string_agg but the server is SQL Server 2012 and beyond my control. So I'm trying a python approach. I have a dataframe of shape [20225 rows x 7 columns], and there a bit ...
Maxcot's user avatar
  • 123

1
2 3 4 5
13