Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1
vote
0answers
48 views

Python speed up dataframe writes

I'm aware that iterating frames in pandas is a very slow operation. I've heard vectorization speeds things up. I found a project that I'm using for sentiment analysis on reddit. The Calculate ...
1
vote
1answer
61 views

I have code that loops through a column of a dataframe. How can I make this code faster? (Python/Pandas)

I have datasetA with 90,000 rows and datasetB with 5,000 rows. Each dataset has a column called "ID" with employee IDs. My goal is to to create another column in datasetA that identifies ...
2
votes
0answers
57 views

Python REST API Data Extraction with requests library

Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
1
vote
1answer
39 views

Processing CSV files with filtering

I have 4 CSV files that i need to process (filter out Part segment and then merge them) but the problem is that they do not fit in my memory. So I decided to: [open - filter - write out] each one of ...
1
vote
1answer
33 views

Pandas transform series values until condition is met without for loop

I have a pandas Series contain 0s and 1s. Now I want to convert all the 0s to 1s which come before the first 1, and rest of the 1s to 0s. I can achieve that using the below code: ...
1
vote
1answer
51 views

How to optimize this majority vote method

I have the following code to do a majority vote for data in a dataframe: ...
0
votes
0answers
30 views

Merge rows in a pandas DataFrame while ignoring specified values and checking for conflicts

The problem I have a set of dataframes where each row should have a unique ID value, but sometimes imported data has multiple rows with the same ID. I want to do a ...
1
vote
0answers
39 views

Resource reservation system

I'm using a function to determine if resources can be used again or not. This is the numpy array I'm using. ...
0
votes
0answers
50 views

Speed-up anomaly detection for pandas dataframes with vectorized operations

Motivation: I very often face a task where I need to check that dataframe's values are correct or not. Based on it I can create another dataframe with True/False values which are based on the first ...
5
votes
1answer
62 views

Filter dataframe based on unique values in multiple columns and drop rows based on index

I'm wondering if there is a more efficient way of filtering a dataframe down based on certain unique values in several columns. Once I have it filtered down, I then want to extract keep one the ...
2
votes
0answers
46 views

Covid-19 World Case Report Beta 0.1

I am a self-learned programmer and this is my first program. I would really appreciate any critique on my approach. The program should automatically download the who data file to the folder these ...
1
vote
1answer
46 views

Pandas DataFrame.sort_values() new method to exclude by row name

I was working on new function with Pandas DataFrame. It's meant to sort values by column name but exclude specific rows from sorting and return sorted dataframe with that has rows with preserved ...
1
vote
1answer
63 views

How to get the indexes of smallest value per row in a DF, as column index and not column name? [closed]

I'd like, from a DataFrame, to retrieve a Series which contains the index of the minimal value per row (so ...
1
vote
1answer
26 views

Fill a pandas dataframe (of predetermined size) with results from a similar dataframe (of size less than or equivalent to the original)

I'd like some feedback/suggestions on how to improve the following. Specifically I want to know if what I'm doing is reliable and fast, or if there is a better way to accomplish this. The problem: I ...
3
votes
0answers
116 views

Implement vectorization instead of nested loops on dataframe

I have a dataset ('sample_data.csv') of the form below: ...
4
votes
1answer
76 views

Reddit bot in Python

I have created a Reddit bot that goes through an "x" amount of posts. For each post it pulls all the comments from that post into a list then a data frame and then it loops over a CSV look ...
-4
votes
1answer
52 views

How can I reduce the TIme Complexity. of the python program

I have 7000 CSV files that I want to do operation on. but the problem is that it takes to much time like it takes 13 seconds to process these files. What I need to know is which operation take more ...
5
votes
1answer
77 views

Getting death row inmates' last statement

Initial version of the code appeared as an answer to an SO question. I refactored the code a bit and it works pretty well, IMHO. I get a solid .csv file with all ...
4
votes
2answers
170 views

Create dummy variables in dataframe using for loop and apply lambda

I like to compute dummy variables based on the existing values in a dataframe and a dictionary with certain "cut off" points. I wonder whether I need to get rid of the for loop in the ...
1
vote
1answer
23 views

Cleaning Netflix-style recommendation engine datasets

I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
4
votes
1answer
57 views

Python : adding a columns with count of missing values by row

I have a big python data-frame and I am trying to add a column to it with average number of missing values by row. I have inherited some code that is working but I'd like to reduce memory usage by ...
4
votes
1answer
48 views

Create DataFrame from dictionary

I request data from a table in a database and each line comes as a dictionary like this one : ...
3
votes
1answer
50 views

ways to code for better speed of processing date comparisions / slicing in python

I wrote the below code to convert a long data frame containing start and end date event spans into a daily df with rows for each day: output: ...
4
votes
1answer
48 views

Best practice for cleaning Pandas dataframe columns

I am cleaning str columns in a Pandas dataframe (see below for an example), and I have been wondering if there are more concise ways or additional inplace methods to do so. What are the general best ...
5
votes
1answer
43 views

Scraper for basketball rosters

I am very new to Python and I am trying to learn through personnal projects and today I needed to collect a lot of baksetball player names and decided this was a good time to learn and practice. The ...
2
votes
1answer
94 views

Stock data REST API extractor with GCP storage options

I'm currently working on an AI-driven trading system, the code below aims to extract ticker data from polygon REST API, this is a paid service so, in order to test the code you will need to subscribe /...
8
votes
3answers
342 views

Priority based categorization using pandas/python

I have invoice and code data in the below Dataframes Invoices ...
2
votes
1answer
31 views

Filtering of maximum value by ID and category sum

I have a task that is quite simple: I need to get the sum of values in the categories of each ID, and keep the category with the highest sum: ...
3
votes
0answers
755 views

Finding the most frequent words in Pandas dataframe

I am new in python coding. I think the code could be written in a better and more compact form. It compiles quite slowly due to the method of removing stop-words. I wanted to find the top 10 most ...
1
vote
1answer
36 views

Dropping a row in a dataframe

I have made a function to drop a row in a pandas.DataFrame. Since, in general, the input dataframe of a function is just a reference to the dataframe. So any change made in that function to the ...
5
votes
2answers
63 views

Get exactly `n` unique randomly sampled rows per category in a Dataframe

I want to get exactly n unique randomly sampled rows per category in a Dataframe. This proved to be involve more steps than the description would lead you to ...
2
votes
0answers
29 views

Populating 2 columns in a dataframe with frequencies from a dictionary of dataframes

I have a pandas dataframe, my_calculation, that is a Cartesian product of 4 different categories (Name, Freqset, Formula, and Location) and two additional ...
4
votes
1answer
82 views

Multiple Pandas Ranking Operations within a Loop - Better Optimization and Performance

I have implemented the following code which works as intended. However, I would like to improve my code in terms of performance and efficiency Code in Question ...
4
votes
1answer
276 views

Converting location names into country names in pandas dataframe

I have a dataframe df such that: print(df['user_location'].value_counts()) ...
1
vote
0answers
64 views

Computing on two colums

I have this snippet, which computes on two columns, and if I get some increasing pattern then I will consider it a valid Key. How can I optimize this in pandas? Sample Code Here there is only one ...
1
vote
1answer
35 views

Pandas : Apply Merge operations from a column

I have a data frame, which contains three columns: ...
3
votes
2answers
45 views

Iterating through, and editing a dataframe, using outputs from a collision detector's neighbor-list

Say the beginning dataframe is this: ...
3
votes
0answers
71 views

Reorder the Columns in a CSV File in Descending Order

I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like ...
5
votes
1answer
158 views

Concatenate several CSV files in a single dataframe

I have currently 600 CSV files (and this number will grow) of 50K lines each i would like to put in one single dataframe. I did this, it works well and it takes 3 minutes : ...
2
votes
0answers
53 views

How can I shorten the runtime of my simulation?

Before you read the code below, note the following explanation: I have three classes: Driver, Vehicle, and ...
2
votes
1answer
87 views

Creating a weekly sheet for employees

I am looking for any feedback for this first script I have written. It is a mix of openpyxl and pandas, where it takes employee times from different Excel workbooks, then creates a weekly sheet for ...
1
vote
0answers
50 views

A program that performs logistic regression based on Iris dataset

I am a newcomer in Machine Learning and I have wrote a simple program for logistic regression based on Iris dataset. I would like for experts to tell me about its drawbacks and bugs, and if it can be ...
6
votes
0answers
72 views

Code optimisation: Converting dataframe to numpy's ndarray

I am working with a dataframe of over 21M rows. ...
5
votes
1answer
61 views

Parse bed file with Pandas

For the sections between count=1s and the start and end; combine overlapping positions and output the median of the counts. Input ...
2
votes
1answer
61 views

Load multiple worksheets from Excel file to multiple DataFrames

The spreadsheet can be found just above the references section at this link. https://www.sciencedirect.com/science/article/pii/S1872497316301429 It's a publication for STR DNA locus frequencies to ...
6
votes
2answers
829 views

Python - use a list of names to find exact match in pandas column containing emails

I have an Excel file containing a free-form text column (that sometimes is structured like an email), where I need to find all first and last names and add an extra columns saying TRUE/FALSE to these ...
1
vote
0answers
36 views

Interpolation across different dataframes with Pandas

Context of the Problem I am running a discrete event simulation where at the end of each event I store the state of the system in a row of a dataframe. Each simulation run has a clock which determines ...
6
votes
0answers
44 views

Subtracting elements of datasets of an HDF5 file

I am trying to solve the following problem: Input: Input is two arrays (Nx4, sorted in column-2) stored in datasets-1 and 2 in HDF5 file (input.h5). N is huge (...
4
votes
1answer
75 views

Determine table structure in pdf using whitespace between coordinates

I'm looking to see if there are any better / faster ways of identifying table structures on a page without gridlines. The text is extracted from the file and the coordinates of each block of text are ...
2
votes
1answer
34 views

Merge multiple pandas dataframe along with np.where to process values

I have a pandas dataframe, df1: id level val 1 l1 0.2 1 l2 0.1 and another dataframe which contains type for <...

1
2 3 4 5
10