Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
0answers
21 views

Compute damages and savings for motor insurance claims

I am working on some claims data, where I have features such as The descriptions of the columns are: (A) the year of claim, (B) who's the insuring company (C) whether the vehicle is driven, (D) ...
1
vote
0answers
50 views

How to Better Parse CSV file with Pandas

I am working on a CSV data Sheet and want to parse and filter the data out it, While working on a code I found a similar code someone has asked on SO POST there and ...
2
votes
1answer
47 views

Find the most similar pair of rows from a table

I have a list of n cities and hourly weather prediction for each one of them for h hours. I'm writing a script to find a pair of ...
0
votes
0answers
26 views

How to most efficiently make repeated calls to API with Pandas/Python

Looking for some help on how to speed up the code below as I'm still relatively new to using Pandas. The playoff_stats function makes thousands of calls to the PlayerCareerStats API - and while it ...
0
votes
1answer
143 views

create weekly data for stocks faster based on daily data

daily data: ...
4
votes
2answers
65 views

For each of the visitors, find if they were present within a time period: code is running slow

I'm working with a pandas dataframe which describes the start and end time of visits. A toy example of the data could be as follows: ...
3
votes
0answers
38 views

Extracting Korean text for sentiment analysis

It is quite long so I'm not sure I can ask you to review my code. But I have no choice indeed. What I'm doing is analyzing the Korean language. To do this, first I collect data and loaded the data via ...
0
votes
0answers
41 views

How do I make my html object-to-table array script run faster? (Pandas)

I have a data frame that has several columns, one of which contains html objects (containing tables). I want a column of table arrays. My problem is that this piece of code takes a long time to run. ...
0
votes
1answer
24 views

Get a “train or test” slicing elegantly after splitting

I am learning numpy, pandas, and sklearn for machine learning now. My teacher gives me code like below, but I guess it is ugly. This code is for splitting the data into training set and testing set, ...
1
vote
1answer
40 views

Check if any of the coordinates are within a certain distance using apply in Python pandas with two data frames

I wrote this code that takes the vessel coordinates and checks them against every possible port coordinates. If the distance between the vessel and the port coordinates is within x meters (in this ...
0
votes
0answers
26 views

Plotting 2 types of data from DataFrame with different colors

This is the task: ...
1
vote
1answer
42 views

Pandas merge a “%Y%M%D” date column with a “%H:%M:%S” time column

I have a dataframe with a column composed by date object and a column composed by time ...
0
votes
0answers
29 views

Keras: Using Autoencoders for prediction

I have a dataset which I divided into two sections horizontally. Column A of first section is the input variable and Column A of second section is the target variable. I am trying to build a Denoising ...
0
votes
1answer
440 views

William Fractal technical indicator implementation

I'm really new to both Python, data analysis and Panda and for gathering a bit of familiarity with this components and trying to replicate a trading strategy I'm creating a function that give me the ...
5
votes
1answer
107 views

Reading and analyzing budget and transaction data from user input and .csv

I'm looking for ideas on how to optimize the following code. I don't really know how to condense anything, I'm still very new to coding and this is the first time I've really created anything besides &...
3
votes
1answer
53 views

Filtering a DataFrame based on two logical conditions, first one numpy array values, second one current day based

The following is just a working example. I have a DataFrame containings a monotonous growing function. Some of the values are actuals, some are forecasted. I need to filter specific actuals values ...
3
votes
1answer
72 views

Pandas replace rare values in a pipeline

A common preprocessing in machine learning consists in replacing rare values in the data by a label stating "rare". So that subsequent learning algorithms will not try to generalize a value ...
2
votes
2answers
53 views

Pandas Filtering based on the length of the same kind variables in a column

In my real case I have a set of time series related to different IDs stored in a single DataFrame Some are composed by 400 samples, some by 1000 samples, some by ...
3
votes
2answers
434 views

Parsing a text file into a pandas DataFrame

I have a .txt file that looks like this: ...
0
votes
1answer
25 views

Function to perform checks on data

I'm currently trying to do data quality checks in python primarily using Pandas. The code below is fit for purpose. The idea is to do the checks in Python and ...
3
votes
0answers
36 views

Modified Gantt Chart

this is my very first pandas/matplotlib code and I would be grateful if you can review my code and share your thoughts or ideas for improvement. Please focus on code, not the features of resulting ...
3
votes
1answer
55 views

Manipulating Pandas Dataframe with vaccination data from CSV to display on matplotlib

I have some code that manipulates a Pandas Dataframe containing Covid-19 vaccine data and displays it on Matplotlib. The data is here: https://covid.ourworldindata.org/data/owid-covid-data.csv (...
2
votes
1answer
47 views

Martix of Counts of Regex Hits Over a List of Strings

I have: A dataframe with Identifiers (TermId) and Search Terms (SearchTerm). A list of text strings (...
1
vote
1answer
64 views

T-test Independence for a Pandas DataFrame

I want to calculate the scipy.stats.ttest_ind() for numeric columns in a pandas DataFrame with the binary target variable. ...
0
votes
0answers
33 views

Optimize Yahoo Finance Code for Analysis

I am trying to analyze a number of companies using financial data I gathered from Yahoo Finance. I am also using the yfinance API to get some more details about the company using functions. Since I am ...
1
vote
0answers
37 views

Asking for feedback, when utilizing Pandas, why is it better to use pandas.Series than lists data type?

I found out the hard way, when utilizing pandas is better to use pandas.Series than lists data type to manipulate data. Even if ...
2
votes
2answers
95 views

Downloading COVID data and uploading graphs

So this is a project I have been working on for the last few weeks. Just started learning Python. Started out with bash scripting and got the itch to learn more. Anyway code fetches covid-19 data from ...
1
vote
0answers
51 views

Python speed up dataframe writes

I'm aware that iterating frames in pandas is a very slow operation. I've heard vectorization speeds things up. I found a project that I'm using for sentiment analysis on reddit. The Calculate ...
1
vote
1answer
67 views

I have code that loops through a column of a dataframe. How can I make this code faster? (Python/Pandas)

I have datasetA with 90,000 rows and datasetB with 5,000 rows. Each dataset has a column called "ID" with employee IDs. My goal is to to create another column in datasetA that identifies ...
2
votes
0answers
68 views

Python REST API Data Extraction with requests library

Is there a better way to write this piece of code to extract data from a REST API with the requests library? It currently takes about 10 minutes to finish extracting the data from the REST API. I ...
1
vote
1answer
44 views

Processing CSV files with filtering

I have 4 CSV files that i need to process (filter out Part segment and then merge them) but the problem is that they do not fit in my memory. So I decided to: [open - filter - write out] each one of ...
1
vote
1answer
42 views

Pandas transform series values until condition is met without for loop

I have a pandas Series contain 0s and 1s. Now I want to convert all the 0s to 1s which come before the first 1, and rest of the 1s to 0s. I can achieve that using the below code: ...
1
vote
1answer
57 views

How to optimize this majority vote method

I have the following code to do a majority vote for data in a dataframe: ...
0
votes
0answers
63 views

Merge rows in a pandas DataFrame while ignoring specified values and checking for conflicts

The problem I have a set of dataframes where each row should have a unique ID value, but sometimes imported data has multiple rows with the same ID. I want to do a ...
1
vote
0answers
40 views

Resource reservation system

I'm using a function to determine if resources can be used again or not. This is the numpy array I'm using. ...
0
votes
0answers
61 views

Speed-up anomaly detection for pandas dataframes with vectorized operations

Motivation: I very often face a task where I need to check that dataframe's values are correct or not. Based on it I can create another dataframe with True/False values which are based on the first ...
4
votes
1answer
271 views

Filter dataframe based on unique values in multiple columns and drop rows based on index

I'm wondering if there is a more efficient way of filtering a dataframe down based on certain unique values in several columns. Once I have it filtered down, I then want to extract keep one the ...
2
votes
1answer
58 views

Covid-19 World Case Report Beta 0.1

I am a self-learned programmer and this is my first program. I would really appreciate any critique on my approach. The program should automatically download the who data file to the folder these ...
1
vote
1answer
67 views

Pandas DataFrame.sort_values() new method to exclude by row name

I was working on new function with Pandas DataFrame. It's meant to sort values by column name but exclude specific rows from sorting and return sorted dataframe with that has rows with preserved ...
1
vote
1answer
100 views

How to get the indexes of smallest value per row in a DF, as column index and not column name? [closed]

I'd like, from a DataFrame, to retrieve a Series which contains the index of the minimal value per row (so ...
1
vote
1answer
32 views

Fill a pandas dataframe (of predetermined size) with results from a similar dataframe (of size less than or equivalent to the original)

I'd like some feedback/suggestions on how to improve the following. Specifically I want to know if what I'm doing is reliable and fast, or if there is a better way to accomplish this. The problem: I ...
3
votes
0answers
121 views

Implement vectorization instead of nested loops on dataframe

I have a dataset ('sample_data.csv') of the form below: ...
4
votes
1answer
84 views

Reddit bot in Python

I have created a Reddit bot that goes through an "x" amount of posts. For each post it pulls all the comments from that post into a list then a data frame and then it loops over a CSV look ...
-4
votes
1answer
73 views

How can I reduce the TIme Complexity. of the python program

I have 7000 CSV files that I want to do operation on. but the problem is that it takes to much time like it takes 13 seconds to process these files. What I need to know is which operation take more ...
5
votes
1answer
81 views

Getting death row inmates' last statement

Initial version of the code appeared as an answer to an SO question. I refactored the code a bit and it works pretty well, IMHO. I get a solid .csv file with all ...
4
votes
2answers
394 views

Create dummy variables in dataframe using for loop and apply lambda

I like to compute dummy variables based on the existing values in a dataframe and a dictionary with certain "cut off" points. I wonder whether I need to get rid of the for loop in the ...
1
vote
1answer
29 views

Cleaning Netflix-style recommendation engine datasets

I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
4
votes
1answer
66 views

Python : adding a columns with count of missing values by row

I have a big python data-frame and I am trying to add a column to it with average number of missing values by row. I have inherited some code that is working but I'd like to reduce memory usage by ...
4
votes
1answer
51 views

Create DataFrame from dictionary

I request data from a table in a database and each line comes as a dictionary like this one : ...
3
votes
1answer
50 views

ways to code for better speed of processing date comparisions / slicing in python

I wrote the below code to convert a long data frame containing start and end date event spans into a daily df with rows for each day: output: ...

1
2 3 4 5
11