Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Pandas is a Python data analysis library.

0
votes
0answers
18 views

Find and replace set of words over multiple files [closed]

I have 10 different csv files wherein Market_Name column is common. There are four names which i want to replace with other names for all 10 files. I can do this with map function over single ...
1
vote
1answer
23 views

Display the difference between DataFrames' dtypes?

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having ...
0
votes
2answers
32 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...
2
votes
1answer
37 views

Check values in datatables, with validation rules that vary according to the data source

Background I receive dataframes from several sources (below I call these sources s1, s2 and ...
2
votes
1answer
58 views

Function for setting a streak of strings in a Pandas

I have a df with artist names in one column, I want to find how many times the artists repeats through the column. Example: ...
1
vote
0answers
27 views

Calculating how much $ goes into each defaulting bucket

I have the following code which I use to determine how much of each repayment goes into each late payments bucket (i.e. 0-7 days past due, 7-15 days past due... etc) For example, the repayment amount ...
2
votes
0answers
30 views

Function that plots data frames, supporting several variants

I defined a function that generates tables and creates countplots and barplots based on the arguments that it receives. But the snippets of code inside the function are repeated several times, which ...
4
votes
1answer
60 views

Python program computing some statistics on Scottish geographic areas

This simple script computes some basic descriptive statistics, like mean, standard deviation, kurtosis, etc. on data column imported from a CSV file with use of ...
0
votes
0answers
20 views

Reading an Excel sheet in PANDAS and appending to cells

Please check my code below for any tweaks that can make the output cleaner or make the function run faster. ...
0
votes
1answer
170 views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
3
votes
1answer
46 views

turn a dict that may contain a pandas dataframe to several dicts

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows). I want to create a new dict for each row in the DataFrame's, ...
4
votes
1answer
36 views

Subtable timestamps in python pandas

I have a dataframe like: ...
2
votes
0answers
110 views

Finding peaks in a DataFrame

I would like to compare the min/max values of a time-series with a test time-series. Additionally, I would like to compare the time of the "peaks". However, I'm having trouble extracting these ...
2
votes
1answer
40 views

Pivot DataFrame with DateTimeIndex

I have some data, which I've faked using: ...
4
votes
0answers
136 views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
2
votes
0answers
69 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
3
votes
0answers
110 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
2
votes
1answer
89 views

Convert list of dictionaries to iterable list

Due to another module coupled to my function, I can only receive the input to my part in the form of a JSON object structured roughly like this: ...
1
vote
1answer
79 views

Implementing a faster inbound neighbour similarity in Networkx

Given a directed graph G of 20+ million edges and approximately 2 million unique nodes, can this function/process be made faster on a dataset of 40k edges? The current speed is at ~1.8 seconds per ...
2
votes
1answer
139 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
91 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
0
votes
1answer
271 views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
3
votes
0answers
338 views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
3
votes
2answers
90 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...
5
votes
1answer
197 views

Spearman correlations between Numpy array and every Pandas DataFrame row

I want to efficiently calculate Spearman correlations between a Numpy array and every Pandas DataFrame row: ...
7
votes
2answers
94 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
4
votes
1answer
50 views

Python pandas: Take column of counts and create DataFrame with a row per count

I have a DataFrame with values in columns a and b and a third column with the count of that row. I would like to convert this ...
3
votes
1answer
41 views

Pivot and annotate Pandas DataFrame

Given the CSV data: ,fan1,fan2,foil1,foil2 0,0.0,0.0,0.0,0.125 1,0.0625,0.0,0.0625,0.125 2,0.0625,0.0,0.0,0.3125 Which I want to turn into a kind of annotated ...
4
votes
2answers
148 views

Iterate over a list of list names as file names

Interrogating a web API using query-url's, for each query I can get either zero hits, one hit, or multiple hits. Each of those categories needs to go into a separate CSV file for manual review or ...
0
votes
0answers
89 views

Slow cursor.fetchall

I am trying to get +10M records from a DB: ...
1
vote
1answer
66 views

Importing text into PANDAS and counting certain words

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
5
votes
1answer
40 views

Non normalized set difference algorithm

I'm trying to set up an algorithm in python for getting all sets from a set (DataSet1) less any instances of data in a second set (DataSet2). Objective: ...
4
votes
1answer
257 views

Pandas/Tkinter GUI Excel report generator

I built a GUI tool that takes excel files and outputs a finished report to help automate a report at work. It was a fantastic learning experienced and I feel much more comfortable with pandas and ...
0
votes
0answers
19 views

Processing reservations file for analytical purposes using pandas

I am trying to process reservations file for analytical purposes. I have used pandas for the same. The reservations.csv file is available on GitHub. ...
1
vote
0answers
29 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
3
votes
1answer
268 views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
1
vote
1answer
83 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
0
votes
0answers
20 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
2
votes
0answers
39 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
3
votes
0answers
76 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
1answer
31 views

Getting the average score for hotel scores in different countries using pandas

I am diving into Data Analysis with pandas, and I have just written this Python script to calculate the average of hotel review scores of each country. The dataset contains an individual average score ...
2
votes
1answer
718 views

Loading .txt file content and filename into pandas dataframe

Is there a better way to read in the name of a textfile and the content of a text file into a dataframe? (Or is my implementation even okay?) Can I avoid storing the data in lists? ...
2
votes
1answer
299 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
4
votes
1answer
54 views

Class made to do some basic RGB analysis

I created this class to load in an image and get some specific values I needed. ...
7
votes
0answers
155 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
17
votes
1answer
319 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
0
votes
1answer
2k views

Multiply two columns of Census data and groupby

I have census data that looks like this ...
3
votes
1answer
66 views

Code runs Speed Test CLI using the python wrapper and then stores values within CSV

I wanted to make use of the speedtest-cli while also incorporating pandas into the project. My initial goal, which I am posting here, was to take the values of the speed test and incorporate them into ...
1
vote
0answers
49 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
2
votes
0answers
30 views

Determine file provider based on column headers, replace columns and process data

Each day I receive files from different customers relaying the same information, but the customers use proprietary formats. For example, ...