Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Pandas is a Python data analysis library.

1
vote
0answers
14 views

Finding peaks in a DataFrame

I would like to compare the min/max values of a time-series with a test time-series. Additionally, I would like to compare the time of the "peaks". However, I'm having trouble extracting these ...
2
votes
1answer
38 views

Pivot DataFrame with DateTimeIndex

I have some data, which I've faked using: ...
4
votes
0answers
49 views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
2
votes
0answers
52 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
3
votes
0answers
78 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
2
votes
1answer
62 views

Convert list of dictionaries to iterable list

Due to another module coupled to my function, I can only receive the input to my part in the form of a JSON object structured roughly like this: ...
1
vote
1answer
56 views

Implementing a faster inbound neighbour similarity in Networkx

Given a directed graph G of 20+ million edges and approximately 2 million unique nodes, can this function/process be made faster on a dataset of 40k edges? The current speed is at ~1.8 seconds per ...
2
votes
1answer
66 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
81 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
0
votes
1answer
45 views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
3
votes
0answers
140 views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
3
votes
2answers
87 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...
5
votes
1answer
86 views

Spearman correlations between Numpy array and every Pandas DataFrame row

I want to efficiently calculate Spearman correlations between a Numpy array and every Pandas DataFrame row: ...
7
votes
2answers
93 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
4
votes
1answer
43 views

Python pandas: Take column of counts and create DataFrame with a row per count

I have a DataFrame with values in columns a and b and a third column with the count of that row. I would like to convert this ...
3
votes
1answer
35 views

Pivot and annotate Pandas DataFrame

Given the CSV data: ,fan1,fan2,foil1,foil2 0,0.0,0.0,0.0,0.125 1,0.0625,0.0,0.0625,0.125 2,0.0625,0.0,0.0,0.3125 Which I want to turn into a kind of annotated ...
4
votes
2answers
144 views

Iterate over a list of list names as file names

Interrogating a web API using query-url's, for each query I can get either zero hits, one hit, or multiple hits. Each of those categories needs to go into a separate CSV file for manual review or ...
0
votes
0answers
41 views

Slow cursor.fetchall

I am trying to get +10M records from a DB: ...
1
vote
1answer
57 views

Importing text into PANDAS and counting certain words

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
5
votes
1answer
37 views

Non normalized set difference algorithm

I'm trying to set up an algorithm in python for getting all sets from a set (DataSet1) less any instances of data in a second set (DataSet2). Objective: ...
4
votes
1answer
105 views

Pandas/Tkinter GUI Excel report generator

I built a GUI tool that takes excel files and outputs a finished report to help automate a report at work. It was a fantastic learning experienced and I feel much more comfortable with pandas and ...
0
votes
0answers
19 views

Processing reservations file for analytical purposes using pandas

I am trying to process reservations file for analytical purposes. I have used pandas for the same. The reservations.csv file is available on GitHub. ...
1
vote
0answers
27 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
3
votes
1answer
128 views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
1
vote
1answer
53 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
0
votes
0answers
19 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
2
votes
0answers
39 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
3
votes
0answers
39 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
1answer
29 views

Getting the average score for hotel scores in different countries using pandas

I am diving into Data Analysis with pandas, and I have just written this Python script to calculate the average of hotel review scores of each country. The dataset contains an individual average score ...
2
votes
1answer
382 views

Loading .txt file content and filename into pandas dataframe

Is there a better way to read in the name of a textfile and the content of a text file into a dataframe? (Or is my implementation even okay?) Can I avoid storing the data in lists? ...
2
votes
1answer
211 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
4
votes
1answer
46 views

Class made to do some basic RGB analysis

I created this class to load in an image and get some specific values I needed. ...
7
votes
0answers
122 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
17
votes
1answer
303 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
0
votes
1answer
1k views

Multiply two columns of Census data and groupby

I have census data that looks like this ...
3
votes
1answer
52 views

Code runs Speed Test CLI using the python wrapper and then stores values within CSV

I wanted to make use of the speedtest-cli while also incorporating pandas into the project. My initial goal, which I am posting here, was to take the values of the speed test and incorporate them into ...
1
vote
0answers
46 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
2
votes
0answers
30 views

Determine file provider based on column headers, replace columns and process data

Each day I receive files from different customers relaying the same information, but the customers use proprietary formats. For example, ...
3
votes
1answer
270 views

Compute conditional median of PANDAS dataframe

I am new to Python/Pandas. Consider the following code: ...
0
votes
1answer
34 views

Filter if any value in group is null

The following code works, but the filter operation is bottlenecking my analysis. Do you have an idea how I can make the filter operation faster? Do you have other comments about the codestyle? ...
0
votes
0answers
67 views

Normalize JSON for Single Level Children

The goal here is to normalize or flatten JSON: [{ "parent_key": { "child_key_1": "child_value_1", "child_key_2": "child_value_2" } }] To ...
2
votes
0answers
37 views

Average coverage between each sphere of each spheres groups

Let me start with a litle bit of context. I have Spheres, a large pandas DataFrame with positions and radius of multiple spheres during time. (in real life 10 000 ...
3
votes
1answer
46 views

One minute cron Python script reads from MySQL and updates Redis based on calculations

The following code works and will be put into cron on a AWS server (not sure what specifications, cores, kernels etc) at a every 1 minute frequency. However, it's my first time putting a script into ...
2
votes
1answer
52 views

Pythonic way of managing data for plotting 2D variance plots

I have collected large dataset of trajectories in 3D space. Below is the information about dataset- Each trajectory (shape: n x 4) is saved into CSV file with the following header: ...
3
votes
1answer
213 views

Processing large time-series data

I am currently processing some large time-series data with pandas, and I have a function that is intolerably slow, and I'm sure it could be done faster. The problem is: I'm studying a factory that ...
4
votes
1answer
68 views

Munging select rows in CSV files using Pandas

I'm learning Pandas. I have a project where I need to munge some CSV files and resave them as CSV. I can use dictionaries and CSV module, but decided to use DataFrames to get more exposure and ...
3
votes
1answer
235 views

Subclassing pandas DataFrame for an ETL

I'm writing a small ETL, which loads data from CSVs, cleans each table a bit, and then loads everything into a PostgreSQL database. I was planning to use pandas for its built-in capabilities, but am ...
3
votes
1answer
154 views

Permutations with a sum constraint

I have made the following function that gives the combinations permuted of n elements with a constrain/rule. This consist on creating a dataframe with all combination of numbers from 0 to n that sums ...
2
votes
1answer
33 views

Count number of registers in interval & location

Recently I asked how one could count the number of registers by the interval as answered in https://stackoverflow.com/questions/49240140/count-number-of-registers-in-interval. The solution works ...
1
vote
1answer
80 views

Pandas dataframe custom forward fillna optimisation

I have a function that synchronizes according to some specifications columns of a dataframe. The functions works, nevertheless I was wondering how to: Improve ...