Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
4
votes
0answers
1k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
2
votes
0answers
93 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
3
votes
0answers
951 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
2
votes
1answer
233 views

Convert list of dictionaries to iterable list

Due to another module coupled to my function, I can only receive the input to my part in the form of a JSON object structured roughly like this: ...
2
votes
1answer
191 views

Implementing a faster inbound neighbour similarity in Networkx

Given a directed graph G of 20+ million edges and approximately 2 million unique nodes, can this function/process be made faster on a dataset of 40k edges? The current speed is at ~1.8 seconds per ...
3
votes
1answer
623 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
160 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
0
votes
1answer
4k views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
4
votes
0answers
1k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
3
votes
2answers
386 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...
5
votes
1answer
798 views

Spearman correlations between Numpy array and every Pandas DataFrame row

I want to efficiently calculate Spearman correlations between a Numpy array and every Pandas DataFrame row: ...
7
votes
2answers
104 views

Counting SKUs that have not appeared in previous orders

I need help with optimizing my Python code. I have a bunch of orders and each order has several SKUs. This is what my data looks like: ...
4
votes
1answer
67 views

Python pandas: Take column of counts and create DataFrame with a row per count

I have a DataFrame with values in columns a and b and a third column with the count of that row. I would like to convert this ...
3
votes
1answer
52 views

Pivot and annotate Pandas DataFrame

Given the CSV data: ,fan1,fan2,foil1,foil2 0,0.0,0.0,0.0,0.125 1,0.0625,0.0,0.0625,0.125 2,0.0625,0.0,0.0,0.3125 Which I want to turn into a kind of annotated ...
4
votes
2answers
165 views

Iterate over a list of list names as file names

Interrogating a web API using query-url's, for each query I can get either zero hits, one hit, or multiple hits. Each of those categories needs to go into a separate CSV file for manual review or ...
1
vote
1answer
96 views

Importing text into PANDAS and counting certain words [closed]

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
5
votes
1answer
45 views

Non normalized set difference algorithm

I'm trying to set up an algorithm in python for getting all sets from a set (DataSet1) less any instances of data in a second set (DataSet2). Objective: ...
4
votes
1answer
2k views

Pandas/Tkinter GUI Excel report generator

I built a GUI tool that takes excel files and outputs a finished report to help automate a report at work. It was a fantastic learning experienced and I feel much more comfortable with pandas and ...
1
vote
0answers
35 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
3
votes
1answer
1k views

Matching rows between two dataframes

I have managed to write this piece of code but as I am using it on big data sets it ends up being quite slow. I am pretty sure it would be possible to optimize it but I am very knew to coding and I ...
1
vote
1answer
443 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
0
votes
1answer
44 views

Manipulating DataFrames on pandas

I recently failed to finish a code for a job interview. One of the problems is that I decided to use Pandas (it made sense) but I was unfamiliar with it (however I know Python Scipy and Numpy), so it ...
2
votes
0answers
61 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
3
votes
0answers
165 views

Pandas data extraction task taking too much memory. How to optimize for memory usage?

I need to process some data (one of its columns contains a json/dict with params- I need to extract those params to individual columns of their own; catch- some rows have some parameters, others have ...
3
votes
1answer
42 views

Getting the average score for hotel scores in different countries using pandas

I am diving into Data Analysis with pandas, and I have just written this Python script to calculate the average of hotel review scores of each country. The dataset contains an individual average score ...
2
votes
1answer
2k views

Loading .txt file content and filename into pandas dataframe

Is there a better way to read in the name of a textfile and the content of a text file into a dataframe? (Or is my implementation even okay?) Can I avoid storing the data in lists? ...
2
votes
1answer
399 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
4
votes
1answer
71 views

Class made to do some basic RGB analysis

I created this class to load in an image and get some specific values I needed. ...
7
votes
1answer
279 views

Analysis of airport utilization using PANDAS

I am trying to find a vectorized (or more efficient) solution to an iteration problem, where the only solution I found requires row by row iteration of a DataFrame with multiple loops. The actual data ...
16
votes
1answer
379 views

Warhammer: How many of my attacks will succeed?

Me and a couple of mates sometimes play a game called Warhammer. When playing the game you have options of what each model attacks. This can lead to situations where you know if you shoot with 100% of ...
1
vote
1answer
4k views

Multiply two columns of Census data and groupby

I have census data that looks like this ...
3
votes
1answer
147 views

Code runs Speed Test CLI using the python wrapper and then stores values within CSV

I wanted to make use of the speedtest-cli while also incorporating pandas into the project. My initial goal, which I am posting here, was to take the values of the speed test and incorporate them into ...
1
vote
0answers
58 views

Aggregate conditional logic test results iterated over changing row selections

This for-loop is the biggest bottleneck in my project, and I'm curious whether there is an opportunity to speed it up by improving the grammar or by using a ...
2
votes
0answers
33 views

Determine file provider based on column headers, replace columns and process data

Each day I receive files from different customers relaying the same information, but the customers use proprietary formats. For example, ...
3
votes
1answer
1k views

Compute conditional median of PANDAS dataframe

I am new to Python/Pandas. Consider the following code: ...
0
votes
1answer
40 views

Filter if any value in group is null

The following code works, but the filter operation is bottlenecking my analysis. Do you have an idea how I can make the filter operation faster? Do you have other comments about the codestyle? ...
2
votes
0answers
41 views

Average coverage between each sphere of each spheres groups

Let me start with a litle bit of context. I have Spheres, a large pandas DataFrame with positions and radius of multiple spheres during time. (in real life 10 000 ...
3
votes
1answer
137 views

One minute cron Python script reads from MySQL and updates Redis based on calculations

The following code works and will be put into cron on a AWS server (not sure what specifications, cores, kernels etc) at a every 1 minute frequency. However, it's my first time putting a script into ...
2
votes
1answer
62 views

Pythonic way of managing data for plotting 2D variance plots

I have collected large dataset of trajectories in 3D space. Below is the information about dataset- Each trajectory (shape: n x 4) is saved into CSV file with the following header: ...
2
votes
1answer
724 views

Processing large time-series data

I am currently processing some large time-series data with pandas, and I have a function that is intolerably slow, and I'm sure it could be done faster. The problem is: I'm studying a factory that ...
4
votes
1answer
129 views

Munging select rows in CSV files using Pandas

I'm learning Pandas. I have a project where I need to munge some CSV files and resave them as CSV. I can use dictionaries and CSV module, but decided to use DataFrames to get more exposure and ...
3
votes
1answer
565 views

Subclassing pandas DataFrame for an ETL

I'm writing a small ETL, which loads data from CSVs, cleans each table a bit, and then loads everything into a PostgreSQL database. I was planning to use pandas for its built-in capabilities, but am ...
4
votes
1answer
1k views

Permutations with a sum constraint

I have made the following function that gives the combinations permuted of n elements with a constrain/rule. This consist on creating a dataframe with all combination of numbers from 0 to n that sums ...
2
votes
1answer
34 views

Count number of registers in interval & location

Recently I asked how one could count the number of registers by the interval as answered in https://stackoverflow.com/questions/49240140/count-number-of-registers-in-interval. The solution works ...
1
vote
1answer
231 views

Pandas dataframe custom forward fillna optimisation

I have a function that synchronizes according to some specifications columns of a dataframe. The functions works, nevertheless I was wondering how to: Improve ...
3
votes
0answers
102 views

Custom Day of the Week Manipulation

The function serves two purposes as follows: When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
3
votes
1answer
207 views

Complete missing data in timeseries pandas

I collected data using a smartphone. The phone could run out of battery or the collection software could fail, leading to missing data. For this example let's say I already compute the number of ...
6
votes
1answer
789 views

Bank saving account class in Python (pandas)

I am setting up a kids bank account class to teach them how to save, and myself how to write a class in Python. The original idea was to create a pandas DataFrame that will show how the money gains ...
1
vote
1answer
8k views

Extracting specific words from PANDAS dataframe

I'm new to data analysis and doing some online training. I have a task to extract specific words from specific column in data frame, then count those words and then just to do some min/max/mean ant ...
5
votes
1answer
201 views

Linear Regression on random data

Wrote a simple script to implement Linear regression and practice numpy/pandas. Uses random data, so obviously weights (thetas) have no significant meaning. Looking for feedback on Performance Python ...