Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
1answer
264 views

Implementing a faster inbound neighbour similarity in Networkx

Given a directed graph G of 20+ million edges and approximately 2 million unique nodes, can this function/process be made faster on a dataset of 40k edges? The current speed is at ~1.8 seconds per ...
2
votes
2answers
15k views

Using Pandas to merge .csv files

Hope you can help me out with this one because it is really slow. Is there a way to do this without loading the whole .csv into memory? The thing is... I have files containing timeseries data with 10 ...
2
votes
1answer
4k views

Faster way of replacing strings in large pandas dataframe with regex

I want to replace all instances of a location with just the location keyword itself, eg "Foxclore Road" with "road", "Chesture Avenue" with "avenue". The file is several GBs, with many million rows. ...
2
votes
1answer
7k views

Create quantile slices from dataframe columns

I have the following, working code: ...
2
votes
1answer
57 views

Faster solution for data grouping to determine highest frequency than iteration with for loops

The code below allows me to determine what the most common main dish and the most common method of preparation for the most common main dish, for each US Region. It uses data obtained from '...
2
votes
1answer
80 views

Create new DataFrame containing mean information

I have a dataframe with the columns "Action", "Filt" and "Val" which measure the resulting error of filtering an action. I want to calculate the mean of the error for each "Action" and "Filt". ...
2
votes
1answer
574 views

Compressing time series by removing repeated samples

I work on a project with time series data. So there are samples (\$y\$), and each sample has a timestamp (\$x\$). The data will be visualized, but often there are time series which contain samples ...
2
votes
1answer
3k views

Python Pandas Apply with a Lambda Function

I have a table in pandas that has two columns, QuarterHourDimID and StartDateDimID ; these columns give me an ID for each date / ...
2
votes
0answers
24 views

Python Wrapper for Dynamic SQL

...
2
votes
0answers
42 views

Faster way to replace strings in pandas series using dictionary in Python

I'm working with a dictionary with 11830 keys/values, that have to loop over 13905 strings, that can have up to 50 words each. My series looks like this: ...
2
votes
0answers
325 views

Python, Selenium, Pandas project; for populating web form page on loop

I have inherited this nightmare Python, Selenium, and Pandas project. I am learning Python as I go; and have been cleaning up and refactoring the code where I see necessary - It was worst if you can ...
2
votes
0answers
50 views

Efficiently/pythonically calculating new Dataframe rows

I'm new to Pandas, and slightly new to Python as well (but not development in general). I've got a chunk of code that works, but it feels like I'm missing out on the Pandas/Python way of doing ...
2
votes
0answers
71 views

Sum, validate, and plot monthly inventory levels

The idea is that I have a subset of the movements of material of a factory (over 700k rows), and I want to separate these movements by the reference of each product to plot the stock of each product ...
2
votes
0answers
32 views

Function that plots data frames, supporting several variants

I defined a function that generates tables and creates countplots and barplots based on the arguments that it receives. But the snippets of code inside the function are repeated several times, which ...
2
votes
0answers
334 views

Finding peaks in a DataFrame

I would like to compare the min/max values of a time-series with a test time-series. Additionally, I would like to compare the time of the "peaks". However, I'm having trouble extracting these ...
2
votes
0answers
94 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
2
votes
0answers
72 views

Regression on Pandas DataFrame

I am working on the following assignment and I am a bit lost: Build a regression model that will predict the rating score of each product based on attributes which correspond to some very common ...
2
votes
1answer
441 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
0answers
33 views

Determine file provider based on column headers, replace columns and process data

Each day I receive files from different customers relaying the same information, but the customers use proprietary formats. For example, ...
2
votes
0answers
41 views

Average coverage between each sphere of each spheres groups

Let me start with a litle bit of context. I have Spheres, a large pandas DataFrame with positions and radius of multiple spheres during time. (in real life 10 000 ...
2
votes
1answer
35 views

Count number of registers in interval & location

Recently I asked how one could count the number of registers by the interval as answered in https://stackoverflow.com/questions/49240140/count-number-of-registers-in-interval. The solution works ...
2
votes
1answer
2k views

Writing data to an Excel sheet using openpyxl [closed]

I am currently writing a program to update and copy data from one spreadsheet to another. The code I have written works fine, but it takes way too long for it to be practical. In total, it takes about ...
2
votes
1answer
2k views

Parsing HTML table into Pandas DataFrame

There is a text (link clickable) file with HTML table. The table is a bank statement. I'd like to parse it into pandas DataFrame. Is there a way to do it more gracefully? I've started to learn Python ...
2
votes
0answers
164 views

Network traffic flow detection using pandas dataframe

I have a pandas dataframe containing network traffic of multiple hosts: ...
2
votes
0answers
126 views

Football Web Scraper Part 2

A revised version of the code in this question. Things I have done so far: Adjusted some formatting things like constant & variable naming and indentation Wrapped most of the functions into a ...
2
votes
0answers
74 views

Back-testing algorithm for equity

I've written a code on python which back tests on equities data, to compare result of strategy against a benchmark. I want someone to review to assure that program works as desired, and any changes ...
2
votes
0answers
40 views

Pull out max from each column in a DataFrame in turns

I have a DataFrame like this: ...
2
votes
0answers
364 views

Set up of pandas dataframe with mixed datatypes including arrays from .mat file in a pythonic/numpy way

Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe. I'm importing a .mat file from ...
2
votes
1answer
80 views

How to speed up ordering of output file

I have a process where I take several .lst flatfiles, apply filters and reformat the data, then append the results to a single file. I loop through every file in a subdirectory, applying the ...
2
votes
0answers
191 views

Grouping and aggregating using Pandas

This was inspired by Aggregate loans report without using Python standard aggregate or group functions question, but I've decided to approach it using pandas. To ...
2
votes
1answer
2k views

Apply the smallest possible datatype for each column in a pandas dataframe to reduce RAM use

This function is for Python Pandas users. I noticed a lot of RAM can be saved by applying smaller datatypes to columns. The function's input is a Pandas DataFrame. The function output is the compact ...
2
votes
0answers
340 views
2
votes
0answers
69 views

Creating multiple observations from single observation depending on conditions

I have written a function in python with three loops which is time consuming. Is it possible to do the same operation in less time with some other way. Here are my code and sample data you can run at ...
2
votes
0answers
227 views

Using pandas.io.pytables.TableIterator for several iterations

I'm writing method that takes a pandas iterator and then computes something based on that. The problem is that some of these computations require iterating twice over the data. For example the ...
2
votes
0answers
40 views

Grouping logs and providing counts

The code is supposed to group start and end time logs and provide log counts and unique ID counts. The grouping will be variable 1 hour, 6 hours, 12 hours, 24 hours, etc. What is the better way to ...
2
votes
0answers
67 views

Analysis of TV station preferences for various demographic groups

I have a notebook here which details my code and may be more legible. I worked on a project that flattens a table that initially looked like this (this is table1): ...
2
votes
0answers
979 views

Speeding up filtering function in Pandas

I have a CSV file with 400 000 rows and the following headers: ...
2
votes
0answers
808 views

Speed up projection of a bipartitie network for a big file using NetworkX and Pandas

I have a pretty big file (3 million lines) with each line being a person-to-event relationship. Ultimate, I want to project this bipartite network onto a single-mode, weighted, network, and write it ...
1
vote
1answer
6k views

Groupby and moving average function in pandas works but is slow

I have some time series data collected for a lot of people (over 50,000) over a two year period on 1 day intervals. I want to applying a exponential weighted moving average function for each person ...
1
vote
1answer
77 views

My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report

I have a requirement to write a Python 3.6 script which should be able to detect following case within a time period and be able to create a text file report that lists the condition violated and ...
1
vote
1answer
2k views

Parse Bloomberg Excel/CSV with Pandas DataFrame

I retrieved Bloomberg data using the Excel API. In the typical fashion, the first row contains tickers in every fourth column, and the second row has the labels Date, PX_LAST, [Empty Column], Date, ...
1
vote
1answer
43 views

Elegant way to do the dataframe operations twice

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the ...
1
vote
2answers
236 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
1
vote
1answer
47 views

Efficient way to add a colon before and after every word in strings inside a Series

I was working with twitter data, and extracted every emoji from the data. But, when I passed that data through CountVectorizer, colons where substracted from the ...
1
vote
1answer
50 views

Calls many stored dataframes, change them and combine them

I call many dataframes (using pickle), then I extract the interesting values of each one and finally combine them. Is there a better way to do it? For example with a for loop call them automatically ...
1
vote
1answer
31 views

tablename alias in attribute name

I have dataframe like this: ...
1
vote
1answer
56 views

Efficient way to get value from a dataframe and append new dataframe

I have a dataframe that have about 200 million rows. the example of dataframe is like this: ...
1
vote
2answers
53 views

Function to count words in Python

I am new to developing in Python and would like to see comments about my word count solution. Please indicate my mistakes or best solutions. ...
1
vote
2answers
55 views

Groupby, apply custom function to data, return results in new columns

I want to group by id, apply a custom function to the data, and create a new column with the results. It seems there must be a faster/more efficient way to do this ...
1
vote
1answer
551 views

Customer segmentation using RFM analysis [closed]

Currently, my code works perfectly well but i would like to make it cleaner for other users by removing the duplicate and similar lines of code into functions or for loops. Because I am still new ...