Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
3
votes
1answer
192 views

Python/Pandas code creating 12 month product balance forecast

I have an input DataFrame with one row per product with balances, rates, etc. I want to be able to take that starting balance, and other inputs from the row (characteristics of the product), and ...
4
votes
0answers
942 views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
2
votes
2answers
14k views

Using Pandas to merge .csv files

Hope you can help me out with this one because it is really slow. Is there a way to do this without loading the whole .csv into memory? The thing is... I have files containing timeseries data with 10 ...
10
votes
3answers
2k views

Table of Tribonacci sequence using NumPy and PANDAS

What I am trying to accomplish is calculate the Tribonacci numbers and the ratio. I can do this in Excel easy like this. So I tried do the same using Python 3 now. It works and gives such output. ...
4
votes
1answer
803 views

Twitter data mining project

I've written this simple code, that is possible to find on my github profile (it simply downloads some twitter data and store in an excel file). It is one of my first experiments of using OOP in ...
1
vote
1answer
3k views

Create all lags of given columns

I'm creating a pandas.DataFrame out of lags from an existing pandas.DataFrame, using ...
3
votes
1answer
69 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
9
votes
2answers
52k views

Dropping rows from a PANDAS dataframe where some of the columns have value 0

I am dropping rows from a PANDAS dataframe when some of its columns have 0 value. I got the output by using the below code, but I hope we can do the same with less code — perhaps in a single line. df:...
7
votes
2answers
356 views

Appointment Schedule

In the below code there are two data frames given, free_schedule and appointments which represent a pre-defined available ...
2
votes
1answer
2k views

Writing data to an Excel sheet using openpyxl [closed]

I am currently writing a program to update and copy data from one spreadsheet to another. The code I have written works fine, but it takes way too long for it to be practical. In total, it takes about ...
2
votes
1answer
2k views

Parsing HTML table into Pandas DataFrame

There is a text (link clickable) file with HTML table. The table is a bank statement. I'd like to parse it into pandas DataFrame. Is there a way to do it more gracefully? I've started to learn Python ...
13
votes
1answer
30k views

Parse complex text files using Python

I'm looking for a simple way of parsing complex text files into a pandas DataFrame. Below is a sample file, what I want the result to look like after parsing, and my current method. Is there any way ...
3
votes
0answers
338 views

Compute distance matrix using DTW acceptable for scipy.cluster.hierarchy

I am new to both data science and python. I have a dataset of the time-dependent samples, which I want to run agglomerative hierarchical clustering on them. I have found that Dynamic Time Warping (DTW)...
4
votes
1answer
928 views

Calculate distance between points and price per area in Pandas

I'm trying to find distances between all points (latitude, longitude), and for each point, get the average price_area (price/area) of the closest points around it. This code is taking too long: ...
6
votes
2answers
2k views

Python Pandas Dataframe code takes too long to finish

I have a Pandas DataFrame with users subscription dates in the following format: UserId, StartingDate, EndingDate And I try to calculate the Churn Rate metric for ...
2
votes
0answers
155 views

Network traffic flow detection using pandas dataframe

I have a pandas dataframe containing network traffic of multiple hosts: ...
2
votes
0answers
101 views

Football Web Scraper Part 2

A revised version of the code in this question. Things I have done so far: Adjusted some formatting things like constant & variable naming and indentation Wrapped most of the functions into a ...
2
votes
0answers
73 views

Back-testing algorithm for equity

I've written a code on python which back tests on equities data, to compare result of strategy against a benchmark. I want someone to review to assure that program works as desired, and any changes ...
2
votes
1answer
3k views

Faster way of replacing strings in large pandas dataframe with regex

I want to replace all instances of a location with just the location keyword itself, eg "Foxclore Road" with "road", "Chesture Avenue" with "avenue". The file is several GBs, with many million rows. ...
5
votes
0answers
65 views

Comparing the size of tumors over time using PANDAS

I have a dataframe in pandas like this ...
1
vote
1answer
5k views

Groupby and moving average function in pandas works but is slow

I have some time series data collected for a lot of people (over 50,000) over a two year period on 1 day intervals. I want to applying a exponential weighted moving average function for each person ...
1
vote
0answers
28 views

Writing excelsheet from python dataframe

I am reading dataframe and writing exccel, I cannot use direct df.to_excel method as I am writing data in a very particular way. Looping on dataframe is taking much time, could anything in below ...
2
votes
0answers
39 views

Pull out max from each column in a DataFrame in turns

I have a DataFrame like this: ...
0
votes
1answer
90 views

Finding the difference in time between two events for a specific ID

Details about code: Dataframe (x) shape: (20494463, 31) In this Dataframe, each id shows up multiple times with different events and the event dates. What I'm looking for is the time between the ...
0
votes
1answer
5k views

Searching for keywords in dataframe

Given a dataframe with three columns of text blobs to search through, which can be found in this Gist. And three keywords that I want to identify in this text: ...
6
votes
1answer
4k views

Show weekly and daily variations in time-series data

I've got some time-series data. I want to examine the weekly and daily variation of that data. For weekly data I can make a plot like this, with the days along the horizontal axis: For daily data I ...
3
votes
1answer
86 views

Fetch and append player statistics to an Excel spreadsheet

I am a new programmer in python, and I need a bit of help to structure my code. ...
1
vote
1answer
3k views

Convert sum from monthly to quarterly values

I'm attempting to aggregate multiple columns of monthly data into quarterly chunks. Currently, I'm applying a rolling sum to the columns and then selecting only every third row to 'delete' or hide the ...
5
votes
0answers
165 views

Code for training machine learning linear regression and SVM

Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used ...
2
votes
0answers
332 views

Set up of pandas dataframe with mixed datatypes including arrays from .mat file in a pythonic/numpy way

Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe. I'm importing a .mat file from ...
2
votes
1answer
77 views

How to speed up ordering of output file

I have a process where I take several .lst flatfiles, apply filters and reformat the data, then append the results to a single file. I loop through every file in a subdirectory, applying the ...
2
votes
1answer
6k views

Create quantile slices from dataframe columns

I have the following, working code: ...
3
votes
1answer
51 views

Function that fills a time series row-by-row by using the values in the row before

Summary: performance issue with for-loop, vectorization possible? I am processing a time series of securities price data, stored in a pandas dataframe df_buy_sell. ...
6
votes
1answer
2k views

String split for street addresses

I have 856k addresses of pattern: 660 1st Ave New York, NY 10016 (with 3 spaces between Ave and New York) I need to parse and store parts as: ...
3
votes
1answer
321 views

Monte Carlo Simulation of P-Value

I'm testing Python 3 code to perform a Monte Carlo simulation based on the result of an statistical test. I currently have the result of the statistical test in a pandas dataframe, like this. ...
3
votes
1answer
8k views

Calculating sum of multiple columns in pandas

I have data from one data provider in very thin demographic units: Adults_18_21,Adults_22_24,Adults_25_27, etc. Then in the original data frame that is built like this: ...
3
votes
0answers
49 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
1
vote
0answers
69 views

Dependency resolving functions

Written in Python 3.6 and I must admit, this might be too far... Tell me what you think of this code. There must be a better way... It's (another decorator...) that allows you to compose functions ...
7
votes
1answer
7k views

Remove duplicates from csv based on conditions

The task is basically this: I am given the following csv file with lots of duplicate email addresses ...
2
votes
0answers
183 views

Grouping and aggregating using Pandas

This was inspired by Aggregate loans report without using Python standard aggregate or group functions question, but I've decided to approach it using pandas. To ...
4
votes
0answers
396 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
6
votes
1answer
291 views

Python API for Crimson Hexagon

I'd like some feedback on the readability, style, and potential problems or issues. In particular I'm not too happy with how I handle ratelimits. ...
4
votes
2answers
4k views

Download, unzip and parse xml into pandas dataframe

I have a python script that downloads, unzip and parses an XML file published by a Canadian institution. Only some very specific tags are extracted and then all put into a pandas dataframe for later ...
7
votes
3answers
219 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
3
votes
0answers
397 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
5
votes
2answers
7k views

Gradient descent for linear regression using numpy/pandas

I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
7
votes
1answer
658 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
3
votes
1answer
2k views

Python - Faster random business date generation

I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
2
votes
1answer
56 views

Faster solution for data grouping to determine highest frequency than iteration with for loops

The code below allows me to determine what the most common main dish and the most common method of preparation for the most common main dish, for each US Region. It uses data obtained from '...
3
votes
0answers
420 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...