Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
4
votes
1answer
35 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
4
votes
1answer
69 views

Google Analytic Webcrawler for every country code

At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
4
votes
1answer
224 views

Aggregating text and deleting duplicates with pandas to consolidate a table for reporting

At work, I use software for analyzing administrative data and daily I generate a CSV that looks like: ...
4
votes
1answer
2k views

Calculating speed from a Pandas Dataframe with Time, X, and Y columns

I'm trying to calculate speed between consecutive timepoints given data that is in a '.csv' file with the following columns: "Time Elapsed", "x", and "y". The ultimate goal is to get the data into a ...
4
votes
1answer
13k views

Subtract multiple columns in PANDAS DataFrame by a series (single column)

Background I have tons of very large pandas DataFrames that need to be normalized with the following operation; log2(data) - mean(log2(data)) Example Data The example DataFrame ...
4
votes
1answer
492 views

Data cleansing and formatting script

This is a script that creates a base dataframe from a sqlite database, adds data to it (also from SQLite), cleanse it and formats it all to an Excel file. I feel like it is incredibly verbose and my ...
4
votes
1answer
62 views

Python script for to evaluate business data

The following script is part of a further education I'm currently enrolled into. Not all of the code is written by myself. Function signatures for example. Therefore I have put the sections, written ...
4
votes
1answer
1k views

Permutations with a sum constraint

I have made the following function that gives the combinations permuted of n elements with a constrain/rule. This consist on creating a dataframe with all combination of numbers from 0 to n that sums ...
4
votes
1answer
784 views

Formatting Excel data into smaller pivot tables

The code below (Distribution.py) reads the results I get from a simulation I am running and formats them into smaller pivot tables. The data comes out as excel ...
4
votes
1answer
967 views

Calculate distance between points and price per area in Pandas

I'm trying to find distances between all points (latitude, longitude), and for each point, get the average price_area (price/area) of the closest points around it. This code is taking too long: ...
4
votes
1answer
3k views

PANDAS code for calculating distance between waypoints

I've written some python code designed to take a csv of waypoints for a series of trips, and calculate the distance of each trip by the sum of the distance between the waypoints. An example csv might ...
4
votes
1answer
80 views

Backing up files to remote servers

I have code that backup files to remote servers. There are 2 pandas data frames: storedfiles: that points to the files already stored in previous runs ...
4
votes
1answer
763 views

Fetching, processing, and storing Mixpanel analytics data to SQLite

I'm a self-taught Python programmer and I never really learned the fundamentals of programming, so I want to see how to improve upon this script and make it adhere to best practices. The script has ...
4
votes
1answer
77 views

Optimising iteration and substitution over large dataset

I have the following code: ...
4
votes
1answer
277 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
4
votes
1answer
67 views

Python program computing some statistics on Scottish geographic areas

This simple script computes some basic descriptive statistics, like mean, standard deviation, kurtosis, etc. on data column imported from a CSV file with use of ...
4
votes
1answer
2k views

Pandas/Tkinter GUI Excel report generator

I built a GUI tool that takes excel files and outputs a finished report to help automate a report at work. It was a fantastic learning experienced and I feel much more comfortable with pandas and ...
4
votes
1answer
71 views

Class made to do some basic RGB analysis

I created this class to load in an image and get some specific values I needed. ...
4
votes
1answer
129 views

Munging select rows in CSV files using Pandas

I'm learning Pandas. I have a project where I need to munge some CSV files and resave them as CSV. I can use dictionaries and CSV module, but decided to use DataFrames to get more exposure and ...
4
votes
1answer
2k views

Downloads historical data from a yahoo finance csv and returns a pandas DataFrame

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to ...
4
votes
1answer
52 views

Terminal handling charges for different countries

I have the following problem that I solved using Python (Numpy/Pandas) and the code is provided later on. I mainly program in Java and wrote this program for a job interview as Python developer. I ...
4
votes
1answer
677 views

Analysis of call center employee performance

This code basically calculates a few metrics based on numerators and denominators and then it bins the data into quintiles. This is a common operation that I perform for various data sets. My issue is ...
4
votes
1answer
1k views

PANDAS spatial clustering

I'am writing on a spatial clustering algorithm using pandas and scipy's kdtree. I profiled the code and the .loc part takes most time for bigger datasets. I wonder ...
4
votes
0answers
29 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
4
votes
0answers
43 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
4
votes
1answer
52 views

Set of one-hot encoders in Python

In the absence of feature-complete and easy-to-use one-hot encoders in the Python ecosystem I've made a set of my own. This is intended to be a small library, so I want to make sure it's as clear and ...
4
votes
0answers
1k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
4
votes
0answers
1k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
4
votes
0answers
988 views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...
4
votes
0answers
397 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
4
votes
0answers
402 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
4
votes
0answers
1k views

Python Script to query OSRM (drive-times)

The below script takes two arguments Path to the OSRM Map Path to a .csv containing the columns ...
3
votes
2answers
385 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...
3
votes
2answers
938 views

Slow code pandas 8 million rows

I have the following code which takes 10-15 minutes to execute. That is way too slow considering that my database is growing daily. Is there any chance to make it faster? ...
3
votes
3answers
5k views

Plotting from a Pandas dataframe

I want to improve my code. Is it possible to get the plot without repeating the same instructions multiple lines? The data comes from a Pandas' dataframe, but I am only plotting the last column (...
3
votes
1answer
47 views

Calculate percentage based on dummy variable in Pandas

Today I had a simple task: I have products, quantities and a dummy, and I needed to know which percentage of the total quantities of that product the dummy represented. My DataFrame looked like this: ...
3
votes
1answer
1k views

Compute conditional median of PANDAS dataframe

I am new to Python/Pandas. Consider the following code: ...
3
votes
1answer
41 views

Python: Combining Two Rows with Pandas read_excel

I am reading an Excel file using Pandas and I feel like there has to be a better way to handle the way I create column names. This is something like the Excel file I'm reading: ...
3
votes
1answer
70 views

Checking whether fund prices changed over multiple CSV files

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes. 1st download - 2 Fund ...
3
votes
1answer
53 views

Django module to transform data from database to be displayed in Slickgrid tables

This is a Django tag with 2 classes, but can be viewed as a module: What does the module do? it loads a database, performs some transformations on the data, returns the output as a dictionary of json ...
3
votes
1answer
42 views

Getting the average score for hotel scores in different countries using pandas

I am diving into Data Analysis with pandas, and I have just written this Python script to calculate the average of hotel review scores of each country. The dataset contains an individual average score ...
3
votes
1answer
147 views

Code runs Speed Test CLI using the python wrapper and then stores values within CSV

I wanted to make use of the speedtest-cli while also incorporating pandas into the project. My initial goal, which I am posting here, was to take the values of the speed test and incorporate them into ...
3
votes
1answer
70 views

Compute leaderboard of competitive riders throughout a season

I have a table in my DB that contains Riders as rows and their points throughout the season as columns. Here are the columns: ...
3
votes
1answer
622 views

Pivot pandas DataFrame while removing index duplicates

The goal of my code is to pivot a pandas DataFrame which is shown below. The idea is to use the Individual column as the column IDs and the ...
3
votes
1answer
86 views

Fetch and append player statistics to an Excel spreadsheet

I am a new programmer in python, and I need a bit of help to structure my code. ...
3
votes
1answer
54 views

Function that fills a time series row-by-row by using the values in the row before

Summary: performance issue with for-loop, vectorization possible? I am processing a time series of securities price data, stored in a pandas dataframe df_buy_sell. ...
3
votes
1answer
62 views

Filling in gym membership prices by joining PANDAS dataframes

I have two dataframes, df which contains a list of members in addition to the type of contract that they purchased on a given date, ...
3
votes
1answer
108 views

Excel Laboratory Data Entry from Python 2.7

I've written a script to automate the entry of laboratory instrument data into an Excel spreadsheet using pandas and win32com. I've got the script functioning correctly, but it is painfully slow. In ...
3
votes
1answer
6k views

Working with pandas dataframes for stock backtesting exercise

I'm attempting to apply a long set of conditions and operations onto a pandas dataframe (see the dataframe below with VTI, upper, lower, etc). I attempted to use apply, but I was having a lot of ...