Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
4
votes
2answers
167 views

Iterate over a list of list names as file names

Interrogating a web API using query-url's, for each query I can get either zero hits, one hit, or multiple hits. Each of those categories needs to go into a separate CSV file for manual review or ...
4
votes
2answers
89 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
4
votes
2answers
2k views

Groupby count, then sum and get the percentage

I wrote this code. It works, but I think there is a more elegant and Pythonic way to this task. Groupby and count the different occurences Get the sum of all the occurences Divide each occurrence ...
4
votes
1answer
63 views

Subtable timestamps in python pandas

I have a dataframe like: ...
4
votes
1answer
839 views

Twitter data mining project

I've written this simple code, that is possible to find on my github profile (it simply downloads some twitter data and store in an excel file). It is one of my first experiments of using OOP in ...
4
votes
2answers
4k views

Download, unzip and parse xml into pandas dataframe

I have a python script that downloads, unzip and parses an XML file published by a Canadian institution. Only some very specific tags are extracted and then all put into a pandas dataframe for later ...
4
votes
1answer
4k views

Conditional Concatenation of a Pandas DataFrame

I am concatenating columns of a Python Pandas Dataframe and want to improve the speed of my code. My data has the following structure: ...
4
votes
1answer
306 views

Find if items in a list of list matches to another list

We have two lists: ...
4
votes
3answers
124 views

Load recurring (but not strictly identical) sets of Key, Values into a DataFrame from text files

I am reading text files that contain data from observations. The format is not Fixed Width or Delimited, so I built a generator ...
4
votes
2answers
2k views

Matplotlib-venn and keeping lists of the entries

Having come upon the wonderful little module of matplotlib-venn I've used it for a bit, I'm wondering if there's a nicer way of doing things than what I have done so far. I know that you can use the ...
4
votes
2answers
167 views

Long expression to sum data in a multi-dimensional model

I am porting a linear optimization model for power plants from GAMS to Pyomo. Models in both frameworks are a collection of sets (both elementary or tuple sets), parameters (fixed values, defined over ...
4
votes
1answer
49 views

Get minimum of each group based on hour criteria using pandas

I have a requirement which I posted in SO. However, I have already written code and works fine. I request your review to know whether it is fool proof/elegant and efficient. Please find the details ...
4
votes
1answer
466 views

Cumulative counts of items in a Pandas dataframe

i'm using a dataframe full of headers/field names to develop a schema and I need unique values for each header - but each header should be represented. I want to loop though the dataframe list of ...
4
votes
2answers
4k views

Count values in one dataframe that are less than all the values in another

I currently have 2 dataframes, A and B. These dataframes are generated in runtime, and increase in size according to parameters in the program execution. I need to evaluate how many times a value in ...
4
votes
1answer
211 views

Applying different equations to a Pandas DataFrame

I wrote a task using pandas and I'm wondering if the code can be optimized. The code pretty much goes as: All the dataframes are 919 * 919. socio is a dataframe ...
4
votes
2answers
386 views

Pulling financial data via IEX Cloud API

I am pulling some financial data from the IEX Cloud API. Since it is a paid API w/ rate limits, I would like to do this as efficiently as possible (i.e., as few calls as possible). Given that the <...
4
votes
2answers
65 views

Grouping a Pandas Dataframe by two parameters

I have a .csv file of 8k+ rows which looks like this: ...
4
votes
1answer
37 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
4
votes
1answer
70 views

Google Analytic Webcrawler for every country code

At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
4
votes
1answer
231 views

Aggregating text and deleting duplicates with pandas to consolidate a table for reporting

At work, I use software for analyzing administrative data and daily I generate a CSV that looks like: ...
4
votes
1answer
2k views

Calculating speed from a Pandas Dataframe with Time, X, and Y columns

I'm trying to calculate speed between consecutive timepoints given data that is in a '.csv' file with the following columns: "Time Elapsed", "x", and "y". The ultimate goal is to get the data into a ...
4
votes
1answer
14k views

Subtract multiple columns in PANDAS DataFrame by a series (single column)

Background I have tons of very large pandas DataFrames that need to be normalized with the following operation; log2(data) - mean(log2(data)) Example Data The example DataFrame ...
4
votes
1answer
501 views

Data cleansing and formatting script

This is a script that creates a base dataframe from a sqlite database, adds data to it (also from SQLite), cleanse it and formats it all to an Excel file. I feel like it is incredibly verbose and my ...
4
votes
2answers
76 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
1answer
54 views

Load excel sheet to data frame

I have a bare-bones example of how I plan to load excel sheets into pandas data frames. However, the code runs unexpectedly slow. I'm open to suggestions on ways I can speed this up, even converting ...
4
votes
1answer
80 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing code to automate some weekly reports. I had help over on Stack Overflow. I have code that works for the most part, however there are a few things that I just can't seem to fix. In ...
4
votes
1answer
119 views

Python script for to evaluate business data

The following script is part of a further education I'm currently enrolled into. Not all of the code is written by myself. Function signatures for example. Therefore I have put the sections, written ...
4
votes
1answer
79 views

Class made to do some basic RGB analysis

I created this class to load in an image and get some specific values I needed. ...
4
votes
1answer
1k views

Permutations with a sum constraint

I have made the following function that gives the combinations permuted of n elements with a constrain/rule. This consist on creating a dataframe with all combination of numbers from 0 to n that sums ...
4
votes
1answer
847 views

Formatting Excel data into smaller pivot tables

The code below (Distribution.py) reads the results I get from a simulation I am running and formats them into smaller pivot tables. The data comes out as excel ...
4
votes
1answer
1k views

Calculate distance between points and price per area in Pandas

I'm trying to find distances between all points (latitude, longitude), and for each point, get the average price_area (price/area) of the closest points around it. This code is taking too long: ...
4
votes
1answer
3k views

PANDAS code for calculating distance between waypoints

I've written some python code designed to take a csv of waypoints for a series of trips, and calculate the distance of each trip by the sum of the distance between the waypoints. An example csv might ...
4
votes
1answer
82 views

Backing up files to remote servers

I have code that backup files to remote servers. There are 2 pandas data frames: storedfiles: that points to the files already stored in previous runs ...
4
votes
1answer
778 views

Fetching, processing, and storing Mixpanel analytics data to SQLite

I'm a self-taught Python programmer and I never really learned the fundamentals of programming, so I want to see how to improve upon this script and make it adhere to best practices. The script has ...
4
votes
1answer
80 views

Optimising iteration and substitution over large dataset

I have the following code: ...
4
votes
1answer
115 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
4
votes
1answer
640 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
4
votes
1answer
69 views

Python program computing some statistics on Scottish geographic areas

This simple script computes some basic descriptive statistics, like mean, standard deviation, kurtosis, etc. on data column imported from a CSV file with use of ...
4
votes
1answer
3k views

Pandas/Tkinter GUI Excel report generator

I built a GUI tool that takes excel files and outputs a finished report to help automate a report at work. It was a fantastic learning experienced and I feel much more comfortable with pandas and ...
4
votes
1answer
132 views

Munging select rows in CSV files using Pandas

I'm learning Pandas. I have a project where I need to munge some CSV files and resave them as CSV. I can use dictionaries and CSV module, but decided to use DataFrames to get more exposure and ...
4
votes
1answer
2k views

Downloads historical data from a yahoo finance csv and returns a pandas DataFrame

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to ...
4
votes
1answer
52 views

Terminal handling charges for different countries

I have the following problem that I solved using Python (Numpy/Pandas) and the code is provided later on. I mainly program in Java and wrote this program for a job interview as Python developer. I ...
4
votes
1answer
715 views

Analysis of call center employee performance

This code basically calculates a few metrics based on numerators and denominators and then it bins the data into quintiles. This is a common operation that I perform for various data sets. My issue is ...
4
votes
1answer
2k views

PANDAS spatial clustering

I'am writing on a spatial clustering algorithm using pandas and scipy's kdtree. I profiled the code and the .loc part takes most time for bigger datasets. I wonder ...
4
votes
0answers
35 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
4
votes
0answers
69 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
4
votes
0answers
37 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
4
votes
0answers
54 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
4
votes
1answer
58 views

Set of one-hot encoders in Python

In the absence of feature-complete and easy-to-use one-hot encoders in the Python ecosystem I've made a set of my own. This is intended to be a small library, so I want to make sure it's as clear and ...
4
votes
0answers
1k views

Pandas/Numpy statistical odds ratio test

I'm looking for a faster way to do odds ratio tests on a large dataset. I have about 1200 variables (see var_col) I want to test against each other for mutual exclusion/ co-occurrence. An odds ratio ...