Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
0
votes
1answer
6k views

Searching for keywords in dataframe

Given a dataframe with three columns of text blobs to search through, which can be found in this Gist. And three keywords that I want to identify in this text: ...
6
votes
1answer
5k views

Show weekly and daily variations in time-series data

I've got some time-series data. I want to examine the weekly and daily variation of that data. For weekly data I can make a plot like this, with the days along the horizontal axis: For daily data I ...
3
votes
1answer
88 views

Fetch and append player statistics to an Excel spreadsheet

I am a new programmer in python, and I need a bit of help to structure my code. ...
1
vote
1answer
3k views

Convert sum from monthly to quarterly values

I'm attempting to aggregate multiple columns of monthly data into quarterly chunks. Currently, I'm applying a rolling sum to the columns and then selecting only every third row to 'delete' or hide the ...
5
votes
0answers
180 views

Code for training machine learning linear regression and SVM

Ok , for my final year project I've wrote this piece of code to train my machine learning model on a this dataset , here the code i used ...
2
votes
0answers
352 views

Set up of pandas dataframe with mixed datatypes including arrays from .mat file in a pythonic/numpy way

Coming from a heavy MATLAB background so excuse my un-pythonic ways. I'd like to get an idea of how a python coder would approach this simple set-up of a dataframe. I'm importing a .mat file from ...
2
votes
1answer
80 views

How to speed up ordering of output file

I have a process where I take several .lst flatfiles, apply filters and reformat the data, then append the results to a single file. I loop through every file in a subdirectory, applying the ...
2
votes
1answer
7k views

Create quantile slices from dataframe columns

I have the following, working code: ...
3
votes
1answer
55 views

Function that fills a time series row-by-row by using the values in the row before

Summary: performance issue with for-loop, vectorization possible? I am processing a time series of securities price data, stored in a pandas dataframe df_buy_sell. ...
6
votes
1answer
3k views

String split for street addresses

I have 856k addresses of pattern: 660 1st Ave New York, NY 10016 (with 3 spaces between Ave and New York) I need to parse and store parts as: ...
3
votes
1answer
368 views

Monte Carlo Simulation of P-Value

I'm testing Python 3 code to perform a Monte Carlo simulation based on the result of an statistical test. I currently have the result of the statistical test in a pandas dataframe, like this. ...
3
votes
1answer
8k views

Calculating sum of multiple columns in pandas

I have data from one data provider in very thin demographic units: Adults_18_21,Adults_22_24,Adults_25_27, etc. Then in the original data frame that is built like this: ...
3
votes
0answers
50 views

Hotpatching pandas.DataFrame with extended insert method

pandas.DataFrame.insert can only handle an int for it's loc kwarg this is inconvenient for ...
1
vote
0answers
72 views

Dependency resolving functions

Written in Python 3.6 and I must admit, this might be too far... Tell me what you think of this code. There must be a better way... It's (another decorator...) that allows you to compose functions ...
7
votes
1answer
8k views

Remove duplicates from csv based on conditions

The task is basically this: I am given the following csv file with lots of duplicate email addresses ...
2
votes
0answers
190 views

Grouping and aggregating using Pandas

This was inspired by Aggregate loans report without using Python standard aggregate or group functions question, but I've decided to approach it using pandas. To ...
4
votes
0answers
405 views

Speed up calculating relative strength index

I am trying to program the calculation of the relative strength index indicator described on a stockcharts.com page. On the same page there are Excel formulas used for calculations. My code below is ...
6
votes
1answer
310 views

Python API for Crimson Hexagon

I'd like some feedback on the readability, style, and potential problems or issues. In particular I'm not too happy with how I handle ratelimits. ...
4
votes
2answers
4k views

Download, unzip and parse xml into pandas dataframe

I have a python script that downloads, unzip and parses an XML file published by a Canadian institution. Only some very specific tags are extracted and then all put into a pandas dataframe for later ...
7
votes
3answers
220 views

Linking two databases based on street addresses

For my work, I wrote a python script to link 2 files. Since I am an autodidact and since no one of my colleagues writes code, I ask the question here. My code takes an unbelievable time to run. Is it ...
3
votes
0answers
435 views

Sklearn custom transformer (label encoder and imputer)

I have a pandas.Dataframe with a single (new) record and the following function: ...
6
votes
3answers
8k views

Gradient descent for linear regression using numpy/pandas

I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
7
votes
1answer
733 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
3
votes
1answer
2k views

Python - Faster random business date generation

I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
2
votes
1answer
57 views

Faster solution for data grouping to determine highest frequency than iteration with for loops

The code below allows me to determine what the most common main dish and the most common method of preparation for the most common main dish, for each US Region. It uses data obtained from '...
3
votes
0answers
441 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
4
votes
1answer
2k views

Downloads historical data from a yahoo finance csv and returns a pandas DataFrame

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to ...
2
votes
1answer
2k views

Apply the smallest possible datatype for each column in a pandas dataframe to reduce RAM use

This function is for Python Pandas users. I noticed a lot of RAM can be saved by applying smaller datatypes to columns. The function's input is a Pandas DataFrame. The function output is the compact ...
2
votes
1answer
50 views

Calculate Twitter Follower Changes Over Time

I have a code that grabs quickly the twitter followers for 12 different users. After appending to a Pandas DataFrame, the data is compared to the same file pulled ...
2
votes
1answer
79 views

Create new DataFrame containing mean information

I have a dataframe with the columns "Action", "Filt" and "Val" which measure the resulting error of filtering an action. I want to calculate the mean of the error for each "Action" and "Filt". ...
4
votes
1answer
231 views

Aggregating text and deleting duplicates with pandas to consolidate a table for reporting

At work, I use software for analyzing administrative data and daily I generate a CSV that looks like: ...
6
votes
2answers
129 views

Listing page visits for each visitor

I have some data in the following format: ...
3
votes
0answers
77 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
1
vote
1answer
46 views

Applying function to entire dataframe [closed]

I'm wondering if there is a faster way to do the following code, which works fine. It needs to be applied to every row of the DataFrame df. ...
5
votes
1answer
100 views

Drawing a regression line with a timeseries

My code is using seaborn to draw a regression line with a timeseries data as the x axis, and whatever you want the y-axis to be. Since seaborn does not support this function directly, a dummy column ...
5
votes
0answers
3k views

Calculate relationship between 2 categorical variables in a pandas Dataset with chi square test

I'm working on a Machine Learning project and I'm in Data Exploration step, and my dataset has both categorical and continuous attributes. I decided to compute a chi square test between 2 categorical ...
8
votes
1answer
198 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
4
votes
4answers
9k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
3
votes
2answers
999 views

Slow code pandas 8 million rows

I have the following code which takes 10-15 minutes to execute. That is way too slow considering that my database is growing daily. Is there any chance to make it faster? ...
5
votes
1answer
5k views

Importing database of 4 million rows into Pandas DataFrame

I am using the following code to import database table into a DataFrame: ...
5
votes
1answer
271 views

Plot president's approval rating of president with lowest approval rating by day of presidency

I was curious which U.S. president had the lowest approval rating for each day in their presidency. For example, which president had the lowest approval rating on day 42, and what was the rating. I ...
1
vote
1answer
127 views

Faster function iteration on Pandas df with axis=1

I have the following function that allows me to calculate the marketing spend on my advertising campaigns. The problem is that it's now taking too long (over 15 minutes) to apply the function to ...
4
votes
2answers
3k views

Count values in one dataframe that are less than all the values in another

I currently have 2 dataframes, A and B. These dataframes are generated in runtime, and increase in size according to parameters in the program execution. I need to evaluate how many times a value in ...
8
votes
1answer
308 views

Average across direction-specific average speeds and probabilities

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-...
6
votes
2answers
1k views

Python code to split csv into smaller csvs, not splitting IDs

I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like ...
4
votes
1answer
52 views

Terminal handling charges for different countries

I have the following problem that I solved using Python (Numpy/Pandas) and the code is provided later on. I mainly program in Java and wrote this program for a job interview as Python developer. I ...
4
votes
1answer
2k views

Calculating speed from a Pandas Dataframe with Time, X, and Y columns

I'm trying to calculate speed between consecutive timepoints given data that is in a '.csv' file with the following columns: "Time Elapsed", "x", and "y". The ultimate goal is to get the data into a ...
3
votes
0answers
37 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
2
votes
1answer
50 views

Check value existence in two lists and gather the result in a csv

I have two lists of protein sequences, I have to check every entry's existence in the two lists, say like ...