Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
5
votes
2answers
7k views

Gradient descent for linear regression using numpy/pandas

I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
7
votes
1answer
657 views

PANDAS nearest site algorithm

I have got CSVs full of property transactions in the UK from 1995 to 2017, separated by year such as "RS2015.csv". I have a 2nd CSV with a list of wind turbines in the UK. Both have coordinates in WGS ...
3
votes
1answer
2k views

Python - Faster random business date generation

I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
2
votes
1answer
56 views

Faster solution for data grouping to determine highest frequency than iteration with for loops

The code below allows me to determine what the most common main dish and the most common method of preparation for the most common main dish, for each US Region. It uses data obtained from '...
3
votes
0answers
420 views

pandas time-based qcut

I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
4
votes
1answer
2k views

Downloads historical data from a yahoo finance csv and returns a pandas DataFrame

Not too long ago, yahoo-finance shut down its public API for downloading historical data. There are however works around and the below code is on of them. I started looking for solutions of how to ...
2
votes
1answer
2k views

Apply the smallest possible datatype for each column in a pandas dataframe to reduce RAM use

This function is for Python Pandas users. I noticed a lot of RAM can be saved by applying smaller datatypes to columns. The function's input is a Pandas DataFrame. The function output is the compact ...
2
votes
1answer
49 views

Calculate Twitter Follower Changes Over Time

I have a code that grabs quickly the twitter followers for 12 different users. After appending to a Pandas DataFrame, the data is compared to the same file pulled ...
2
votes
1answer
78 views

Create new DataFrame containing mean information

I have a dataframe with the columns "Action", "Filt" and "Val" which measure the resulting error of filtering an action. I want to calculate the mean of the error for each "Action" and "Filt". ...
4
votes
1answer
221 views

Aggregating text and deleting duplicates with pandas to consolidate a table for reporting

At work, I use software for analyzing administrative data and daily I generate a CSV that looks like: ...
6
votes
2answers
129 views

Listing page visits for each visitor

I have some data in the following format: ...
3
votes
0answers
75 views

Optimize this loop to access if a value was in previous line with one condition

I successfully create a for loop to generate a new column in my data. This column access if there is retention of the customer on a Radio station. The data looks like this: ...
1
vote
1answer
46 views

Applying function to entire dataframe [closed]

I'm wondering if there is a faster way to do the following code, which works fine. It needs to be applied to every row of the DataFrame df. ...
5
votes
1answer
100 views

Drawing a regression line with a timeseries

My code is using seaborn to draw a regression line with a timeseries data as the x axis, and whatever you want the y-axis to be. Since seaborn does not support this function directly, a dummy column ...
5
votes
0answers
3k views

Calculate relationship between 2 categorical variables in a pandas Dataset with chi square test

I'm working on a Machine Learning project and I'm in Data Exploration step, and my dataset has both categorical and continuous attributes. I decided to compute a chi square test between 2 categorical ...
8
votes
1answer
193 views

Summarize a document as a key-phrase or key-words

A few days ago I finished a coding challenge for a potential job. I was super happy with my code, till I got the response that my code wasn't good enough. :( So, apparently I'm still making mistakes, ...
4
votes
4answers
8k views

Cleaning and extracting meaningful text from tweets

I have a dataset of around 200,000 tweets. I am running a classification task on them. Dataset has two columns - class label and the tweet text. In the preprocessing step I am passing the dataset ...
3
votes
2answers
901 views

Slow code pandas 8 million rows

I have the following code which takes 10-15 minutes to execute. That is way too slow considering that my database is growing daily. Is there any chance to make it faster? ...
5
votes
1answer
5k views

Importing database of 4 million rows into Pandas DataFrame

I am using the following code to import database table into a DataFrame: ...
5
votes
1answer
264 views

Plot president's approval rating of president with lowest approval rating by day of presidency

I was curious which U.S. president had the lowest approval rating for each day in their presidency. For example, which president had the lowest approval rating on day 42, and what was the rating. I ...
1
vote
1answer
127 views

Faster function iteration on Pandas df with axis=1

I have the following function that allows me to calculate the marketing spend on my advertising campaigns. The problem is that it's now taking too long (over 15 minutes) to apply the function to ...
4
votes
2answers
3k views

Count values in one dataframe that are less than all the values in another

I currently have 2 dataframes, A and B. These dataframes are generated in runtime, and increase in size according to parameters in the program execution. I need to evaluate how many times a value in ...
7
votes
1answer
293 views

Average across direction-specific average speeds and probabilities

I have a wind rose with 72 evenly-spaced directions, spanning 360°, each describing a direction-specific average wind speed and associated probability. We must condense this information to 36 evenly-...
6
votes
2answers
991 views

Python code to split csv into smaller csvs, not splitting IDs

I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like ...
4
votes
1answer
52 views

Terminal handling charges for different countries

I have the following problem that I solved using Python (Numpy/Pandas) and the code is provided later on. I mainly program in Java and wrote this program for a job interview as Python developer. I ...
4
votes
1answer
2k views

Calculating speed from a Pandas Dataframe with Time, X, and Y columns

I'm trying to calculate speed between consecutive timepoints given data that is in a '.csv' file with the following columns: "Time Elapsed", "x", and "y". The ultimate goal is to get the data into a ...
3
votes
0answers
37 views

Calculating frequencies of each obs in the data

I am currently attempting to make some code more maintainable for a research project I am working on. I am definitely looking to create some more functions, and potentially create a general class to ...
4
votes
0answers
1k views

Sanitize and standardize street addresses using Google Maps lookups

Take in incomplete user input street addresses, clean it, segregate based on word count and run it into Google maps places api and output completed & standardised addresses based on Json object ...
2
votes
1answer
50 views

Check value existence in two lists and gather the result in a csv

I have two lists of protein sequences, I have to check every entry's existence in the two lists, say like ...
4
votes
0answers
378 views

Efficiently writing string comparison functions in Python

Let's say I work for a company that hands out different types of loans. We are getting our loan information from from a big data mart from which I need to calculate some additional things to calculate ...
3
votes
1answer
282 views

Categorization algorithm for discrete variables

I am trying to categorize some data. For that I check the distribution of the data. Then I split based on the number of appearance of each value. The algorithm I have is working so far but really ...
1
vote
0answers
49 views

bullet-proof way to generate a list of object attributes (if object has them)

Background I wrote a a couple functions that make it possible to generate a list of attributes for any given object that has them. I use the list that is ...
6
votes
1answer
394 views

Expanding url from shortened url obtained from tweet

I have a twitter data set. I have extracted all the expanded urls from the json and now am trying to resolve the shortened ones. Also, I need to check which urls are still working and only keep those. ...
6
votes
1answer
67 views

Undoing corrections to a big dataframe

I have 2 dataframes. The first one (900 lines) contains corrections that have been applied to a deal. The second dataframe (140,000 lines) contains the list of deals with corrected values. What I am ...
6
votes
2answers
1k views

Code for creating combinations taking a long time to finish

I have the following code that I'm using to create combinations of elements in my dataset: ...
8
votes
2answers
142 views

Developing an investment strategy based on stock movements

Below is my code ...
7
votes
1answer
184 views

Coalesce consecutive failures in a DataFrame of hourly sensor readings

I have a PANDAS DataFrame that contains sensor data that is recorded every hour (sample included below). It is important to note that every hour is not necessarily in the dataframe, as sometimes the ...
4
votes
1answer
13k views

Subtract multiple columns in PANDAS DataFrame by a series (single column)

Background I have tons of very large pandas DataFrames that need to be normalized with the following operation; log2(data) - mean(log2(data)) Example Data The example DataFrame ...
8
votes
2answers
750 views

Extract unique terms from a PANDAS series

Background I have process tons of DataFrames with shapes of ~230 columns x ~2000-50000+ rows. Here is an extremely simplified example; ...
8
votes
1answer
96 views

Calculating distance and time between waypoints for large files

This is very similar to other code I've posted, however this is designed for very large CSV files, for example 35GB. The files typically look like this: ...
9
votes
2answers
264 views

Curried function

From the question and since I'm currently learning functional programming I was inspired to write the following (curried) function: ...
5
votes
1answer
365 views

Clustering points on a sphere

I have written a short Python program which does the following: loads a large data file (\$10^9+\$ rows) where each row is a point on a sphere. The code then loads a pre-determined triangular grid on ...
3
votes
1answer
1k views

Optimize calculation for the amount of time between when a value changes in Pandas

I have some sensor data that contains timestamps from when a machine is turned on and an indicator variable showing whether or not the machine is actively running. The data is mostly recorded every ...
6
votes
0answers
2k views

Calculating T-Test within Large Pandas Dataframes

The below code runs a t-statistic within a large dataframe (rnadf) based on masked values from another dataframe (cnvdf_maked). ...
4
votes
1answer
2k views

PANDAS code for calculating distance between waypoints

I've written some python code designed to take a csv of waypoints for a series of trips, and calculate the distance of each trip by the sum of the distance between the waypoints. An example csv might ...
4
votes
1answer
3k views

Conditional Concatenation of a Pandas DataFrame

I am concatenating columns of a Python Pandas Dataframe and want to improve the speed of my code. My data has the following structure: ...
0
votes
2answers
12k views

Interpret YYYYMMDD as the nth day of the year [closed]

I am provided a bunch of dates in the format (YYYYMMDD) such as: date='20170503' My goal is to convert that date into the ...
0
votes
1answer
142 views

Calls one column of a dataframe, turns it into an array and plots it

My code calls 1 column of a dataframe, turns it into an array and plots it. I want to able to do this for all the columns without having to repeat the code many times. How can I do this? ...
3
votes
2answers
81k views

Reading from a .txt file to a pandas dataframe

Having a text file './inputs/dist.txt' as: ...
4
votes
1answer
195 views

Applying different equations to a Pandas DataFrame

I wrote a task using pandas and I'm wondering if the code can be optimized. The code pretty much goes as: All the dataframes are 919 * 919. socio is a dataframe ...