Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
1answer
46 views

Pandas updating/adding columns to rows incrementally using dictionary key values

I want to populate columns of a dataframe (df) by iteratively looping over a list (A_list) generating a dictionary where the keys are the names of the desired columns of df (in the example below the ...
3
votes
1answer
24 views

Parse table names and attributes from SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
7
votes
1answer
152 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
-1
votes
0answers
19 views

Filter and clean up a dataframe of metereological data [on hold]

I have written code that will filter streaming data once they are imported and unzipped. The code works, but looks quite messy and unprofessional. Someone will inherit it in the future. I want to be ...
-1
votes
0answers
16 views

How do I wrap up all my dataframe filter in a def function [duplicate]

I have written codes that will filter streaming data once they are imported and unzipped. The code works, but looks quite messy and unprofessional. Someone will inherit them in the future. I want to ...
1
vote
0answers
22 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
-2
votes
0answers
22 views

Volatility drift-independent (Yang & Zhang) [closed]

I am not sure if the following code for implementing Yang & Zhang volatility model works correctly - for the AMZN stock I get an output btw 0.0 and 4.0: ...
0
votes
1answer
25 views
0
votes
1answer
29 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
2
votes
1answer
62 views

Parsing data from JSON API in Pandas

I made a little program to parse data from an API. I do not have experience with Pandas. It is working but I would like to know how to do that better and more efficiently. ...
1
vote
2answers
35 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
1
vote
1answer
43 views

Efficient way to get value from a dataframe and append new dataframe

I have a dataframe that have about 200 million rows. the example of dataframe is like this: ...
4
votes
2answers
71 views

Groupby count, then sum and get the percentage

I wrote this code. It works, but I think there is a more elegant and Pythonic way to this task. Groupby and count the different occurences Get the sum of all the occurences Divide each occurrence ...
1
vote
1answer
19 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
4
votes
2answers
51 views

Grouping a Pandas Dataframe by two parameters

I have a .csv file of 8k+ rows which looks like this: ...
5
votes
1answer
94 views

Building a frequency dictionary from a Pandas dataframe

I need to make a frequency dictionary from a pandas series (from the 'amino_acid' column in dataframe below) that also adds an adjacent row for each entry in the dictionary (from 'templates' column). ...
1
vote
0answers
28 views

Sortings stocks into quantiles based on their signal

This function sorts stocks into quantiles. To do so I used this function that accepts sig_df (dataframe with the timeseries of stocks signal) and number of ...
3
votes
2answers
896 views

Slow code pandas 8 million rows

I have the following code which takes 10-15 minutes to execute. That is way too slow considering that my database is growing daily. Is there any chance to make it faster? ...
3
votes
0answers
51 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
6
votes
1answer
52 views

Get nearest major city of an image (using EXIF GPS data)

I am writing a program that will take images in a folder, get the GPS Coordinates (from the EXIF data), then return the closest "major city". (I define a major city as one with an airport, so ...
3
votes
1answer
39 views

Calculate percentage based on dummy variable in Pandas

Today I had a simple task: I have products, quantities and a dummy, and I needed to know which percentage of the total quantities of that product the dummy represented. My DataFrame looked like this: ...
1
vote
2answers
45 views

Function to count words in Python

I am new to developing in Python and would like to see comments about my word count solution. Please indicate my mistakes or best solutions. ...
4
votes
0answers
28 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
2
votes
0answers
33 views

Faster way to replace strings in pandas series using dictionary in Python

I'm working with a dictionary with 11830 keys/values, that have to loop over 13905 strings, that can have up to 50 words each. My series looks like this: ...
2
votes
1answer
33 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
3
votes
1answer
41 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
1
vote
1answer
98 views

Comparing columns in pandas different data frames and fill in a new column

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists ...
1
vote
0answers
28 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
4
votes
1answer
126 views

Munging select rows in CSV files using Pandas

I'm learning Pandas. I have a project where I need to munge some CSV files and resave them as CSV. I can use dictionaries and CSV module, but decided to use DataFrames to get more exposure and ...
2
votes
1answer
80 views

Importing text into PANDAS and counting certain words

Aim: To improve the speed of the following code. Current timing is about 80~ hours :0 Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns ...
3
votes
0answers
19 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
3
votes
1answer
69 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
1answer
574 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
1
vote
1answer
47 views

Efficient way to add a colon before and after every word in strings inside a Series

I was working with twitter data, and extracted every emoji from the data. But, when I passed that data through CountVectorizer, colons where substracted from the ...
2
votes
1answer
166 views

Faster way to 3 nested for loop python

I am calculating parameters for a time series and the data size is very big. How can I make my function faster? The following function, param, calculates parameters ...
2
votes
1answer
69 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
4
votes
1answer
35 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
1
vote
1answer
4k views
1
vote
0answers
18 views

Requests / Facebook API - Dealing with Errors

I work in marketing and I am currently trying to aggregate data for 50 facebook pages on the same Business Page. I have a loop ...
2
votes
1answer
387 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
2answers
57 views

Fastest way to find dataframe indexes of column elements that exist as lists

I asked this question here: https://stackoverflow.com/q/55640147/5202255 and was told to post on this forum. I would like to know whether my solution can be improved or if there is another approach to ...
3
votes
1answer
37 views

Python: Combining Two Rows with Pandas read_excel

I am reading an Excel file using Pandas and I feel like there has to be a better way to handle the way I create column names. This is something like the Excel file I'm reading: ...
4
votes
0answers
39 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
3
votes
0answers
26 views

Feedback on Object-Oriented ETL Class Structure for Computing Profit Summary Data

I am using Python 3 and pandas 0.24.2 to do some data processing and ETL flows. I have followed this pattern a couple of times and it is really bothering me but I am not sure which direction to go to ...
2
votes
1answer
167 views

Pandas nested loop which checks if values from one series are in another series

I have two different series in pandas that I have created a nested for loop which checks if the values of the first series are in the other series. But this is time consuming in pandas and I cannot ...
7
votes
1answer
437 views

PANDAS DataFrame operations to analyze top Server Fault tags

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
1
vote
1answer
28 views

Applying a dataframe function to a pandas groupby object

I am trying to apply a function to each group in a pandas dataframe where the function requires access to the entire group (as opposed to just one row). For this I am iterating over each group in the ...
2
votes
0answers
37 views

Compare every combination of variables within the powerset

I have some survey data that represents individual's responses to multiple survey questions. There are about 10,000 people in my actual dataset, and each person answered 35 questions. From these 35 ...