Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
5
votes
1answer
48 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...
3
votes
1answer
85 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
0answers
30 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
1
vote
1answer
28 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
5
votes
1answer
73 views

Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)

I would like to aggregate rows of my dataframe together, following those rules: Rows are grouped by "Id", and sorted by anticipation One row can only be aggregated with rows next to it, with the ...
2
votes
1answer
41 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...
4
votes
0answers
55 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
2
votes
0answers
28 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing a code to automate some weekly reports. I had help over on overflow, My original question. I have a code that works for the most part, however there are few things that I just cant ...
1
vote
1answer
39 views

Elegant way to do the dataframe operations twice

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the ...
4
votes
2answers
96 views

Pulling financial data via IEX Cloud API

I am pulling some financial data from the IEX Cloud API. Since it is a paid API w/ rate limits, I would like to do this as efficiently as possible (i.e., as few calls as possible). Given that the <...
4
votes
1answer
45 views

Get minimum of each group based on hour criteria using pandas

I have a requirement which I posted in SO. However, I have already written code and works fine. I request your review to know whether it is fool proof/elegant and efficient. Please find the details ...
5
votes
2answers
58 views

A/B testing using chi-square to calculate the significance in an elegant way

The definition of ABtest Goal: For input data, each pair row is a group such as 0 and 1, 2 and 3. I want to test whether each group is significant using chi-square. If result is significant and ...
2
votes
1answer
57 views

Loop to extract json from dataframe and storing in a new dataframe

I have a dataframe (obtained from a csv saved from mysql) with several columns, and one of them consist of a string which is the representation of a json. The data looks like: ...
6
votes
1answer
66 views

Graphing refugee movements using plotly in Python

I've created a simple data visualization in python that shows the movement of refugees around the world. The code works but I feel it is a bit slow. it takes about 10 s (on my hardware at least) to ...
1
vote
0answers
27 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...
0
votes
0answers
29 views

Search a better way to work with pandas dataframe when i use lamba and apply

I have this code to take prices from a database into pandas, and what i'm doing here is group my prices based on itemnumber and price group, and then i want to push it to a dataframe where i use apply....
4
votes
2answers
61 views

Ways to differentiate comments in source code (PyCharm)

I want to differentiate between comments I make for (1) myself, (2) team members, (3) on business rules, (4) on watchout notes for hardcoded constants, (5) on ideas for improvements, and (6) on ...
2
votes
1answer
26 views

Optimal way to filter DataFrame by many arguments, without usage of multiple ifs and for loops

I am learning about survival analysis and I made some implementation in Python using lifelines and plotly. However depending on ...
3
votes
2answers
81 views

Checking whether customers in a dataframe have social media accounts

I am using an API to check if customers have social media profiles. Depending on whether they have phone or email or both there's a different search type. There's a lot of conditions in this ...
1
vote
0answers
37 views

is there way to make faster to write to a file?

...
3
votes
2answers
88 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
0
votes
0answers
19 views

Grouping by on Pandas df with focus on performance

I am doing a group by on a subset of a dataframe. I am working with 9m+ rows atm, subject to sharp increase. I tried different approaches already and wanted to know if i could squeeze a bit more ...
3
votes
1answer
67 views

SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
2
votes
1answer
69 views

Pandas updating/adding columns to rows incrementally using dictionary key values

I want to populate columns of a dataframe (df) by iteratively looping over a list (A_list) generating a dictionary where the keys are the names of the desired columns of df (in the example below the ...
1
vote
0answers
29 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
4
votes
1answer
79 views

Optimising iteration and substitution over large dataset

I have the following code: ...
0
votes
1answer
36 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
2
votes
1answer
87 views

Parsing data from JSON API in Pandas

I made a little program to parse data from an API. I do not have experience with Pandas. It is working but I would like to know how to do that better and more efficiently. ...
1
vote
2answers
100 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
1
vote
1answer
31 views

tablename alias in attribute name

I have dataframe like this: ...
1
vote
1answer
56 views

Efficient way to get value from a dataframe and append new dataframe

I have a dataframe that have about 200 million rows. the example of dataframe is like this: ...
1
vote
1answer
22 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
5
votes
1answer
122 views

Building a frequency dictionary from a Pandas dataframe

I need to make a frequency dictionary from a pandas series (from the 'amino_acid' column in dataframe below) that also adds an adjacent row for each entry in the dictionary (from 'templates' column). ...
1
vote
0answers
34 views

Sortings stocks into quantiles based on their signal

This function sorts stocks into quantiles. To do so I used this function that accepts sig_df (dataframe with the timeseries of stocks signal) and number of ...
3
votes
0answers
79 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
6
votes
1answer
62 views

Get nearest major city of an image (using EXIF GPS data)

I am writing a program that will take images in a folder, get the GPS Coordinates (from the EXIF data), then return the closest "major city". (I define a major city as one with an airport, so ...
4
votes
2answers
127 views

Groupby count, then sum and get the percentage

I wrote this code. It works, but I think there is a more elegant and Pythonic way to this task. Groupby and count the different occurences Get the sum of all the occurences Divide each occurrence ...
3
votes
1answer
128 views

Calculate percentage based on dummy variable in Pandas

Today I had a simple task: I have products, quantities and a dummy, and I needed to know which percentage of the total quantities of that product the dummy represented. My DataFrame looked like this: ...
1
vote
2answers
52 views

Function to count words in Python

I am new to developing in Python and would like to see comments about my word count solution. Please indicate my mistakes or best solutions. ...
4
votes
2answers
59 views

Grouping a Pandas Dataframe by two parameters

I have a .csv file of 8k+ rows which looks like this: ...
4
votes
0answers
33 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
2
votes
0answers
40 views

Faster way to replace strings in pandas series using dictionary in Python

I'm working with a dictionary with 11830 keys/values, that have to loop over 13905 strings, that can have up to 50 words each. My series looks like this: ...
2
votes
1answer
33 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
3
votes
1answer
42 views

Subject group creation/selection using multiple filter tags derived from multiple databases

This is part of my research project with the main objective to select patients from various characteristics that are modeled as filters. The main filters include ...
1
vote
1answer
102 views

Comparing columns in pandas different data frames and fill in a new column

I have two dataframes: One contains of company and its corresponding texts. The texts are in lists ...
1
vote
0answers
30 views

Flagging out-of-stock items

I wrote a function that iterates over 3 parameters to spot out-of-stock items. To illustrate this I'll draw an example: suppose that the product 10 in store 2 for an offer named "super5" is following ...
3
votes
0answers
26 views

TensorFlow regressor to predict probability of winning, given a current score

This is a TensorFlow regressor that tells me that if you get x score, you will have x% to win a game in x game(game is not important). Doing this as a project to learn tensorflow. ...
1
vote
1answer
47 views

Efficient way to add a colon before and after every word in strings inside a Series

I was working with twitter data, and extracted every emoji from the data. But, when I passed that data through CountVectorizer, colons where substracted from the ...
2
votes
1answer
74 views

Analyzing patient treatment data using Pandas

I work in the population health industry and get contracts from commercial companies to conduct research on their products. This is the general code to identify target patient groups from a provincial ...
4
votes
1answer
35 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...