Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
-2
votes
0answers
20 views

Dataframe containing value 0 should be removed or replaced? [migrated]

I have a question that mostly I get stuck at. I was looking at the data for diabetes patients and found that most of the rows have 0 values under most of their columns. Reference for the url https://...
6
votes
0answers
78 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
2
votes
0answers
39 views

A python script to plot a joint probability mass function table

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
5
votes
1answer
70 views

Fetch, plot, and send security incident statistics with Excel

TLDR: I don't develop for a living. If someone could point me in the right direction to make my script more readable/pythonic I would appreciate the assistance. In particular I'm struggling with the ...
3
votes
1answer
45 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing a code to automate some weekly reports. I had help over on overflow, My original question. I have a code that works for the most part, however there are few things that I just cant ...
2
votes
1answer
40 views

K_nearest_neighbors from scratch [on hold]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
8
votes
1answer
1k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
6
votes
1answer
2k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
5
votes
1answer
51 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...
3
votes
1answer
86 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
0answers
30 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
1
vote
1answer
28 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
5
votes
1answer
74 views

Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)

I would like to aggregate rows of my dataframe together, following those rules: Rows are grouped by "Id", and sorted by anticipation One row can only be aggregated with rows next to it, with the ...
4
votes
0answers
55 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
2
votes
1answer
42 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...
6
votes
2answers
89 views

Comparing the size of tumors over time using PANDAS

I have a dataframe in pandas like this ...
4
votes
2answers
118 views

Pulling financial data via IEX Cloud API

I am pulling some financial data from the IEX Cloud API. Since it is a paid API w/ rate limits, I would like to do this as efficiently as possible (i.e., as few calls as possible). Given that the <...
5
votes
2answers
60 views

A/B testing using chi-square to calculate the significance in an elegant way

The definition of ABtest Goal: For input data, each pair row is a group such as 0 and 1, 2 and 3. I want to test whether each group is significant using chi-square. If result is significant and ...
1
vote
1answer
40 views

Elegant way to do the dataframe operations twice

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the ...
4
votes
1answer
46 views

Get minimum of each group based on hour criteria using pandas

I have a requirement which I posted in SO. However, I have already written code and works fine. I request your review to know whether it is fool proof/elegant and efficient. Please find the details ...
2
votes
1answer
58 views

Loop to extract json from dataframe and storing in a new dataframe

I have a dataframe (obtained from a csv saved from mysql) with several columns, and one of them consist of a string which is the representation of a json. The data looks like: ...
3
votes
1answer
136 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
3
votes
1answer
74 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
2answers
90 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
6
votes
1answer
68 views

Graphing refugee movements using plotly in Python

I've created a simple data visualization in python that shows the movement of refugees around the world. The code works but I feel it is a bit slow. it takes about 10 s (on my hardware at least) to ...
5
votes
1answer
766 views

KNN pipeline w/ cross_validation_scores

Using the wine quality dataset, I'm attempting to perform a simple KNN classification (w/ a scaler, and the classifier in a pipeline). It works, but I've never used ...
1
vote
0answers
28 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...
0
votes
0answers
29 views

Search a better way to work with pandas dataframe when i use lamba and apply

I have this code to take prices from a database into pandas, and what i'm doing here is group my prices based on itemnumber and price group, and then i want to push it to a dataframe where i use apply....
6
votes
3answers
8k views

Gradient descent for linear regression using numpy/pandas

I currently follow along Andrew Ng's Machine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using ...
1
vote
1answer
5k views
2
votes
1answer
422 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
4
votes
2answers
61 views

Ways to differentiate comments in source code (PyCharm)

I want to differentiate between comments I make for (1) myself, (2) team members, (3) on business rules, (4) on watchout notes for hardcoded constants, (5) on ideas for improvements, and (6) on ...
7
votes
1answer
448 views

PANDAS DataFrame operations to analyze top Server Fault tags [closed]

I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global ...
5
votes
2answers
21k views

Excel's SUMIFS implemented using PANDAS, the Python Data Analysis Library

I've implemented Excel's SUMIFS function in Pandas using the following code. Is there a better — more Pythonic — implementation? ...
2
votes
1answer
26 views

Optimal way to filter DataFrame by many arguments, without usage of multiple ifs and for loops

I am learning about survival analysis and I made some implementation in Python using lifelines and plotly. However depending on ...
3
votes
2answers
81 views

Checking whether customers in a dataframe have social media accounts

I am using an API to check if customers have social media profiles. Depending on whether they have phone or email or both there's a different search type. There's a lot of conditions in this ...
1
vote
1answer
22 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
0
votes
0answers
19 views

Grouping by on Pandas df with focus on performance

I am doing a group by on a subset of a dataframe. I am working with 9m+ rows atm, subject to sharp increase. I tried different approaches already and wanted to know if i could squeeze a bit more ...
3
votes
1answer
69 views

SQL queries in a dataframe

I want to get table names and column names from queries in a dataframe. The dataframe is like this: ...
2
votes
1answer
73 views

Pandas updating/adding columns to rows incrementally using dictionary key values

I want to populate columns of a dataframe (df) by iteratively looping over a list (A_list) generating a dictionary where the keys are the names of the desired columns of df (in the example below the ...
7
votes
1answer
198 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
1
vote
0answers
29 views

Parsing and plotting with numpy (or pandas)

I am trying to improve the following code to make it more broadly applicable and quick to use. ...
1
vote
1answer
31 views
0
votes
1answer
36 views

Code scrapes, queries against a database and intersects

The code base is split between three different files odbc.py, scrape.py, and dataprocessor.py. For ODBC, the job of this file is to take scraped data and determine whether or not those results are in ...
2
votes
1answer
88 views

Parsing data from JSON API in Pandas

I made a little program to parse data from an API. I do not have experience with Pandas. It is working but I would like to know how to do that better and more efficiently. ...
1
vote
2answers
110 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
1
vote
1answer
56 views

Efficient way to get value from a dataframe and append new dataframe

I have a dataframe that have about 200 million rows. the example of dataframe is like this: ...
4
votes
2answers
143 views

Groupby count, then sum and get the percentage

I wrote this code. It works, but I think there is a more elegant and Pythonic way to this task. Groupby and count the different occurences Get the sum of all the occurences Divide each occurrence ...