Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
-1
votes
0answers
11 views

python pandas iterating rows of two different columns and returning the repeated one once and corresponding values of repeated values in single row [closed]

for instance, I have a .csv file with 1000s of rows like below: [column 1] [column 2] 1992 Alex 1992 Anna 1993 Max 1993 Bob 1993 Tom . . . ...
2
votes
0answers
20 views

Sklearn Custom Imputer from prediction

I asked this question on Stackoverflow, but thought it might be a better fit here. I'm playing around with the titanic dataset, and wanted to create a custom imputer to fill in missing ...
0
votes
0answers
27 views

Script for matching regex matches between two lists

gist of this project is that I have two datasets, 1. dfCore, which contains station_code and station_name 2. dfMetrix, which contains station_name the ...
2
votes
2answers
39 views

Any tips to refactor the following Pandas code?

I am doing a Pandas project on alcohol consumption. For your information, the dataset has the following columns: | Continent | Country | Beer | Spirit | Wine | The following is my code: ...
2
votes
0answers
31 views

Trying to optimize code that creates a running tally of a team's seasonal stats and adds it to their boxscore

I'm trying to compile stats for college basketball games. I have two inputs: regular_season_boxscore_df: A DataFrame consiststing of regular season boxscores. It ...
3
votes
0answers
23 views

Get Asset Returns for multiple Tickers and Holding Periods

I've written some code that will take multiple stock tickers and calculate their returns over yearly, monthly, bi-weekly, and weekly holding periods. The results are written to an xlsx file. I'm ...
1
vote
1answer
35 views

Reduce the redundancy in my pandas highlight function

After a lot of browsing and experimenting I have managed to achieve to read excel file as a Dataframe, apply some style to it and export it to html. My main concern is that the highlight_max function ...
2
votes
1answer
37 views

Re-write custom feature encoding function python

I have a question regarding code quality and best practices. My task - to write a feature encoding function that will encode categorical labels, boolean labels as one-hot encoding, timestamps for ...
0
votes
1answer
40 views

Calculate the statistical mean of different files and save it into a CSV file

In this code I calculate the statistical functions of values from different files and save them into CSV files (one filer for each statistical function). ...
1
vote
1answer
44 views

Memory efficient way to merge multiple Pandas DataFrames [closed]

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote: ...
5
votes
2answers
127 views

Loop through group and difference lags

I have time-series data (t) that I want to loop through each t and take the difference of the previous five lags ...
2
votes
0answers
57 views

Pythonic way to cast a dictionary into a pd.DataFrame with two columns?

I have a dictionary where for each key, a single value is stored. Say import pandas as pd dd = {'Alice': 40, 'Bob': 50, 'Charlie': 35} Now, I want to ...
2
votes
2answers
74 views

Reduce repetition in Pandas DataFrame column assignments [closed]

Could the aggregate functions after using grouping functions like groupby() and resample() be looped for cleaner code ____DateTime.____ | Steps | day_segment 2017-10-25 00:00:00| 4 | night 2017-...
4
votes
0answers
11 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
4
votes
1answer
64 views

Reading multiple csv files in a single dataframe

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now: ...
7
votes
0answers
142 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
3
votes
0answers
72 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
2answers
53 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
4
votes
2answers
82 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
2answers
215 views

Pandas - 3 iterations per second on simple conditional

The function removes rows from a pandas df if that row doesn't have the value of important_1 inside of important_2. For example ...
2
votes
1answer
106 views

Forex Simulator in Python using pandas and matplotlib

Description: This would make a first part of a currency exchange simulator The first phase includes downloading currency historical data from histdata.com api, plotting given period and given time ...
1
vote
0answers
22 views

DataSet Scaler with pandas

This is a Python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
4
votes
2answers
95 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
8
votes
3answers
625 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
0
votes
1answer
84 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
2
votes
1answer
45 views

Faster way to concat dataframes with different columns vertically

I am trying to concat two huge dataframes (around 15GB) like the following: a_df: ...
2
votes
1answer
184 views

Pandas iteration over rows for features calculation

I have a pandas data frame and I want to calculate some features based on some short_window, long_window and ...
6
votes
1answer
135 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
2
votes
1answer
61 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
0
votes
0answers
73 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
2
votes
1answer
46 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
5
votes
1answer
96 views

Remove words from an exclude list in each row of a pandas.Series

I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach. ...
5
votes
1answer
55 views

Average row id volume

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's ...
3
votes
1answer
58 views

Map price values to bins where bins are of a larger price range

I have two different pandas data frames where in the first data frame (price), I have two columns. The first column named value has some values in it and the second column amount has the available ...
4
votes
0answers
35 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
2
votes
0answers
29 views

Python Wrapper for Dynamic SQL

...
4
votes
1answer
57 views

Load excel sheet to data frame

I have a bare-bones example of how I plan to load excel sheets into pandas data frames. However, the code runs unexpectedly slow. I'm open to suggestions on ways I can speed this up, even converting ...
2
votes
2answers
54 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
6
votes
0answers
146 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
1
vote
0answers
77 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
5
votes
1answer
77 views

Fetch, plot, and send security incident statistics with Excel

TLDR: I don't develop for a living. If someone could point me in the right direction to make my script more readable/pythonic I would appreciate the assistance. In particular I'm struggling with the ...
2
votes
1answer
50 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
5
votes
1answer
145 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...
3
votes
1answer
96 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
0answers
30 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
0
votes
1answer
35 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
5
votes
1answer
85 views

Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)

I would like to aggregate rows of my dataframe together, following those rules: Rows are grouped by "Id", and sorted by anticipation One row can only be aggregated with rows next to it, with the ...
2
votes
1answer
55 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...

1 2 3 4 5 8