Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
4
votes
1answer
67 views

Code for wireless communication work

I need some pointers on how I can speed up my code, as of now it is incredible slow for larger inputs. The way the code works is that the file Loc_Circle_50U.txt contains the true locations of 50 ...
1
vote
0answers
20 views

Improving loop through xml files ad add to DF

With this code, i loop through similar xml files and add them to a DataFrame. How can i make this code better so it is more pythonic? ...
1
vote
1answer
22 views

Finding matches in Excel files, adding to a Pandas Dataframe

I have a business requirement where I have to read excel files and match each cell with a given keyword list and find the matches and write it into a dataframe. I have written the code, but when the ...
3
votes
1answer
72 views

Building optimal portfolios for all combinations of stocks

Hi I recently wrote some code in python that does the following: 1.) Pulls stock closing data from yahoo finance for x number of stock 2.) finds all possible combinations of x stocks in groups of y ...
7
votes
2answers
93 views

Parsing Addresses with GeoPanda's GeoDataFrame

Long time reader of Code Review with my first question. I'm self-taught and suspect this code can be improved. The project is really important to me and the team and I could really use the help. I'm ...
6
votes
4answers
728 views

Basic function to convert Country name to ISO code using Pycountry

I am working on a dataset which contains a column with common country names. Task: To convert country name into standard ISO names I have written a basic function which converts country names into ...
4
votes
1answer
77 views

Scraping Data From Multiple URLs into Single Dataframe

I have a Class that: goes to a url grabs a link from that page, and a date (filing_date) navigates to the link, and writes the table from that page to a ...
1
vote
0answers
35 views
1
vote
0answers
15 views

ML Pipeline with OneHotEncoder and model

can you please provide some advice on my codes below? There is 7 categorical features and 1 categorical target. This is a classification problem and I'm using RandomForest to train the data. I have ...
2
votes
0answers
26 views

Merge two DataFrames with key over-write

I have implemented a custom merge of 2 pandas DataFrames, but only mutates the first. The new columns will be the union of the original columns and the new keys will be union of original keys. ...
2
votes
0answers
28 views

Sklearn Custom Imputer from prediction

I asked this question on Stackoverflow, but thought it might be a better fit here. I'm playing around with the titanic dataset, and wanted to create a custom imputer to fill in missing ...
2
votes
2answers
46 views

Any tips to refactor the following Pandas code?

I am doing a Pandas project on alcohol consumption. For your information, the dataset has the following columns: | Continent | Country | Beer | Spirit | Wine | The following is my code: ...
0
votes
0answers
33 views

Script for matching regex matches between two lists

gist of this project is that I have two datasets, 1. dfCore, which contains station_code and station_name 2. dfMetrix, which contains station_name the ...
2
votes
0answers
36 views

Trying to optimize code that creates a running tally of a team's seasonal stats and adds it to their boxscore

I'm trying to compile stats for college basketball games. I have two inputs: regular_season_boxscore_df: A DataFrame consiststing of regular season boxscores. It ...
3
votes
0answers
26 views

Get Asset Returns for multiple Tickers and Holding Periods

I've written some code that will take multiple stock tickers and calculate their returns over yearly, monthly, bi-weekly, and weekly holding periods. The results are written to an xlsx file. I'm ...
1
vote
1answer
38 views

Reduce the redundancy in my pandas highlight function

After a lot of browsing and experimenting I have managed to achieve to read excel file as a Dataframe, apply some style to it and export it to html. My main concern is that the highlight_max function ...
0
votes
1answer
40 views

Calculate the statistical mean of different files and save it into a CSV file

In this code I calculate the statistical functions of values from different files and save them into CSV files (one filer for each statistical function). ...
2
votes
1answer
39 views

Re-write custom feature encoding function python

I have a question regarding code quality and best practices. My task - to write a feature encoding function that will encode categorical labels, boolean labels as one-hot encoding, timestamps for ...
1
vote
1answer
48 views

Memory efficient way to merge multiple Pandas DataFrames [closed]

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote: ...
2
votes
1answer
46 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
2
votes
1answer
121 views

Forex Simulator in Python using pandas and matplotlib

Description: This would make a first part of a currency exchange simulator The first phase includes downloading currency historical data from histdata.com api, plotting given period and given time ...
7
votes
0answers
144 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
2
votes
2answers
75 views

Reduce repetition in Pandas DataFrame column assignments [closed]

Could the aggregate functions after using grouping functions like groupby() and resample() be looped for cleaner code ____DateTime.____ | Steps | day_segment 2017-10-25 00:00:00| 4 | night 2017-...
5
votes
2answers
130 views

Loop through group and difference lags

I have time-series data (t) that I want to loop through each t and take the difference of the previous five lags ...
2
votes
0answers
58 views

Pythonic way to cast a dictionary into a pd.DataFrame with two columns?

I have a dictionary where for each key, a single value is stored. Say import pandas as pd dd = {'Alice': 40, 'Bob': 50, 'Charlie': 35} Now, I want to ...
4
votes
0answers
12 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
1
vote
0answers
22 views

DataSet Scaler with pandas

This is a Python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
4
votes
1answer
142 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
2
votes
1answer
195 views

Pandas iteration over rows for features calculation

I have a pandas data frame and I want to calculate some features based on some short_window, long_window and ...
4
votes
1answer
74 views

Reading multiple csv files in a single dataframe

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now: ...
3
votes
0answers
77 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
2answers
55 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
4
votes
2answers
84 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
2answers
217 views

Pandas - 3 iterations per second on simple conditional

The function removes rows from a pandas df if that row doesn't have the value of important_1 inside of important_2. For example ...
4
votes
2answers
139 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
3
votes
1answer
313 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
8
votes
3answers
629 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
3
votes
1answer
75 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
2answers
245 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
0
votes
1answer
84 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
6
votes
1answer
141 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
2
votes
1answer
46 views

Faster way to concat dataframes with different columns vertically

I am trying to concat two huge dataframes (around 15GB) like the following: a_df: ...
0
votes
2answers
12k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
1
vote
2answers
707 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
3
votes
1answer
495 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
1answer
62 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
0
votes
0answers
79 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
2
votes
1answer
37 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...

1
2 3 4 5
8