Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
0
votes
1answer
40 views

Calculate the statistical mean of different files and save it into a CSV file

In this code I calculate the statistical functions of values from different files and save them into CSV files (one filer for each statistical function). ...
2
votes
1answer
34 views

Re-write custom feature encoding function python

I have a question regarding code quality and best practices. My task - to write a feature encoding function that will encode categorical labels, boolean labels as one-hot encoding, timestamps for ...
1
vote
1answer
43 views

Memory efficient way to merge multiple Pandas DataFrames

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote: ...
2
votes
1answer
46 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
2
votes
1answer
101 views

Forex Simulator in Python using pandas and matplotlib

Description: This would make a first part of a currency exchange simulator The first phase includes downloading currency historical data from histdata.com api, plotting given period and given time ...
7
votes
0answers
142 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
2
votes
2answers
72 views

Reduce repetition in Pandas DataFrame column assignments [closed]

Could the aggregate functions after using grouping functions like groupby() and resample() be looped for cleaner code ____DateTime.____ | Steps | day_segment 2017-10-25 00:00:00| 4 | night 2017-...
5
votes
2answers
125 views

Loop through group and difference lags

I have time-series data (t) that I want to loop through each t and take the difference of the previous five lags ...
2
votes
0answers
57 views

Pythonic way to cast a dictionary into a pd.DataFrame with two columns?

I have a dictionary where for each key, a single value is stored. Say import pandas as pd dd = {'Alice': 40, 'Bob': 50, 'Charlie': 35} Now, I want to ...
4
votes
0answers
11 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
1
vote
0answers
22 views

DataSet Scaler with pandas

This is a Python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
4
votes
1answer
127 views

Optimising Probabilistic Weighted Moving Average (PEWMA) df.iterrows loop in Pandas

I'm implementing the Probabilistic Exponentially Weighted Mean for real time prediction of sensor data in pandas but have issues with optimising the pandas notebook for quick iterations. Is there a ...
2
votes
1answer
174 views

Pandas iteration over rows for features calculation

I have a pandas data frame and I want to calculate some features based on some short_window, long_window and ...
4
votes
1answer
63 views

Reading multiple csv files in a single dataframe

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now: ...
3
votes
0answers
71 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
2answers
51 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
4
votes
2answers
79 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
2answers
213 views

Pandas - 3 iterations per second on simple conditional

The function removes rows from a pandas df if that row doesn't have the value of important_1 inside of important_2. For example ...
4
votes
2answers
95 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
3
votes
1answer
272 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
8
votes
3answers
623 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
3
votes
1answer
75 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
2answers
198 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
0
votes
1answer
84 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
6
votes
1answer
135 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
2
votes
1answer
45 views

Faster way to concat dataframes with different columns vertically

I am trying to concat two huge dataframes (around 15GB) like the following: a_df: ...
0
votes
2answers
10k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
1
vote
2answers
514 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
3
votes
1answer
475 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
1answer
61 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
0
votes
0answers
72 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
2
votes
1answer
35 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
5
votes
1answer
90 views

Remove words from an exclude list in each row of a pandas.Series

I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach. ...
5
votes
1answer
54 views

Average row id volume

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's ...
4
votes
1answer
81 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing code to automate some weekly reports. I had help over on Stack Overflow. I have code that works for the most part, however there are a few things that I just can't seem to fix. In ...
3
votes
1answer
55 views

Map price values to bins where bins are of a larger price range

I have two different pandas data frames where in the first data frame (price), I have two columns. The first column named value has some values in it and the second column amount has the available ...
4
votes
0answers
35 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
4
votes
1answer
56 views

Load excel sheet to data frame

I have a bare-bones example of how I plan to load excel sheets into pandas data frames. However, the code runs unexpectedly slow. I'm open to suggestions on ways I can speed this up, even converting ...
2
votes
2answers
54 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
7
votes
4answers
24k views

Extracting contents of dictionary contained in Pandas dataframe to make new dataframe columns

I created a Pandas dataframe from a MongoDB query. c = db.runs.find().limit(limit) df = pd.DataFrame(list(c)) Right now one column of the dataframe corresponds ...
6
votes
0answers
145 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
1
vote
0answers
73 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
5
votes
1answer
77 views

Fetch, plot, and send security incident statistics with Excel

TLDR: I don't develop for a living. If someone could point me in the right direction to make my script more readable/pythonic I would appreciate the assistance. In particular I'm struggling with the ...
2
votes
1answer
48 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
8
votes
1answer
2k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
6
votes
1answer
2k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
5
votes
1answer
134 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...

1 2 3 4 5 8