Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
2
votes
1answer
168 views

Pandas iteration over rows for features calculation

I have a pandas data frame and I want to calculate some features based on some short_window, long_window and ...
7
votes
0answers
123 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
3
votes
1answer
49 views

Reading multiple csv files in a single dataframe

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now: ...
3
votes
0answers
32 views

Order of columns in XML to CSV Parsing

Code takes a directory where multiple XML files are located and parses them into a csv file. ...
3
votes
0answers
67 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
2answers
50 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
4
votes
2answers
74 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
2answers
207 views

Pandas - 3 iterations per second on simple conditional

The function removes rows from a pandas df if that row doesn't have the value of important_1 inside of important_2. For example ...
0
votes
1answer
52 views

Forex Simulator in Python using pandas and matplotlib

Description: This would make a first part of a currency exchange simulator written in Python. The first phase includes downloading currency historical data from histdata.com api, plotting given ...
-1
votes
0answers
38 views

Financial Neural Network [closed]

This is a financial stock predicting neural network, and I would like to get a code review for it because I'm not so good at PEP8 code styling conventions. Thanks in advance! ...
1
vote
0answers
18 views

DataSet Scaler with pandas

This is a python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
4
votes
2answers
86 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
3
votes
1answer
235 views

Producing a Pandas Data Frame from Long to Wide format Efficiently

I have a time series data in long format which looks like as follows: ...
8
votes
3answers
621 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
3
votes
1answer
75 views

Separate scrap and production by area

I'm looking for feedback on things that can be streamlined in each section. The data it creates is all needed whether it's used in a graph or not so I can't get rid of anything, just combine steps ...
3
votes
2answers
162 views

Web-scraping investment fund data into Pandas

Just starting out trying to do some python for data analysis. I am a total beginner and have figure out how to brute force somethings, but I know it is inefficient but don't know how to do it without ...
0
votes
1answer
72 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
6
votes
1answer
134 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
2
votes
1answer
43 views

Faster way to concat dataframes with different columns vertically

I am trying to concat two huge dataframes (around 15GB) like the following: a_df: ...
0
votes
2answers
9k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
1
vote
2answers
401 views

Parse date format in Pandas using Python

I have a column in a Pandas Dataframe containing birth dates in object/string format: 0 16MAR39 1 21JAN56 2 18NOV51 3 05MAR64 4 05JUN48 I want to ...
3
votes
1answer
457 views

Subset pandas DataFrame based on two columns ignoring what order the match happens in

I have two Pandas DataFrames and I want to subset df_all based on the values within to_keep. Unfortunately this isn't straight ...
2
votes
1answer
60 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
0
votes
0answers
65 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
2
votes
1answer
34 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
2
votes
1answer
42 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
5
votes
1answer
73 views

Remove words from an exclude list in each row of a pandas.Series

I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach. ...
5
votes
1answer
53 views

Average row id volume

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's ...
4
votes
1answer
80 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing code to automate some weekly reports. I had help over on Stack Overflow. I have code that works for the most part, however there are a few things that I just can't seem to fix. In ...
3
votes
1answer
52 views

Map price values to bins where bins are of a larger price range

I have two different pandas data frames where in the first data frame (price), I have two columns. The first column named value has some values in it and the second column amount has the available ...
4
votes
0answers
35 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
4
votes
1answer
53 views

Load excel sheet to data frame

I have a bare-bones example of how I plan to load excel sheets into pandas data frames. However, the code runs unexpectedly slow. I'm open to suggestions on ways I can speed this up, even converting ...
2
votes
2answers
51 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
7
votes
4answers
23k views

Extracting contents of dictionary contained in Pandas dataframe to make new dataframe columns

I created a Pandas dataframe from a MongoDB query. c = db.runs.find().limit(limit) df = pd.DataFrame(list(c)) Right now one column of the dataframe corresponds ...
6
votes
0answers
139 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
1
vote
0answers
66 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
5
votes
1answer
77 views

Fetch, plot, and send security incident statistics with Excel

TLDR: I don't develop for a living. If someone could point me in the right direction to make my script more readable/pythonic I would appreciate the assistance. In particular I'm struggling with the ...
2
votes
1answer
47 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
8
votes
1answer
2k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
6
votes
1answer
2k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
5
votes
1answer
94 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...
3
votes
1answer
94 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
0answers
30 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
0
votes
1answer
35 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
5
votes
1answer
81 views

Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)

I would like to aggregate rows of my dataframe together, following those rules: Rows are grouped by "Id", and sorted by anticipation One row can only be aggregated with rows next to it, with the ...
4
votes
0answers
68 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
2
votes
1answer
52 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...