Questions tagged [pandas]

Pandas is a Python data analysis library.

Filter by
Sorted by
Tagged with
1
vote
1answer
57 views

Memory efficient way to merge multiple Pandas DataFrames [closed]

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote: ...
5
votes
2answers
131 views

Loop through group and difference lags

I have time-series data (t) that I want to loop through each t and take the difference of the previous five lags ...
2
votes
0answers
60 views

Pythonic way to cast a dictionary into a pd.DataFrame with two columns?

I have a dictionary where for each key, a single value is stored. Say import pandas as pd dd = {'Alice': 40, 'Bob': 50, 'Charlie': 35} Now, I want to ...
2
votes
2answers
77 views

Reduce repetition in Pandas DataFrame column assignments [closed]

Could the aggregate functions after using grouping functions like groupby() and resample() be looped for cleaner code ____DateTime.____ | Steps | day_segment 2017-10-25 00:00:00| 4 | night 2017-...
4
votes
0answers
17 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
4
votes
1answer
194 views

Reading multiple csv files in a single dataframe

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now: ...
7
votes
0answers
146 views

Multithreaded HD Image Processing + Logistic reg. Classifier + Visualization

[I'm awaiting suggestions for improvement/optimization/more speed/general feedback ...] This code takes a label and a folder path of subfolders as input that have certain labels ex: trees, cats with ...
3
votes
0answers
83 views

Compare historical data

I have the following function that is supposed to compare current and historical transactions for each id. The logic seems to work ok, but the performance is extremely low for the amount of data I'm ...
3
votes
2answers
64 views

Convert rows with multiple values to row with single value

I have a dataframe with measurement data of different runs at same conditions. Each row contains both the constant conditions the experiment was conducted and all the results from the different runs. ...
4
votes
2answers
97 views

First program in Python for data analysis

I'm just starting my adventure with python and I wanted to share my first project with you and ask for feedback and advice. This is a script for my friend to automate all calculations and database ...
4
votes
2answers
220 views

Pandas - 3 iterations per second on simple conditional

The function removes rows from a pandas df if that row doesn't have the value of important_1 inside of important_2. For example ...
2
votes
1answer
148 views

Forex Simulator in Python using pandas and matplotlib

Description: This would make a first part of a currency exchange simulator The first phase includes downloading currency historical data from histdata.com api, plotting given period and given time ...
1
vote
0answers
25 views

DataSet Scaler with pandas

This is a Python program that takes data from Quandl Databases and formats it for my Neural Network to Train. The code works but I'm not so confident in my code styling conventions. If you would like ...
4
votes
2answers
237 views

Xml to Csv Python Code too long

The code takes a very simple xml and converts it into a dataframe in csv file. This is the Xml file ...
8
votes
3answers
632 views

Modifying Titration Data analysis results

This is my first script that I've written. As a result, I'm sure there are extra lines that are unneeded, or maybe better more concise ways of doing things than I have done here. I have tried to add ...
0
votes
1answer
86 views

Try and except clauses for appending elements in lists

I have a list that can have up to three elements, and I want to append them to three list, because I get this list from another element within a loop. My final goal is to have a dataframe with the ...
2
votes
1answer
57 views

Faster way to concat dataframes with different columns vertically

I am trying to concat two huge dataframes (around 15GB) like the following: a_df: ...
2
votes
1answer
210 views

Pandas iteration over rows for features calculation

I have a pandas data frame and I want to calculate some features based on some short_window, long_window and ...
6
votes
1answer
145 views

Slow parsing of mutiple XML files to CSV

The code below is written as part of a parser, which files would be also added here: XMLParser, parser, Name, GenericElement, attrib It reads an XML files from a folder, turn them into a dataframe ...
2
votes
1answer
62 views

Finding the market price for each order at the time it was placed

I have two datasets. The first dataset “insideMarke1” contains the Best Bid / Best Ask (=BBA) data for a given day, starting at 8.00 am. Every time the BBA is updated, there is a new row with a new ...
0
votes
0answers
88 views

Parsing large JSON files

I have the following code for reading a json file into pandas dataframe and parsing the fields, but it is too slow for large files. Is there a way to make the code more efficient? ...
2
votes
1answer
52 views

Python - Automate code involving suffix [closed]

I am trying to automate this portion of code in python (code shown for 2 polygons, I am trying to do the same for 10 000 polygons). Any ideas on how to implement this using ...
2
votes
1answer
47 views

Optimizing a TFxIDF vectorization - Python 3.x [closed]

I am attempting to train many classifiers to test their performance with classifying tweets as being from a political bot, or not (a binary 0 or 1 classifier). My ...
5
votes
1answer
308 views

Remove words from an exclude list in each row of a pandas.Series

I am trying to remove all occurrences of the words I collect in the list exclude from every text note. This is my approach. ...
5
votes
1answer
56 views

Average row id volume

After a test run with LoadRunner I wanted to know the average time for each transaction for each VU volume step, the LR let you create a graph for this but don't give you the point values... so it's ...
3
votes
1answer
88 views

Map price values to bins where bins are of a larger price range

I have two different pandas data frames where in the first data frame (price), I have two columns. The first column named value has some values in it and the second column amount has the available ...
4
votes
0answers
36 views

Insert multiple columns in specific location based on two conditions in other columns

need to rewrite some code I already have. It insert columns where a 'Standard Deviation' column is missing. To do that it checks if the preceding column is a "variable" column, and if there already is ...
2
votes
0answers
37 views

Python Wrapper for Dynamic SQL

...
4
votes
1answer
59 views

Load excel sheet to data frame

I have a bare-bones example of how I plan to load excel sheets into pandas data frames. However, the code runs unexpectedly slow. I'm open to suggestions on ways I can speed this up, even converting ...
2
votes
2answers
58 views

Standardizes Rows of a Pandas.DataFrame Object

Please take a look at this little snippet of code and explicate on whether there are an efficiency enhancements that you'd make for it. It standards a row of a pandas.DataFrame by adding zeros to it ...
6
votes
0answers
169 views

How to make my groupby and transpose operations efficient?

I have a DataFrame of size 3,745,802 rows and 30 columns. I would like to perform certain groupby and ...
1
vote
0answers
115 views

A python script to plot a joint probability mass function table [closed]

Consider this example Suppose that 3 balls are randomly selected from an urn containing 3 red, 4 white, and 5 blue balls. If we let X and Y denote, respectively, the number of red and white ...
5
votes
1answer
77 views

Fetch, plot, and send security incident statistics with Excel

TLDR: I don't develop for a living. If someone could point me in the right direction to make my script more readable/pythonic I would appreciate the assistance. In particular I'm struggling with the ...
2
votes
1answer
51 views

K_nearest_neighbors from scratch [closed]

I wanted to create a script that will perform the k_nearest_neighbors algorithm on the well-known iris dataset. This was mainly for me to better understand the algorithm and process. I think it works, ...
5
votes
1answer
282 views

GMAIL API -Create/Send/Search messages for keyword

I have written below code using GMAIL API reference which creates a basic message, sends it and also lists all the messages from user's mailbox matching a keyword.It works fine. Looking for experts ...
3
votes
1answer
98 views

How do I efficiently create a dataframe from a list of dictionaries?

I used this API to download some data. The code is supposed to create a dataframe table using specific fields of an object, and it seems to work, but running slow. I think it would run faster if I ...
3
votes
0answers
31 views

Python Dataframe column value updates in for loop [closed]

I have a dataset on which I need to do entity resolution. Hence in this dataset some records have links some records have address. I need to build a logic for it to identify each record and put the ...
0
votes
1answer
35 views

Pulling data from multiple excel files in Python [closed]

I understand this code is running properly and is grabbing from each excel file individually, but I want to create a for loop for using this data across multiple excel files ...
5
votes
1answer
90 views

Looking for performance improvement of my custom dataframe aggregation function (Python/Pandas)

I would like to aggregate rows of my dataframe together, following those rules: Rows are grouped by "Id", and sorted by anticipation One row can only be aggregated with rows next to it, with the ...
2
votes
1answer
81 views

Changes all floats in a pandas DataFrame to string

Below I created a function to format all the floats in a pandas DataFrame to a specific precision (6 d.p) and convert to string for output to a GUI (hence why I didn't just change the pandas display ...
4
votes
0answers
79 views

Data profiling in a dataframe based on an input file-pandas

I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. ...
4
votes
1answer
84 views

Automating a set a of weekly reports, including graphs and delivery of reports

I have been writing code to automate some weekly reports. I had help over on Stack Overflow. I have code that works for the most part, however there are a few things that I just can't seem to fix. In ...
1
vote
1answer
46 views

Elegant way to do the dataframe operations twice

I am trying to do certain operations in dataframe as shown in my code below. Though this works fine.However I am repeating the same statements twice or following a dirty approach to perform the ...
4
votes
2answers
1k views

Pulling financial data via IEX Cloud API

I am pulling some financial data from the IEX Cloud API. Since it is a paid API w/ rate limits, I would like to do this as efficiently as possible (i.e., as few calls as possible). Given that the <...
4
votes
1answer
52 views

Get minimum of each group based on hour criteria using pandas

I have a requirement which I posted in SO. However, I have already written code and works fine. I request your review to know whether it is fool proof/elegant and efficient. Please find the details ...
5
votes
2answers
68 views

A/B testing using chi-square to calculate the significance in an elegant way

The definition of ABtest Goal: For input data, each pair row is a group such as 0 and 1, 2 and 3. I want to test whether each group is significant using chi-square. If result is significant and ...
2
votes
1answer
131 views

Loop to extract json from dataframe and storing in a new dataframe

I have a dataframe (obtained from a csv saved from mysql) with several columns, and one of them consist of a string which is the representation of a json. The data looks like: ...
6
votes
1answer
108 views

Graphing refugee movements using plotly in Python

I've created a simple data visualization in python that shows the movement of refugees around the world. The code works but I feel it is a bit slow. it takes about 10 s (on my hardware at least) to ...
1
vote
0answers
152 views

Extract log file and convert into panadas dataframe [closed]

I am getting log file which contain other information along with leakage information into following format Log_file ...

1
2
3 4 5
9