Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pandas]

Pandas is a Python data analysis library.

2
votes
1answer
50 views

Finding recessions in US GDP data using pandas

For an assignment, I am identifying the first quarter of the 2008 recession in the United States. The Excel data I'm using can be downloaded here: gdplev.xls. How can I improve this pandas code to ...
1
vote
0answers
12 views

how to take the linspace of values between 2 columns in pandas [on hold]

I am trying to take the linspace of between 2 columns of a dataset. Heres an example. a b C D 0 3 13 3 13 1 4 14 4 14 2 5 15 5 15 ...
1
vote
0answers
45 views

Optimize Application of Python Gensim Search Functions

I have a DataFrame that has a text column. I am splitting the DataFrame into two parts based on the value in another column. One of those parts is indexed into a gensim similarity model. The other ...
4
votes
1answer
51 views

NFL data scraper

I made a Python 3 class that scrapes data from Pro Football Reference. It uses requests and beautifulsoup4 to gather the data ...
0
votes
0answers
25 views

Support Vector Machine Classifier [closed]

I have the dataframe all_data which contains two columns: Event_Summary: Which is a textual description of the event and ...
2
votes
1answer
36 views

Unit test for a function that transforms a PANDAS dataframe

I am currently working on a project that uses many dataframe wrangling. The intention is to write test to check that the output functions are correct. I have written a test for a dataframe function ...
1
vote
1answer
100 views

Customer segmentation using RFM analysis [on hold]

Currently, my code works perfectly well but i would like to make it cleaner for other users by removing the duplicate and similar lines of code into functions or for loops. Because I am still new ...
0
votes
0answers
14 views

Create an interpolant based on data in a csv

I have a csv file with many columns (e.g. A,B,C,D,...
1
vote
1answer
27 views

Pandas: updating cells with new value plus old value

I have a folder with zip files that contain CSVs for each day. Each CSV contains rows with IPs and a company ID. I want to calculate a distribution of the number of firms accessed. Meaning that I want ...
1
vote
1answer
39 views

BTC price checker with web scraping, regex and Tkinter GUI

The purpose of this was to practice web scraping, regex and GUI. I use a popular bitcoin website to scrape the names and prices of the top 16 cryptocurrencies. I then create a data frame and make a ...
2
votes
2answers
56 views

Pandas calculations per columns and per rows for very big datasets

I am using a panda's dataframe and I am doing filtering and some calculations per column and per row. The dataframe looks this: ...
0
votes
1answer
34 views

Linear Regression on Pandas

I'm working on a simple statistics problem with Pandas and sklearn. I'm aware that my code is ugly, but how can I improve it? <...
0
votes
1answer
50 views

Get all node descendants in a tree

I have a CSV holding a "flat" table of a tree's edges (NOT binary, but a node cannot have two parents), ~1M edges: ...
4
votes
1answer
64 views

Google Analytic Webcrawler for every country code

At the moment I'd like recommendation where I can change and what I can use to improve on simplicity and modularity. If I'm practicing good naming conventions as to provide clean readable code. Any ...
3
votes
1answer
42 views

Set of one-hot encoders in Python

In the absence of feature-complete and easy-to-use one-hot encoders in the Python ecosystem I've made a set of my own. This is intended to be a small library, so I want to make sure it's as clear and ...
2
votes
2answers
38 views

Compare an array with a file and form groups from elements of an array

I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. ...
2
votes
1answer
56 views

Generating scatter plot from a CSV file

Assume you have the following data in the form of a csv-file. The content looks something like this: ...
5
votes
1answer
53 views

Calculating time deltas between rows in a Pandas dataframe

I am trying to compute the difference in timestamps and make a delta time column in a Pandas dataframe. This is the code I am currently using: ...
2
votes
0answers
61 views

Sum, validate, and plot monthly inventory levels

The idea is that I have a subset of the movements of material of a factory (over 700k rows), and I want to separate these movements by the reference of each product to plot the stock of each product ...
4
votes
1answer
42 views

Python script for to evaluate business data

The following script is part of a further education I'm currently enrolled into. Not all of the code is written by myself. Function signatures for example. Therefore I have put the sections, written ...
3
votes
1answer
25 views

Django module to transform data from database to be displayed in Slickgrid tables

This is a Django tag with 2 classes, but can be viewed as a module: What does the module do? it loads a database, performs some transformations on the data, returns the output as a dictionary of json ...
0
votes
1answer
49 views

How to better process csv file with pandas and further dealing with set

I have below working code with pandas and python, i'm looking if there is an improvement or simplification which can be done. ...
1
vote
0answers
23 views

Groupby Count on user defined time periods pandas

I have a data frame like: ...
0
votes
1answer
16 views

Assigning zero to random cells in a pandas.DataFrame

I am trying to assign 0 to random cells in a one dimensional pandas.DataFrame. The below code is the best way I could think of ...
5
votes
1answer
179 views

Making a dataframe of parent IDs

I am quite new to scripting, and here is my first code I wrote. The goal of the code is to extract a parent ID based on a child ID. I would appreciate some constructive criticism and aggressive ...
1
vote
1answer
71 views

My Python 3.6 script to detect stamp card redeem fraud and prepare a csv report

I have a requirement to write a Python 3.6 script which should be able to detect following case within a time period and be able to create a text file report that lists the condition violated and ...
2
votes
1answer
49 views

twitter data mining script in python

I have written a simple script that searches twitter for keywords and saves them to a csv file if they contain those words. It can be found on my github here. How can I improve this code to generally ...
1
vote
0answers
15 views

Batch gradient descent and stochastic gradient descent

I'm trying to implement logistic regression and I believe my batch gradient descent is correct or at least it works well enough to give me decent accuracy for the dataset I'm using. When I use ...
1
vote
1answer
31 views

Display the difference between DataFrames' dtypes?

I recently updated Pandas and noticed that it is no longer possible to merge DataFrames with mismatched dtypes. I have a script which relied on merging and seemed to work in the past despite having ...
0
votes
2answers
37 views

Python Pandas - finding duplicate names and telling them apart

(Somewhat) new to python and coding in general! Using python 3.7 and pandas, I'm running code to create a searchable list of people in my dataframe, and I feel like my way of telling duplicates apart ...
2
votes
1answer
44 views

Check values in datatables, with validation rules that vary according to the data source

Background I receive dataframes from several sources (below I call these sources s1, s2 and ...
2
votes
1answer
66 views

Function for setting a streak of strings in a Pandas

I have a df with artist names in one column, I want to find how many times the artists repeats through the column. Example: ...
1
vote
0answers
29 views

Calculating how much $ goes into each defaulting bucket

I have the following code which I use to determine how much of each repayment goes into each late payments bucket (i.e. 0-7 days past due, 7-15 days past due... etc) For example, the repayment amount ...
2
votes
0answers
31 views

Function that plots data frames, supporting several variants

I defined a function that generates tables and creates countplots and barplots based on the arguments that it receives. But the snippets of code inside the function are repeated several times, which ...
4
votes
1answer
62 views

Python program computing some statistics on Scottish geographic areas

This simple script computes some basic descriptive statistics, like mean, standard deviation, kurtosis, etc. on data column imported from a CSV file with use of ...
0
votes
1answer
1k views

Lookup closest value in Pandas DataFrame

I have a DataFrame that contains the data shown below: ...
3
votes
1answer
50 views

turn a dict that may contain a pandas dataframe to several dicts

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows). I want to create a new dict for each row in the DataFrame's, ...
4
votes
1answer
52 views

Subtable timestamps in python pandas

I have a dataframe like: ...
2
votes
0answers
122 views

Finding peaks in a DataFrame

I would like to compare the min/max values of a time-series with a test time-series. Additionally, I would like to compare the time of the "peaks". However, I'm having trouble extracting these ...
2
votes
1answer
43 views

Pivot DataFrame with DateTimeIndex

I have some data, which I've faked using: ...
4
votes
0answers
451 views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
2
votes
0answers
74 views

Pandas: missing value imputation

I want to find a more efficient solution to the following problem: My dataset is about online games. Each player may have multiple plays (rows in the df). Each play has its corresponding timestamp. ...
3
votes
0answers
283 views

Function to calculate distances and nearest points between 2 GeoPandas dataframes

I have 2 geoPandas frames and want to calculate the distance and the nearest point (see functions below) from the geoSeries geometry from dataframe 1 (containing ...
2
votes
1answer
148 views

Convert list of dictionaries to iterable list

Due to another module coupled to my function, I can only receive the input to my part in the form of a JSON object structured roughly like this: ...
1
vote
1answer
95 views

Implementing a faster inbound neighbour similarity in Networkx

Given a directed graph G of 20+ million edges and approximately 2 million unique nodes, can this function/process be made faster on a dataset of 40k edges? The current speed is at ~1.8 seconds per ...
2
votes
1answer
292 views

Concordance index calculation

I am trying to calculate a customized concordance index for survival analysis. Below is my code. It runs well for small input dataframe but extremely slow on a dataframe with one million rows (>30min)....
3
votes
1answer
108 views

Pytest fixture for testing a vertex-parsing function

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy ...
0
votes
1answer
1k views

Create new columns using groupby in pandas [closed]

I'm running into a large bottleneck in my program that takes hours to perform. I have a Dataframe that is very large. I need to take the columns of the Dataframe and create new columns within same ...
3
votes
0answers
592 views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
3
votes
2answers
140 views

Reading set of large log files line by line and count how many hostnames appear on each line

I have around 300 log files in a directory and each log file contains around 3300000 lines. I need to read through each file line by line and count how many hostnames that appear on each line. I wrote ...