All Questions
23 questions
2
votes
1
answer
872
views
Extract the last row in a dataframe with a timestamp before some given time
Related questions: I need to take specific row from different df to generate new df. How can I make my code faster?
I want to take the last data before the specified time from different time intervals ...
- 177
2
votes
2
answers
203
views
Generate date_range for string intervals like 09:00-10:00/11:00-14:00/15:00-18:00 and calculate the working minutes
I'm a bit new in python and I'm trying to achieve what I can't with VBA, get things faster.
I have large datasets like this in the source(just a sample):
This is done also by python and it's a CSV ...
- 198
0
votes
1
answer
192
views
Extraction of subsets by date and intervals from a dataframe with a specific date format
Given the following dataframe:
...
- 153
2
votes
1
answer
87
views
simulation of updating a dataset from real time data
In this code (reproducible example), I want to generate a dataframe composed by random dates in the first col, and by random numbers (in reality they will be generate from a socket) in the second ...
- 129
4
votes
2
answers
78
views
For each of the visitors, find if they were present within a time period: code is running slow
I'm working with a pandas dataframe which describes the start and end time of visits. A toy example of the data could be as follows:
...
- 141
1
vote
1
answer
501
views
Pandas merge a "%Y%M%D" date column with a "%H:%M:%S" time column
I have a dataframe with a column composed by date object and a column composed by time ...
- 579
3
votes
1
answer
56
views
ways to code for better speed of processing date comparisions / slicing in python
I wrote the below code to convert a long data frame containing start and end date event spans into a daily df with rows for each day:
output:
...
- 31
1
vote
1
answer
44
views
Check that elements in list follow a specified date-format schema and are not null/na/nan [closed]
I'm practicing some data-analysis and I'm currently checking the integrity of my data. The elements that do not follow my date-schema, should be funneled to a new separate list so I can work on ...
2
votes
2
answers
2k
views
Parse date format in Pandas using Python
I have a column in a Pandas Dataframe containing birth dates in object/string format:
0 16MAR39
1 21JAN56
2 18NOV51
3 05MAR64
4 05JUN48
I want to ...
- 151
4
votes
1
answer
86
views
2
votes
1
answer
2k
views
Processing large time-series data
I am currently processing some large time-series data with pandas, and I have a function that is intolerably slow, and I'm sure it could be done faster.
The problem is: I'm studying a factory that ...
- 23
3
votes
0
answers
116
views
Custom Day of the Week Manipulation
The function serves two purposes as follows:
When dealing with the day of the week, Python will always assign 0 to Sunday and 6 to Saturday and I want to change that. The first part of the code is ...
- 363
7
votes
1
answer
9k
views
Show weekly and daily variations in time-series data
I've got some time-series data. I want to examine the weekly and daily variation of that data. For weekly data I can make a plot like this, with the days along the horizontal axis:
For daily data I ...
- 201
3
votes
1
answer
3k
views
Python - Faster random business date generation
I wrote a program that generates some synthetic data. Everything runs fine, but right now the program is so slow that I am only generating 1000 rows of data (and it still takes 3 or so minutes). ...
- 33
3
votes
0
answers
505
views
pandas time-based qcut
I have a dataframe pairs where each row is a transaction with corresponding price and date. Now, I need to assign each transaction to a quintile (add a new column), ...
- 131
7
votes
1
answer
263
views
Coalesce consecutive failures in a DataFrame of hourly sensor readings
I have a PANDAS DataFrame that contains sensor data that is recorded every hour (sample included below). It is important to note that every hour is not necessarily in the dataframe, as sometimes the ...
- 610
5
votes
1
answer
3k
views
Optimize calculation for the amount of time between when a value changes in Pandas
I have some sensor data that contains timestamps from when a machine is turned on and an indicator variable showing whether or not the machine is actively running. The data is mostly recorded every ...
- 610
0
votes
2
answers
20k
views
Interpret YYYYMMDD as the nth day of the year [closed]
I am provided a bunch of dates in the format (YYYYMMDD) such as:
date='20170503'
My goal is to convert that date into the ...
- 476
3
votes
1
answer
2k
views
Calculate working minutes between two timestamps
I've created a function to calculate the working minutes between two timestamps.
I class working minutes as those between 9am - 5pm and not on weekends or national holidays.
The holidays are those ...
- 289
3
votes
1
answer
109
views
Filling in gym membership prices by joining PANDAS dataframes
I have two dataframes, df which contains a list of members in addition to the type of contract that they purchased on a given date, ...
2
votes
0
answers
46
views
Grouping logs and providing counts
The code is supposed to group start and end time logs and provide log counts and unique ID counts. The grouping will be variable 1 hour, 6 hours, 12 hours, 24 hours, etc.
What is the better way to ...
- 121
11
votes
3
answers
1k
views
Project Euler #19: Counting Sundays in the 20th century using Pandas
Project Euler #19 asks:
How many Sundays fell on the first of the month during the twentieth century (1 Jan 1901 to 31 Dec 2000)?
I'm hoping I wasn't too off course from the spirit of the exercise ...
- 641
5
votes
1
answer
69
views
Identifying surface events happening at specific time intervals
Here is some code I wrote to surface Mint.com transactions that occur at monthly intervals, in order to identify subscriptions I may be paying for without realizing it.
I'd like to have some friends ...
- 151