7
\$\begingroup\$

I've got some time-series data. I want to examine the weekly and daily variation of that data. For weekly data I can make a plot like this, with the days along the horizontal axis:

For daily data I can make a plot like this, with the hours of the day along the horizontal axis and the different colors corresponding to different days:

I've got code defined to get this information and then to plot it, but it feels cumbersome to me, like there must be some better, smaller, clearer way to do this. Can you see a way to simplify the code and make it smaller?

code:

setup

import datetime
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
plt.rcParams["figure.figsize"] = (17, 8)
import pandas as pd
import seaborn as sns
sns.set(context = "paper", font = "monospace")
from sklearn.preprocessing import MinMaxScaler
import sqlite3
import warnings
warnings.filterwarnings("ignore")
pd.set_option("display.max_rows", 500)
pd.set_option("display.max_columns", 500)

df = pd.read_csv("data.csv")

df["datetime"] = pd.datetools.to_datetime(df["datetime"])

add time variables

df.index                    = df["datetime"]
df["weekday"]               = df["datetime"].dt.weekday
df["weekday_name"]          = df["datetime"].dt.weekday_name
df["time_through_day"]      = df["datetime"].map(lambda x: x - datetime.datetime.combine(x.date(), datetime.time()))
df["fraction_through_day"]  = df["time_through_day"].map(lambda x: x / datetime.timedelta(hours = 24))
df["hours_through_day"]     = df["fraction_through_day"] * 24
df["days_through_week"]     = df.apply(lambda row: row["weekday"] + row["fraction_through_day"], axis = 1)
df["fraction_through_week"] = df["days_through_week"] / 24

rescale for plotting

variables_rescale = ["hash_rate", "shares"]
scaler = MinMaxScaler()
df[variables_rescale] = scaler.fit_transform(df[variables_rescale])

weekly variation

weeks = []
week  = []
previous_days_through_week = 0
for days_through_week, hash_rate in zip(df["days_through_week"], df["hash_rate"]):
    if abs(days_through_week - int(previous_days_through_week)) < 6:
        week.append([days_through_week, hash_rate])
    else:
        weeks.append(week)
        week = []
    previous_days_through_week = days_through_week
if not weeks: # < 1 week data
    weeks.append(week)

for week in weeks:
    plt.plot([datum[0] for datum in week], [datum[1] for datum in week], linestyle = "-", linewidth = 1)
plt.ylabel("hash rate")
plt.xticks(
    [     0.5,       1.5,         2.5,        3.5,      4.5,        5.5,      6.5],
    ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
);

daily variation

days = []
day  = []
previous_hours_through_day = 0
for hours_through_day, hash_rate in zip(df["hours_through_day"], df["hash_rate"]):
    if hours_through_day < previous_hours_through_day:
        days.append(day)
        day = []
    previous_hours_through_day = hours_through_day
    day.append([hours_through_day, hash_rate])
if not days: # < 1 day data
    days.append(day)

for day in days:
    plt.plot([datum[0] for datum in day], [datum[1] for datum in day], linestyle = "-", linewidth = 1)

CSV data:

datetime,hash_rate,shares
2017-10-19 23:25:14,14619488,3715761
2017-10-20 00:15:45,14995173,14170275
2017-10-20 01:06:14,14351142,24030987
2017-10-20 01:56:29,14061329,3832722
2017-10-20 02:46:43,14469215,14071308
2017-10-20 03:37:02,14887834,4120626
2017-10-20 04:27:21,14576553,14530155
2017-10-20 05:17:42,14844899,24723756
2017-10-20 06:08:10,15375848,35857727
2017-10-20 06:59:26,15588730,1857548
2017-10-20 07:52:36,14519547,8854300
2017-10-20 08:45:37,15177078,361400
2017-10-20 09:37:41,14562665,11085945
2017-10-20 10:28:43,14433315,7182825
2017-10-20 11:20:22,14681236,17735705
2017-10-20 12:12:01,14400977,28116920
2017-10-20 13:04:52,14692015,39160952
2017-10-20 13:58:18,14805181,49943940
2017-10-20 14:50:31,14915132,60771876
2017-10-20 15:46:03,14249498,71992428
2017-10-20 16:38:56,14126233,82345092
2017-10-20 17:30:58,14459050,93049044
2017-10-20 18:22:45,14668601,103536024
2017-10-20 19:14:54,13657824,113764704
2017-10-20 20:07:58,14496030,124716624
2017-10-20 21:00:04,14680928,123984
2017-10-20 21:52:23,15161664,11179224
2017-10-20 22:42:59,14483703,21335580
2017-10-20 23:33:24,13263374,30717036
2017-10-21 00:23:47,14237172,41007708
2017-10-21 01:14:07,14730234,51112404
2017-10-21 02:04:38,15026071,61630380
2017-10-21 02:54:53,14224845,71580096
2017-10-21 03:45:15,14569989,81912096
2017-10-21 04:35:40,15716359,93018996
2017-10-21 05:26:08,14415618,8516770
2017-10-21 06:16:29,14115803,18676585
2017-10-21 07:06:59,14238747,28519990
2017-10-21 07:57:33,14105674,38660835
2017-10-21 08:48:18,14028049,7519655
2017-10-21 09:39:10,14249836,17586140
2017-10-21 10:30:11,14859751,28256800
2017-10-21 11:21:02,15203522,39336440
2017-10-21 12:13:49,13204020,2339337
2017-10-21 13:10:03,14643361,13651196
2017-10-21 14:04:14,14329230,10815636
2017-10-21 14:59:08,15114558,121524
2017-10-21 15:52:47,14425886,10866271
2017-10-21 16:44:24,14715853,4020419
2017-10-21 17:35:00,14546705,14380340
2017-10-21 18:26:35,14389640,7848425
2017-10-21 19:17:44,14872919,18502029
2017-10-21 20:08:25,15259542,29358173
2017-10-21 20:59:19,14437968,39738348
2017-10-21 21:50:16,14655443,49895729
2017-10-21 22:40:32,14788345,60508825
2017-10-21 23:30:51,15525346,71354842
2017-10-22 00:21:21,14184246,81340064
2017-10-22 01:11:41,14377558,91406302
2017-10-22 02:02:05,14957493,102059906
2017-10-22 02:52:30,14701035,112366488
2017-10-22 03:42:48,14863432,122746593
2017-10-22 04:33:06,14647824,132957273
2017-10-22 05:23:20,14620873,4450230
2017-10-22 06:13:50,13771920,14401125
2017-10-22 07:04:40,14459168,6946425
2017-10-22 07:56:27,14674775,17428185
2017-10-22 08:47:25,14863432,28181025
2017-10-22 09:39:02,14566972,38436885
2017-10-22 10:31:35,14607398,49212315
2017-10-22 11:25:46,15025137,6902378
2017-10-22 12:19:55,14783087,5743332
2017-10-22 13:15:06,14649431,2087244
2017-10-22 14:13:13,14929857,7690836
2017-10-22 15:06:37,15086895,18935628
2017-10-22 15:57:27,14615780,29503476
2017-10-22 16:47:47,14727951,39826872
2017-10-22 17:38:02,14694300,50084454
2017-10-22 18:28:34,14941074,60633498
2017-10-22 19:19:04,15053244,8988312
2017-10-22 20:09:43,14099797,18907422
2017-10-22 21:00:15,14335355,6882264
2017-10-22 21:50:42,14784036,17393700
2017-10-22 22:41:00,15311236,28196598
2017-10-22 23:31:20,14626997,38313150
2017-10-23 00:22:49,13920325,48288672
2017-10-23 01:14:04,13538946,56412
2017-10-23 02:04:33,14638214,10445622
2017-10-23 02:54:48,14335355,20458752
2017-10-23 03:45:07,14268053,30481284
2017-10-23 04:35:20,14032495,4371930
2017-10-23 05:25:35,14256836,1532526
2017-10-23 06:15:53,14761602,4306116
2017-10-23 07:06:07,15591661,15250044
2017-10-23 07:56:21,15558010,26128158
2017-10-23 08:46:39,14537261,36310524
2017-10-23 09:36:54,14974725,46897176
2017-10-23 10:27:19,14514827,1081230
2017-10-23 11:17:24,14896206,4954854
Improve this question
\$\endgroup\$
2
  • \$\begingroup\$ Which Python version are you using? \$\endgroup\$
    – Phrancis
    Oct 23, 2017 at 19:31
  • \$\begingroup\$ Python 3 is the preference, but I'd try to be compatible with 2.7 if it's easy to do so. \$\endgroup\$
    – BlandCorporation
    Oct 23, 2017 at 19:33

1 Answer 1

Reset to default
2
\$\begingroup\$

Here a compact and flexible implementation using resample and plot pandas.Dataframe methods.

import pandas as pd

data = pd.read_csv('data.csv')
data['datetime'] = pd.to_datetime(data.datetime)
data = data.set_index('datetime')


data.resample('d').mean().plot()

data['day'] = data.index.day
data['hour'] = data.index.hour
data_by_day = data.resample('h').mean().set_index(['day', 'hour']).unstack('day')
data_by_day['hash_rate'].plot()
data_by_day['shares'].plot()

Output

pandas.Dataframe.resample method documentation

Improve this answer
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.