I am reading a large data file where the time is given in number of days since some epoch. I am currently converting this to Python's datetime format using this function:
import datetime as dt def days2dt(days_since_epoch): epoch = dt.datetime(1980, 1, 6) datelist = [epoch + dt.timedelta(days=x) for x in days_since_epoch] return datelist # run with sample data (might be larger in real life, in worst case multiply # the list by 40 instead of 6) import numpy as np sample = list(np.arange(0, 3/24., 1/24./3600./50.))*6 dates = days2dt(sample)
Running this function takes 5x longer than reading the entire file using
pandas.read_csv() (perhaps because the listcomp performs an addition for each element). The returned list is used immediately as the index of the pandas DataFrame, though interestingly, using a generator expression instead of a listcomp as above improves performance by ~35% (why?).
Aside from using a generator expression, can the performance of this function be improved in any way, e.g. by not performing this date conversion per-element or by using some NumPy feature I'm not aware of?