As a component of some trading software, I wrote function parse_ticks()
, as part of an Exchange
class that models the Bitcoin derivatives exchange BitMEX.
The purpose of parse_ticks()
is to convert the previous minutes stored tick data (a list of individual trades, obtained by calling all_ticks = self.ws.get_ticks()
) into a dictionary which aggregates the OHLCV values from the list of previous minutes ticks. The list of trades is a list of dict's, each dict containing a timestamp, size of trade, if the tick was a buy or sell, etc.
parse_ticks()
gets invoked once per instrument when minute elapses. In this scenario, I am watching two instruments, so parse_ticks()
gets called twice at the start of each minute.
I timed parse_ticks()
run time for 2400 minutes (2400 observations) with as many background processes and programs disabled as possible, and obtained these results (in seconds):
mean run time: 0.0318425
min run time: 0.00458
max run time: 0.07958
std dev: 0.02276709988
Theres a massive range here, with the min and max substantially different, and the std dev is almost as large as the mean.
How would I lower the std. dev of parse_ticks()
run time, and get the mean run time closer the minimum observation of 0.00458?
Is this kind of optimisation within the scope of python?
EDIT: To answer the questions in comments (thank you all for your help):
ws.get_ticks()
has no outbound API calls, it looks like this:
def get_ticks(self):
return self.data['trade']
where data['trade']
is a list containing ticks saved from a websocket stream in realtime. The 'trade'
list is trimmed by 30% if it goes above 10000 elements, so that aspect (constantly-increasing amount of data to parse) is constant, once the limit is reached. So the data is already available when parse_ticks()
is called.
The amount of ticks DOES vary, so some variance can be explained by that. But surely not the extreme range of 0.075? (min - max).
The run times are seemingly random, there are long and short runs interspersed throughout all the observations.
self.bars = {}
self.symbols = ["XBTUSD", "ETHUSD"]
self.ws = Bitmex_WS()
def parse_ticks(self):
"""Return a 1-min OHLCV dict, given a list of the previous
minutes tick data."""
all_ticks = self.ws.get_ticks()
target_minute = datetime.datetime.utcnow().minute - 1
ticks_target_minute = []
tcount = 0
# search from end of tick list to grab newest ticks first
for i in reversed(all_ticks):
try:
ts = i['timestamp']
if type(ts) is not datetime.datetime:
ts = parser.parse(ts)
except Exception:
self.logger.debug(traceback.format_exc())
# scrape prev minutes ticks
if ts.minute == target_minute:
ticks_target_minute.append(i)
ticks_target_minute[tcount]['timestamp'] = ts
tcount += 1
# store the previous-to-target-minute bar's last
# traded price to use as the open price for target bar
if ts.minute == target_minute - 1:
ticks_target_minute.append(i)
ticks_target_minute[tcount]['timestamp'] = ts
break
ticks_target_minute.reverse()
# reset bar dict ready for new bars
self.bars = {i: [] for i in self.symbols}
# build 1 min bars for each symbol
for symbol in self.symbols:
ticks = [i for i in ticks_target_minute if i['symbol'] == symbol]
bar = self.build_OHLCV(ticks, symbol)
self.bars[symbol].append(bar)
# self.logger.debug(bar)