Aggregating recent trading data from Bitcoin derivatives exchange

Question

As a component of some trading software, I wrote function parse_ticks(), as part of an Exchange class that models the Bitcoin derivatives exchange BitMEX.

The purpose of parse_ticks() is to convert the previous minutes stored tick data (a list of individual trades, obtained by calling all_ticks = self.ws.get_ticks()) into a dictionary which aggregates the OHLCV values from the list of previous minutes ticks. The list of trades is a list of dict's, each dict containing a timestamp, size of trade, if the tick was a buy or sell, etc.

parse_ticks() gets invoked once per instrument when minute elapses. In this scenario, I am watching two instruments, so parse_ticks() gets called twice at the start of each minute.

I timed parse_ticks() run time for 2400 minutes (2400 observations) with as many background processes and programs disabled as possible, and obtained these results (in seconds):

mean run time: 0.0318425

min run time: 0.00458

max run time: 0.07958

std dev: 0.02276709988

Theres a massive range here, with the min and max substantially different, and the std dev is almost as large as the mean.

How would I lower the std. dev of parse_ticks() run time, and get the mean run time closer the minimum observation of 0.00458?

Is this kind of optimisation within the scope of python?

EDIT: To answer the questions in comments (thank you all for your help):

ws.get_ticks() has no outbound API calls, it looks like this:

def get_ticks(self):
        return self.data['trade']

where data['trade'] is a list containing ticks saved from a websocket stream in realtime. The 'trade' list is trimmed by 30% if it goes above 10000 elements, so that aspect (constantly-increasing amount of data to parse) is constant, once the limit is reached. So the data is already available when parse_ticks() is called.

The amount of ticks DOES vary, so some variance can be explained by that. But surely not the extreme range of 0.075? (min - max).

The run times are seemingly random, there are long and short runs interspersed throughout all the observations.

self.bars = {}
self.symbols = ["XBTUSD", "ETHUSD"]
self.ws = Bitmex_WS()

def parse_ticks(self):
    """Return a 1-min OHLCV dict, given a list of the previous
       minutes tick data."""
    
    all_ticks = self.ws.get_ticks()
    target_minute = datetime.datetime.utcnow().minute - 1
    ticks_target_minute = []
    tcount = 0

    # search from end of tick list to grab newest ticks first
    for i in reversed(all_ticks):
        try:
            ts = i['timestamp']
            if type(ts) is not datetime.datetime:
                ts = parser.parse(ts)
        except Exception:
            self.logger.debug(traceback.format_exc())
        
        # scrape prev minutes ticks
        if ts.minute == target_minute:
            ticks_target_minute.append(i)
            ticks_target_minute[tcount]['timestamp'] = ts
            tcount += 1

        # store the previous-to-target-minute bar's last
        # traded price to use as the open price for target bar
        if ts.minute == target_minute - 1:
            ticks_target_minute.append(i)
            ticks_target_minute[tcount]['timestamp'] = ts
            break
    ticks_target_minute.reverse()

    # reset bar dict ready for new bars
    self.bars = {i: [] for i in self.symbols}

    # build 1 min bars for each symbol
    for symbol in self.symbols:
        ticks = [i for i in ticks_target_minute if i['symbol'] == symbol]
        bar = self.build_OHLCV(ticks, symbol)
        self.bars[symbol].append(bar)
        # self.logger.debug(bar)

Answer 1

We (and it sounds like you as well) do not have enough information to answer this question. But some good news first: I cannot see any region where your method is obviously wasting time (like searching with in for some item in a growing list).

Before you concern yourself more with this, consider that even if the variance is large, your maximum time is less than a tenth of a second, for two instruments. In other words, you could be monitoring more than 1500 instruments before you get close to a maximum processing time on the order of your frequency (one minute). So ask yourself if you are performing needless premature optimization. Will this code run with \$\mathcal{O}(1000)\$ instruments, for longer than two days? If not, you can stop right here. If yes, or as an academic exercise, continue on.

---

As far as I can see, the runtime of this code will be largely determined by two factors:

1. How long self.ws.get_ticks() takes. Does this method connect to the internet to get it's data? Then the variance might actually be the variance in establishing the connection and getting the data. This could be influenced by your internet connection, but also the current load on the server you are trying to connect to. In this case there is nothing you can do.

2. How many elements are returned. The actual function getting the data might take longer for more elements, but also the processing would, since you iterate over all elements of the list.

The only way to know is to gather more data. Individually time the call to self.ws.get_ticks(), collect the len(all_ticks) and plot everything in a time-ordered way. Maybe this will help you to discover something interesting.

Here are some possibilities of what you could discover:

• The server where you get the information also has some updating frequency, so only every five minutes will there be data for you.
• They have a rate limit, which makes all requests in-between fail (which is faster than transmitting a bunch of data).
• The call actually returns all data every time, and you just break when you have reached the point of the last minute. In this case each successive call is more expensive, so of course this increases the standard deviation. Try to find a way to pass the start date to the call.

---

And here is a small class that can help you keep track of different timers I just came up with:

from time import perf_counter
from statistics import mean, median, stdev
from collections import defaultdict

class Timer:
    durations = defaultdict(list)

    def __init__(self, name=""):
        self.name = name

    def __enter__(self):
        self.start = perf_counter()

    def __exit__(self, *args):
        Timer.durations[self.name].append(perf_counter() - self.start)

    @staticmethod
    def calc_stats(x):
        return {"mean": mean(x),
                "std": stdev(x),
                "median": median(x),
                "min": min(x),
                "max": max(x)}

    @staticmethod
    def stats():
        return {name: Timer.calc_stats(x) for name, x in Timer.durations.items()}

With some example usage:

from time import sleep
import pandas as pd
import matplotlib.pyplot as plt

from timer import Timer

for n in range(10):
    with Timer("a"):
        sleep(0.5)
    with Timer("b"):
        sleep(0.1 * n)

print(pd.DataFrame(Timer.stats()))
#                a         b
# mean    0.500559  0.450504
# std     0.000021  0.303079
# median  0.500552  0.450502
# min     0.500534  0.000006
# max     0.500602  0.901003

plt.plot(Timer.durations["a"], label="a")
plt.plot(Timer.durations["b"], label="b")
plt.legend()
plt.xlabel("Iteration")
plt.ylabel("Time [s]")
plt.show()

Some other comments:

• Instead of self.bars = {i: [] for i in self.symbols} you can use self.bars = defaultdict(list), like I did in the Timer class.
• Try to avoid single letter variables. They are OK in a few cases, but i (and n) are usually reserved for integers. Use x, or maybe even better, tick.
• Don't directly compare type, use isinstance(datetime.datetime, s) instead. This also allows subclasses.
• Don't lie in your docstring. You say "Return a 1-min OHLCV dict, given a list of the previous minutes tick data.", but almost none of this is true. The method does not return anything and it also does not take any parameters as input!
• If you know the exception to expect, catch only that. At least you don't have a bare except, but except KeyError and maybe some specific error from the parser would be better. This way you don't miss an unexpected error. You should also ask yourself what your code does if an error occurs. I think it will just use the previous iterations ts, which would duplicate data. Just hope that you never have a problem in the first timestamp!