0
\$\begingroup\$

As a component of some trading software, I wrote function parse_ticks(), as part of an Exchange class that models the Bitcoin derivatives exchange BitMEX.

The purpose of parse_ticks() is to convert the previous minutes stored tick data (a list of individual trades, obtained by calling all_ticks = self.ws.get_ticks()) into a dictionary which aggregates the OHLCV values from the list of previous minutes ticks. The list of trades is a list of dict's, each dict containing a timestamp, size of trade, if the tick was a buy or sell, etc.

parse_ticks() gets invoked once per instrument when minute elapses. In this scenario, I am watching two instruments, so parse_ticks() gets called twice at the start of each minute.

I timed parse_ticks() run time for 2400 minutes (2400 observations) with as many background processes and programs disabled as possible, and obtained these results (in seconds):

mean run time: 0.0318425

min run time: 0.00458

max run time: 0.07958

std dev: 0.02276709988

Theres a massive range here, with the min and max substantially different, and the std dev is almost as large as the mean.

How would I lower the std. dev of parse_ticks() run time, and get the mean run time closer the minimum observation of 0.00458?

Is this kind of optimisation within the scope of python?

EDIT: To answer the questions in comments (thank you all for your help):

ws.get_ticks() has no outbound API calls, it looks like this:

def get_ticks(self):
        return self.data['trade']

where data['trade'] is a list containing ticks saved from a websocket stream in realtime. The 'trade' list is trimmed by 30% if it goes above 10000 elements, so that aspect (constantly-increasing amount of data to parse) is constant, once the limit is reached. So the data is already available when parse_ticks() is called.

The amount of ticks DOES vary, so some variance can be explained by that. But surely not the extreme range of 0.075? (min - max).

The run times are seemingly random, there are long and short runs interspersed throughout all the observations.

self.bars = {}
self.symbols = ["XBTUSD", "ETHUSD"]
self.ws = Bitmex_WS()

def parse_ticks(self):
    """Return a 1-min OHLCV dict, given a list of the previous
       minutes tick data."""
    
    all_ticks = self.ws.get_ticks()
    target_minute = datetime.datetime.utcnow().minute - 1
    ticks_target_minute = []
    tcount = 0

    # search from end of tick list to grab newest ticks first
    for i in reversed(all_ticks):
        try:
            ts = i['timestamp']
            if type(ts) is not datetime.datetime:
                ts = parser.parse(ts)
        except Exception:
            self.logger.debug(traceback.format_exc())
        
        # scrape prev minutes ticks
        if ts.minute == target_minute:
            ticks_target_minute.append(i)
            ticks_target_minute[tcount]['timestamp'] = ts
            tcount += 1

        # store the previous-to-target-minute bar's last
        # traded price to use as the open price for target bar
        if ts.minute == target_minute - 1:
            ticks_target_minute.append(i)
            ticks_target_minute[tcount]['timestamp'] = ts
            break
    ticks_target_minute.reverse()

    # reset bar dict ready for new bars
    self.bars = {i: [] for i in self.symbols}

    # build 1 min bars for each symbol
    for symbol in self.symbols:
        ticks = [i for i in ticks_target_minute if i['symbol'] == symbol]
        bar = self.build_OHLCV(ticks, symbol)
        self.bars[symbol].append(bar)
        # self.logger.debug(bar)
\$\endgroup\$
6
  • 2
    \$\begingroup\$ If that short amount of time is important to you, then questioning whether Python is suitable is warranted. However, the folks over at softwareengineering.stackexchange.com would be better suited in answering that. To optimize for speed this way, you may need a lower-level language and correspondingly invest more time in development. \$\endgroup\$
    – Gloweye
    Oct 4, 2019 at 13:18
  • \$\begingroup\$ Could it be branch prediction? \$\endgroup\$
    – Mast
    Oct 4, 2019 at 13:29
  • 3
    \$\begingroup\$ Is the amount of ticks constant or not? If the amount of work you have to do varies, then that would also cause the runtime to vary. \$\endgroup\$
    – JAD
    Oct 4, 2019 at 13:32
  • \$\begingroup\$ The current question title, which states your concerns about the code, is too general to be useful here. Please edit to the site standard, which is for the title to simply state the task accomplished by the code. Please see How to get the best value out of Code Review: Asking Questions for guidance on writing good question titles. \$\endgroup\$ Oct 4, 2019 at 13:42
  • 2
    \$\begingroup\$ If you examine the run times, are they random or do they get bigger as the 2400 minutes pass? If so then I would suggest that searching for the newest tick starting at the oldest is going to take progressively longer and you may want to find some way of remembering where you are at each pass. Similarly the last loop, building the bars, will get progressively longer for the same reason, more data to process each time. \$\endgroup\$ Oct 4, 2019 at 14:10

1 Answer 1

1
\$\begingroup\$

We (and it sounds like you as well) do not have enough information to answer this question. But some good news first: I cannot see any region where your method is obviously wasting time (like searching with in for some item in a growing list).

Before you concern yourself more with this, consider that even if the variance is large, your maximum time is less than a tenth of a second, for two instruments. In other words, you could be monitoring more than 1500 instruments before you get close to a maximum processing time on the order of your frequency (one minute). So ask yourself if you are performing needless premature optimization. Will this code run with \$\mathcal{O}(1000)\$ instruments, for longer than two days? If not, you can stop right here. If yes, or as an academic exercise, continue on.


As far as I can see, the runtime of this code will be largely determined by two factors:

  1. How long self.ws.get_ticks() takes. Does this method connect to the internet to get it's data? Then the variance might actually be the variance in establishing the connection and getting the data. This could be influenced by your internet connection, but also the current load on the server you are trying to connect to. In this case there is nothing you can do.

  2. How many elements are returned. The actual function getting the data might take longer for more elements, but also the processing would, since you iterate over all elements of the list.

The only way to know is to gather more data. Individually time the call to self.ws.get_ticks(), collect the len(all_ticks) and plot everything in a time-ordered way. Maybe this will help you to discover something interesting.

Here are some possibilities of what you could discover:

  • The server where you get the information also has some updating frequency, so only every five minutes will there be data for you.
  • They have a rate limit, which makes all requests in-between fail (which is faster than transmitting a bunch of data).
  • The call actually returns all data every time, and you just break when you have reached the point of the last minute. In this case each successive call is more expensive, so of course this increases the standard deviation. Try to find a way to pass the start date to the call.

And here is a small class that can help you keep track of different timers I just came up with:

from time import perf_counter
from statistics import mean, median, stdev
from collections import defaultdict

class Timer:
    durations = defaultdict(list)

    def __init__(self, name=""):
        self.name = name

    def __enter__(self):
        self.start = perf_counter()

    def __exit__(self, *args):
        Timer.durations[self.name].append(perf_counter() - self.start)

    @staticmethod
    def calc_stats(x):
        return {"mean": mean(x),
                "std": stdev(x),
                "median": median(x),
                "min": min(x),
                "max": max(x)}

    @staticmethod
    def stats():
        return {name: Timer.calc_stats(x) for name, x in Timer.durations.items()}

With some example usage:

from time import sleep
import pandas as pd
import matplotlib.pyplot as plt

from timer import Timer

for n in range(10):
    with Timer("a"):
        sleep(0.5)
    with Timer("b"):
        sleep(0.1 * n)

print(pd.DataFrame(Timer.stats()))
#                a         b
# mean    0.500559  0.450504
# std     0.000021  0.303079
# median  0.500552  0.450502
# min     0.500534  0.000006
# max     0.500602  0.901003

plt.plot(Timer.durations["a"], label="a")
plt.plot(Timer.durations["b"], label="b")
plt.legend()
plt.xlabel("Iteration")
plt.ylabel("Time [s]")
plt.show()

enter image description here

Some other comments:

  • Instead of self.bars = {i: [] for i in self.symbols} you can use self.bars = defaultdict(list), like I did in the Timer class.
  • Try to avoid single letter variables. They are OK in a few cases, but i (and n) are usually reserved for integers. Use x, or maybe even better, tick.
  • Don't directly compare type, use isinstance(datetime.datetime, s) instead. This also allows subclasses.
  • Don't lie in your docstring. You say "Return a 1-min OHLCV dict, given a list of the previous minutes tick data.", but almost none of this is true. The method does not return anything and it also does not take any parameters as input!
  • If you know the exception to expect, catch only that. At least you don't have a bare except, but except KeyError and maybe some specific error from the parser would be better. This way you don't miss an unexpected error. You should also ask yourself what your code does if an error occurs. I think it will just use the previous iterations ts, which would duplicate data. Just hope that you never have a problem in the first timestamp!
\$\endgroup\$
2
  • \$\begingroup\$ Thanks @Graphier, updated the question to address a few points raised. In regards to maximum processing time, i'd be monitoring 50-100 instruments in this manner (parsing tick data), with all parsing needing to be completed as close to the turn of each new minute as possible (i.e all instruments done in > 0.1 sec). The other way I'd thought to scale is use more machines, running maybe 10 instruments in this way, per machine, if it is possible to get individual parse times under 0.01, as the minimum score of 0.00458 would suggest. Or as someone suggested, perhaps python is the wrong choice. \$\endgroup\$
    – s_dbq
    Oct 4, 2019 at 22:34
  • \$\begingroup\$ @s_dbq Usually we disallow code edits after answers have arrived. However, since Graipher admits to a lack of information and his is the only answer, I'll leave it up to him whether or not this is acceptable. Do note a mess might be created if other answers pop-up in the meantime. \$\endgroup\$
    – Mast
    Oct 5, 2019 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.