This code is about instantiating object attributes Just in Time.
I'm working on a REST API, where I currently create Python objects from the JSON data returned by the server, though not all attributes of the created objects will be called upon in user code. Hence only instantiating the attributes the user access via dot notation seems like a valid way to improve performance.
lazy_evaluate is a function, with a signature similar to
lazy_evaluate returns a function that when called multiple times only computes the result once.
I then take advantage of the builtin
property function to create a getter that calls the function returned by
There is clearly a trade off here. This only helps performance if a subset of the attributes are required in user code. Otherwise the extra code will slow it down.
The core of the idea is to use Python scoping to mutate a list in the scope outside the function
evaluate. I dislike this the most.
- Do you think it's a good idea?
- Do you have a better solution?
- Do you think the complexity is worth it?
I left this out as I felt it was a distraction from the concept.
JSON from the server does get parsed into a python dict. Though
aiohttp does this for me.
When the server sends JSON:
- All attributes are converted to snake_case via a dictionary lookup
- Time data is turned into a pandas.TimeStamp (requires extra parsing)
In some cases there may be 5000 objects to instantiate or objects may be nested around 3 deep. Objects get instantiated in this module.
This example runs a benchmark:
from time import time class Time(object): def __enter__(self): self.start = time() return self def __exit__(self, exc_type, exc_val, exc_tb): self.end = time() self.interval = self.end - self.start def lazy_evaluate(func, *args, **kwargs): acc =  def evaluate(): if acc == : print('EVALUATING') acc.append(func(*args, **kwargs)) return acc return evaluate class JitAttributes(object): foo = property(lambda self: getattr(self, '_foo')()) bar = property(lambda self: getattr(self, '_bar')()) baz = property(lambda self: getattr(self, '_baz')()) def __init__(self, foo, bar, baz): self._foo = foo self._bar = bar self._baz = baz class Attributes(object): def __init__(self, foo, bar, baz): self.foo = foo self.bar = bar self.baz = baz a = 500 # Careful making this too big! b = a * 2 c = b * 2 print('RUNNING JitAttributes') repeats = range(100000) with Time() as t: for _ in repeats: JitAttributes( lazy_evaluate(list, range(a)), lazy_evaluate(list, range(b)), lazy_evaluate(list, range(c)), ) print('TOOK ', t.interval, ' seconds') print('RUNNING Attributes') with Time() as t: for _ in repeats: Attributes( list(range(a)), list(range(b)), list(range(c)), ) print('TOOK ', t.interval, ' seconds')
RUNNING JitAttributes TOOK 0.2956113815307617 seconds RUNNING Attributes TOOK 4.880501985549927 seconds
>>> jit_instance = JitAttributes( ... lazy_evaluate(list, range(a)), ... lazy_evaluate(list, range(b)), ... lazy_evaluate(list, range(c)), ... ) >>> jit_instance.foo # EVALUATING <- Only evaluates once :) # [0, 1, 2, 3, ... ] >>> jit_instance.foo # [0, 1, 2, 3, ... ] >>> non_jit_instance = Attributes( ... list(range(a)), ... list(range(b)), ... list(range(c)), ... ) >>> non_jit_instance.foo # [0, 1, 2, 3, ...] >>> non_jit_instance.foo # [0, 1, 2, 3, ...]
New review following on from this one