This code is about instantiating object attributes Just in Time.
I'm working on a REST API, where I currently create Python objects from the JSON data returned by the server, though not all attributes of the created objects will be called upon in user code. Hence only instantiating the attributes the user access via dot notation seems like a valid way to improve performance.
lazy_evaluate is a function, with a signature similar to functools.partial.
lazy_evaluate returns a function that when called multiple times only computes the result once.
I then take advantage of the builtin property function to create a getter that calls the function returned by lazy_evaluate.
There is clearly a trade off here. This only helps performance if a subset of the attributes are required in user code. Otherwise the extra code will slow it down.
The core of the idea is to use Python scoping to mutate a list in the scope outside the function evaluate. I dislike this the most.
- Do you think it's a good idea?
- Do you have a better solution?
- Do you think the complexity is worth it?
I will try and create more context here as to why I want this. I have implemented this idea in my package async_v20 found here. I have written docs here.
I left this out as I felt it was a distraction from the concept.
JSON from the server does get parsed into a python dict. Though aiohttp does this for me.
When the server sends JSON:
- All attributes are converted to snake_case via a dictionary lookup
- Time data is turned into a pandas.TimeStamp (requires extra parsing)
In some cases there may be 5000 objects to instantiate or objects may be nested around 3 deep. Objects get instantiated in this module.
This example runs a benchmark:
from time import time
class Time(object):
def __enter__(self):
self.start = time()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.end = time()
self.interval = self.end - self.start
def lazy_evaluate(func, *args, **kwargs):
acc = []
def evaluate():
if acc == []:
print('EVALUATING')
acc.append(func(*args, **kwargs))
return acc[0]
return evaluate
class JitAttributes(object):
foo = property(lambda self: getattr(self, '_foo')())
bar = property(lambda self: getattr(self, '_bar')())
baz = property(lambda self: getattr(self, '_baz')())
def __init__(self, foo, bar, baz):
self._foo = foo
self._bar = bar
self._baz = baz
class Attributes(object):
def __init__(self, foo, bar, baz):
self.foo = foo
self.bar = bar
self.baz = baz
a = 500 # Careful making this too big!
b = a * 2
c = b * 2
print('RUNNING JitAttributes')
repeats = range(100000)
with Time() as t:
for _ in repeats:
JitAttributes(
lazy_evaluate(list, range(a)),
lazy_evaluate(list, range(b)),
lazy_evaluate(list, range(c)),
)
print('TOOK ', t.interval, ' seconds')
print('RUNNING Attributes')
with Time() as t:
for _ in repeats:
Attributes(
list(range(a)),
list(range(b)),
list(range(c)),
)
print('TOOK ', t.interval, ' seconds')
Outputs:
RUNNING JitAttributes
TOOK 0.2956113815307617 seconds
RUNNING Attributes
TOOK 4.880501985549927 seconds
Console demonstration:
>>> jit_instance = JitAttributes(
... lazy_evaluate(list, range(a)),
... lazy_evaluate(list, range(b)),
... lazy_evaluate(list, range(c)),
... )
>>> jit_instance.foo
# EVALUATING <- Only evaluates once :)
# [0, 1, 2, 3, ... ]
>>> jit_instance.foo
# [0, 1, 2, 3, ... ]
>>> non_jit_instance = Attributes(
... list(range(a)),
... list(range(b)),
... list(range(c)),
... )
>>> non_jit_instance.foo
# [0, 1, 2, 3, ...]
>>> non_jit_instance.foo
# [0, 1, 2, 3, ...]
Update
New review following on from this one
