In my machine learning programs I have the typical situation where I need to perform certain operations on data, which results in the following graph:
- data --op A--> intermediate_state --op B--> final result type 1
- data --op A--> intermediate_state --op C--> final result type 2
- data --op A--> intermediate_state --op D--> final result type 3
So in order to not have to repeat operation A all the time I provide a "fit" method, which implements operation A. Then, I implement transformation methods that implement the other operations. However, since I usually only need some of the final results in the graph leaves, I want to calculate those on the one hand lazily, but then cache the result.
Since the operations require a parameter, I provide this one in the constructor.
I also need to pass a parametrized object with the same parametrization to different parts of the the program, where the data of operation A changes, without the parametrization changing.
That is the reason why
fit is it's own method and not part of the constructor.
Also, because of that I cannot simply cache the results of the operations that are not A, without resetting them after invoking a new
Here is my result in Python:
class MyClass: def __init__(self, parameter): self.parameter = parameter # expensive machine learning fitting def fit(self, data) -> MyClass: self.intermediate_data_ = data + self.parameter + 2 self._complex_transform1 = None self._complex_transform2 = None return self # expensive machine learning operation version 1 @property def complex_transform1(self): if self._complex_transform1 is None: self._complex_transform1 = self.intermediate_data_ / self.parameter / 2 return self._complex_transform1 # expensive machine learning operation version 2 @property def complex_transform2(self): if self._complex_transform2 is None: self._complex_transform2 = self.intermediate_data_ / self.parameter / 5 return self._complex_transform2
The problem I have with this approach is that I am repeating quite a bit of boiler plate code. (Note, that I have many more operations that are not A.) Namely:
if self._complex_transform1 is None:could be reduced from
if self._complex_transform1 is None: self._complex_transform1 = self.intermediate_data_ / self.parameter / 2 return self._complex_transform1
return self.intermediate_data_ / self.parameter / 2
and I would save a nesting.
I was thinking of defining a container class that contains the results and is reset by fit. But I do not see how to implement this without basically introducing the same about of boilerplate code into the transform methods.
Some decorator-approach might be great, but I am not sure how to approach this.