5
\$\begingroup\$

I have a performance issue causing test->code->test cycle to be really slow. The slowness is hard to avoid, since I am doing some heavy image processing, and I am trying to accelerate things a bit by not running functions when it is not needed. For ex: compute some numbers from big images -> serialize results in text files -> resume computations from text files.

Currently, I commit the text files containing the results to share them with the team, and let other people run the tests quickly. But I have to make sure that whenever one dependency is updated (src images, preprocessing functions, etc), those value are recomputed. I looked for a tool which would allow me to describe dependencies and production rules to automate the process and avoid committing data in the codebase. I thought about makefile, but I read a lot of negative advice against it. I found stuff like Scons and other build automation tools, but those are for building software, and they do not seem adapted for my task.

I decided to write a small lib to tackle this issue (my environment is mainly Python and some C). The goal was to have objects knowing the production rule for their own output, and aware of the other objects they depend upon. An object knows if its output is up-to-date by comparing the current MD5 checksum of the file against the last one (stored somewhere in a small temp file).

Am I reinventing a wheel here? Is there a tool out there I should use instead of a custom lib? If not, is this a good pattern for taking care of this problem?

import os
import path
from md5 import md5 as stdMd5

# path.root is the root of our repo
hashDir = os.path.join(path.root, '_hashes')

def md5(toHash):
    return stdMd5(toHash).hexdigest()

class BaseRule(object):
    '''base object, managing work like checking if results are up to date, and
    calling production rules if they are not'''

    outPath = None

    def __init__(self, func):
        self.func = func

    @property
    def hashPath(self):
        return os.path.join(hashDir, md5(self.outPath))

    def getOutHash(self):
        with open(self.outPath) as f:
            fileHash = md5(f.read())
        return fileHash

    def isUpToDate(self):
        if not os.path.exists(self.outPath):
            return False
        if not os.path.exists(self.hashPath):
            return False
        with open(self.outPath) as f:
            fileHash = self.getOutHash()
        with open(self.hashPath) as f:
            storedHash = f.read().strip()
        return storedHash == fileHash

    def storeHash(self):
        if not os.path.exists(hashDir):
            os.makedirs(hashDir)
        with open(self.hashPath, 'w') as f:
            f.write(self.getOutHash())

    def get(self):
        inputPathes = dict([(key, inp.get()) for key, inp in self.inputs.items()])
        if not self.isUpToDate():
            self.func(outPath=self.outPath, **inputPathes)
            self.storeHash()
        return self.outPath

class StableSrc(BaseRule):
    'source file that never change'

    inputs = {}

    def __init__(self, path):
        self.outPath = path

    def isUpToDate(self):
        return True


def rule(_inputs, _outPath):
    'decorator used to declare dependencies'

    class Rule(BaseRule):
        inputs = _inputs
        outPath = _outPath

    return Rule

def copyTest():
    'test function'

    import shutil

    @rule({'inp' : StableSrc('test.txt')}, 'test2.txt')
    def newTest(inp, outPath):
        print 'copy'
        shutil.copy(inp, outPath)


    @rule({'inp' : newTest}, 'test3.txt')
    def newNewTest(inp, outPath):
        print 'copy2'
        shutil.copy(inp, outPath)

    return newNewTest.get()

if __name__ == '__main__':
    copyTest() # will copy test.txt to test2.txt and test3.txt. If ran a second time, won't copy anything
\$\endgroup\$
  • \$\begingroup\$ "Am I reinventing a wheel here?" Yes. "Is there a tool out there I should use instead of a custom lib?" Yes. SCons. "those are for building softwares". False. They're for processing dependencies. The default configuration builds software, since that's the 80% use case. You're the other 20%. You have to write customized rules. \$\endgroup\$ – S.Lott Sep 20 '11 at 15:42
  • \$\begingroup\$ Thank you. I will have a closer look to Scons. Does it integrate well in a python codebase, or should I maintain a strict separation between scons and python scripts? \$\endgroup\$ – Simon Bergot Sep 20 '11 at 17:08
  • \$\begingroup\$ What? "a strict separation between scons and python scripts"? What can this mean? Also. You should ask for details on Scons on Stackoverflow. Not here. That's not a code review question at all. \$\endgroup\$ – S.Lott Sep 20 '11 at 17:10
  • \$\begingroup\$ Re: SCons: or nmake, GNU make, Ant, NAnt etc. everything that does dependency checking \$\endgroup\$ – sehe Sep 26 '11 at 10:21
2
\$\begingroup\$
# path.root is the root of our repo
hashDir = os.path.join(path.root, '_hashes')

Python style guide recommends that global constants are ALL_CAPS

class BaseRule(object):
    '''base object, managing work like checking if results are up to date, and
    calling production rules if they are not'''

    outPath = None

Python style guide recommends that class level constants are ALL_CAPS

    def isUpToDate(self):
        if not os.path.exists(self.outPath):
            return False
        if not os.path.exists(self.hashPath):
            return False
        with open(self.outPath) as f:
            fileHash = self.getOutHas()

You don't use the file you open here, instead you open the file inside getOutHash

        with open(self.hashPath) as f:
            storedHash = f.read().strip()
        return storedHash == fileHash


    def get(self):
        inputPathes = dict([(key, inp.get()) for key, inp in self.inputs.items()])

No need for the square brackets. They make it so that it produces a list. But since you are creating a dictionary there is no need to make a list first.

        if not self.isUpToDate():
            self.func(outPath=self.outPath, **inputPathes)
            self.storeHash()
        return self.outPath

class StableSrc(BaseRule):
    'source file that never change'

    inputs = {}

I know you'll never modify this, but inputs is an object level attribute above, and its clearest if you keep it that way.

    def __init__(self, path):
        self.outPath = path

    def isUpToDate(self):
        return True



def copyTest():
    'test function'

    import shutil

    @rule({'inp' : StableSrc('test.txt')}, 'test2.txt')
    def newTest(inp, outPath):
        print 'copy'
        shutil.copy(inp, outPath)

Usually you've got a small number of possible actions with different inputs/outputs for each use. Your decorator method assumes that each action will be different.

  1. You aren't tracking dependencies amongst the rules. So you can't have some like .swig produces .cpp produces .o which is linked to produce .exe
  2. Creating a class for every rule is odd. It would make more sense to create an object for each rule.
  3. The only thing you've got over makefiles is your use of a hash rather then a timestamp
\$\endgroup\$
  • \$\begingroup\$ thank you for your comments. Note that rules may be used several times on the same function without the decorator syntax, so I can define a small set of functions, and apply rule on them. The class creation seems more natural to me, because a decorator must be callable. Here newTest is an object, not a class. Do you think I should go for makefiles? \$\endgroup\$ – Simon Bergot Sep 20 '11 at 12:53
  • \$\begingroup\$ @Simon, functions and objects can also be callable. Creating various copies of a class at runtime isn't really expected behavior. Classes are expected to more or less be created once at import time and then used. Unless I'm misreading the code (possible) newTest is a class. The decorator replaces the function you define with a class. Makefiles: reusing existing solutions is usually the best choice. Makefiles are very simple but when they work, they work very well. I'd check to see whether the serve your purposes and if so, use them. \$\endgroup\$ – Winston Ewert Sep 20 '11 at 14:05
  • \$\begingroup\$ if we express the decorator differently, we get newTest=rule(args)(newTest). rule(args) is a class. rule(args)(func) is an object. I can make rule(args) a callable object which return a Rule instance when called with a function, but I think it is too complex compared to the class approach. Still, I don't think that this lib is a good idea. \$\endgroup\$ – Simon Bergot Sep 20 '11 at 17:13
  • \$\begingroup\$ @Simon, you are correct, I forgot how decorators worked. I'd probably define the class outside and use functools.partial rather then define a new class for every rule. But that a style preference I can't argue its clearly better. \$\endgroup\$ – Winston Ewert Sep 20 '11 at 17:35
  • \$\begingroup\$ I see what you mean. Accepting this answer, since my original question is answered. \$\endgroup\$ – Simon Bergot Sep 21 '11 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.