I sometimes do experiments at work and separate the computation and the analysis so I can do the computation on a cluster and the analysis locally and sometimes in a Jupyter notebook. I wrote a class which allows me to save results to a hidden file as if it was a dictionary. The idea is to create an object specifying the name of the experiment and from there you can use it as a dictionary, and it is saved to disk so you can access it from other python files. I'd appreciate any thoughts since IO isn't my forte. I used python 2.7 but I think it should work for python 3.0

import os
import cPickle as pickle

class FileDict():

    def __init__(self, name, default = None):
        self.fpath = '.{}.fd'.format(name)
        self.default = default

    def __getitem__(self, key):
        if os.path.isfile(self.fpath):
            d = pickle.load(open(self.fpath))
            if key in d:
                return d[key]
            return self.default

    def __setitem__(self, key, value):
        if os.path.isfile(self.fpath):
            d = pickle.load(open(self.fpath))
            d[key] = value
            d = {key : value}
        pickle.dump(d, open(self.fpath, 'w'))

if __name__ == '__main__':
    test = FileDict('test', 0)
    test[1] = 'thing'

2 Answers 2

  • I think you should rather returns a default value if file exists but key is not found in it.
  • Opening file every time you want to get or set a value is expensive. Consider reading it in __init__ method, saving the data in handler registered using atexit module and adding a flush() method if for some reason you'd like to dump data right now. You could also add some no_cache option in __init__ to force saving thins right away?
  • pickle module is potentially insecure. Consider famous example import pickle; pickle.loads("cantigravity\n") If you open a maliciously prepared file with your class a weird things can happen. It should be used for internal used only, not for a general purpose class that can read any input.
  • What if multiple instances of FileDict will use the same file as source? Consider using tempfile module or generating unique names with uuid module, and then allowing returning generated the name with some method
  • Following collections.defaultdict example you might want to use a callable to generate a default value. You can always pass lambda: 0 if you want to have only one value returned

There are two things I don't like about this implementation:

  1. It has a very incomplete dict interface. You cannot explore the dict at all, since neither len, keys, nor values is implemented. You also cannot use dictionary unpacking (**fd) because of this. (You would only need keys defined for that, in addition to __getitem__, which you already have. See here for more details.)

  2. It performs a lot of unnecessary disk reads and writes. It would be better if it had a cache in memory and kept that cache in sync with the file on disk. This specifically means, if the file on disk has not changed and we are asked to retrieve a key for the second time, we can just serve it from the cache. You might either want to keep the whole dict in this cache, or a cache with some maximum size, like a lru_cache. See this post for multiple ways to implement it in Python 2.7 (Python 3.x has it in the functools module).

I will post some example implementation as soon as I get back to my PC, it is a bit too much on a mobile...

  • \$\begingroup\$ I hadn't considered implementing the entire interface, but this seems sensible. The disk read and writes is definitely what I was after with this questions. Thanks for the suggestions @Graipher \$\endgroup\$
    – HBeel
    Mar 17, 2017 at 11:27
  • \$\begingroup\$ @HBeel I'm still fiddling around with my implementation of that, Of course it's not as easy as I thought it was :) \$\endgroup\$
    – Graipher
    Mar 17, 2017 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.