15
\$\begingroup\$

I'm using this code in an app that has hundreds of daily users. The code works, and passes all of the test cases I have.

Occasionally some users report a problem that I don't get right to the bottom of, and there is a possibility it is related to this PermaDict. Can you see anything I may have missed in terms of robustness, or corner cases that may cause it to fail? (And of course, any other Code Review feedback is welcome).

import pickle, json, csv, os, shutil

class PermaDict(dict):
    ''' Persistent dictionary with an API compatible with shelve and anydbm.

    The dict is kept in memory, so the dictionary operations run as fast as
    a regular dictionary.

    Write to disk is delayed until close or sync (similar to gdbm's fast mode).

    Input file format is automatically discovered.
    Output file format is selectable between pickle, json, and csv.
    All three serialization formats are backed by fast C implementations.

    '''

    def __init__(self, filename, flag='c', mode=None, format='json', *args, **kwds):
        self.flag = flag                    # r=readonly, c=create, or n=new
        self.mode = mode                    # None or an octal triple like 0644
        self.format = format                # 'csv', 'json', or 'pickle'
        self.filename = filename
        if flag != 'n' and os.access(filename, os.R_OK):
            fileobj = open(filename, 'rb') # 'rb' if format=='pickle' else 'r')
            with fileobj:
                self.load(fileobj)
        dict.__init__(self, *args, **kwds)

    def sync(self):
        'Write dict to disk'
        if self.flag == 'r':
            return
        filename = self.filename
        tempname = filename + '.tmp'
        try:
            with open(tempname, 'wb') as fileobj: # if self.format=='pickle' else 'w') as fileobj:
                self.dump(fileobj)
        except Exception:
            os.remove(tempname)
            raise
        shutil.move(tempname, self.filename)    # atomic commit
        if self.mode is not None:
            os.chmod(self.filename, self.mode)

    def close(self):
        self.sync()

    def __enter__(self):
        return self

    def __exit__(self, *exc_info):
        self.close()

    def dump(self, fileobj):
        if self.format == 'csv':
            csv.writer(fileobj).writerows(self.items())
        elif self.format == 'json':
            json.dump(self, fileobj, separators=(',', ':'))
        elif self.format == 'pickle':
            pickle.dump(dict(self), fileobj, 2)
        else:
            raise NotImplementedError('Unknown format: ' + repr(self.format))

    def load(self, fileobj):
        # try formats from most restrictive to least restrictive
        for loader in (pickle.load, json.load, csv.reader):
            fileobj.seek(0)
            try:
                return self.update(loader(fileobj))
            except Exception:
                pass
        raise ValueError('File not in a supported format')

Example usage of a "PermaDict" is:

Opening/initialising application state:

    state_filename = os.path.join(app_pathname,"AppState.txt")

    try:
        state = permadict.PermaDict(state_filename)
    except:
        pub.sendMessage(b"ERROR_MESSAGE", msg = "I can't figure out how to read your saved information!")
        raise

    state['global']['root_dir_name'] = root_dir_name
    state['global']['app_dir_name'] = app_dir_name
    state.sync()

Other usage:

def _CallForRefresh(self):
    # we're going to do our stuff, processing files etc...
    # while we're doing that, we don't want to interrupt ourselves
    # due to making a file system change 
    self._eventHandler.SetQueueMode()

    # Do the stuff...
    self._mainWindow.Refresh()
    self._appState.sync()

Down in _mainWindow.Refresh(), we have stuff like:

    wx.Dialog.__init__(self, None, -1, _("Messaging"),
                       style=wx.DEFAULT_FRAME_STYLE,
                       pos=(app_state["Chat"]["pos_x"], 
                       app_state["Chat"]["pos_y"]),
                       size=(app_state["Chat"]["width"], 
                       app_state["Chat"]["height"]) )

and

def OnResize(self, event):
    (self._appState['Chat']['width'], self._appState['Chat']['height']) = event.GetSize()
    event.Skip()

And of course:

# Here we handle incoming events from the GUI
def OnAppEvent(self, event, value = None):
    # most important thing: if we have to die, then do it
    if event == AppEvents.APP_EXIT or event == AppEvents.APP_AUTO_EXIT:
        if event == AppEvents.APP_AUTO_EXIT:
            wx.Sound.PlaySound('autoExit.wav')
            wx.MilliSleep(500)
        self._appState.close()
        self._mainWindow.ShowStatusIcon(False)
        sys.exit()

One obvious way to screw up usage of this storage is to fail to sync() it and then bomb out without going through OnExit() ... I think I have that covered (and at least, it's something I'm aware of, as opposed to anything suspect in the PermaDict code that I might not be aware of!)

\$\endgroup\$
  • 2
    \$\begingroup\$ If these permadicts are shared across threads or users, you may have a problem with one thread or user stomping on the state of the other. \$\endgroup\$ – Sam Bayer Jun 21 '14 at 22:28
  • \$\begingroup\$ It's been almost four years. Maybe you want to review your own code using your new experience? \$\endgroup\$ – Zeta Apr 1 '18 at 7:10
  • 1
    \$\begingroup\$ This code is still in the app, and it still fails from time to time when Windows crashes. However, the application status is "no longer maintained", and the workaround instructions are "delete that state file and start again, sorry". I have never been able to see why this doesn't work robustly. I don't think there is sharing going on. \$\endgroup\$ – GreenAsJade Apr 2 '18 at 2:39
3
\$\begingroup\$

This is overall look quite good for me. I personally prefer """ for comments.


Dependency Injection

Problem I see with PermaDict is that it needs to know about json/pickle/csv load and dump methods. This makes it harder for you to add any new serialisation without changing the code.

In Python functions are objects, so you can pass them as parameters to construction. You can even pass whole modules/classes, etc too.

def __init__(self, filename, flag='c', mode=None, format='json', *args, **kwds):

Instead of specifying 'json' as a string. We can simply pass json or any other object with .load and .dump methods.

 def __init__(self, filename, flag='c', mode=None, serializer=json, *args, **kwds):

Now we just use self.serializer.dump and self.serializer.load. This way you can easily support multiple serializers.

for loader in (pickle.load, json.load, csv.reader):

I don't personally like this kind of try all loading. I'd avoid pickle and stick with json if it is possible due to security concerns.

Warning The pickle module is not secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

However since it is for your own pickled content it might be fine.

Designing for resilience

I suggest you give LMDB a try. You can find more information here: https://symas.com/lmdb/. There are also python bindings available. This should be easier than trying to come up with your own resilient data store.

To be resilient is hard work. To have some resilience maybe store 2 copies of same data, store data checksums or create backups that you can recover from. All hardwares, softwares, OSes, Python interpreters and C runtimes. are subject to failure.

\$\endgroup\$
  • 1
    \$\begingroup\$ Thanks. After all these years, I never did work out how the permadict gets corrupted :( \$\endgroup\$ – GreenAsJade Apr 21 at 9:06
  • \$\begingroup\$ @GreenAsJade you might need to think of a way to write and load from multiple redundant files to fight corruption. \$\endgroup\$ – bhathiya-perera Apr 21 at 10:52
  • \$\begingroup\$ Yeah I guess so. What I balk at is not understanding why/how that corruption occurs. Why isn't this code itself bullet proof? \$\endgroup\$ – GreenAsJade Apr 24 at 1:16
  • \$\begingroup\$ To be resilient is hard work. To have some resilience maybe store 2 copies of same data, store data checksums, backups that you can recover from, etc. All hardware, software, OSes, cables, Python interpreter, C runtime, etc.. are subject to failure. \$\endgroup\$ – bhathiya-perera Apr 24 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.