I want a 4-bit resource to share status between processes and to retain status after shutdown and reboot. If the system were a microprocessor, I'd use a 4-bit register in non-volatile memory.

Process interaction with the shared resource is typically minutes to hours between interactions, however processes may attempt to access it at exactly the same moment. So there is a simple file locking mechanism. A process that finds the resource locked, will back off for a short while and then try again.

There are only 4 or 5 processes that deal with the resource. The short while (timeout) is 15 milliseconds plus random between 0 to 9 milliseconds. This is to prevent race conditions, processes locking each other, etc.

Processes will be daemonised, and they will use the functions below to access the pickle.

To simulate different processes, I'm using a few command prompt windows, started at about the same time. See the code.

All of Python's multitasking libraries, threading, Queue and DMTCP seem way too elaborate for my tiny 4-bit register. Hence this solution. Please comment.

import cPickle as pickle
import math
from time import sleep
import sys

    There are 4 switches, A to D. Each can be On or Off.
    Switches must retain state on shutdown and/or reboot. 

    Switch state is stored in a pickle in switch_state.p
    Switch state is a 4-bit number, ie max 15 decimal (all On)
    Switch A is bit 0, switch B is bit 1, and so on. 

    Several processes may interact with switches. 
    So, also in the pickle is a boolean 'locked'.
    All processes, when switching, must load the pickle, and first check if it's locked. 
    If it is, it should wait until it becomes unlocked. 
    If it's not, it should lock it, do its thing, and then unlock it. 

def new_state(statefile='switch_state.p'):
        switch = pickle.load(open(statefile, 'rb'))
        switch = {'state':0, 'locked': False}
        pickle.dump(switch, open(statefile, 'wb'))

# functions to turn state on and off
# using bitwise operations, so no need to check current state
# i.e. if you turn on a switch that's already on, nothing will change

def turn_on(cur_state, switchID):
    ON_bits = pow(2, ord(switchID) - ord('A'))
    return cur_state | ON_bits

def turn_off(cur_state, switchID):
    OFF_bits = ~pow(2, ord(switchID) - ord('A'))
    return cur_state & OFF_bits

def update_state(switchID, state, statefile='switch_state.p'):
    switch = pickle.load(open(statefile, 'rb'))

    while switch['locked']:
        # some other process is switching, wait till it finishes
        sleep(0.015 + random.random()/100)
        switch = pickle.load(open(statefile, 'rb'))

    if not switch['locked']:
        new_switch = {'state': switch['state'], 'locked': True}
        pickle.dump(new_switch, open(statefile, 'wb'))
        if state == 'On':
            newstate = turn_on(switch['state'], switchID)
        elif state == 'Off':
            newstate = turn_off(switch['state'], switchID)
            #shouldn't ever get here
            print 'unknown state'

        new_switch = {'state': newstate, 'locked': False}
        pickle.dump(new_switch, open(statefile, 'wb'))
        report = 'switch {} is now {}, switchstate {}'
        print report.format(switchID, state, newstate)


# test the above in three or four command shells
# python switchreg.py [switchID] [delaytime]
# switchID may be A to D
# shorter delaytimes simulate race-conditions better 

switchID = sys.argv[1]
delaytime = float(sys.argv[2])
endrange = int(30/delaytime) # so time taken up by iterations is about the same

for i in range(0, endrange):
    update_state(switchID, 'On')
    update_state(switchID, 'Off')

3 Answers 3


The one major problem is the lack of synchronisation between processes, even if you randomise the sleep interval, this is still an invitation for race conditions (unless you can somehow proof that the randomisation is good enough).

Using a library like lockfile or directly using e.g. fcntl.lock will take care of that. If you are worried about the performance, well you are using Python, with cPickle (good point that). Otherwise some IPC mechanism, maybe a shared semaphore between processes, would suffice.

Next, I think you can leave the file simply open and rewind to the start, the read again instead of constantly closing and opening. I'm also not sure, but you could investigate file (region) locking via fcntl instead to provide the necessary synchronisation.

The turn_on/turn_off seems very complicated if you could just write out a dict, sequence, whatever, instead of doing bit manipulation. If, however, you really want that part, then be consistent and simply write two bytes to the file instead of using pickle. The mixture of very high-level serialisation and low-level composition of four bits doesn't seem necessary at all.

Otherwise looks fine. Some common expressions could be shared and the written data structure is mostly used without any abstraction (like you did with new_state, a updated_state function to create the new dict could be nice), but I guess for a small script that hardly matters.

  • \$\begingroup\$ Many thanks, I had not come across lockfile nor fcntl.lock yet nor IPC yet. Will certainly look into those. (Being self-taught with Python, I tend to play with code more than I google for the appropriate theory) \$\endgroup\$
    – RolfBly
    Nov 30, 2014 at 17:05

The most serious problem here is that such synchronization just doesn't work: there is a gap between loading a file and testing a state. During this gap (or even during the load), another process may also load the file, and both of them will see the state unlocked.

Using pow is sure an overkill. 1 << ord(switchID) - ord('A') works as well. I would also recommend to move an index calculation(ord(switchID) - ord('A')) out of turn_on and turn_off functions. They are too low level to worry about names of switches.

  • \$\begingroup\$ Thank you. Any suggestions for a better sync scheme? Second point (1 << etc) accepted as well, good idea! (Why didn't I think of that?) Re third point: switches are these big remote controlled AC switches, clearly labelled A, B etc., so using their names throughout is just more user-friendly. I'll admit it does get a bit geeky here. \$\endgroup\$
    – RolfBly
    Nov 30, 2014 at 16:53

I found a similar approach to the same problem on SO. The best answer imho is this one, in that same thread:

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don't want to solve.

Even with few processes attempting infrequently to access the shared file, a file locking scheme with time-out on collision will only work if power never fails. But power does fail, sometimes. The chance may be slim, but it's not zero. So you may find a broken transaction in the shared file, or a false lock on it, or whatever - as Kevin said: "all kinds of issues I don't want to solve."

So I dropped the shared file idea, I can do without it altogether for now. When a similar requirement arises, I think Queue is the way to go.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.