set can hit a race condition if you use them together to achieve something like locking because it is not atomic. A Simple
memcache locking logic using
set would look like:
def access(resource): if memcache.get(resource): return "Access Denied" else: memcache.set(resource, timestamp) return "Access Granted and Lock Obtained"
If this code is running in multi-threaded web applications, you can have a scenario where two
get can happen together. Hence both will get the access the resource.
add doesn't get into this problem because it just sets the
key only if the
key does not exists. It also makes only a single roundtrip and hence it is atomic by default.
The same API with
add would look like:
def access(resource): if memcache.add(resource, timestamp): return "Access Granted and Lock Obtained" else: return "Acess Denied"
Now, I am writing a script to simulate this race condition while using
set and prove that using
add solves the concurrent access problem with this naive
memcache lock implementation. I am using files as my resource here. Multiple users can see the same file in the web application front-end and move it to another location if they get a
lock to that file. However, sometimes
set race condition happens giving access to the file for two users leading to
file not found error for one of the users. Using
add doesn't lead to this problem.
The script uses
threading to simulate this scenario. If a
file not found error happens then its considered a failure. I have two variants of the
move function, one using
set and another using
add which can be used in the
run method of the
thread to demonstrate both functionalities.
Can anyone review this and tell me if this code is fine? A high level logical review is also fine if it is hard to set this up in your machine. One thing I notice is that the ratio of error is pretty high, but is this because I am actually doing
threading which really is not random.
Pre-requisites: You need
memcache module and
memcached server running on port
5551 to run this. You also need a folder called
archive to move the files.
import memcache import signal import sys import shutil import os import threading import time mc = memcache.Client(['127.0.0.1:5551'], debug=0) cwd = os.getcwd() error = 0 processed = 0 def touch(fname, times=None): with open(fname, 'a'): os.utime(fname, times) def signal_handler(signal, frame): print('You pressed Ctrl+C!') sys.exit(0) signal.signal(signal.SIGINT, signal_handler) print("Ctrl-C exits") def create_files(): for fname in range(1,11): touch(str(fname) + ".txt") def move_files_get_set(): global error global processed files = [file for file in os.listdir(".") if file.endswith(".txt")] for file in files: if os.path.isfile(file) and not mc.get(file): processed = processed + 1 mc.set(file, int(round(time.time() * 1000))) try: shutil.move(os.path.join(cwd, file), os.path.join(cwd, "archive", "%s" % file)) except Exception as e: print(e) processed = processed - 1 error = error + 1 print("%s errors happened with %s" % (error, processed)) mc.set(file, None) def move_files_add(): global error global processed files = [file for file in os.listdir(".") if file.endswith(".txt")] for file in files: if os.path.isfile(file) and mc.add(file, int(round(time.time() * 1000))): processed = processed + 1 try: shutil.move(os.path.join(cwd, file), os.path.join(cwd, "archive", "%s" % file)) except Exception as e: print(e) processed = processed - 1 error = error + 1 print("%s errors happened with %s" % (error, processed)) mc.set(file, None) class myThread (threading.Thread): def __init__(self, threadID, name, counter): threading.Thread.__init__(self) self.threadID = threadID self.name = name self.counter = counter def run(self): print "Starting " + self.name move_files_get_set() print "Exiting " + self.name def trigger(): while True: create_files() t1 = myThread("1", "Thread 1", "1") t2 = myThread("2", "Thread 2", "2") t1.start() t2.start() t1.join() t2.join()