Multi-threading upload tool

Question

We have about 14G files and each is about 150k.

I'm writing a script to upload them to Azure Blob storage and would like to run an upload() function (each with own file to upload) in 5 threads.

Here is how I realized it and it looks like it works - but I still have doubts about how correct this code is.

...
class Loader:

    def __init__(self):

        self.account = 'accname'
        #self.container = 'userdata'
        self.container = 'bar1'
        key = 'DQ4***A=='

        self.base_url = 'http://' + self.account + '.blob.core.windows.net'
        self.blob = BlobService(account_name=self.account, account_key=key)

    def uploader(self, filepath, userfile, locker):

        print 'Uploading file: {}'.format(userfile)

        self.blob.put_block_blob_from_path(self.container,
                                           userfile,
                                           filepath,
                                           max_connections=1,
                                           max_retries=5,
                                           retry_wait=1.0)

        locker.release()

    def upload(self, path):

        print('Upload files from {} to {}'.format(path, self.base_url))

        locker = threading.BoundedSemaphore(5)

        for root, dirs, files in os.walk(path):
            print root
            for userfile in files:
                #print 'Uploading {}'.format(os.path.join(root, userfile))
                locker.acquire()
                t = threading.Thread(target=self.uploader, args=(os.path.join(root, userfile), userfile, locker))
                t.start()
...

And script in action:

$ ./storage_upload.py -u --path /tmp/bartest/
Upload files from /tmp/bartest/ to http://accname.blob.core.windows.net
/tmp/bartest/
Uploading file: 15.txt
Uploading file: 12.txt
Uploading file: 7.txt
Uploading file: 13.txt
Uploading file: 19.txt

Answer 1

Looks okay if a bit sloppy with different formatting for some print x vs. print(x) calls (the latter being preferred really); you probably should also use new-style classes, i.e. class Loader(object):.

Other than that the main concern I'd have is that the semaphore should be protected against exceptions. This is mostly a concern for bigger scripts, but it's a good habit regardless. Thus, the release method should be called regardless of whether an exception was raised by anything else in the thread - otherwise the program could just get stuck there, which is fine for a one-off script probably.

You should probably also check whether the threading actually does improve the throughput, considering Pythons global interpreter lock.