This is a batch program to xcopy from host PC to remote destination with multi-processing.

Kepler, Python-3.x are my environment.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# based on Carnival http://ask.python.kr/users/6970/carnival/

import os
import os.path
import csv
import re

from multiprocessing import Process, Lock

class Server:
    def __init__(self, addr, path):
        self.addr = addr
        self.path = path

def distribute_file(server_list, pathname):
    print("Read from {}".format(pathname))  

    with open(pathname, 'rb') as inFile:
        buffer = inFile.read()

    for server in server_list:
        remotepath = "//%s/%s/%s" % (server.addr, server.path, pathname)
        print ("Write to {}".format(remotepath))
        with open(remotepath, 'wb') as outFile:

def multi_distribute_file(lo, server, dirpath, filename, path):

    pathname = os.path.join(dirpath, filename)    
    print("Read from {}".format(pathname)) 

    with open(pathname, 'rb') as inFile:
        buffer = inFile.read()

    if(path == ''):
        remotepath = "//%s/%s/%s" % (server.addr, server.path, filename)
        remotepath = "//%s/%s/%s/%s" % (server.addr, server.path, path, filename)

    print ("Write to {}".format(remotepath))
    with open(remotepath, 'wb') as outFile:


def make_dir(server_list, path):
    for server in server_list:    
        dirpath = "//%s/%s/%s" % (server.addr, server.path, path)
        d = os.path.dirname(dirpath)
        if not os.path.exists(d):
            print ("Make dir {}".format(d))         
            print ("dir {} already exists.".format(d))

        if not os.path.exists(dirpath):
            print ("Make dir {}".format(dirpath))          
            print ("dir {} already exists.".format(dirpath))

def dist_files(server_list, subdir):
    for dirpath, dirnames, filenames in os.walk(subdir):     

        l = re.findall(r"[\w']+",subdir)
        m = re.findall(r"[\w']+",dirpath)        
        path = ''
        if(l != m):
            path = m[len(m)-1]

        make_dir(server_list, path)            
        for filename in filenames:
            for server in server_list:
                lock = Lock()
                Process(target=multi_distribute_file,args=(lock, server, dirpath, filename, path)).start()                                

def get_server_list(filename):    
    mydictionary = []
    csvFile = csv.reader(open(filename, "r"))
    for row in csvFile:
      mydictionary.append(Server(row[0], row[1]))
      print("{}, {}".format(row[0], row[1]))

    return mydictionary

if __name__ == '__main__':        
    server_list = get_server_list('client_list.csv')
    dist_files(server_list, 'C:\\Users\\Public\\Test')


  •, Test
  •, Test
  •, Test

1 Answer 1


Technical issues:

  • You are reading whole files into memory at once. Consider what happens if the directory contains huge files.
  • You use multiprocessing, apparently in an attempt to parallelize things, and use locks to synchronize.
  • First, why do you need a lock anyway? Which data do you want to protect? The only shared data are in the filesystems, which could be changed by other processes at any time anyway.
  • Next, each call to multi_distribute_file gets its own lock instance, that is only used once by this call. So there is no synchronization happening at all.
  • On the other hand, if you would use a single lock for all calls to multi_distribute_file, the processes will execute one after another, as each acquires the lock for its entire runtime. Your parallelizing efforts would be in vain.
  • You are spawning a new process for each file being copied. This can create a huge number of processes very quickly. Consider what this will do to your system performance.
  • If something goes wrong in multi_distribute_file and raises an exception, the lock will never be released. And things will go wrong occassionaly as you are communicating over a network. Use a with statement instead.
  • Don't use string manipulation (format and re) to manipulate paths. Use os.path instead. This is saver and more reliable.

Style issues:

  • There are no docstrings and no comments (apart from the header)
  • Avoid single-letter and other abbreviated names like d, l, m, and lo.
  • mydictionary is a meaningless and misleading name. Meaningless, as it doesn't tell us anything about the purpose of the variable, and misleading as it is actually a list, not a dictionary. A better name might be server_list (like you do elsewhere).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.