#Coding style 1. get consistent: most of your functions and variable names are lower_snake_case except `direcToData` (why not `directToData`?), you should stick to lower_snake_case which is recommended by PEP8. <!-- ol/ul separator --> * get consistent: you should have spacing around all your affectations (`=`). * use `except ValueError as e:`, the form using a coma is being deprecated and a syntax error in Python 3; * use `pass` instead of `{}` to create a no-op statement; * try to be consistent: indent with 4 spaces anywhere, not 9 as in your `except`; * `myloop` (why not `my_loop`?) is set to `True` and never changed, thus `if myloop` is unnecessary; * two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between `pool =` and `direcToData =`. Oh, and you use both `'` and `"` as string delimiter. You should try to use only one for consistency. #Directory traversal You may be able to simplify your file management using `os.walk`. If `"/FullData/RowData"` recursively contains only the files you wish to process, then you can write: for path, dirnames, filenames in os.walk('/FullData/RowData'): if filenames: # files exist in traversed directory full_names = map(lambda f, path=path: os.path.join(path, f), filenames) pool.map(process_file, full_names, 16) # Using @holroy advices The `process_file` function would then be: def process_file(filename): producer = SimpleProducer(client) with open(filename) as source_file: for line in source_file: try: jrec = json.loads(line.strip()) producer.send_messages('twitter2613', json.dumps(jrec)) except ValueError: # `as` nothing since we don't use the value pass This will also help process things faster since it will create only one producer per file instead of one per line.