You have strange error handling in here:
except ValueError, e:
Why do you create a redundant dictionary? If you want nothing to happen, you can use the
pass keyword. It's designed to be placed where Python expects blocks of code so it can just do nothing. This avoids the syntax error in trying to leave out a block of code. It's much more readable to do that way than creating an empty dictionary for no reason.
If you do have a good reason, then leave a comment explaining it as it's not clear in the current context.
Also don't use
except ValueError, e because it's ambiguous (and specifically invalid syntax in Python 3). Consider if you needed to also catch an
except ValueError, IOError, e:
Now how is Python supposed to parse that? The comma separation doesn't make it clear what is a new exception and what should be the exception's name, so in Python 2 you should use the more explicit
except ValueError as e form, which is also more readable. Incidentally this makes it easier to now use multiple exceptions:
except (ValueError, IOError) as e:
myloop value seems to always be
True, nothing in this code ever changes it. So why is it even necessary?
You're also using nested loops to find all the file paths that are inside
glob could do this in one function call.
from glob import iglob
files = iglob('/FullData/RowData/*/*')
glob is the normal function that returns a list but
iglob returns a generator, which suits your needs better as it's more efficient when you're just iterating over it)
*s are wildcards, so any folder/file in those places will be matched. But you could make it more specific by adding say a filename at the end:
files = iglob('/FullData/RowData/*/*.data')
.data files in the nested folders will match. You know your filenames better than me, so you could figure out a good pattern. This means that you having a single loop:
if __name__ == "__main__":
client = KafkaClient('10.62.84.35:9092')
pool = Pool(30)
for f in iglob('/FullData/RowData/*/*.data'):
with open(f) as source_file:
# chunk the work into batches of 4 lines at a time
results = pool.map(process_line, source_file, 30)