I have a backup job that walks across a huge directory with potentially millions of files. Python's os.walk() works just fine.
Now, I have implemented a feature to ignore files based in a black list of patterns. I just want it to behave like a .gitignore file.
Currently I do this:
import os from fnmatch import fnmatch some_dir = '/' ignore_list = ['*.tmp', 'tmp/', '*.py'] for dirname, _, filenames in os.walk(some_dir): for filename in filenames: should_ignore = False for pattern in ignore_list: if fnmatch(filename, pattern): should_ignore = True if should_ignore: print 'Ignore', filename continue
It works, but I failed to write it in terms of iterators.
Can this routine be written in terms of python iterators? Can it be improved in terms of iterators or generators?