Check if a file path matches any of the patterns in a blacklist

Question

I'm processing a list of files, and want to check the filenames against a list of regex, like:

IGNORE_FILES = [
    re.compile('^./Vendor.*'),
    re.compile('^./Pods.*'), 
    …
]

def in_blacklist(file):
    return len(list(filter(lambda r: r.match(file) != None, IGNORE_FILES))) > 0

def in_whitelist(file):
    return SWIFT_FILE_REGEX.match(file) != None

def files():
    valid_files = []
    for root, dirs, files in os.walk('.'):
        for file in files:
            if in_whitelist(file) and not in_blacklist(root):
                valid_files.append("%s/%s" % (root, file))
    return valid_files

I'm looking for a smoother way to write:

len(list(filter(lambda r: r.match(root) != None, IGNORE_FILES))) == 0

I feel that it's not so easy to read, and especially with the list(..)

Answer 1

You can build one regex out of the list of regexes by

'|'.join(IGNORE_FILES) 

(Well, before running re.compile on them...)

Then your in_blacklist function becomes as easy to write and read as in_whitelist.

I further assume that you don't want to ignore paths that start with any letter, but those starting with a literal dot (which is what os.walk('.') yields). So you need to escape those leading dots.

Also, in your files() function, you should move your in_blacklist check out of the inner loop. No need to check the same root directory multiple times.

So:

IGNORE_FILES = re.compile('|'.join([
    r'^\./Vendor.*',
    r'^\./Pods.*', 
]))

def in_blacklist(file):
    return IGNORE_FILES.match(file) != None

def in_whitelist(file):
    return SWIFT_FILE_REGEX.match(file) != None

def files():
    valid_files = []
    for root, dirs, files in os.walk('.'):
        if in_blacklist(root):
            continue
        for file in files:
            if in_whitelist(file):
                valid_files.append("%s/%s" % (root, file))
    return valid_files

And for more portability, you could replace "%s/%s" % (root, file) by os.path.join(root, file), but I wouldn't insist on this too hard...

Answer 2

I find your use of "blacklist" and "whitelist" problematic.

What is a "whitelist"? Which takes precedence — a whitelist or a blacklist? (Adding an e-mail address to a whitelist should let it bypass a spam filter.)

As @JoeWallis has already pointed out, if the directory is in the blacklist, then there is no need to iterate through its contents. To add to the confusion, the blacklist works on directories, but the whitelist works on filenames. It would be clearer to name them exclude_dirs and filename_filter. If, as I suspect, you are using the filename filter to look for a .swift extension, consider using fnmatch as a more natural alternative to a regex.

Answer 3

You can use a comprehension. Comprehensions are like easier to write maps and filters. Your in_blacklist can actually be a generator comprehension. One that yields rather than array.appends. Wrapped in an any. This is if there is at least one match it's True. And you don't really want to check the rest.

You can also change your files function to use one, all you need to do is move the discouraged printf string format to the beginning and keep the rest the same. You can also move if not in_blacklist(root) up so that if the root is not allowed then you're not spending time on the children.

IGNORE_FILES = [
    re.compile('^./Vendor.*'),
    re.compile('^./Pods.*'), 
    …
]

def in_blacklist(string):
    return any(r.match(string) is not None for r in IGNORE_FILES)

def in_whitelist(file):
    return SWIFT_FILE_REGEX.match(file) is not None

def files():
    return [
        '{}/{}'.format(root, file)
        for root, dirs, files in os.walk('.')
        if not in_blacklist(root)
        for file_name in files
        if in_whitelist(file_name)
    ]

Answer 4

Type hints

Since you are using Python 3.x you could take advantage of the new type hints. According to PEP 484:

This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information.

Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.

Even if you don't use static code analysis at the moment, type hints still have the advantage of making the code easier to read and understand.

---

In your case, adding type hints (along with adding a verb like is_something instead of something) to functions would make the code more clear. Since this is about file names, I'd also suggest using file_name instead of file, so it's clear we're dealing with a string and not an actual file. For example:

def is_in_whitelist(file_name: str) -> bool:
    return SWIFT_FILE_REGEX.match(file_name) != None

---

In the same vein, IGNORE_FILES may be better named as IGNORED_ROOT_NAMES since that is what they actually are. pixelbrei's answer addressed a great way to simplify it.