List directories, subdirectories and files, while ignoring some dirs

Question

In my application, the user may or may not want to ignore some directories. I can do that, but it seems like I am repeating myself. Does anyone have an idea to refactor that?

from os import walk, path
exclude = ['dir1/foo']
for root, dirs, files in walk('.', topdown=True):
    if exclude != None:
        dirs[:] = [d for d in dirs if d not in exclude]
        for name in files:
            for excluded in exclude:
                if excluded not in root:
                    print path.join(root, name)
    else:
        for name in files:
            print path.join(root, name)

exclude is None when there are no dirs to exclude. I thought of setting it to an empty list, but then, this loop for excluded in exclude: won't execute at all. My ambition was to avoid such a big if/else. Any ideas?

---

Example:

gsamaras@pc:~/mydir$ ls */*
dir1/bar:
test.txt

dir1/foo:
test.txt

dir2/bar:
test.txt

dir2/foo:
test.txt

I am getting:

./dir2/foo/test.txt
./dir2/bar/test.txt

./dir1/bar/test.txt

If I want, I can do an exclude = ['foo'], and then get:

./dir2/bar/test.txt
./dir1/bar/test.txt

meaning that I ignored all directories named "foo".

Answer 1

You are already modifiyin the list of directories, so that should be enough. But your exclude includes the full path, so your check in the list comprehension does not actually filter the excluded directories, you only do that in the for loop below when already having descended into those excluded directories (wasting time).

So, this should work:

from os import walk, path

exclude = {'./dir1/foo'}

for root, dirs, files in walk('.'):
    if exclude is not None:
        dirs[:] = [d for d in dirs if path.join(root, d) not in exclude]
    for name in files:
        print path.join(root, name)

Note that exclude needs to contain paths starting with the starting point of os.walk, so in this case ..

I also made exclude a set (\$\mathcal{O}(1)\$ in lookup), used the fact that topdown=True by default and used is not instead of != for comparison to None.

---

If you want to instead exclude folder names (regardless of their position in the directory tree), you can do that as well like this:

from os import walk, path

exclude = {'foo'}

for root, dirs, files in walk('.'):
    if exclude is not None:
        dirs[:] = [d for d in dirs if d not in exclude]
    for name in files:
        print path.join(root, name)

What is not possible with either of these two approaches is to include foo only in sub-directories of dir1, like in your example. However, I think this is more consistent behaviour, so you should choose one of the two, IMO.

---

As a last point, you should probably switch to Python 3 sooner rather than later if at all possible, because support for Python 2 will end in a bit more than a year (at the time of writing this).