Improve file-path finding method

Question

I am using a recursive algorithm to find all of the file-paths in a given directory: it returns a dictionary like this: {'Tkinter.py': 'C:\Python27\Lib\lib-tk\Tkinter.py', ...}.

I am using this in a script to open modules by solely given the name. Currently, the whole process (for everything in sys.path) takes about 9 seconds. To avoid doing this every time, I have it save to a .pkl file and then just load this in my module-opener program.

The original recursive method took too long and sometimes gave me a MemoryError, so what I did was create a helper method to iterate through the subfolders (using os.listdir), and then call the recursive method.

Here is my code:

import os, os.path
def getDirs(path):
    sub = os.listdir(path)
    paths = {}
    for p in sub:
        print p
        pDir = '{}\{}'.format(path, p)
        if os.path.isdir(pDir): 
            paths.update(getAllDirs(pDir, paths))
        else:
            paths[p] = pDir
    return paths

def getAllDirs(mainPath, paths = {}):
    subPaths = os.listdir(mainPath)
    for path in subPaths:
        pathDir = '{}\{}'.format(mainPath, path)
        if os.path.isdir(pathDir):
            paths.update(getAllDirs(pathDir, paths))
        else:
                paths[path] = pathDir
    return paths

Is there any way to make this faster? Thanks!

Answer 1

import os, os.path
def getDirs(path):

Python convention is to use lowercase_with_underscores for function names

    sub = os.listdir(path)

Don't needlessly abbreviate, and at least have it be a plural name.

    paths = {}
    for p in sub:

You don't need to store things in a temporary variable to loop over them

        print p

Do you really want this function printing?

        pDir = '{}\{}'.format(path, p)

Use os.path.join to join paths. That'll make sure it works regardless of your platform.

        if os.path.isdir(pDir): 
            paths.update(getAllDirs(pDir, paths))

You shouldn't both pass it and update it afterwords. That's redundant.

        else:
            paths[p] = pDir
    return paths

def getAllDirs(mainPath, paths = {}):

Don't use mutable objects as default values, they have unexpected behavior.

    subPaths = os.listdir(mainPath)
    for path in subPaths:
        pathDir = '{}\{}'.format(mainPath, path)
        if os.path.isdir(pathDir):
            paths.update(getAllDirs(pathDir, paths))
        else:
            paths[path] = pathDir
    return paths

This whole section is repeated from the previous function. You should combine them.

Take a look at the os.walk function. It does most of the work you're doing here and you could use to simplify your code.