2
\$\begingroup\$

A new function based version of the code from this question as suggested by @Hlib Babii

I write a lot of TODO's but I never keep track of where they are.

This program will search through files (and recursively through folders if needed) and find any TODO comments and their line number.

The details will be printed to the screen and a file will be generated in the folder which is a csv of the files and TODO details. This means there is always a local list of TODOs in the folder and this can be stroed in git..

You can choose different filetypes, comments and TODO phrases to suit , so it is not just for python.

GitHub

from os.path import exists
import sys
from glob import glob

def _get_files(path, ext, recursive=False):
    # return a list of files 
    return glob(path + f"/**/*{ext}", recursive=recursive)    

def _find_todos_in_file(fn, todo_token, comment_start):
    # return a list of todos in the file
    temp_todos = []
    with open(fn, "r") as input_:
        for line_no, line in enumerate(input_):                
            if comment_start in line and todo_token in line:
                # check to make sure that it is a true comment and not a variable name.
                # Avoid false positives like :: `TODOs.append(todo) # there are no todos in this line`
                comment_index = line.find(comment_start)
                todo_index = line.find(todo_token)                
                if todo_index > comment_index:
                    temp_todos.append([fn, f"Line - {line_no+1}  ::  {line.strip()}"])
    return temp_todos

def find_todos(path, ext, todo_token = 'TODO', comment_start = '#', recursive=False):
    # returns a dictionary of todos
    todos = {}
    files = _get_files(path, ext,recursive=recursive)
    if exists(path):
        for x in files:
            try:
                print(f"Searching  ::  {x}")
                result = _find_todos_in_file(x, todo_token, comment_start)
                if result:
                    todos[x] = result
            except PermissionError:
                pass # not a ext file (possible a folder)
    else:
        raise OSError("Path does not exist.")
    return todos

def show_todos(todos: dict):
    # show todos
    line = "-" * 100
    for k, v in todos.items():
        print(f"\n{line}\n\n{k}")
        for x in v:
            print(f">>>{x[1]}")

def save_csv(todos, ext):
    # save todos to a csv file
    import csv
    for k, v in todos.items():       
        k = k.split(ext)[0][:-1]
        with open(f"{k}-TODOS.csv", "w", newline="") as csvfile:
            w = csv.writer(csvfile, delimiter=",", quoting=csv.QUOTE_MINIMAL)
            for row in v:
                w.writerow(row)

if __name__ == "__main__":
    todos = find_todos('.', ext='py', comment_start='#', recursive=True)
    show_todos(todos)
    save_csv(todos, 'py')
\$\endgroup\$
3
\$\begingroup\$

Overall, pretty good job. Here are some observations:

_get_files

path + f"/**/*{ext}"

Might cause trouble if path ends in a / and may be unsafe. If a user inputs an empty path, this could start at the root of the file system, and try to scan every file. Consider checking for such things and maybe using os.path.normpath or path.resolve() in pathlib. pathlib also provides exists() and glob().

find_todos()

if exists(path) should probably come before it is used in the call to _get_files():

if exists(path):
    files = _get_files(path, ext,recursive=recursive)
    ...

A PermissionError might occur when a filed is opened, so I would tend to put the try:...except PermissionError: block in _find_todos_in_file()

_find_todos_in_file()

Each line is scanned at least twice, and maybe four times:

if comment_start in line and todo_token in line: # two scans
        ....
    comment_index = line.find(comment_start)     # one scan
    todo_index = line.find(todo_token)           # one scan

line.find(comment_start) returns -1 if the comment_start isn't in the line. Also, find() takes a parameter that says where to start searching. So the above code can be simplified and made more efficient like so:

comment_index = line.find(comment_start)
if comment_index >= 0:
    todo_index = line.find(todo_token, comment_index)

    if todo_index >= 0:
        temp_todos.append((line_no+1, line.strip))

I changed what gets appended. The file name is redundant, it gets passed in as an argument, so the caller already knows what it is -- find_todos() uses it as a dict key. And I put line_no and the text in a tuple rather than format them into a string. Putting them in a string seems more like a job for an output function. And keeping them as a tuple is more more flexible (e.g. you want to put them in a database, or see if TODOs show up more near the end of a file, etc).

other stuff

find_todos() and find_todos_in_file() basically iterate over all the lines in a bunch of files. This is what the fileinput library does. These two function could be combined and rewritten something like this:

def find_todos(path, ext, todo_token, comment_start, recursive):
    todos = defaultdict(list)

    files = glob("{path}/**/*{ext}", recursive=recursive)

    with fileinput.input(files=files) as f:
        for line in f:
            if f.isfirstline():
                filename = f.filename()
                print(f"Searching: {filename}")

            comment_index = line.find(comment_start)
            if comment_index >= 0:
                todo_index = line.find(todo_token, comment_index)

                if todo_index >= 0:
                    todos[filename].append((f.filelineno(), line.strip))

For a simple command line tool, the argparse library can help with parsing command parameters and options. For more complicated or enhanced tools try a command line tool library like Click or Python Prompt.

def make_arg_parser():
    import argparse

    parser = argparse.ArgumentParser(
                        description='TODO finder/lister.')

    parser.add_argument('path',
                        default='.',
                        help="where to search for TODO's")
    parser.add_argument('ext',
                        default='py',
                        help="kind of files to search")
    parser.add_argument('comment',
                        default='#',
                        help="line comment marker")
    parser.add_argument('todo',
                        default='TODO',
                        help="text to mark TODO items")
    parser.add_argument('--norecurse',
                        action='store_false',
                        dest='recursive',
                        help="don't recurse into subdirs")
    parser.add_argument('--csv',
                        action='store_true',
                        help="save to a csv file")

    return parser

if __name__ == "__main__":
    cli_parser = make_arg_parser()
    args = cli_parser.parse_args()

    todos = find_todos(args.path,
                       ext=args.ext,
                       todo_token=args.todo,
                       comment_start=args.comment,
                       recursive=args.recursive)
show_todos(todos)

if args.csv:
    save_csv(todos, 'py')

The above is just an example. It mirrors the call to find_todos() and adds an argument for todo_token and a flag to save to a csv file.

\$\endgroup\$
  • \$\begingroup\$ Seems like you're missing a " in the final if __name__ == .... \$\endgroup\$ – AlexV Jul 18 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.