Automate the Boring Stuff Chapter 10 - Filling gaps in a sequence

Question

The project outline:

Write a program that finds all files with a given prefix, such as spam001.txt, spam002.txt, and so on, in a single folder and locates any gaps in the numbering (such as if there is a spam001.txt and spam003.txt but no spam002.txt). Have the program rename all the later files to close this gap.

My solution:

import re
from pathlib import Path

def filename_change(marker, stem, extension, pattern_len):
    correct_number = str(marker).zfill(pattern_len)
    new_filename = f"{stem}{correct_number}.{extension}"
    return Path(new_filename)

def check_gaps(basedir, prefix_pattern):
    sequence = []
    marker = 0
    pattern_len = len(prefix_pattern)
    filename_reg = re.compile(r"(.*)(\d{%s})\.(.*)" % pattern_len)
    for filename in basedir.glob("*"):
        match = filename_reg.search(str(filename))
        if not match:
            continue
        sequence.append(filename)
    sequence.sort()
    for filename in sequence:
        match = filename_reg.search(str(filename))
        stem = match.group(1)
        prefix = match.group(2)
        extension = match.group(3)
        marker = marker + 1
        if int(prefix) != marker:
            new_filename = filename_change(marker, stem, extension, pattern_len)
            filename.rename(new_filename)

def main():
    while True:
        basedir = Path(input("Please enter a folder to search: "))
        if not basedir.is_dir():
            print("This path does not exist.")
            continue
        prefix_pattern = input("Enter a pattern to find. It must be zero padded (e.g. '001', '01', not '1'): ")
        check_gaps(basedir, prefix_pattern)

if __name__ == '__main__':
    main()

I assumed that the order of the sequence was important and so opted to rename every file after the gap rather than filling the gaps with files from the end of the list. This seems fine for small sequences but I wondered if there was a more efficient way when dealing with hundreds of files.

Answer 1

Forewords

I do believe that you misunderstood the prefix part of the task and that your implementation treats the number or index of the files as the prefix and that you call the prefix stem in your code. It thus result in spam001.txt, spam101.txt and unrelated007.txt being renamed into spam001.txt, spam002.txt and unrelated003.txt.

A prefix being something that comes before the information, I do believe that spam, in the example, is the prefix of the files we need to rename. And the prefix will act as a filter on which "series" of files the script needs to deal with.

In your implementation, though, you mostly use the user provided information to know how to pad the resulting filename with zeroes. This is an interesting question as the task at hand does not explicit how it should be dealt with but I will talk about it latter.

User input

As you will learn to automate (boring) stuff, you’ll find out that an interactive script as you did that ask the user about further information it needs to proceed is quite tedious to automate or even test in quick succession.

A better and more common approach is to provide options on the command-line and implement a parser within your script. Python provides the argparse module for such tasks. You can easily say that you want to provide a folder to your script and (optionally?) a prefix to filter the files within that folder.

Iteration

sequence = []
for … in …:
    …
    sequence.append(…)

Is a code smell, you should be better off writing a list-comprehension:

sequence = [filename for filename in basedir.glob('*') if filename_reg.search(str(filename))]

You can even use the capabilities of glob to incorporate the prefix in your search and simplify further the extraction of the index by using a simple slice instead of the re module:

sequence = [filename for filename in basedir.glob(prefix + '*') if filename.stem[len(prefix):].isnumeric()]

This makes sure that you match any spamXXX.txt while excluding both morespamYYY.txt and spammerZZZ.txt.

You also use a construct like:

marker = 0
for … in …:
    …
    marker = marker + 1
    …

Which is a convoluted way around enumerate.

Path manipulations

Using pathlib to manipulate filenames is a powerful tool that you don't leverage much. You apply the regex against the whole path instead of only the name or even the stem of the file, which leads to an expression more complex than it needs to be, and more difficult to handle.

When renaming the path, you also recreate a whole Path from scratch instead of only changing the relevant part through means of with_name or with_stem.

This adds up in terms of complexity for the reader of your code compared to, e.g.:

new_name = f'{prefix}{expected_index:03}'
filename.rename(filename.with_stem(new_name))

Also note the :03 format specifier when turning the expected_index (marker in your code) into a string, this feels clearer than str(expected_index).zfill(3). And you can still parametrize it: new_name = f'{prefix}{expected_index:0{padding}}'.

Padding

Because filenames may or may not have their index padded with leading zeroes, I find it quite tricky to have a "one-size-fits-all" kind of approach. Using .isnumeric on the remaining characters of the filename after the prefix is a first step to detect indexes of any length, but this says nothing about whether or not the resulting filenames should be padded or not.

We could compute the len of the digits at the end of the filename and use that as the padding of the resulting filename, but if they aren't padded at all, a filename with a two-digits index (say spam11.txt) being renamed to a one-digit index (spam08.txt) will keep an extra leading zero, which is unexpected.

So I guess the best course of action here is to let the user tell whether or not the script need to guess for the padding to apply or just use no padding at all.

Proposed improvements

"""
Check and fill gaps in a directory.

Providing a prefix, scan a folder for files numbered after
this prefix and remove gaps by re-numbering them if necessary.
"""


import argparse
from pathlib import Path


def check_and_fill_gaps(folder: Path, prefix: str = '', ignore_padding: bool = False) -> None:
    files = sorted(
            (int(index), path)
            for path in folder.glob(prefix + '*')
            if (index := path.stem[len(prefix):]).isnumeric()
    )
    for expected_index, (index, path) in enumerate(files, start=1):
        if index != expected_index:
            padding = 0 if ignore_padding else len(path.stem) - len(prefix)
            path.rename(path.with_stem(f'{prefix}{expected_index:0{padding}}'))


def folder(value: str) -> Path:
    f = Path(value)
    if not f.is_dir():
        raise argparse.ArgumentError(f'{value} is not an existing directory')
    return f


def command_line_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument('folder', type=folder, help='folder in which run the scan and replace process')
    parser.add_argument('-p', '--prefix', default='', help='search for files of the form `prefixXXX.ext` where XXX is any amount of digits')
    parser.add_argument('-n', '--ignore-padding', action='store_true', help='do not try to 0-pad resulting filenames when renaming')

    return parser


if __name__ == '__main__':
    args = command_line_parser().parse_args()
    check_and_fill_gaps(**vars(args))

Note the use of the walrus operator to avoid extracting the index from path.stem twice.

Example usage

Using the following test folder:

$ ls -1 testing
morespam42.txt
spam103.txt
spam10.txt
spam11.txt
spam19.txt
spam1.txt
spam21.txt
spam23.csv
spam2.txt
spam3.txt
spam6.txt
spam7.txt
spammer007.xls

We can filter with padding guessed:

$ python fill_gaps.py -p spam testing; ls -1 testing
morespam42.txt
spam011.txt
spam06.txt
spam07.txt
spam08.txt
spam09.txt
spam10.csv
spam1.txt
spam2.txt
spam3.txt
spam4.txt
spam5.txt
spammer007.xls

Or without:

$ python fill_gaps.py -p spam -n testing; ls -1 testing
morespam42.txt
spam10.csv
spam11.txt
spam1.txt
spam2.txt
spam3.txt
spam4.txt
spam5.txt
spam6.txt
spam7.txt
spam8.txt
spam9.txt
spammer007.xls

Handling adding a fixed padding to every filename is left as an exercise to the reader.

Answer 2

Regex is pretty expensive, and you're looping over the list twice. If you can assume that the operating system is going to order them, you can get away with just renaming as soon as you run into one that doesn't fit into the sequence:

from pathlib import Path


def close_gaps(directory: Path, prefix: str = "spam") -> None:
    for i, file in enumerate(directory.listdir()):
        if not file.stem.endswith(str(i)):
            new_name = Path(file.parent / f"{prefix}{str(i).zfill(3)}")
            file.rename(new_name)


parser = ArgumentParser()
parser.add_argument("directory", nargs=1, type=Path, help="target directory to close gaps in")
args = parser.parse_args()

close_gaps(**vars(args))