Python Function that Replicates a Directory's File-Tree for Specific File Types while Changing the Parent Directory

Question

Problem Context: I needed to make a script that would search through all the files in a GitHub Repo and find all the README files within it, then replicate the File-Tree of the Repo to a specified destination directory.

Solution: I made a function that takes in two directories (source, destination). The directory inputs are Path objects. The source_dir input is validated in the main() function to exist.

import os # For walking through a filetree.
import shutil # For copying files from src to dest
import sys # For CLI argument handling
from pathlib import Path # For proper handling of file paths regardless of OS back/forward slash convention

def find_readmes(source_dir, dest_dir):
    readme_filepath_list_original = [] # We need to store a list of all the README filepaths we found to preserve their filetree while changing their *common* parent directory later
    readme_filepath_list_new = [] # Then we need to store the new file paths that will serve as the respective destinations where we copy the old files to
    for root, dirs, files in os.walk(source_dir): # Note: On Python 3.5+ os.walk() is implemented using os.scandir() resulting in a massive performance boost (5~20x) compared to python 3.4-
        for file in files:
            if file.endswith("README.md"):
                readme_filepath = os.path.join(root, file)
                readme_filepath_list_original.append(readme_filepath) # Now that we found a README.md file, add it the list mentioned above
    for path in readme_filepath_list_original: # We need to build a new version of the list above with the common path changed to our destination directory 
        if path.startswith(str(source_dir)):
            readme_filepath_list_new.append(path.replace(str(source_dir), str(dest_dir), 1))
    for src_path, dest_path in zip(readme_filepath_list_original, readme_filepath_list_new): # Because we appended to our new list in exactly the same order as the old one, we can iterate simultaneously over both and copy files
        os.makedirs(os.path.dirname(dest_path), exist_ok=True) # shutil.copy() expects folders that any files will be copied to, to already exist or it throws an error
        print("| Original Path: {}\n| New Path: {}\n".format(src_path, dest_path))
        shutil.copy(src_path, dest_path)
    print("CREATED OUTPUT FOLDER '{}' which contains any detected README.md files, preserved in their original file structure.".format(dest_dir))

Question: I do not know if this is the most effective way to replicate the file tree with a changed parent directory and I was hoping to see if there was a better way to do this.

Answer 1

This is more complicated than it needs to be, and doesn't make good use of pathlib. You should be using rglob instead of doing a nested loop.

This:

if path.startswith(str(source_dir)):

is a problem, because

1. it should never evaluate to false, since you've acquired path from source_dir
2. if it did evaluate to false, then the sequences over which you zip are now misaligned and you'll be copying to the wrong place.

find_readmes does not do what's on the tin: it both finds and copies. Rename it, and/or separate it.

Suggested

from pathlib import Path
from shutil import copy
from typing import Iterator, Iterable


def find_readmes(source_dir: Path, dest_dir: Path) -> Iterator[tuple[Path, Path]]:
    for source in source_dir.rglob('*README.md'):
        dest = dest_dir / source.relative_to(source_dir)
        yield source, dest


def copy_readmes(paths: Iterable[tuple[Path, Path]]) -> None:
    for source, dest in paths:
        print(f'| Original Path: {source}\n'
              f'| New Path: {dest}')
        dest.parent.mkdir(parents=True, exist_ok=True)
        copy(source, dest)


copy_readmes(find_readmes(Path('../some-source'), Path('./dest-readmes')))