Python Function that Replicates a Directory's File-Tree for Specific File Types while Changing the Parent Directory

Problem Context: I needed to make a script that would search through all the files in a GitHub Repo and find all the README files within it, then replicate the File-Tree of the Repo to a specified destination directory.

Solution: I made a function that takes in two directories (source, destination). The directory inputs are Path objects. The source_dir input is validated in the main() function to exist.

import os # For walking through a filetree.
import shutil # For copying files from src to dest
import sys # For CLI argument handling
from pathlib import Path # For proper handling of file paths regardless of OS back/forward slash convention

readme_filepath_list_original = [] # We need to store a list of all the README filepaths we found to preserve their filetree while changing their *common* parent directory later
readme_filepath_list_new = [] # Then we need to store the new file paths that will serve as the respective destinations where we copy the old files to
for root, dirs, files in os.walk(source_dir): # Note: On Python 3.5+ os.walk() is implemented using os.scandir() resulting in a massive performance boost (5~20x) compared to python 3.4-
for file in files:
for path in readme_filepath_list_original: # We need to build a new version of the list above with the common path changed to our destination directory
if path.startswith(str(source_dir)):
for src_path, dest_path in zip(readme_filepath_list_original, readme_filepath_list_new): # Because we appended to our new list in exactly the same order as the old one, we can iterate simultaneously over both and copy files
os.makedirs(os.path.dirname(dest_path), exist_ok=True) # shutil.copy() expects folders that any files will be copied to, to already exist or it throws an error
print("| Original Path: {}\n| New Path: {}\n".format(src_path, dest_path))
shutil.copy(src_path, dest_path)
print("CREATED OUTPUT FOLDER '{}' which contains any detected README.md files, preserved in their original file structure.".format(dest_dir))


Question: I do not know if this is the most effective way to replicate the file tree with a changed parent directory and I was hoping to see if there was a better way to do this.

This is more complicated than it needs to be, and doesn't make good use of pathlib. You should be using rglob instead of doing a nested loop.

This:

if path.startswith(str(source_dir)):


is a problem, because

1. it should never evaluate to false, since you've acquired path from source_dir
2. if it did evaluate to false, then the sequences over which you zip are now misaligned and you'll be copying to the wrong place.

find_readmes does not do what's on the tin: it both finds and copies. Rename it, and/or separate it.

Suggested

from pathlib import Path
from shutil import copy
from typing import Iterator, Iterable

def find_readmes(source_dir: Path, dest_dir: Path) -> Iterator[tuple[Path, Path]]:
dest = dest_dir / source.relative_to(source_dir)
yield source, dest

def copy_readmes(paths: Iterable[tuple[Path, Path]]) -> None:
for source, dest in paths:
print(f'| Original Path: {source}\n'
f'| New Path: {dest}')
dest.parent.mkdir(parents=True, exist_ok=True)
copy(source, dest)