3
\$\begingroup\$

Problem Context: I needed to make a script that would search through all the files in a GitHub Repo and find all the README files within it, then replicate the File-Tree of the Repo to a specified destination directory.

Solution: I made a function that takes in two directories (source, destination). The directory inputs are Path objects. The source_dir input is validated in the main() function to exist.

import os # For walking through a filetree.
import shutil # For copying files from src to dest
import sys # For CLI argument handling
from pathlib import Path # For proper handling of file paths regardless of OS back/forward slash convention

def find_readmes(source_dir, dest_dir):
    readme_filepath_list_original = [] # We need to store a list of all the README filepaths we found to preserve their filetree while changing their *common* parent directory later
    readme_filepath_list_new = [] # Then we need to store the new file paths that will serve as the respective destinations where we copy the old files to
    for root, dirs, files in os.walk(source_dir): # Note: On Python 3.5+ os.walk() is implemented using os.scandir() resulting in a massive performance boost (5~20x) compared to python 3.4-
        for file in files:
            if file.endswith("README.md"):
                readme_filepath = os.path.join(root, file)
                readme_filepath_list_original.append(readme_filepath) # Now that we found a README.md file, add it the list mentioned above
    for path in readme_filepath_list_original: # We need to build a new version of the list above with the common path changed to our destination directory 
        if path.startswith(str(source_dir)):
            readme_filepath_list_new.append(path.replace(str(source_dir), str(dest_dir), 1))
    for src_path, dest_path in zip(readme_filepath_list_original, readme_filepath_list_new): # Because we appended to our new list in exactly the same order as the old one, we can iterate simultaneously over both and copy files
        os.makedirs(os.path.dirname(dest_path), exist_ok=True) # shutil.copy() expects folders that any files will be copied to, to already exist or it throws an error
        print("| Original Path: {}\n| New Path: {}\n".format(src_path, dest_path))
        shutil.copy(src_path, dest_path)
    print("CREATED OUTPUT FOLDER '{}' which contains any detected README.md files, preserved in their original file structure.".format(dest_dir))

Question: I do not know if this is the most effective way to replicate the file tree with a changed parent directory and I was hoping to see if there was a better way to do this.

\$\endgroup\$

1 Answer 1

3
\$\begingroup\$

This is more complicated than it needs to be, and doesn't make good use of pathlib. You should be using rglob instead of doing a nested loop.

This:

if path.startswith(str(source_dir)):

is a problem, because

  1. it should never evaluate to false, since you've acquired path from source_dir
  2. if it did evaluate to false, then the sequences over which you zip are now misaligned and you'll be copying to the wrong place.

find_readmes does not do what's on the tin: it both finds and copies. Rename it, and/or separate it.

Suggested

from pathlib import Path
from shutil import copy
from typing import Iterator, Iterable


def find_readmes(source_dir: Path, dest_dir: Path) -> Iterator[tuple[Path, Path]]:
    for source in source_dir.rglob('*README.md'):
        dest = dest_dir / source.relative_to(source_dir)
        yield source, dest


def copy_readmes(paths: Iterable[tuple[Path, Path]]) -> None:
    for source, dest in paths:
        print(f'| Original Path: {source}\n'
              f'| New Path: {dest}')
        dest.parent.mkdir(parents=True, exist_ok=True)
        copy(source, dest)


copy_readmes(find_readmes(Path('../some-source'), Path('./dest-readmes')))
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.