Recursive os.scandir() for copying/moving files preserving subdirectories

Question

This is modified from the general recursive solution here: https://stackoverflow.com/a/54544211/42346 to, more specifically, copy/move files with a given extension from one directory to another whilst preserving subdirectories.

I'm reluctant to use .rename() for this purpose as it seems slightly less intuitive than copying or moving.

Any other ideas you have about how to clean this up are greatly appreciated.

import os, shutil
from pathlib import Path

def recur(path,destination,file_ext,is_subdir=False):
    if not is_subdir:
        os.chdir(path)
    for entry in os.scandir(path): 
        if os.path.splitext(entry.name)[-1].lower() == file_ext:
            subdir_path = entry.path.replace(os.getcwd() + os.sep,'')
            new_path = Path(destination) / subdir_path 
            os.makedirs(os.path.dirname(new_path), exist_ok=True) 
            shutil.copy(entry.path,new_path)
        if entry.is_dir():
            recur(Path(path).joinpath(entry.name),destination,file_ext,is_subdir=True)

Answer 1

You use pathlib.Path, which is great, but it can do a lot more:

• Operations like os.path.splitext should be modernized using the suffix attribute of Path objects; for a full list of those designated replacements, see the documentation.
• os.scandir can in this case be replaced by Path.rglob. This is where the recursion you mentioned comes into play. But that's it: Recursive globbing is an irrelevant implementation detail here. The point is that we do not have to do it "manually". rglob takes a pattern as its string argument, which can be the suffix you mentioned.
• Further, os.makedirs can be Path.mkdir with parents=True for the latter. This is not as big a win as the others, but nice.

Eventually, it turns out the os import can be done without altogether, and recursion is neatly tucked away:

from pathlib import Path
from shutil import copy2  # "identical to copy() but attempts to preserve to file metadata"

COPY_FROM = Path("source")
COPY_TO = Path("dest")

SUFFIX = "*.py"

for source in COPY_FROM.rglob(SUFFIX):
    subpath = source.relative_to(COPY_FROM)
    destination = COPY_TO.joinpath(subpath)
    destination.parent.mkdir(parents=True, exist_ok=True)
    copy2(source, destination)

Note the use of copy2, which is identical to copy but attempts to preserve metadata. This can be convenient for e.g. music files.

The parent attribute is the logical parent path to the found file, so essentially all path elements except for the found file itself. Like in your solution, those path elements (directories) have to be created first. In mkdir, exist_ok=False is the default and would prohibit us from most desired operations, like copying files in subdirectories.

The above gives the following result, which is hopefully what you are aiming for:

~$ tree
.
├── dest
│   ├── file1.py
│   ├── test1
│   │   ├── file2.py
│   │   └── file3.py
│   └── test2
│       ├── file4.py
│       └── test3
│           └── file5.py
└── source
    ├── file1.py
    ├── test1
    │   ├── file2.py
    │   └── file3.py
    └── test2
        ├── file4.py
        └── test3
            └── file5.py