Resolving Paths relative to script file, independent of calling location

Question

Idea

Consider the following snippet:

import inspect
from pathlib import Path


def path_relative_to_caller_file(*pathparts: str) -> Path:
    """Provides a new path as a combination of the caller's directory and a subpath.

    When creating a Path like Path("resources", "log.txt"), the containing Python script
    has to be called from within the directory where the subdirectory "resources" is
    found. Otherwise, the relative path breaks with an error.
    This function provides a new path to always assure those relative paths are found,
    no matter from where the script containing the relative path definition is called.
    Example:

    A/
    └── B/
        └── C/
            ├── script.py
            └── D/
                └── E/
                    └── config.txt

    File script.py contains a Path("D", "E", "config.txt") to access data in that config
    file. If the script is called from the "C" directory, the relative path resolves
    fine, since the "D" subdirectory is immediately visible.
    However, if the script is called from e.g. "A", it breaks, because there is no
    "D/E/config.txt" in "A". If the script uses this function instead, the paths get
    resolved correctly to absolute paths.
    Here, Path("D", "E", "config.txt") is the 'subpath'.

    Note: this function can also be called without an argument to get the caller's
    file's containing directory, or with ".." (as many as needed) to move up.
    If called with one argument, a file name, this script can replace
    `Path(__file__).with_name("new_file")` to get a new file path in the same directory
    as the caller's file, while being much clearer in syntax.

    Attributes:
        pathparts: As many arguments as the subpath needs. The new Path will be
            created as e.g. Path("x", "y", "z"). This gets rid of ambiguities
            surrounding usage of forward or backward slashes.
    """
    current_frame = inspect.currentframe()  # This function's frame
    previous_frame = current_frame.f_back  # One up: the caller's frame
    caller_filename = inspect.getsourcefile(previous_frame)
    # Construct a Path relative to the caller's directory:
    caller_dir = Path(caller_filename).parent
    sub_path = Path(*pathparts)  # Can be anything: file, dir, links
    return caller_dir.joinpath(sub_path).resolve()

The docstring explains it in detail. The synopsis is: there is a Python file, here script.py, that relies on a file that is found in a fixed location relative to it. In this case, D/E/config.txt. It can be any relative path, including .. parts.

The script.py file can be called from anywhere. If it is called from anywhere but the C/ directory, the discovery for config.txt can break easily with many naive approaches.

As such, a script.py file can import path_relative_to_caller_file. It can do so from wherever, that part should not matter. The function fully resolves paths relative to the file from which it is called.

Directory Structure

The directory structure is as follows:

~$ tree A
A
└── B
    └── C
        ├── D
        │   └── E
        │       └── config.txt
        ├── path_relative_to_caller_file.py
        └── script.py

where path_relative_to_caller_file.py contains only the code shown above. script.py is:

from pathlib import Path

from path_relative_to_caller_file import path_relative_to_caller_file

paths = [
    path_relative_to_caller_file(),
    path_relative_to_caller_file(".."),
    path_relative_to_caller_file("..", ".."),
    path_relative_to_caller_file("D"),
    path_relative_to_caller_file("D", "E"),
    path_relative_to_caller_file("D", "E", "config.txt"),
    path_relative_to_caller_file("nonexistent_directory"),
]

plain_path = Path("D", "E", "config.txt")

print("path_relative_to_caller_file:")
for path in paths:
    print("\t", path, path.exists())

print("Plain path in script.py:")
print("\t", plain_path, plain_path.exists())

Output

This works if called from the parent directory of A/, so ~:

~$ python3 A/B/C/script.py
path_relative_to_caller_file:
         /home/hansA/B/C True
         /home/hans/A/B True
         /home/hans/A True
         /home/hans/A/B/C/D True
         /home/hans/A/B/C/D/E True
         /home/hans/A/B/C/D/E/config.txt True
         /home/hans/A/B/C/nonexistent_directory False
Plain path in script.py:
         D/E/config.txt False

The plain, "naive" approach works only if called from ~/A/B/C/:

 ~/A/B/C$ python3 script.py
path_relative_to_caller_file:
         /home/hansA/B/C True
         /home/hans/A/B True
         /home/hans/A True
         /home/hans/A/B/C/D True
         /home/hans/A/B/C/D/E True
         /home/hans/A/B/C/D/E/config.txt True
         /home/hans/A/B/C/nonexistent_directory False
Plain path in script.py:
         D/E/config.txt True

The above approach keeps working however. It even works when navigating up the tree:

 ~/A/B/C/D/E$ python3 ../../script.py
path_relative_to_caller_file:
         /home/hans/A/B/C True
         /home/hans/A/B True
         /home/hans/A True
         /home/hans/A/B/C/D True
         /home/hans/A/B/C/D/E True
         /home/hans/A/B/C/D/E/config.txt True
         /home/hans/A/B/C/nonexistent_directory False
Plain path in script.py:
         D/E/config.txt False

Now, the inspect module seems a bit overkill for this. I also wonder about security (can the frame be injected maliciously by the caller?) and performance (an entire inspection for what is not much more than some string-fu) issues. When looking at the problem at a distance, it seems like there should be an easier solution.

It is also possible I got this entirely backwards and am missing the bigger picture.

Alternative

An obvious alternative would be to just have a function that requires a Path object to do the relative work on. Callers of that function would then just pass their __file__ variable (part of globals()), followed by the same *pathargs that will work relative on that __file__ path and return the (resolved) result. This would be straighforward. In fact, this is how I had it at first. However, then all function calls have __file__ as their first argument. As such, I came up with the above to rid the code of this perceived redundancy (DRY).

Tested on Python 3.7.7 (Debian) and Python 3.8.2 (Win10).

Answer 1

importlib.resources

As of Python 3.7 you can make put the data files in packages and use importlib.resources

A/
└── B/
    └── C/
        ├── script.py
        └── D/
            ├── __init__.py              <== could be an empty file
            └── E/
                ├── __init__.py          <== could be an empty file
                └── config.txt

Then use read_binary or read_text to read a file. open_binary and open_text return a file-like object.

from importlib.resources import open_text

with open_text("D.E", "config.txt") as f:
    for line in f:
        print(line)

I believe it will work going up the directory tree if they are all packages (have __init__.py files), but I haven't tried it.

Answer 2

I think you understand the tradeoffs: you can either use __file__ everywhere or trust the stack to rewind to the caller's frame of reference. The former looks redundant, but the latter makes your function kinda weird and may behave oddly if the user isn't aware of what's going on. For example, if they decorated your function, then they might get the file path relative to the decorator's file rather than the caller's file. There are ways around this (namely, taking an explicit argument for how far back the call stack to go), but it gets ugly fast.

Python doesn't have macros, so there's no super-clean way to create a function whose global closure is that of the caller rather than where that function is defined. There are, of course, ways to do this, but it adds complexity to the calling code which is what you're trying to avoid.

I'd argue that passing __file__ each time doesn't break DRY, because even though that variable has the same name in every context, it means something different each time. You're not repeating yourself, you just have the different value in the same name each time. Doing this makes the calling code more explicit, reduces the opportunity for bugs, simplifies your code, and increases the usability of your function. E.g., the calling code could ask for the file path relative to a parent module if the project was set up such that, say, a function was imported and exposed via an __init__.py up a folder or two.

So in summary, I think the simpler approach that requires passing __file__ doesn't break DRY and does conform to KISS.