Managing file paths in a python Data Science project

Question

When doing Data Science projects, I often have to load data and metadata, and output results, plots, logs, etc.

Therefore I have to handle all the file paths from where to load the input and write the output.

I don't think that hardcoding the paths in the script is a good practice, therefore I write them in a JSON file like this (UPDATED in order to make the below code run)

{"raw_data_analysis": {"log_path": "outputs/data_analysis/raw/data_analysis_log.txt",
    "dataset_path": "data/raw/raw_dataset.csv",
    "variables_metadata_path": "data/raw/variables_metadata.json",
    "csv_output_folder": "outputs/data_analysis/raw/csv_files",
    "markdown_output_folder": "outputs/data_analysis/raw/markdown_tables"
  },
  "clean_data_analysis": {
    "dataset": "data/clean/clean_dataset.csv",
    "variables_metadata_path": "data/clean/intermediate_variables_metadata.json",
    "csv_output_folder": "outputs/data_analysis/clean/csv_files",
    "table_output_folder": "outputs/data_analysis/cleaned/markdown_tables",
    "plots_output_folder": "outputs/data_analysis/cleaned/plots"
  }
}

Then i made a function that, given the path of the JSON file as command line argument with --paths_configuration_path, and the key of the paths i want to read, returns a dictionary with as keys the "description" of the path and as value the path. This function also accepts a value for a default path of the JSON file, in case i don't give it from command line.

import argparse
import json
from pathlib import Path

def get_paths_config_from_cl_arguments(default_config_path: Path | None, default_key:str=None)->dict:
    parser = argparse.ArgumentParser()
    parser.add_argument("--paths_configuration_path", type=str,
                        default=default_config_path,
                        help="Path to the paths configuration file")
    parser.add_argument("--key", type=str,
                        default=default_key,
                        help="key for the paths in the configuration file")

    args = parser.parse_args()
    paths_configuration_path=args.paths_configuration_path
    key=args.key

    paths = get_paths_from_configuration(paths_configuration_path=paths_configuration_path, key=key)

    return paths

def get_paths_from_configuration(paths_configuration_path:Path,key:str|None)->dict:
    with open(paths_configuration_path,'r') as file:
        paths_configuration=json.load(file)
    if key:
        paths_configuration=paths_configuration[key]
    return {key:Path(value) for key,value in paths_configuration.items()}

At this point, in the python scripts that i would actually run, I call this function giving as default path the one that i usually need, and then I extract the paths from the dictionary (In order for the velow code to run, you need the above JSON at the path configurations/data_analysis/data_analysis_paths_config.json)

def main():
    paths = get_paths_config_from_cl_arguments(
        default_config_path=Path('configurations/data_analysis/data_analysis_paths_config.json'),
        default_key='clean_data_analysis')
    data_path = paths['dataset_path']
    variables_metadata_path = paths['variables_description_path']
    csv_output_folder = paths['csv_output_folder']
    markdown_output_folder = paths['table_output_folder']
    plots_output_folder = paths['plots_output_folder']

if __name__=='__main__':
    main()

In this way I am free to run my script with the default paths, or i can just create a new JSON file with other paths and just pass the path to that JSON file as command line argument when i run the script.

What do you think about this approach? Is it clean and readable? Am I introducing too many functions and complicating a rather trivial problem? Is there any established design pattern that i should use? What is the standard approach for handling file paths?

Answer 1

argparse

When I run the code without passing a command-line argument, it dies badly with error messages. It would be better if it died with a helpful message which clearly shows the intended usage.

Also, when just using the --help option, it shows all arguments as being optional. However, as mentioned above, the code does require the user to specify something on the command line. argparse can be configured to indicate required arguments.

Option names are typically short, especially in a case like this where there are just 3 options. It would be more convenient for the user to rename paths_configuration_path as something like paths.

Simpler

This code creates several unnecessary variables:

paths_configuration_path=args.paths_configuration_path
key=args.key

paths = get_paths_from_configuration(paths_configuration_path=paths_configuration_path, key=key)

return paths

It can be simplified as:

return get_paths_from_configuration(
    paths_configuration_path = args.paths_configuration_path,
    key = args.key
)

Whitespace

The code uses inconsistent whitespace around the = operators; some have space, while others do not. The black program can be used to automatically format your code.

Naming

The function named get_paths_config_from_cl_arguments seems unnecessarily long. I think get_paths_config is sufficient. You can add a docstring to summarize its purpose:

def get_paths_config(default_config_path: Path | None, default_key:str=None)->dict:
    """ Use command-line arguments to read configuration file. """

Test

It is great that you provided a sample JSON input file in the question, but I get errors on these Python code lines when I run the code:

data_path = paths['dataset_path']
variables_metadata_path = paths['variables_description_path']

Answer 2

JSON usage

JSON was never developed as a human-readable config language. Yes, a lot of popular projects (esp. in JS ecosystem) decided to abuse it for such purposes, and it has bitten almost everyone working with such systems. JSON does not support comments. JSON does not support trailing commas (read "unclean git diffs and painful merges", not "ohno, I'll have to type a comma next time"). JSON does not support line breaks (so there's no way to break a 400 chars string into several lines for readability - yikes). JSON is painful for humans to work with. Don't be like JSON.

There are plenty of other options available: TOML (my personal favourite here, it was specifically developed with humans in mind and gained a wide adoption, including built-in python support since 3.11), INI (old-style configparser module, but still very convenient) or YAML (flawed and inconvenient IMO, but very popular and more human), to name a few. Consider using one of those instead.

Composition

If you want to change one destination, you need to copy the whole JSON config to edit one entry. Being able to provide such overrides easier (by merging config files, with an env variable or with a CLI argument) would be a quality of life improvement.

General problem

Honestly, I'm not sure if you're solving the right problem at the right level. What I see as a requirement: you're trying to build a fixed file hierarchy for use in multiple projects. That's a great idea if the projects are similar, nothing wrong yet. However, you assume that all projects need approximately the same file paths, which is really unlikely to be true - you'll eventually need "just one more file", I swear. JSON is a forgiving language, so you can add whatever you want there, but it's breaking the pretty "one size fits all" solution.

I can suggest another approach: build a reusable parser instead. Here's how I would approach this myself (quick draft, of course). I'll be using a designated library to support parsing CLI arguments, config files and envvars at the same time: ConfigArgParse (no affiliation).

import configargparse


def file_paths_parser(
    *names: str, section: str = "raw_data_analysis", default_file: str = "paths.toml"
) -> configargparse.ArgParser:
    parser = configargparse.ArgParser(
        # You can also add a global file here (e.g. ~/my_ml_paths.toml)
        default_config_files=[default_file],
        config_file_parser_class=configargparse.TomlConfigParser([section]),
    )
    parser.add_argument(
        "--paths-config",
        is_config_file=True,
        help="file paths config",
        env_var="PATHS_CONFIG",
    )
    for name in names:
        parser.add_argument(
            "--" + name.replace("_", "-"), env_var=name.upper(), required=True
        )
    return parser


def main() -> None:
    parser = file_paths_parser("log_path", "dataset_path", section="raw_data_analysis")
    parser.add_argument("-v", "--verbose", action="store_true", help="emit more output")
    args = parser.parse_args()
    print(f"{args.log_path=}")
    print(f"{args.dataset_path=}")

This way every program starts by explicitly declaring paths it needs. It's good for future maintenance: you won't have to look through the code to find that out.

You can also add more args (-v in the example) when you don't abstract "build and use parser" to a single function together.

This enables a bunch of available options. Let's create a "seed" paths.toml (it doesn't have to contain all keys, if some are missing - they must be passed as CLI args or env vars instead, so you can still use the shared file, but add one more path easily):

[raw_data_analysis]
log-path = "outputs/data_analysis/raw/data_analysis_log.txt"
dataset-path = "data/raw/raw_dataset.csv"

And now a few invocation options:

$ DATASET_PATH=foo python s.py
args.log_path='outputs/data_analysis/raw/data_analysis_log.txt'
args.dataset_path='foo'

$ DATASET_PATH=foo python s.py --log-path bar
args.log_path='bar'
args.dataset_path='foo'

$ python s.py
args.log_path='outputs/data_analysis/raw/data_analysis_log.txt'
args.dataset_path='data/raw/raw_dataset.csv'

$ python s.py -h
usage: s.py [-h] [--paths-config PATHS_CONFIG] --log-path LOG_PATH --dataset-path DATASET_PATH [-v]

options:
  -h, --help            show this help message and exit
  --paths-config PATHS_CONFIG
                        file paths config [env var: PATHS_CONFIG]
  --log-path LOG_PATH   [env var: LOG_PATH]
  --dataset-path DATASET_PATH
                        [env var: DATASET_PATH]
  -v, --verbose         emit more output

Args that start with '--' can also be set in a config file (paths.toml or specified via --paths-config). Config file syntax is Tom's Obvious, Minimal Language. See https://github.com/toml-lang/toml/blob/v0.5.0/README.md for details.
In general, command-line values override environment variables which override config file values which override defaults.

You'll need to pip install configargparse toml to test this exact snippet, but I invite you to read the whole documentation and pick the file format and configuration options that fit you most.