Send an array of files and return the first one that exists

Question

I have a little project, chuy, which is basically a copy of Make.

By request of users I have added support for toml configuration, since originally it only accepted json.

The files that chuy have to check are: ["chuy.json","chuy.toml","pyproject.toml"].

The order of priority is the same order in which they are placed. (If it is true that chuy.json exists, it no longer matters whether chuy.toml or pyproject.toml exist.)

So I wrote:

def get_config_file() -> str:
    try:
        with open("chuy.json", mode="r", encoding="utf-8"):
            return "json"
    except FileNotFoundError:
        try:
            with open("pyproject.toml", mode="r", encoding="utf-8"):
                return "pyproject"
        except FileNotFoundError:
            try:
                with open("chuy.toml", mode="r", encoding="utf-8"):
                    return "toml"
            except FileNotFoundError:
                raise BaseException("I can't find your configuration file :(")

Which is horrible, some refactor later:

def get_config_file(posible_config_files = ["chuy.json", "chuy.toml", "pyproject.toml"])-> str:
    
    for file in posible_config_files:
        try:
            with open(file, mode="r", encoding="utf-8"):
                return file
        except FileNotFoundError:
            continue

Which is a bit better, but I still don't think it's a good solution.

Any ideas?

Answer 1

Thanks Eliaz for your question, I'll give you a few pointers to help you. These are pretty standard boiler-plate code patterns in any Python project. For example:

• For configuration files, you should utilise ConfigParser.
• Using argparse for user input or default choices like with unattended script running
• Using logging to help figure out what happens during execution (maybe it loaded a different file to what they expected?).

You might be wondering why I'm giving all these additions to your question - and that's because your question actually is very common as a software developer.

If you try using this in your github project, you will see how it can remove a lot of code used for printing status updates (such as changing the logging level for screen (warning/info) verses for file (debug)) and argparse and configparser makes life easier to manage input and standardise config files.

This will allow use-cases where they have two or more config files, but they wish to use a specific config file for a particular run/exec.

Entry Point

This is the start of the logic for your script, and it helps other coders to understand what your program does, and how it does it. Let's establish the program flow to make it clear to the readers of your code what the script is doing:

if __name__ == "__main__":
    log = activate_logging()
    log.debug('Log activated')
    params = parse_command_line()
    log.debug(f"Received these options: {params.file_list}")
    config_file = load_first_config(params)
    log.debug(f"Selected config file: {config_file}")

That makes it quite clear what it's attempting to do. Enabling logging, reading the parameters from the command line, then loading the appropriate config file.

Logging

I've found that using logging rather than step-by-step debugging is faster because the system will tell you what it's doing (as long as you write the correct statements). It saves a lot of time, and removes the questions like "how did that happen?" This is a standard logging pattern, and it will log to the screen when it runs. If you refer to the Python documentation, you can modify this to make it log to a file and screen, or only to a file.

def activate_logging():
    logger = logging.getLogger()
    handler = logging.StreamHandler()
    formatter = logging.Formatter("%(asctime)s %(name)-12s %(levelname)-8s %(message)s")
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    logger.setLevel(logging.DEBUG)
    return logger

Parsing Choices

One way to get user input is via the command line, but you can have some defaults, like what you've done. So we use argparse to handle user choices or make our own selection from the default items we specified. Here is another standard coding pattern:

def parse_command_line():
    desc = "Chuy: Set alias to long commands and speed up your workflow."
    parser = argparse.ArgumentParser(description=desc)

    parser.add_argument("-f", "--file", action="store", dest="file_list",
                        type=str, nargs="*",
                        default=["chuy.json", "pyproject.toml", "chuy.toml"],
                        help="Examples: -i chuy.json pyproject.toml, -i chuy.toml")
    args = parser.parse_args()
    log.debug(f"List of items: {args.file_list}")
    return args

argparse is great because it can do a lot of boilerplate things such as automatic help:

ch@ubuntu:~/PycharmProjects/testing$ python chuy.py -h
2021-09-11 03:22:19,612 root  DEBUG    Log activated
usage: chuy.py [-h] [-f [FILE_LIST [FILE_LIST ...]]]

Chuy: Set alias to long commands and speed up your workflow.

optional arguments:
  -h, --help            show this help message and exit
  -f [FILE_LIST [FILE_LIST ...]], --file [FILE_LIST [FILE_LIST ...]]
                        Examples: -f chuy.json pyproject.toml, -f chuy.toml

Configuration Selection

Finally, we use configparser to handle looking for the choices, and returning the first choice.

def load_first_config(args):
    parser = ConfigParser()
    candidates = args.file_list
    found = parser.read(candidates)
    missing = set(candidates) - set(found)
    log.debug(f"Found config files: {found}")
    log.debug(f"Missing files     : {missing}")
    if found:
        return found[0]
    return None

If you're using an [ini] style config file, configparser can also read those and return a config object (more information can be found in the documentation).

Running The Code

Including the imports at the top of the file:

import logging
import argparse
from configparser import ConfigParser

We can then run it:

2021-09-11 03:19:51,510 root  DEBUG    Log activated
2021-09-11 03:19:51,511 root  DEBUG    List of items: ['chuy.json', 'pyproject.toml', 'chuy.toml']
2021-09-11 03:19:51,511 root  DEBUG    Received these options: ['chuy.json', 'pyproject.toml', 'chuy.toml']
2021-09-11 03:19:51,511 root  DEBUG    Found config files: []
2021-09-11 03:19:51,511 root  DEBUG    Missing files     : ['chuy.json', 'chuy.toml', 'pyproject.toml']
2021-09-11 03:19:51,511 root  DEBUG    Selected config file: None

Now we create a dummy file chuy.toml (touch chuy.toml) and re-run the app:

2021-09-11 03:20:04,702 root  DEBUG    Log activated
2021-09-11 03:20:04,703 root  DEBUG    List of items: ['chuy.json', 'pyproject.toml', 'chuy.toml']
2021-09-11 03:20:04,703 root  DEBUG    Received these options: ['chuy.json', 'pyproject.toml', 'chuy.toml']
2021-09-11 03:20:04,703 root  DEBUG    Found config files: ['chuy.toml']
2021-09-11 03:20:04,703 root  DEBUG    Missing files     : ['chuy.json', 'pyproject.toml']
2021-09-11 03:20:04,703 root  DEBUG    Selected config file: chuy.toml

We can also use command lines:

ch@ubuntu:~/PycharmProjects/testing$ python chuy.py -f pyproject.toml
2021-09-11 03:32:00,024 root  DEBUG    Log activated
2021-09-11 03:32:00,025 root  DEBUG    List of items: ['pyproject.toml']
2021-09-11 03:32:00,025 root  DEBUG    Received these options: ['pyproject.toml']
2021-09-11 03:32:00,025 root  DEBUG    Found config files: []
2021-09-11 03:32:00,025 root  DEBUG    Missing files     : {'pyproject.toml'}
2021-09-11 03:32:00,026 root  DEBUG    Selected config file: None

I hope this helps - please drop any questions below in the comments. Cheers!

Answer 2

• Don't raise BaseException
• Typo: posible -> possible
• Don't type-hint a list; instead use List[str] in the case of your posible_config_files
• Consider making a reusable solution that separates "all existing config files" from "the first existing config file"

Suggested

from pathlib import Path
from typing import Iterable


def config_files() -> Iterable[Path]:
    for name in ("chuy.json", "chuy.toml", "pyproject.toml"):
        path = Path(name)
        if path.is_file():
            yield path


def config_file() -> Path:
    try:
        return next(config_files())
    except StopIteration:
        raise FileNotFoundError('No configuration file was found')