How to handle configuration from a config file and global variables transparently in python?

Question

This is how I handle configuration in my python code, currently, but I guess it might look magic to people, and it is cumbersome to code. How can this be improved? Are there better solutions?

The purpose is to have a general working configuration stored in UPPER CASE in global variables at the beginning of the file. When first run, it creates a config.json file with that configuration. That can be edited by the user, and it would supersede the default configuration, if run again. In order to still benefit from typing hints in modern IDEs, the cfg[] dict is not used directly, but the configuration is written back to the global variables.

This is cumbersome to code, and it contains a lot of repetition. Every variable is touched multiple times.

How can I improve this system and make it more elegant and transparent?

import json
from typing import Any, Dict

CONFIG_FILE = 'config.json'
GLOBAL_CONFIG_EXAMPLE_PORT = 3


def read_config() -> Dict[str, Any]:
    try:
        with open(CONFIG_FILE) as config_file:
            return json.load(config_file)
    except FileNotFoundError:
        pass
    # generate standard config file
    cfg = {
        'example': {
            'port': GLOBAL_CONFIG_EXAMPLE_PORT,
        },
    }
    with open(CONFIG_FILE, 'w') as f:
        json.dump(cfg, f)

    set_global_variables(cfg)
    return cfg


def set_global_variables(cfg: Dict[str, Any]):
    global GLOBAL_CONFIG_EXAMPLE_PORT
    GLOBAL_CONFIG_EXAMPLE_PORT = cfg['example']['port']


def main():
    cfg: Dict[str, Any] = read_config()
    print(cfg['example']['port'], GLOBAL_CONFIG_EXAMPLE_PORT)


if __name__ == "__main__":
    main()

A real-life config file would be this:

{
  "mqtt": {
    "host": "10.21.1.77",
    "port": 1883,
    "topic": "EXTENSE/Lab/XYZ/move/#",
    "topic_ack": "EXTENSE/Lab/XYZ/move/ack"
  },
  "signal": {
    "save": true,
    "length": 60
  },
  "sensor": {
    "num": 5,
    "trigger": 4
  },
  "logging": {
    "level": 10,
    "filename": "log.txt",
    "console": true,
    "format": "%(asctime)s %(levelname)s: %(message)s"
  }
}

PS: How would you go about writing a test for this, efficiently?

Addendum: Is it feasible and sensible to do some magic on the variable name which is all upper case, split them by underscore, and create the dict automatically?

Answer 1

read_config does not do what's on the tin: it attempts to read the config, conditionally writes out a default config, and sets globals. Those obligations should be separated.

Also, your example is a mystery: is it a section? Currently it offers no value and a flat dictionary would be simpler to manipulate.

GLOBAL_CONFIG_EXAMPLE_PORT on its own is not a useful global. Consider instead moving the entire default configuration dictionary to a global constant.

Something like:

import json
from typing import Any, Dict

CONFIG_FILE = 'config.json'

DEFAULT_CONFIG = {
    'port': 3,
}


def read_config() -> Dict[str, Any]:
    with open(CONFIG_FILE) as f:
        return json.load(f)


def write_config(config: Dict[str, Any]) -> None:
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f)


def load_or_default_config() -> None:
    try:
        config = read_config()
    except FileNotFoundError:
        config = DEFAULT_CONFIG
        write_config(config)

    globals().update(config)

Answer 2

I don't really get that part:

In order to still benefit from typing hints in modern IDEs

You might not know about TypedDict?

1. Define a lookup order of precedence from which the config is loaded. This should be stated in a user manual, e.g. in a "configuration" section where you can also specify which config format to use. I suggest: (1) Load package defaults (2) load user-specific file (3) environment variables
2. Separation of Concerns: Your read_config does too much.
3. Code formatting: Apply black formatter (online version)

Which gives:

import json
import os
from typing import Any, Dict, TypedDict

CONFIG_FILE = "config.json"
GLOBAL_CONFIG_EXAMPLE_PORT = 3


class MqttConfig(TypedDict):
    host: str
    port: int
    topic: str
    topic_ack: str


class SignalConfig(TypedDict):
    save: bool
    length: int


class SensorConfig(TypedDict):
    num: int
    trigger: int


class LoggingConfig(TypedDict):
    level: int
    filename: str
    console: bool
    format: str


class Config(TypedDict):
    mqtt: MqttConfig
    signal: SignalConfig
    sensor: SensorConfig
    logging: LoggingConfig


def main():
    cfg = get_config()
    if not os.path.exists(CONFIG_FILE):
        write_config(CONFIG_FILE, cfg)
    set_global_variables(cfg)
    print(cfg["example"]["port"], GLOBAL_CONFIG_EXAMPLE_PORT)


def get_config() -> Config:
    # You could have the dict directly in here. However, I think it's cleaner
    # to have this in it's own function for those reasons:
    # (1) get_config should only deal with the order in which configuration is
    #     loaded and the error cases that might happen
    # (2) The default config could also be loaded from a file. This way you
    #     would have an example directly in your code. Loading the default would
    #     then look exactly the same as loading a user config, except the
    #     path is different
    # (3) This function might get pretty lengthy is the dict is directly in here
    default_config = get_default_config()

    try:
        user_config = read_config(
            CONFIG_FILE
        )  # you might want to make this path absolute
    except FileNotFoundError:
        user_config = {}

    config = default_config
    config.update(user_config)  # type: ignore

    return config


def get_default_config() -> Config:
    return {
        "mqtt": {
            "host": "10.21.1.77",
            "port": 1883,
            "topic": "EXTENSE/Lab/XYZ/move/#",
            "topic_ack": "EXTENSE/Lab/XYZ/move/ack",
        },
        "signal": {"save": True, "length": 60},
        "sensor": {"num": 5, "trigger": 4},
        "logging": {
            "level": 10,
            "filename": "log.txt",
            "console": True,
            "format": "%(asctime)s %(levelname)s: %(message)s",
        },
    }


def read_config(config_file_path: str) -> Dict[str, Any]:
    with open(config_file_path) as config_file:
        cfg = json.load(config_file)
    # Sadly, PEP 655 is still missing so missing keys lead to this dict not
    # being a Config TypedDict as specified
    #
    # You might run some validation here, e.g. that the values have the excpeted
    # types or that there are no keys present which you didn't expect.
    #
    # Pydantic is pretty awesome. I would use pydantic instead of TypedDict
    return cfg


def write_config(config_file_path: str, config: Config) -> None:
    with open(config_file_path, "w") as f:
        json.dump(config, f)


def set_global_variables(cfg: Config):
    global GLOBAL_CONFIG_EXAMPLE_PORT
    GLOBAL_CONFIG_EXAMPLE_PORT = cfg["mqtt"]["port"]


if __name__ == "__main__":
    main()

You might be interested in my cfg_load package.

Answer 3

Well, if you want to see the code, here it is.

It is buggy and oversimplification of the actual code, but it does something to that effect.

Code:

import json
from pathlib import Path

CONFIG_FILE = Path(__file__).parent / 'config.py'

DEFAULT_CONFIG = {
  "mqtt": {
    "host": "10.21.1.77",
    "port": 1883,
    "topic": "EXTENSE/Lab/XYZ/move/#",
    "topic_ack": "EXTENSE/Lab/XYZ/move/ack"
  },
  "signal": {
    "save": True,
    "length": 60
  },
  "sensor": {
    "num": 5,
    "trigger": 4
  },
  "logging": {
    "level": 10,
    "filename": "log.txt",
    "console": True,
    "format": "%(asctime)s %(levelname)s: %(message)s"
  }
}

def write_config(path):
    path.write_text('config = ' + json.dumps(DEFAULT_CONFIG, indent=4).replace('true', 'True').replace('false', 'False').replace('null', 'None'))

if __name__ == '__main__':
    if CONFIG_FILE.is_file():
        from config.py import config
    else:
        write_config(CONFIG_FILE)
        config = DEFAULT_CONFIG

Now this is slightly better version, this method of storing configuration variables has an advantage over json based one: in this way you can store multiple objects in the same file, but using json you can only save one object.

Also json is not Python, Python has many objects that aren't json serializable (set, frozenset, byte, bytearry, complex...), and some objects when deserialized will change their datatypes, so if you want to keep the identity of the objects you'd better use native serializers, for starters you can use repr, it basically guarantees eval(repr(val)) == val (don't run it, val isn't defined, it is basically true for any object).

Now you can put multiple config objects in the same file like this:

from pathlib import Path


CONFIG_FILE = Path(__file__).parent / 'config.py'

USER_ACCOUNT = {'account': 'John Smith', 'password': 'No+Password'}
SERVER = {'address': '127.0.0.1', 'port': '8080', 'protocol': 'HTTPS'}
PLATFORM = {'OS': 'Windows 11', 'Architecture': 'x86-64'}

def write_config(d: dict, path: Path) -> None:
    path.write_text('\n'.join(f'{k} = {v!r}' for k,v in d.items())

if __name__ == '__main__':
    if CONFIG_FILE.is_file():
        from config.py import *
    else:
        write_config({'account': USER_ACCOUNT, 'server': SERVER, 'platform': PLATFORM}, CONFIG_FILE)
        account, server, platform = USER_ACCOUNT, SERVER, PLATFORM

In this way you can just import the objects directly from the file without having to deserialize the objects. But this is actually a bad practice...