3
\$\begingroup\$

This is how I handle configuration in my python code, currently, but I guess it might look magic to people, and it is cumbersome to code. How can this be improved? Are there better solutions?

The purpose is to have a general working configuration stored in UPPER CASE in global variables at the beginning of the file. When first run, it creates a config.json file with that configuration. That can be edited by the user, and it would supersede the default configuration, if run again. In order to still benefit from typing hints in modern IDEs, the cfg[] dict is not used directly, but the configuration is written back to the global variables.

This is cumbersome to code, and it contains a lot of repetition. Every variable is touched multiple times.

How can I improve this system and make it more elegant and transparent?

import json
from typing import Any, Dict

CONFIG_FILE = 'config.json'
GLOBAL_CONFIG_EXAMPLE_PORT = 3


def read_config() -> Dict[str, Any]:
    try:
        with open(CONFIG_FILE) as config_file:
            return json.load(config_file)
    except FileNotFoundError:
        pass
    # generate standard config file
    cfg = {
        'example': {
            'port': GLOBAL_CONFIG_EXAMPLE_PORT,
        },
    }
    with open(CONFIG_FILE, 'w') as f:
        json.dump(cfg, f)

    set_global_variables(cfg)
    return cfg


def set_global_variables(cfg: Dict[str, Any]):
    global GLOBAL_CONFIG_EXAMPLE_PORT
    GLOBAL_CONFIG_EXAMPLE_PORT = cfg['example']['port']


def main():
    cfg: Dict[str, Any] = read_config()
    print(cfg['example']['port'], GLOBAL_CONFIG_EXAMPLE_PORT)


if __name__ == "__main__":
    main()

A real-life config file would be this:

{
  "mqtt": {
    "host": "10.21.1.77",
    "port": 1883,
    "topic": "EXTENSE/Lab/XYZ/move/#",
    "topic_ack": "EXTENSE/Lab/XYZ/move/ack"
  },
  "signal": {
    "save": true,
    "length": 60
  },
  "sensor": {
    "num": 5,
    "trigger": 4
  },
  "logging": {
    "level": 10,
    "filename": "log.txt",
    "console": true,
    "format": "%(asctime)s %(levelname)s: %(message)s"
  }
}

PS: How would you go about writing a test for this, efficiently?

Addendum: Is it feasible and sensible to do some magic on the variable name which is all upper case, split them by underscore, and create the dict automatically?

\$\endgroup\$
10
  • 1
    \$\begingroup\$ Can you include an example JSON file? \$\endgroup\$
    – Reinderien
    Jun 19, 2021 at 12:12
  • \$\begingroup\$ certainly, @Reinderien :-) \$\endgroup\$ Jun 19, 2021 at 12:45
  • \$\begingroup\$ @Reinderien i updated the config file, perhaps this illustrates the use a little better. \$\endgroup\$ Jun 21, 2021 at 17:52
  • 1
    \$\begingroup\$ I don't know if it's just me, but I just store configurations in .py files and import the variables from the files, and I used .__name__, repr() and json.dumps() and quotes and some code to print these configurations to python files. I know it's dirty but it is very effective. \$\endgroup\$ Jun 23, 2021 at 13:04
  • 1
    \$\begingroup\$ If you really want to know, I will post an answer tomorrow, but not today. \$\endgroup\$ Jun 23, 2021 at 13:53

3 Answers 3

4
\$\begingroup\$

read_config does not do what's on the tin: it attempts to read the config, conditionally writes out a default config, and sets globals. Those obligations should be separated.

Also, your example is a mystery: is it a section? Currently it offers no value and a flat dictionary would be simpler to manipulate.

GLOBAL_CONFIG_EXAMPLE_PORT on its own is not a useful global. Consider instead moving the entire default configuration dictionary to a global constant.

Something like:

import json
from typing import Any, Dict

CONFIG_FILE = 'config.json'

DEFAULT_CONFIG = {
    'port': 3,
}


def read_config() -> Dict[str, Any]:
    with open(CONFIG_FILE) as f:
        return json.load(f)


def write_config(config: Dict[str, Any]) -> None:
    with open(CONFIG_FILE, 'w') as f:
        json.dump(config, f)


def load_or_default_config() -> None:
    try:
        config = read_config()
    except FileNotFoundError:
        config = DEFAULT_CONFIG
        write_config(config)

    globals().update(config)
\$\endgroup\$
5
  • \$\begingroup\$ thanks for telling me about globals()! I like your approach a lot, and it's a lot more elegant than mine. However, it lacks the list of global variables at the beginning of the program which I thought to be a part of a good python coding style, and replaces it with a dict. Do you think that's a step forward? \$\endgroup\$ Jun 21, 2021 at 18:42
  • \$\begingroup\$ How would your approach handle multi-level global constants, like in the real-life config example above? @reinderien \$\endgroup\$ Jun 21, 2021 at 18:55
  • \$\begingroup\$ Spin up a @dataclass for each section so that the section variables have types; have a global class instance for each section \$\endgroup\$
    – Reinderien
    Jun 21, 2021 at 19:01
  • \$\begingroup\$ and how would you combine those dataclasses into one? could you do the example in the (real life) config file and update your code, perhaps? \$\endgroup\$ Jun 21, 2021 at 19:07
  • \$\begingroup\$ i mean: in my example you would have four global classes, because there are four sections. how do you combine those four classes into one dict to write into a config file, and how do you read them back into the right class, if there is a config file that you parsed and need to split up into the correct classes? \$\endgroup\$ Jun 21, 2021 at 19:26
2
\$\begingroup\$

I don't really get that part:

In order to still benefit from typing hints in modern IDEs

You might not know about TypedDict?

  1. Define a lookup order of precedence from which the config is loaded. This should be stated in a user manual, e.g. in a "configuration" section where you can also specify which config format to use. I suggest: (1) Load package defaults (2) load user-specific file (3) environment variables
  2. Separation of Concerns: Your read_config does too much.
  3. Code formatting: Apply black formatter (online version)

Which gives:

import json
import os
from typing import Any, Dict, TypedDict

CONFIG_FILE = "config.json"
GLOBAL_CONFIG_EXAMPLE_PORT = 3


class MqttConfig(TypedDict):
    host: str
    port: int
    topic: str
    topic_ack: str


class SignalConfig(TypedDict):
    save: bool
    length: int


class SensorConfig(TypedDict):
    num: int
    trigger: int


class LoggingConfig(TypedDict):
    level: int
    filename: str
    console: bool
    format: str


class Config(TypedDict):
    mqtt: MqttConfig
    signal: SignalConfig
    sensor: SensorConfig
    logging: LoggingConfig


def main():
    cfg = get_config()
    if not os.path.exists(CONFIG_FILE):
        write_config(CONFIG_FILE, cfg)
    set_global_variables(cfg)
    print(cfg["example"]["port"], GLOBAL_CONFIG_EXAMPLE_PORT)


def get_config() -> Config:
    # You could have the dict directly in here. However, I think it's cleaner
    # to have this in it's own function for those reasons:
    # (1) get_config should only deal with the order in which configuration is
    #     loaded and the error cases that might happen
    # (2) The default config could also be loaded from a file. This way you
    #     would have an example directly in your code. Loading the default would
    #     then look exactly the same as loading a user config, except the
    #     path is different
    # (3) This function might get pretty lengthy is the dict is directly in here
    default_config = get_default_config()

    try:
        user_config = read_config(
            CONFIG_FILE
        )  # you might want to make this path absolute
    except FileNotFoundError:
        user_config = {}

    config = default_config
    config.update(user_config)  # type: ignore

    return config


def get_default_config() -> Config:
    return {
        "mqtt": {
            "host": "10.21.1.77",
            "port": 1883,
            "topic": "EXTENSE/Lab/XYZ/move/#",
            "topic_ack": "EXTENSE/Lab/XYZ/move/ack",
        },
        "signal": {"save": True, "length": 60},
        "sensor": {"num": 5, "trigger": 4},
        "logging": {
            "level": 10,
            "filename": "log.txt",
            "console": True,
            "format": "%(asctime)s %(levelname)s: %(message)s",
        },
    }


def read_config(config_file_path: str) -> Dict[str, Any]:
    with open(config_file_path) as config_file:
        cfg = json.load(config_file)
    # Sadly, PEP 655 is still missing so missing keys lead to this dict not
    # being a Config TypedDict as specified
    #
    # You might run some validation here, e.g. that the values have the excpeted
    # types or that there are no keys present which you didn't expect.
    #
    # Pydantic is pretty awesome. I would use pydantic instead of TypedDict
    return cfg


def write_config(config_file_path: str, config: Config) -> None:
    with open(config_file_path, "w") as f:
        json.dump(config, f)


def set_global_variables(cfg: Config):
    global GLOBAL_CONFIG_EXAMPLE_PORT
    GLOBAL_CONFIG_EXAMPLE_PORT = cfg["mqtt"]["port"]


if __name__ == "__main__":
    main()

You might be interested in my cfg_load package.

\$\endgroup\$
7
  • \$\begingroup\$ no, i didnt know about TypedDict. Thanks for introducing me to it! What do you say about what user "Heisenberg" writes above about json being suboptimal for python? I do fancy your package and also your solution. \$\endgroup\$ Dec 25, 2021 at 14:43
  • \$\begingroup\$ you suggest first TypedDict (to help with the hinting in the IDE?) and then write that you would use Pydantic instead of TypedDict. Can you please reconcile that? Like this Pydantic feels like an afterthought and your answer comes across like half-cooked. \$\endgroup\$ Dec 26, 2021 at 9:04
  • \$\begingroup\$ @AndreasSchuldei I cannot see any comment by "Heisenberg". But JSON is a general data exchange format. It's just as good/bad for Python as for other languages. One major flaw I see in JSON as a config file format is that you cannot add comments. That is the reason why I typically use YAML, but some people hate YAML. Some people like TOML. Others prefer pure Python as config files. \$\endgroup\$ Dec 26, 2021 at 11:52
  • \$\begingroup\$ @AndreasSchuldei I've added pydantic as a comment because it is a third-party module. I typically prefer not to rely on 3rd party modules in answers if possible. However, I would use Pydantic for this case if I wrote code for this \$\endgroup\$ Dec 26, 2021 at 11:55
  • \$\begingroup\$ I refer to Xeнεi Ξэnвϵς. above, he wrote: "Also JSON is not Python, Python has many objects that aren't JSON serializable (set, frozenset, byte, bytearry, complex...), and some objects when deserialized will change their datatypes, so if you want to keep the identity of the objects you'd better use native serializer". I am not an expert - I try to ask one expert about an other expert's opinion. :-) feel free to comment on his answer, I would like to read your exchange. (and vice versa) \$\endgroup\$ Dec 26, 2021 at 15:25
1
\$\begingroup\$

Well, if you want to see the code, here it is.

It is buggy and oversimplification of the actual code, but it does something to that effect.

Code:

import json
from pathlib import Path

CONFIG_FILE = Path(__file__).parent / 'config.py'

DEFAULT_CONFIG = {
  "mqtt": {
    "host": "10.21.1.77",
    "port": 1883,
    "topic": "EXTENSE/Lab/XYZ/move/#",
    "topic_ack": "EXTENSE/Lab/XYZ/move/ack"
  },
  "signal": {
    "save": True,
    "length": 60
  },
  "sensor": {
    "num": 5,
    "trigger": 4
  },
  "logging": {
    "level": 10,
    "filename": "log.txt",
    "console": True,
    "format": "%(asctime)s %(levelname)s: %(message)s"
  }
}

def write_config(path):
    path.write_text('config = ' + json.dumps(DEFAULT_CONFIG, indent=4).replace('true', 'True').replace('false', 'False').replace('null', 'None'))

if __name__ == '__main__':
    if CONFIG_FILE.is_file():
        from config.py import config
    else:
        write_config(CONFIG_FILE)
        config = DEFAULT_CONFIG

Now this is slightly better version, this method of storing configuration variables has an advantage over json based one: in this way you can store multiple objects in the same file, but using json you can only save one object.

Also json is not Python, Python has many objects that aren't json serializable (set, frozenset, byte, bytearry, complex...), and some objects when deserialized will change their datatypes, so if you want to keep the identity of the objects you'd better use native serializers, for starters you can use repr, it basically guarantees eval(repr(val)) == val (don't run it, val isn't defined, it is basically true for any object).

Now you can put multiple config objects in the same file like this:

from pathlib import Path


CONFIG_FILE = Path(__file__).parent / 'config.py'

USER_ACCOUNT = {'account': 'John Smith', 'password': 'No+Password'}
SERVER = {'address': '127.0.0.1', 'port': '8080', 'protocol': 'HTTPS'}
PLATFORM = {'OS': 'Windows 11', 'Architecture': 'x86-64'}

def write_config(d: dict, path: Path) -> None:
    path.write_text('\n'.join(f'{k} = {v!r}' for k,v in d.items())

if __name__ == '__main__':
    if CONFIG_FILE.is_file():
        from config.py import *
    else:
        write_config({'account': USER_ACCOUNT, 'server': SERVER, 'platform': PLATFORM}, CONFIG_FILE)
        account, server, platform = USER_ACCOUNT, SERVER, PLATFORM

In this way you can just import the objects directly from the file without having to deserialize the objects. But this is actually a bad practice...

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.