Reading a config file in python

Question

We're wanting to read data in from a config file and provide it to other python modules in a helpful way. We have found a number of ways to accomplish this that range from the module being easy to understand but verbose to use, to the module being a little more complicated but really nice to use.

Below I will present three different methods to read config file data. What are your opinions? Which method would you go with, or would you do something completely different?

The example config file

[section1]
# The trailing slash of this url should be removed.
# (In our actual code, we do a little more work to normalize the urls in the config file)
url = http://example.com/
api_key = 12345

[section2]
# These items are commented out to see how the config loader will handle defaults.
# One of these should have a default, the other should throw a useful error
# if an attempt is made to access it when it is not set.
# with_default =
# without_default =

Method 1

Pros: Very simple to read and understand.

Cons: I have to do things like remove the trailing slash of the url manually outside this module. Also, I am unable to throw very useful errors for the end-user because I'm letting ConfigParser handle errors for me. It is also a little verbose to use this module.

from configparser import ConfigParser

_defaults = {
    'section1': {
        'url': 'http://example.org',
    },
    'section2' : {
        'with_default' : 'default-value',
    },
}

mainconf = ConfigParser()
mainconf.read_dict(_defaults)
mainconf.read('./example.config')

Usage

# Dummy server request function
request = print

from method1 import mainconf
normalize_url = lambda url: url.rstrip('/')
request(normalize_url(mainconf['section1']['url'])+'/v1/endpoint', {
    'api-key': mainconf['section1']['api_key'],
    'data': mainconf['section2']['with_default'],
})

# This should throw an error.
mainconf['section2']['without_default']

Method 2

Pros: I can now provide more helpful errors. I can also access some (but not all) attributes simply by doing things like method2.url, which is much cleaner.

Cons: Inconsistency because only some attributes can be accessed through method2.url while other require using get().

from configparser import ConfigParser

_normalize_url = lambda url: url.rstrip('/')

_defaults = {
    'section1': {
        'url': 'http://example.org',
    },
    'section2' : {
        'with_default' : 'default-value',
    },
}

_mainconf = ConfigParser()
_mainconf.read_dict(_defaults)
_mainconf.read('./example.config')

def get(section, item):
    try:
        ret = _mainconf[section][item]
    except KeyError:
        raise LookupError(f'{item} in section {section} not found in config file') from None

    if section == 'section1' and item == 'url':
        ret = _normalize_url(ret)
    return ret

# Provide items on a module level that has defaults.
# If they don't have defaults, you must go through get() so a good error would
# be thrown if the item is not found.
url = get('section1', 'url')
with_default = get('section2', 'with_default')

Usage

# Dummy server request function
request = print

import method2
request(method2.url+'/v1/endpoint', {
    'api-key': method2.get('section1', 'api_key'),
    'data': method2.with_default,
})

# This should throw a helpful error.
method2.get('section2', 'without_default')

Method 3

Pros: All attributes can be accessed simply by doing mainconf.url, nice! It also throws good errors for missing items.

Cons: The code is a little bit more complected to make this work. Also, it might feel a little too magical to override __getattr__.

from configparser import ConfigParser

_normalize_url = lambda url: url.rstrip('/')

_attr_to_conf = {
    'url': ('section1', 'url'),
    'api_key': ('section1', 'api_key'),
    'with_default': ('section2', 'with_default'),
    'without_default': ('section2', 'without_default'),
}

_defaults = {
    'section1': {
        'url': 'http://example.org',
    },
    'section2' : {
        'with_default' : 'default-value',
    },
}

_mainconf = ConfigParser()
_mainconf.read_dict(_defaults)
_mainconf.read('./example.config')

class MainConf:
    def __getattr__(self, attr):
      section, item = _attr_to_conf[attr]
      try:
          ret = _mainconf[section][item]
      except KeyError:
          raise AttributeError(f"Can't get attribute '{attr}' on 'MainConf' because the config file does not contain '{item}' in the section '{section}'") from None

      if attr == 'url':
          ret = _normalize_url(ret)
      return ret

mainconf = MainConf()

Usage

# Dummy server request function
request = print

from method3 import mainconf
request(mainconf.url+'/v1/endpoint', {
    'api-key': mainconf.api_key,
    'data': mainconf.with_default,
})

# This should throw a helpful error.
mainconf.without_default

Answer 1

Definitely go with Method 1:

• There is no custom code to support. ConfigParser is part of the standard Python library; any programmer can easily look up its documentation to see how it works, if they weren't already familiar with it.

• URL normalization has no business being part of the configuration file parser. Including such normalization would violate the Single Responsibility Principle as well as the Principle of Least Surprise. What next — a rule that automatically converts http: URLs to https:?

  In fact, I'm not convinced that URL normalization is what you want. If you want to construct URLs by appending a path, use urllib.parse.urljoin(). Both of these calls would produce the same result:

  from urllib.parse import urljoin
  
  urljoin('http://example.com/', '/v1/endpoint')
  urljoin('http://example.com', '/v1/endpoint')