Reading configuration variables in a Python project

Question

I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)

I have created .env file at the root of the project and added the following variable to it:

CHROME_WEB_DRIVER=C:/webdrivers/chromedriver.exe 

Note that I have added .env to gitignore as this file changes in different environments.

And I am using decouple package to use this setting from .env file in the constructor of the class:

import scrapy
from decouple import config
from python_json_config import ConfigBuilder
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        # .env variables
        self.CHROME_WEB_DRIVER = config('CHROME_WEB_DRIVER')

        self.job_highlight_dict = {}
        self.wanted_organisations_dict = {}
        self.output_path = ""
        self.start_urls = []
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        self.driver = webdriver.Chrome(self.CHROME_WEB_DRIVER, chrome_options=chrome_options)

        builder = ConfigBuilder()
        config_values = builder.parse_config('config/config.json')
        self.start_urls = config_values.urls
        for o in config_values.organisations:
            self.wanted_organisations_dict[o["brandName"]] = o["organisationName"]
        self.output_path = config_values.output

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        # parse response

Note that I also have a config.json file, but this file contains configuration variables which are consistent across all environments.

This is how the config.json file looks like:

{
    "urls": [
        "https://www.site-to-scrape-1.com",
        "https://www.site-to-scrape-2.com"
    ],
    "organisations": [
        { "brandName" : "brand-1", "organisationName" : "organisation-1" },
        { "brandName" : "brand-2", "organisationName" : "organisation-2" }
    ],
    "output": "../path/to/s3/bucket"
}

I have not included all of spider class, but it is the main class of this project which crawls the given urls and then parse them. I am not sure if the constructor of the spider is the best place to read config.json and .env files?

Answer 1

• Add a declarative format for the config file, possibly. If you encounter errors, the output will look messy today (KeyError is never fun for an end user)
• I would say init is a good time, but unclear if it's the right place. It's looking messy. I would either (1) split out parsing the config file into a helper method or (2) consolodate some lines. (1) is better if you expect to add more config.
• I would consider making the chrome driver at the start of 'parse', as this is more of a "can fail" type operation--it will make writing tests easier.
• Consolidating:
  • 'builder' is used only once, don't declare it
  • don't assign to self.wanted_organizations_dict, or self.output_path, and self.start_urls twice. just initialize them once, correctly, in one line each.
  • self.job_highlight_dict is not initialized. i don't know what this is, but are you sure you want to declare it here?
  • self.CHROME_WEB_DRIVER does not appear to be used outside this, change it to a local variable or inline it, unless you move driver init to parse time
  • declare self.wanted_organisations_dict in a single-line dict comprehension, not 3 lines.
  • Initialize chrome_options in a single line, not 4.