I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)

I have created .env file at the root of the project and added the following variable to it:


Note that I have added .env to gitignore as this file changes in different environments.

And I am using decouple package to use this setting from .env file in the constructor of the class:

import scrapy
from decouple import config
from python_json_config import ConfigBuilder
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        # .env variables
        self.CHROME_WEB_DRIVER = config('CHROME_WEB_DRIVER')

        self.job_highlight_dict = {}
        self.wanted_organisations_dict = {}
        self.output_path = ""
        self.start_urls = []
        chrome_options = Options()
        self.driver = webdriver.Chrome(self.CHROME_WEB_DRIVER, chrome_options=chrome_options)

        builder = ConfigBuilder()
        config_values = builder.parse_config('config/config.json')
        self.start_urls = config_values.urls
        for o in config_values.organisations:
            self.wanted_organisations_dict[o["brandName"]] = o["organisationName"]
        self.output_path = config_values.output

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):

        # parse response

Note that I also have a config.json file, but this file contains configuration variables which are consistent across all environments.

This is how the config.json file looks like:

    "urls": [
    "organisations": [
        { "brandName" : "brand-1", "organisationName" : "organisation-1" },
        { "brandName" : "brand-2", "organisationName" : "organisation-2" }
    "output": "../path/to/s3/bucket"

I have not included all of spider class, but it is the main class of this project which crawls the given urls and then parse them. I am not sure if the constructor of the spider is the best place to read config.json and .env files?

  • \$\begingroup\$ Add more tags, ex. selenium \$\endgroup\$ Nov 4, 2020 at 1:35

1 Answer 1

  • Add a declarative format for the config file, possibly. If you encounter errors, the output will look messy today (KeyError is never fun for an end user)
  • I would say init is a good time, but unclear if it's the right place. It's looking messy. I would either (1) split out parsing the config file into a helper method or (2) consolodate some lines. (1) is better if you expect to add more config.
  • I would consider making the chrome driver at the start of 'parse', as this is more of a "can fail" type operation--it will make writing tests easier.
  • Consolidating:
    • 'builder' is used only once, don't declare it
    • don't assign to self.wanted_organizations_dict, or self.output_path, and self.start_urls twice. just initialize them once, correctly, in one line each.
    • self.job_highlight_dict is not initialized. i don't know what this is, but are you sure you want to declare it here?
    • self.CHROME_WEB_DRIVER does not appear to be used outside this, change it to a local variable or inline it, unless you move driver init to parse time
    • declare self.wanted_organisations_dict in a single-line dict comprehension, not 3 lines.
    • Initialize chrome_options in a single line, not 4.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.