# Reading configuration variables in a Python project

I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)

I have created .env file at the root of the project and added the following variable to it:

CHROME_WEB_DRIVER=C:/webdrivers/chromedriver.exe


Note that I have added .env to gitignore as this file changes in different environments.

And I am using decouple package to use this setting from .env file in the constructor of the class:

import scrapy
from decouple import config
from python_json_config import ConfigBuilder
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

class MySpider(scrapy.Spider):
name = 'my_spider'

def __init__(self):
# .env variables
self.CHROME_WEB_DRIVER = config('CHROME_WEB_DRIVER')

self.job_highlight_dict = {}
self.wanted_organisations_dict = {}
self.output_path = ""
self.start_urls = []
chrome_options = Options()
self.driver = webdriver.Chrome(self.CHROME_WEB_DRIVER, chrome_options=chrome_options)

builder = ConfigBuilder()
config_values = builder.parse_config('config/config.json')
self.start_urls = config_values.urls
for o in config_values.organisations:
self.wanted_organisations_dict[o["brandName"]] = o["organisationName"]
self.output_path = config_values.output

def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):

# parse response



Note that I also have a config.json file, but this file contains configuration variables which are consistent across all environments.

This is how the config.json file looks like:

{
"urls": [
"https://www.site-to-scrape-1.com",
"https://www.site-to-scrape-2.com"
],
"organisations": [
{ "brandName" : "brand-1", "organisationName" : "organisation-1" },
{ "brandName" : "brand-2", "organisationName" : "organisation-2" }
],
"output": "../path/to/s3/bucket"
}


I have not included all of spider class, but it is the main class of this project which crawls the given urls and then parse them. I am not sure if the constructor of the spider is the best place to read config.json and .env files?

• Add more tags, ex. selenium Nov 4 '20 at 1:35