I am new to python, in .NET we use web.config to store environment specific variables... I want to achieve the same for my python program (scraper)
I have created .env file at the root of the project and added the following variable to it:
CHROME_WEB_DRIVER=C:/webdrivers/chromedriver.exe
Note that I have added .env to gitignore as this file changes in different environments.
And I am using decouple
package to use this setting from .env file in the constructor of the class:
import scrapy
from decouple import config
from python_json_config import ConfigBuilder
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
class MySpider(scrapy.Spider):
name = 'my_spider'
def __init__(self):
# .env variables
self.CHROME_WEB_DRIVER = config('CHROME_WEB_DRIVER')
self.job_highlight_dict = {}
self.wanted_organisations_dict = {}
self.output_path = ""
self.start_urls = []
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
self.driver = webdriver.Chrome(self.CHROME_WEB_DRIVER, chrome_options=chrome_options)
builder = ConfigBuilder()
config_values = builder.parse_config('config/config.json')
self.start_urls = config_values.urls
for o in config_values.organisations:
self.wanted_organisations_dict[o["brandName"]] = o["organisationName"]
self.output_path = config_values.output
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
# parse response
Note that I also have a config.json file, but this file contains configuration variables which are consistent across all environments.
This is how the config.json file looks like:
{
"urls": [
"https://www.site-to-scrape-1.com",
"https://www.site-to-scrape-2.com"
],
"organisations": [
{ "brandName" : "brand-1", "organisationName" : "organisation-1" },
{ "brandName" : "brand-2", "organisationName" : "organisation-2" }
],
"output": "../path/to/s3/bucket"
}
I have not included all of spider class, but it is the main class of this project which crawls the given urls and then parse them. I am not sure if the constructor of the spider is the best place to read config.json and .env files?