Implementing web scraping in a function and in a class

Question

The goal of my code is to implement a web scraping routine to obtain the name and the price of a product. I want to put this routine separated from the main program file. Sample url to scrape: https://www.amazon.co.uk/Samsung-MZ-76E1T0B-EU-Solid-State/dp/B078WST5RK/ref=sr_1_1?dchild=1&keywords=samsung+860+evo+1tb&qid=1584446290&sr=8-1. I'm doing this to learn python (I'm quite new).

I didn't know if it was better to use a class or function, so I tried both. I would like to know if any of the 2 implementations could be made more clear, readable or efficient, or if one of the two implementations is better than the other and why.

Implementation with a function:

import json
from collections import namedtuple
import requests
from bs4 import BeautifulSoup

parser = "lxml"
headers = {
    "User-Agent": "I don't put my user-agent in case it contains sensible info."
}
# This dictionary contains multiple entries. They are the supported domains by the program.
# This is just a sample.
HTML_search_attributes = { 
    "amazon.co.uk": {
        "price": ("span", {"id": "priceblock_ourprice"}),
        "name": ("span", {"id": "productTitle"}),
        "JavaScript": False,
    },
    "newegg.com": {
        "price": ("script", {"type": "application/ld+json"}),
        "name": ("div", {"class": "mini-features-desc"}),
        "JavaScript": True,
    },
}

def product(url):
    d = url.replace("https://", "").replace("www.", "")
    domain = d.split("/", maxsplit=1)[0]
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.text, parser)

    # Collect scraping attributes from HTML_search_attributes dictionary.
    try:
        p = HTML_search_attributes[domain]["price"]
        n = HTML_search_attributes[domain]["name"]
        js = HTML_search_attributes[domain]["JavaScript"]
    except KeyError as err:
        raise type(err)(
            f"The domain {domain} is not supported by this app."
        ) from None
    # Find price of the product
    if js is False:
        try:
            price = soup.find(p[0], attrs=p[1]).get_text()
            price= float(price.strip("\r\n\xa0£€EUR* ").replace(",", "."))
        except AttributeError:
            price = "No availability"
    else:
        price = soup.find_all(p[0], attrs=p[1])[-1].get_text()
        price = float(json.loads(price)["offers"]["price"])
    # Find name of the product
    name = soup.find(n[0], attrs=n[1]).get_text()
    name = name.strip("\r\n\xa0 ")

    Product = namedtuple('Product', ['domain', 'name', 'price'])
    return Product(domain, name, price)

Implementation with a Class:

import json
import requests
from bs4 import BeautifulSoup

parser = "lxml"
headers = {
    "User-Agent": "I don't put my user-agent in case it contains sensible info."
}
# This dictionary contains multiple entries. They are the supported domains by the program.
# This is just a sample.
HTML_search_attributes = { 
    "amazon.co.uk": {
        "price": ("span", {"id": "priceblock_ourprice"}),
        "name": ("span", {"id": "productTitle"}),
        "JavaScript": False,
    },
    "newegg.com": {
        "price": ("script", {"type": "application/ld+json"}),
        "name": ("div", {"class": "mini-features-desc"}),
        "JavaScript": True,
    },

}

class Product:
    def __init__(self, url):
        self.url = url

    @property
    def url(self):
        return self._url
    @property
    def domain(self):
        return self._domain
    @property
    def soup(self):
        return self._soup
    @property
    def price(self):
        return self._price
    @property
    def name(self):
        return self._name
    
    @url.setter
    def url(self, url):
        d = url.replace("https://", "").replace("www.", "")
        r = requests.get(url, headers=headers)
        self._domain = d.split("/", maxsplit=1)[0]
        self._soup = BeautifulSoup(r.text, parser)
        self._url = url

        # Collect scraping attributes from HTML_search_attributes dictionary.
        try:
            price = HTML_search_attributes[self._domain]["price"]
            name = HTML_search_attributes[self._domain]["name"]
            js = HTML_search_attributes[self._domain]["JavaScript"]
        except KeyError as err:
            raise type(err)(
                f"The domain {self._domain} is not supported by this app."
            ) from None
        # Find price of the product
        if js is False:
            try:
                self._price = self._soup.find(price[0], attrs=price[1]).get_text()
                self._price = float(self._price.strip("\r\n\xa0£€EUR* ").replace(",", "."))
            except AttributeError:
                self._price = "No availability"
        else:
            self._price = self._soup.find_all(price[0], attrs=price[1])[-1].get_text()
            self._price = float(json.loads(self._price)["offers"]["price"])
        # Find name of the product
        self._name = self._soup.find(name[0], attrs=name[1]).get_text()
        self._name = self._name.strip("\r\n\xa0 ")

Functionality details

In both cases, changing the url changes all related properties. In both cases, I can access the results with result.price, result.name and result.domain. In the class implementation, all properties are read-only except the url. The price extraction routine is different whether or not the webpage uses JavaScript (I've only encountered one case at the moment).

I've measured the computation time of both methods and I've found that the fastest method is the class method if I iterate as follows:

from timeit import default_timer as timer
start = timer()
# urls2bscraped is a list containing multiple urls.
prod = Product(urls2bscraped[0])
print(prod.domain, prod.name, prod.price)
for url in urls2bscraped[1:]:
    prod.url = url
    print(prod.domain, prod.name, prod.price)
end = timer()
print(end - start)

Using the function implementation, the mean computation time (10 samples) is 4.47% slower, and creating a new instance of the class for every iteration is 5.05% slower.

Consulted references:

Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code

Jack Diederich - Stop Writing Classes

Choosing Between Properties and Methods

Answer 1

Data safety

Your current HTML_search_attributes is type-unsafe - it's closer to a serialized format than an in-memory format.

Consider moving those data to a .json file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price, name, and has_javascript. This will go farther to validate your data and increase the confidence in correctness of your code.

URL parsing

Don't do it by hand. This:

    d = url.replace("https://", "").replace("www.", "")

will explode for sites such as

https://foo.com/www.section/

At the least, you should regex-match to ^, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .

The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url is run. The solution to this is to move this block:

    d = url.replace("https://", "").replace("www.", "")
    r = requests.get(url, headers=headers)
    self._domain = d.split("/", maxsplit=1)[0]
    self._soup = BeautifulSoup(r.text, parser)
    self._url = url

into the constructor.

There probably shouldn't even be a public url setter. It only makes sense for it to be initialized once, in the constructor.

The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__; do them in a separate method.

Invert your logic

Rather than

    if js is False:
        # ...
    else:
        # ...

do

if js:
    # ...
else:
    # ...

Answer 2

You are unfairly doing more work in the function approach!

Product = namedtuple('Product', ['domain', 'name', 'price'])

creates a new type every time the statement is executed. Consider:

result1.__class__ == result2.__class__

That will evaluate as False, for different result objects, even when returning results from the same site!

You should move the Product type creation outside of the function, and then rerun your timings.