Data safety
Your current HTML_search_attributes
is type-unsafe - it's closer to a serialized format than an in-memory format.
Consider moving those data to a .json
file. Deserializing it will give you exactly what you have now, though I recommend going one step further. Make a class or at least a named tuple to represent a scraped domain, with attributes of price
, name
, and has_javascript
. This will go farther to validate your data and increase the confidence in correctness of your code.
URL parsing
Don't do it by hand. This:
d = url.replace("https://", "").replace("www.", "")
will explode for sites such as
https://foo.com/www.section/
At the least, you should regex-match to ^
, the beginning of the string. More likely, you should use https://docs.python.org/3/library/urllib.parse.html .
The next problem is your class representation of these URL parts. After a class is initialized, one should be able to assume within reason that its properties are accessible, but yours are not until url
is run. The solution to this is to move this block:
d = url.replace("https://", "").replace("www.", "")
r = requests.get(url, headers=headers)
self._domain = d.split("/", maxsplit=1)[0]
self._soup = BeautifulSoup(r.text, parser)
self._url = url
into the constructor.
There probably shouldn't even be a public url
setter. It only makes sense for it to be initialized once, in the constructor.
The real problem is that the scraping occurs immediately when the class is instantiated. Don't do lengthier operations such as soup calls in __init__
; do them in a separate method.
Invert your logic
Rather than
if js is False:
# ...
else:
# ...
do
if js:
# ...
else:
# ...
into your __init__
.