Scraping webpage elements using Python

Question

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there it might not needed to be at first place.

My goal is basically to have a ready payload before scraping as you will see at the top of the code. Those should be always declared before scraping. What im trying to do basically is to try to scrape those different data. If we don't find the data, then it should skip or set the value to [], None or False (Depending on what we are trying to do). I have read abit regarding getattr and isinstance functions but im not sure if there might be a better way than using lots of Try except as a cover if it doesn't find the element on the webpage.

I would appreciate all kind of helps!

import requests
from bs4 import BeautifulSoup

payload = {
    "name": "Untitled",
    "view": None,
    "image": None,
    "hyperlinks": []
}

site_url = "https://stackoverflow.com/questions/743806/how-to-split-a-string-into-a-list"

response = requests.get(site_url)

bs4 = BeautifulSoup(response.text, "html.parser")

try:
    payload['name'] = "{} {}".format(
        bs4.find('meta', {'property': 'og:site_name'})["content"],
        bs4.find('meta', {'name': 'twitter:domain'})["content"]
    )
except Exception:  # noqa
    pass

try:
    payload['view'] = "{} in total".format(
        bs4.find('div', {'class': 'grid--cell ws-nowrap mb8'}).text.strip().replace("\r\n", "").replace(" ", ""))
except Exception:
    pass

try:
    payload['image'] = bs4.find('meta', {'itemprop': 'image primaryImageOfPage'})["content"]
except Exception:
    pass

try:
    payload['hyperlinks'] = [hyperlinks['href'] for hyperlinks in bs4.find_all('a', {'class': 'question-hyperlink'})]

except Exception:  # noqa
    pass

print(payload)

Answer 1

My take, in short:

• in programming there is nothing worse than silently swallowing exceptions :) And the fact that you are catching any exception, and not just those related to those specific operations, means than all kinds of errors will go unnoticed.
• At the very least, the exceptions should be logged somewhere
• be more specific and catch relevant exceptions instead, usually it will be stuff like HTMLParser.HTMLParseError and also requests.exceptions - experiment a bit
• Note the difference in behavior between find and find_all since you are using both functions:

If find_all() can’t find anything, it returns an empty list. If find() can’t find anything, it returns None

• Armed with that knowledge you can simply test your expressions for None before you try to fetch the value with find - it's better to avoid an exception than having to handle it.
• Likewise, if bs4.find_all('a', {'class': 'question-hyperlink'}) returns an empty list, you don't proceed with the list comprehension but you assign some default value, possibly: payload['hyperlinks'] = None

So the key here is to test the result of every call to find or find_all and act accordingly. Then you can get rid of those try catch blocks. Only one all-purpose exception handler for the whole procedure shall suffice.