2
\$\begingroup\$

I have been trying to improve my knowledge with Python and I think the code is pretty forward. However I do dislike abit the coding style I have done where I use too much try except in a content there it might not needed to be at first place.

My goal is basically to have a ready payload before scraping as you will see at the top of the code. Those should be always declared before scraping. What im trying to do basically is to try to scrape those different data. If we don't find the data, then it should skip or set the value to [], None or False (Depending on what we are trying to do). I have read abit regarding getattr and isinstance functions but im not sure if there might be a better way than using lots of Try except as a cover if it doesn't find the element on the webpage.

I would appreciate all kind of helps!

import requests
from bs4 import BeautifulSoup

payload = {
    "name": "Untitled",
    "view": None,
    "image": None,
    "hyperlinks": []
}

site_url = "https://stackoverflow.com/questions/743806/how-to-split-a-string-into-a-list"

response = requests.get(site_url)

bs4 = BeautifulSoup(response.text, "html.parser")

try:
    payload['name'] = "{} {}".format(
        bs4.find('meta', {'property': 'og:site_name'})["content"],
        bs4.find('meta', {'name': 'twitter:domain'})["content"]
    )
except Exception:  # noqa
    pass

try:
    payload['view'] = "{} in total".format(
        bs4.find('div', {'class': 'grid--cell ws-nowrap mb8'}).text.strip().replace("\r\n", "").replace(" ", ""))
except Exception:
    pass

try:
    payload['image'] = bs4.find('meta', {'itemprop': 'image primaryImageOfPage'})["content"]
except Exception:
    pass

try:
    payload['hyperlinks'] = [hyperlinks['href'] for hyperlinks in bs4.find_all('a', {'class': 'question-hyperlink'})]

except Exception:  # noqa
    pass

print(payload)
\$\endgroup\$
2
\$\begingroup\$

My take, in short:

  • in programming there is nothing worse than silently swallowing exceptions :) And the fact that you are catching any exception, and not just those related to those specific operations, means than all kinds of errors will go unnoticed.
  • At the very least, the exceptions should be logged somewhere
  • be more specific and catch relevant exceptions instead, usually it will be stuff like HTMLParser.HTMLParseError and also requests.exceptions - experiment a bit
  • Note the difference in behavior between find and find_all since you are using both functions:

If find_all() can’t find anything, it returns an empty list. If find() can’t find anything, it returns None

  • Armed with that knowledge you can simply test your expressions for None before you try to fetch the value with find - it's better to avoid an exception than having to handle it.
  • Likewise, if bs4.find_all('a', {'class': 'question-hyperlink'}) returns an empty list, you don't proceed with the list comprehension but you assign some default value, possibly: payload['hyperlinks'] = None

So the key here is to test the result of every call to find or find_all and act accordingly. Then you can get rid of those try catch blocks. Only one all-purpose exception handler for the whole procedure shall suffice.

\$\endgroup\$
3
  • \$\begingroup\$ Thank you very much! I totally agree. There is some places I could use the find without needing any try/expect due to if it doesn't find then it will be None anyways! But what I wish is to have sort of if else function where if its None then we could have a default value such as getattr has but that should work as well with one liner if else statement too \$\endgroup\$ – ProtractorNewbie Feb 21 at 16:39
  • 1
    \$\begingroup\$ You are welcome. My advice to stress-test your program and to make it more robust is to test it in adverse conditions. Try to parse a totally unrelated page and see how it reacts. Not finding the expected elements in page should not crash the application, the exception handler should kick in when other types of issues do occur. Also try to fetch pages from unreachable sites (with a misspelled address for example) to test behavior. A good program should handle exceptions gracefully. \$\endgroup\$ – Anonymous Feb 21 at 16:47
  • \$\begingroup\$ Very well said! I will do that right away! I hoped that I could use getattr since it had a nice function to check if the webelement has a value and u could also have a default exception if its not found for example which I liked but was not able to get it to work with webscraping \$\endgroup\$ – ProtractorNewbie Feb 21 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.