2
\$\begingroup\$

I've written a script to harvest Name and Phone number from yellowpage Canada under pizza category. The web page doesn't show its full content until scrolled downmost. In my crawler's every cycle, it fetches 40 records. I twitched a little in my loop which I've learnt lately to get the full content. Hope I did it the way it should be.

import requests
from lxml import html

Base_url="https://www.yellowpages.ca/search/si/{0}/pizza/Toronto"
with requests.session() as session:
    page_num=1
    while True:
        response = session.get(Base_url.format(page_num))
        if response.status_code==404:
            break
        tree = html.fromstring(response.text)
        for titles in tree.xpath("//div[@itemprop='itemListElement']"):
            try:
                title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
            except IndexError:
                title=""
            try:
                phone= titles.xpath(".//h4[@itemprop='telephone']/text()")[0]
            except IndexError:
                phone=""
            print(title,phone)
        page_num+=1
\$\endgroup\$

1 Answer 1

2
\$\begingroup\$

Applying PEP8 recommendations about naming and formatting, as well as wrapping the main logic in a main function and calling from within a if __name__ == '__main__': guard, the code would become easier to read and possible to import and test:

import requests
from lxml import html

url_format = "https://www.yellowpages.ca/search/si/{0}/pizza/Toronto"


def main():
    with requests.session() as session:
        page_num = 1
        while True:
            response = session.get(url_format.format(page_num))
            if response.status_code == 404:
                break
            tree = html.fromstring(response.text)
            for titles in tree.xpath("//div[@itemprop='itemListElement']"):
                try:
                    title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
                except IndexError:
                    title = ""
                try:
                    phone = titles.xpath(".//h4[@itemprop='telephone']/text()")[0]
                except IndexError:
                    phone = ""
                print(title, phone)
            page_num += 1


if __name__ == '__main__':
    main()

I'm not a huge fan of using exceptions in situations that are not really exceptions, like here:

try:
    title = titles.xpath(".//h3[@itemprop='name']/a/text()")[0]
except IndexError:
    title = ""

What if a titles.xpath(...) call would raise an IndexError due to a bug? It would be inadvertently caught and unnoticed. Catching IndexError here is an indirect way of the real intention, which is getting the first title if it exists. It would be better to write the code in a way that expresses the intention directly, for example:

for item in tree.xpath("//div[@itemprop='itemListElement']"):
    title = get_first_or_empty(item, ".//h3[@itemprop='name']/a/text()")
    phone = get_first_or_empty(item, ".//h4[@itemprop='telephone']/text()")
    print(title, phone)

Where get_first_or_empty is:

def get_first_or_empty(item, xpath):
    matches = item.xpath(xpath)
    if matches:
        return matches[0]
    return ""
\$\endgroup\$
6
  • \$\begingroup\$ Thanks sir janos, for such a nice suggestion along with a workaround about handling error. It's hard for me to follow PEP8 recommendations cause I write code using sublime. Thanks a ton. \$\endgroup\$
    – SIM
    Commented May 28, 2017 at 19:29
  • \$\begingroup\$ How does sublime text prevent you from following PEP8? Genuinely curious. \$\endgroup\$
    – yuri
    Commented May 28, 2017 at 20:27
  • \$\begingroup\$ I'm a big fan of sublime text editor so don't get me wrong. When I use pycharm I can see the warning when i happen to violate PEP8 recommendations but using sublime I don't see that. If my comment hurts you, apology for that. Thanks. \$\endgroup\$
    – SIM
    Commented May 28, 2017 at 20:36
  • \$\begingroup\$ @SMth80 there's also a pep8 command line tool, which you can run on your scripts and it will tell you the violations. You can probably hook up Sublime to run pep8 when you save a file, but that's just a guess, I don't know Sublime. \$\endgroup\$
    – janos
    Commented May 28, 2017 at 20:43
  • \$\begingroup\$ Oh. I thought the core concept of PEP8 suggests a 4-space indent which should be doable in any editor so I was confused how sublime would make that impossible. \$\endgroup\$
    – yuri
    Commented May 28, 2017 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.