I've written a scraper in python using asyncio library to exhaust the name, address and phone number of all category from a webpage. Few days back, when I created a scraper to parse a webpage recursively, I got a suggestion to use asynchronous process to crawl a webpage with enormous data for the sake of optimum performance and avoiding blocking nature. However, this time I tried to follow that process so that my crawler can scrape the webpage asynchronously. It is running specklessly now. I used css selector to make the parser more readable along with acceleration of speed. This is my first time to work with this library, so I suppose there are scopes to take it to the next level whetting it's performance.

Here is what I've written:

import requests
from lxml import html
import asyncio

main_url = "https://www.someweblink/letter/{0}"

async def alphabetical_links(mainurl):

        # scraping all the links with alphbetical index from start page
        for link in [mainurl.format(chr(page)) for page in range(97,123)]:
            response = requests.get(link).text
            tree = html.fromstring(response)

            for container in tree.cssselect(".proSitemapLink a"):
                await sub_links(container.attrib['href'])
                # going for the contents of the sublinks lies within each page with alphabetical index 

async def sub_links(process_links):

        response = requests.get(process_links).text
        root = html.fromstring(response)

        for container in root.cssselect(".proListing"):
            name = container.cssselect("h2 a")[0].text if container.cssselect("h2 a") else ""
            # Used text_content() to get the full address separated by "br" tag
            address = container.cssselect(".proListingLocation")[0].text_content() if container.cssselect(".proListingLocation") else ""
            phone = container.cssselect(".proListingPhone")[0].text if container.cssselect(".proListingPhone") else ""
            profilelink = container.cssselect("h2 a")[0].attrib['href'] if container.cssselect("h2 a") else ""
            await adding_weblink(name, address, phone, profilelink)
            # finally it will go to the profile page to extract website address if available

async def adding_weblink(name, address, phone, profilelink):

        response = requests.get(profilelink).text
        root = html.fromstring(response)
        weblink = root.cssselect(".proWebsiteLink")[0].attrib['href'] if root.cssselect(".proWebsiteLink") else "" 
        if weblink:
            print(name, address, phone, weblink)
            print(name, address, phone)

loop = asyncio.get_event_loop()

1 Answer 1


magic numbers

ASCII numbers are pretty obvious, but I would still rather see you express it in this way:

letters = range(ord('a'), ord('z') + 1)
for link in [mainurl.format(chr(page)) for page in letters:

Alternatively, import string and iterate through string.ascii_lowercase.


I'm not fond of these identifiers, as method name is noun, and argument is verb:

async def sub_links(process_links):

Consider making it:

async def process_links(links):

(or process_links(sub_links) if you prefer). Some folks would use "url" for "link", but that's not a substantive criticism. "Process" is a bit vague - I feel download_links() is better.


Consider breaking out a temp variable:

    anchor = container.cssselect("h2 a")
    name = anchor[0].text if anchor else ""

Better, break that out as a trivial helper function so address and others can concisely use it. I never see such idioms when using BeautifulSoup - perhaps you'd like to try that library in your next project?

Again, I'm not happy with the adding_weblink() identifier, it is much too vague. Your comment suggests the appropriate verb would be download_profile() (or download_profile_page()). Also, PEP8 asks that you name the parameter profile_link - no biggie.

Settle on tree vs. root, just pick one.

Consider writing a brief helper function that returns html.fromstring(requests.get(link).text).

In this context weblink is on the redundant side, probably enough to just call it a link.

The if weblink: test probably isn't needed, as printing blank + empty_string before newline is pretty harmless.

output format

I can't imagine that name, address, etc. are easy to unambiguously parse from your output. It seems like you want a csv.writer, which would let you writerow(name, address, phone, weblink).

Imagine that 1% of all GET requests, or 10%, yielded temporary fail that you should retry. Would the current code grab 99% (or 90%) of all pages? Could a cleanup module conveniently identify links that should be retried?

Overall the code is nice and clear. I am not left with doubts about whether it would do what it is intended to do. (If Houzz changes details like .proListingLocation there will be some fragility issues, but hey, that's the life of a scraper. Definitely store a sample web page in source control, along with corresponding unit tests, so you can debug things in future when the inevitable website update happens. Have Jenkins run a daily test against the website so you notice bit-rot within 24 hours before it drifts "too" far.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.