Parsing a full site using python

Question

I've created a crawler which is scraping name, phone number and web address of each profile from houzz website. Hope I did it the right way. Here is what I've written:

import requests
from lxml import html

url="https://www.houzz.com/professionals/"

def scraper_func(mainurl):
    response = requests.get(mainurl).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//a[@class='sidebar-item-label']"):
        link = titles.xpath(".//@href")
        for item in link:
            paging_stuff(item)

# Done crawling links to the category from left-sided bar
def paging_stuff(process_links):
    response = requests.get(process_links).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//ul[@class='pagination']"):
        link = titles.xpath(".//a[@class='pageNumber']/@href")
        for item in link:
            processing_stuff(item)

# Going to each page to crawl the whole links spread through pagination connected to the profile page
def processing_stuff(procured_links):
    response = requests.get(procured_links).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//div[@class='name-info']"):
        links = titles.xpath(".//a[@class='pro-title']/@href")[0]
        main_stuff(links)

# Going to the profile page of each link
def main_stuff(main_links):
    response = requests.get(main_links).text
    tree = html.fromstring(response)

    def if_exist(titles,xpath):
        info=titles.xpath(xpath)
        if info:
            return info[0]
        return ""

    for titles in tree.xpath("//div[@class='profile-cover']"):
        name = if_exist(titles,".//a[@class='profile-full-name']/text()")
        phone = if_exist(titles,".//a[contains(concat(' ', @class, ' '), ' click-to-call-link ')]/@phone")
        web = if_exist(titles,".//a[@class='proWebsiteLink']/@href")
        print(name,phone,web)

scraper_func(url)

Answer 1

First of all, you should definitely re-use the same session for multiple requests to the same domain - it should result into a performance improvement:

if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

Other Improvements

• improve naming: you are over reusing item and titles variables. Instead, think of more appropriate and meaningful variable names. Also, I don't think the "_stuff" prefix contributes to readability and ease of understanding of the program

• put the main script execution logic to under if __name__ == '__main__': to avoid executing it on import

• you can avoid inner loops and iterate directly over hrefs here:

  for link in tree.xpath("//a[@class='sidebar-item-label']/@href"):
      paging_stuff(link)
  

  And here:

  for link in tree.xpath("//ul[@class='pagination']//a[@class='pageNumber']/@href"):
      processing_stuff(link)
  

• instead of putting comments before the functions, put them into appropriate docstrings

Notes

Note that you should realize that your solution is synchronous - you are processing urls sequentially one by one. If performance matters, consider looking into Scrapy.