Scraping required fields exhausting a full website

Question

I've written some script in python using lxml library which is capable of dealing with a webpage with complicatedly layed-out next page links. My scraper is able to parse all the next page links without going to the next page and hardcoding any number to the lastmost link and scrape the required fields flawlessly. Although I tried to do the whole thing specklessly, any improvement on this will be highly appreciable. Here is what I've tried with:

import requests
from lxml import html

main_link = "https://www.yify-torrent.org/search/1080p/"
base_link = "https://www.yify-torrent.org"

def get_links(item_link):
    response = requests.get(item_link).text
    tree = html.fromstring(response)
    last_page = tree.cssselect('div.pager a:contains("Last")')[0].attrib["href"].split("/")[-2].replace('t-','')
    links = [item_link +"t-{0}/".format(page) for page in range(int(last_page) +1)]
    for link in links:
        process_docs(link)

def process_docs(nextpage_link):
    response = requests.get(nextpage_link).text
    tree = html.fromstring(response)
    for items in tree.cssselect('div.mv'):
        name = items.cssselect('h3 a')[0].text
        try:
            genre = items.xpath('.//div[@class="mdif"]//li[b="Genre:"]/text()')[0] # I don't know If css selector could do this
        except IndexError:
            genre = ""

        try:
            imd = items.xpath('.//div[@class="mdif"]//li[b="Rating:"]/text()')[0] # Used xpath in lieu of css selector
        except IndexError:
            imd = ""

        print(name, genre, imd)

get_links(main_link)

Answer 1

This is quite clean and understandable.

I would pinpoint only a couple of things:

• to avoid code repetition and to generalize getting search result properties, what if you would collect all the properties (genre, quality, size etc) into a dictionary:

  properties = {
      label.text: label.xpath("following-sibling::text()")[0]
      for label in items.xpath(".//div[@class="mdif"]//li/b")
  }
  

  As a bonus, you would not need the try/except part at all.

• as usual, you may reuse a session to boost your HTTP requests part performance

Some other minor notes:

• it looks like that you make an extra request to the page number 0, adjust your range if this is the case
• items variable name is probably a bit misleading, at the very least it should be item since you are iterating over search result items one by one
• put get_links() call to under the if __name__ == '__main__' to avoid the code being executed if the module is ever imported
• I would also return or yield from the get_links and process_docs functions instead of printing things out