I've written some script in python using lxml library which is capable of dealing with a webpage with complicatedly layed-out next page links. My scraper is able to parse all the next page links without going to the next page and hardcoding any number to the lastmost link and scrape the required fields flawlessly. Although I tried to do the whole thing specklessly, any improvement on this will be highly appreciable. Here is what I've tried with:

import requests
from lxml import html

main_link = "https://www.yify-torrent.org/search/1080p/"
base_link = "https://www.yify-torrent.org"

def get_links(item_link):
    response = requests.get(item_link).text
    tree = html.fromstring(response)
    last_page = tree.cssselect('div.pager a:contains("Last")')[0].attrib["href"].split("/")[-2].replace('t-','')
    links = [item_link +"t-{0}/".format(page) for page in range(int(last_page) +1)]
    for link in links:

def process_docs(nextpage_link):
    response = requests.get(nextpage_link).text
    tree = html.fromstring(response)
    for items in tree.cssselect('div.mv'):
        name = items.cssselect('h3 a')[0].text
            genre = items.xpath('.//div[@class="mdif"]//li[b="Genre:"]/text()')[0] # I don't know If css selector could do this
        except IndexError:
            genre = ""

            imd = items.xpath('.//div[@class="mdif"]//li[b="Rating:"]/text()')[0] # Used xpath in lieu of css selector
        except IndexError:
            imd = ""

        print(name, genre, imd)


1 Answer 1


This is quite clean and understandable.

I would pinpoint only a couple of things:

  • to avoid code repetition and to generalize getting search result properties, what if you would collect all the properties (genre, quality, size etc) into a dictionary:

    properties = {
        label.text: label.xpath("following-sibling::text()")[0]
        for label in items.xpath(".//div[@class="mdif"]//li/b")

    As a bonus, you would not need the try/except part at all.

  • as usual, you may reuse a session to boost your HTTP requests part performance

Some other minor notes:

  • it looks like that you make an extra request to the page number 0, adjust your range if this is the case
  • items variable name is probably a bit misleading, at the very least it should be item since you are iterating over search result items one by one
  • put get_links() call to under the if __name__ == '__main__' to avoid the code being executed if the module is ever imported
  • I would also return or yield from the get_links and process_docs functions instead of printing things out
  • \$\begingroup\$ Thanks sir for the review and suggestions. You always lead me to the right direction but I get derailed while writing new stuffs. Btw, what could be the css selector for genre or imd? Thanks again sir, alecxe. \$\endgroup\$
    – SIM
    Jul 28, 2017 at 14:07
  • \$\begingroup\$ @SMth80 Thanks, sir! The actual genre or imd values are coming from text nodes and, from what I recall, you cannot match text nodes with CSS selectors directly. You may go into getting the li values and then parsing the li texts which would contain both the "label" and the "value" of a property - e.g. Genre: drama - then split by :..but, I think this is a good use case for XPaths. \$\endgroup\$
    – alecxe
    Jul 28, 2017 at 14:31
  • \$\begingroup\$ @Sir alecxe, I've already created some css selectors to grab the "Genre" and "Rating" what I couldn't do in my above script. I thought you might like it that is why I drop here a link for you to look into what I did. dropbox.com/s/03iz95cxavp9gug/For%20codereview.txt?dl=0 \$\endgroup\$
    – SIM
    Sep 3, 2017 at 8:07
  • \$\begingroup\$ @Shahin nice looking CSS selectors, good job, thanks. \$\endgroup\$
    – alecxe
    Sep 3, 2017 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.