3
\$\begingroup\$

I have written a crawler in python with the combination of class and function. Few days back I saw a scraper in a tutorial more or less similar to what I did here. I found it hard the necessity of using class here. However, I decided to create one. My scraper is able to traverse all the next pages and print the collected results errorlesly. If there is any suggestion or input to give this scraper a better look, I'm ready to comply with that. Thanks in advance.

Here is what I've written:

import requests
from urllib.request import urljoin
from lxml.html import fromstring            

main_link = "https://www.yify-torrent.org/search/western/"

class movie_doc(object):
    def __init__(self,name,genre,quality,size,rating):
        self.name = name
        self.genre = genre
        self.quality = quality
        self.size = size
        self.rating = rating

film_storage = [] # global variable

def get_links(link):
    root = fromstring(requests.get(link).text)
    for item in root.cssselect(".mv"):
        name = item.cssselect("h3 a")[0].text
        genre = item.cssselect("li b:contains('Genre:')")[0].tail
        quality = item.cssselect("li b:contains('Quality:')")[0].tail
        size = item.cssselect("li b:contains('Size:')")[0].tail
        rating = item.cssselect("li b:contains('Rating:')")[0].tail
        film_details = movie_doc(name,genre,quality,size,rating)
        film_storage.append(film_details)

    next_page = root.cssselect(".pager a:contains('Next')")[0].attrib['href'] if root.cssselect(".pager a:contains('Next')") else ""
    if next_page:
        full_link = urljoin(link,next_page)
        get_links(full_link)

if __name__ == '__main__':  
    get_links(main_link)
    for item in film_storage: # as film_storage variable is declared global I called here directly
        print(item.name,item.genre,item.quality,item.size,item.rating)
\$\endgroup\$
4
\$\begingroup\$

For this kind of a simple "storage" class, it might be more appropriate to use either a simple dictionary or a collections.namedtuple():

from collections import namedtuple

MovieDocument = namedtuple('MovieDocument', ['name', 'genre', 'quality', 'size', 'rating'])

Note, style-wise, the class name should be defined in PascalCase starting with a capital letter.

Also, consider making get_links() a generator and renaming it to, say, crawl() as it is not actually getting the links but crawling and returning movie objects.

I would also use an EAFP approach for the next page link retrieval.

The code with all the changes applied:

from collections import namedtuple
from urllib.request import urljoin

import requests
from lxml.html import fromstring


Movie = namedtuple('MovieDocument', ['name', 'genre', 'quality', 'size', 'rating'])


def crawl(link):
    root = fromstring(requests.get(link).text)
    for item in root.cssselect(".mv"):
        name = item.cssselect("h3 a")[0].text
        genre = item.cssselect("li b:contains('Genre:')")[0].tail
        quality = item.cssselect("li b:contains('Quality:')")[0].tail
        size = item.cssselect("li b:contains('Size:')")[0].tail
        rating = item.cssselect("li b:contains('Rating:')")[0].tail

        yield Movie(name, genre, quality, size, rating)

    try:
        next_page = root.cssselect(".pager a:contains('Next')")[0].attrib['href']
        full_link = urljoin(link,next_page)
        yield from crawl(full_link)
    except IndexError:
        pass


if __name__ == '__main__':
    for movie in crawl("https://www.yify-torrent.org/search/western/"):
        print(movie.name, movie.genre, movie.quality, movie.size, movie.rating)

There is, though, still this repetition in the CSS selectors - consider extracting it into a reusable function parameterized with a text you then use in contains().

\$\endgroup\$
4
  • \$\begingroup\$ No way!! The best approach so far, sir. Two questions come in my mind, though! 1. Was the way i did a valid approach? 2. once again in love with this yield from part but find it hard to understand. Could you please clarify a bit or lead me to follow a link? Thanks a lot once again. \$\endgroup\$ – SIM Oct 28 '17 at 13:34
  • 1
    \$\begingroup\$ camelCase is typically 'first word lowercase', 'next words titlecase', whereas PascalCase is 'all words titlecase'. \$\endgroup\$ – Daniel Oct 28 '17 at 13:47
  • \$\begingroup\$ @Shahin yeah, it was a valid approach overall. The 'yield from' is a way to yield from a subgenerator, which basically helps to avoid looping over the generator and yielding..hope that helps. \$\endgroup\$ – alecxe Oct 28 '17 at 15:03
  • \$\begingroup\$ @Coal_ ah, good catch, applied the change, thanks! \$\endgroup\$ – alecxe Oct 29 '17 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.