Today, while coming across a tutorial made by ScrapingHub on Scrapy about how it usually deals with a webpage while scraping it's content. I could see that the same logic applied in Scrapy can be applied in regular method in python.
So, I tried to make one. Here is what I did. My scraper opens a webpage, parses the 10 tags from it's right sided area then tracks down each tags and then again follows its pagination and parses the whole content. Hope I did it the right way. Here is the code:
import requests ; from lxml import html core_link = "http://quotes.toscrape.com/" def quotes_scraper(base_link): response = requests.get(base_link) tree = html.fromstring(response.text) for titles in tree.cssselect("span.tag-item a.tag"): processing_docs(core_link + titles.attrib['href']) def processing_docs(base_link): response = requests.get(base_link).text root = html.fromstring(response) for soups in root.cssselect("div.quote"): quote = soups.cssselect("span.text").text author = soups.cssselect("small.author").text print(quote, author) next_page = root.cssselect("li.next a").attrib['href'] if root.cssselect("li.next a") else "" if next_page: page_link = core_link + next_page processing_docs(page_link) quotes_scraper(core_link)