2
\$\begingroup\$

I've written a script in python to scrape name, review_star and review_count by going through thousands of links (to different products) stored in a csv file using reverse search. As those links are of amazon site so it is very natural to get the ip address banned for a short time while making use of few links only. However, to retain the continuation it is necessary to filter this process through proxy. This is what I tried to do here and it is running smoothly. For the record: as these proxies are collected from web, they may not last long. Anyways, this scraper is supposed to make requests using each links from the csv file, collect the product name, review_star and review_count from amazon site without being blocked. Considering the space, I only used three proxies in my scraper. I tried my level best to make it flawless and it is working without leaving any complaint at this moment. Any suggestion to make this better will be highly appreciated.

This is the script I've written:

import csv
import requests
from lxml.html import fromstring

def crawl():

    proxy={
        'https': 'https://188.166.154.140:8118',
        'https': 'https://198.52.44.130:53281',
        'https': 'https://46.101.2.115:8118'
        }

    with open("amazon.csv", 'r') as input_file:
        for entry in csv.DictReader(input_file):
            url = entry['Link']
            response = requests.get(url,headers={'User-Agent':'Mozilla/5.0'},proxies=proxy)
            root = fromstring(response.text)
            name = root.cssselect("#productTitle")[0].text.strip() if root.cssselect("#productTitle") else ""
            star = root.cssselect(".AverageCustomerReviews .a-icon-alt")[0].text if root.cssselect(".AverageCustomerReviews .a-icon-alt") else ""
            count = root.cssselect(".AverageCustomerReviews .totalReviewCount")[0].text if root.cssselect(".AverageCustomerReviews .totalReviewCount") else ""
            print("Name: {}\nStar: {}\nCount: {}".format(name,star,count))            

if __name__ == '__main__':
    crawl()

These are the five links out of thousands which are supposed to store in a csv file named amazon.csv containing a header Link:

Link
https://www.amazon.com/dp/B013KBZ9RY
https://www.amazon.com/dp/B004AC6PQW
https://www.amazon.com/dp/B01JJBA06Y
https://www.amazon.com/dp/B001G9G6XY
https://www.amazon.com/dp/B000UCM0P6
\$\endgroup\$
1
\$\begingroup\$

Assuming the proxy dictionary with duplicate keys is just a posting error, here are some of the things I think need to be addressed:

  • as usual, create a requests.Session() instance and re-use - this would not only allow to set headers and proxies once, but also improve performance
  • I would define headers as a constant or go even step further - rotate user agent strings as well - there is a fake-useragent project that would help
  • there is also this DRY principle violation - you are repeating cssselect() calls twice per element - in this case, it's not only the repetition but is also something that unnecessarily slows the HTML parsing down. Define a reusable function that would handle the problem:

    def extract_field(root, selector, default=""):
        try:
            return root.cssselect(selector)[0].text.strip()
        except IndexError:
            return default
    

Here is the code with the above and other modifications applied:

import csv

import requests
from lxml.html import fromstring


def extract_field(root, selector, default=""):
    try:
        return root.cssselect(selector)[0].text.strip()
    except IndexError:
        return default


def crawl():
    proxy = {  # assuming is correct
        'https': 'https://188.166.154.140:8118',
        'https': 'https://198.52.44.130:53281',
        'https': 'https://46.101.2.115:8118'
    }

    with open("amazon.csv", 'r') as input_file, requests.Session() as session:
        session.headers = {'User-Agent': 'Mozilla/5.0'}
        session.proxies = proxy

        for entry in csv.DictReader(input_file):
            url = entry['Link']
            response = session.get(url)
            root = fromstring(response.text)

            name = extract_field(root, "#productTitle")
            star = extract_field(root, ".AverageCustomerReviews .a-icon-alt")
            count = extract_field(root, ".AverageCustomerReviews .totalReviewCount")

            print("Name: {}\nStar: {}\nCount: {}".format(name,star,count))


if __name__ == '__main__':
    crawl()
|improve this answer|||||
\$\endgroup\$
  • \$\begingroup\$ Thanks a lot sir, for your time, a fresh code and invaluable guidelines. I didn't post here that proxy part by mistake; rather, this is my limitation that I know thus far. As it was not producing error and giving me right results I thought what I did was correct. Should I remove anything from that proxy to make it free of duplicates. I used three different proxies to make the scraper fight longer against the blocking process. Thanks a lot again, sir \$\endgroup\$ – SIM Oct 22 '17 at 5:32
  • \$\begingroup\$ proxy dictionary looks odd - you cannot declare a dictionary with the same key. \$\endgroup\$ – DataGreed May 28 '19 at 16:02
  • \$\begingroup\$ @DataGreed yeah, I did not know why the OP had it there but left as is. \$\endgroup\$ – alecxe May 29 '19 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.