Scraping different categories using different links through proxy

Question

I've written a script in python to scrape name, review_star and review_count by going through thousands of links (to different products) stored in a csv file using reverse search. As those links are of amazon site so it is very natural to get the ip address banned for a short time while making use of few links only. However, to retain the continuation it is necessary to filter this process through proxy. This is what I tried to do here and it is running smoothly. For the record: as these proxies are collected from web, they may not last long. Anyways, this scraper is supposed to make requests using each links from the csv file, collect the product name, review_star and review_count from amazon site without being blocked. Considering the space, I only used three proxies in my scraper. I tried my level best to make it flawless and it is working without leaving any complaint at this moment. Any suggestion to make this better will be highly appreciated.

This is the script I've written:

import csv
import requests
from lxml.html import fromstring

def crawl():

    proxy={
        'https': 'https://188.166.154.140:8118',
        'https': 'https://198.52.44.130:53281',
        'https': 'https://46.101.2.115:8118'
        }

    with open("amazon.csv", 'r') as input_file:
        for entry in csv.DictReader(input_file):
            url = entry['Link']
            response = requests.get(url,headers={'User-Agent':'Mozilla/5.0'},proxies=proxy)
            root = fromstring(response.text)
            name = root.cssselect("#productTitle")[0].text.strip() if root.cssselect("#productTitle") else ""
            star = root.cssselect(".AverageCustomerReviews .a-icon-alt")[0].text if root.cssselect(".AverageCustomerReviews .a-icon-alt") else ""
            count = root.cssselect(".AverageCustomerReviews .totalReviewCount")[0].text if root.cssselect(".AverageCustomerReviews .totalReviewCount") else ""
            print("Name: {}\nStar: {}\nCount: {}".format(name,star,count))            

if __name__ == '__main__':
    crawl()

These are the five links out of thousands which are supposed to store in a csv file named amazon.csv containing a header Link:

Link
https://www.amazon.com/dp/B013KBZ9RY
https://www.amazon.com/dp/B004AC6PQW
https://www.amazon.com/dp/B01JJBA06Y
https://www.amazon.com/dp/B001G9G6XY
https://www.amazon.com/dp/B000UCM0P6

Answer 1

Assuming the proxy dictionary with duplicate keys is just a posting error, here are some of the things I think need to be addressed:

• as usual, create a requests.Session() instance and re-use - this would not only allow to set headers and proxies once, but also improve performance
• I would define headers as a constant or go even step further - rotate user agent strings as well - there is a fake-useragent project that would help

• there is also this DRY principle violation - you are repeating cssselect() calls twice per element - in this case, it's not only the repetition but is also something that unnecessarily slows the HTML parsing down. Define a reusable function that would handle the problem:

  def extract_field(root, selector, default=""):
      try:
          return root.cssselect(selector)[0].text.strip()
      except IndexError:
          return default
  

Here is the code with the above and other modifications applied:

import csv

import requests
from lxml.html import fromstring


def extract_field(root, selector, default=""):
    try:
        return root.cssselect(selector)[0].text.strip()
    except IndexError:
        return default


def crawl():
    proxy = {  # assuming is correct
        'https': 'https://188.166.154.140:8118',
        'https': 'https://198.52.44.130:53281',
        'https': 'https://46.101.2.115:8118'
    }

    with open("amazon.csv", 'r') as input_file, requests.Session() as session:
        session.headers = {'User-Agent': 'Mozilla/5.0'}
        session.proxies = proxy

        for entry in csv.DictReader(input_file):
            url = entry['Link']
            response = session.get(url)
            root = fromstring(response.text)

            name = extract_field(root, "#productTitle")
            star = extract_field(root, ".AverageCustomerReviews .a-icon-alt")
            count = extract_field(root, ".AverageCustomerReviews .totalReviewCount")

            print("Name: {}\nStar: {}\nCount: {}".format(name,star,count))


if __name__ == '__main__':
    crawl()