I've written a script in python using "threading" module to scrape two sites simultaneously. It parses the two sites flawlessly. Any insight as to how I can improve this script will be appreciated.

Here is what I did:

import requests ; from lxml import html
import threading ; import time

Yp_link = "https://www.yellowpages.com/search?search_terms=coffee&geo_location_terms=Los%20Angeles%2C%20CA&page=2"
Tuts_link = "http://www.wiseowl.co.uk/videos/"

def create_links(url):
    response = requests.get(url).text
    tree = html.fromstring(response)
    for title in tree.cssselect("div.info"):
        name = title.cssselect("a.business-name span[itemprop=name]")[0].text
        street = title.cssselect("span.street-address")[0].text
        phone = title.cssselect("div[itemprop=telephone]")[0].text if title.cssselect("div[itemprop=telephone]") else ""
        print(name, street, phone)

def process_links(link):
    response = requests.get(link).text
    tree = html.fromstring(response)
    for titles in tree.xpath("//p[@class='woVideoListDefaultSeriesTitle']"):
        title = titles.xpath('.//a')[0]
        print(title.text, title.attrib['href'])

th1 = threading.Thread(target=create_links, args=(Yp_link,))
th2 = threading.Thread(target=process_links, args=(Tuts_link,))


  • 1
    \$\begingroup\$ Do you want to be polite? (You are likely to get blacklisted if your not). Also as this counts as a robot did you want to check the robots.txt file to make sure you are not scrapping pages that are not meant for robots (as this will usually result in your scrapping randomly generated pages that are just junk designed to trap bad robots). \$\endgroup\$ – Martin York Sep 7 '17 at 17:30
  • \$\begingroup\$ Politness: stackoverflow.com/q/8236046/14065 I suppose a sleep of 1 second counts as being polite. \$\endgroup\$ – Martin York Sep 7 '17 at 17:31
  • \$\begingroup\$ Robots: robotstxt.org/robotstxt.html \$\endgroup\$ – Martin York Sep 7 '17 at 17:31
  • \$\begingroup\$ @Loki Astari, I'm dreadfully sorry that I could not respond to your comment instantly cause the internet was down in my end. As I am not from a programming background, sometimes I find it hard to understand stuffs so easily. I knew so far that "robotstxt" is only available in scrapy. However, seeing your comment I got even more confused as to where I find that. Btw, I'm gonna follow your provided link. Thanks. \$\endgroup\$ – MITHU Sep 8 '17 at 7:32
  • 1
    \$\begingroup\$ The robots.txt file (if it exists) is supposed to be at the root of a site. https://www.yellowpages.com/robots.txt and http://www.wiseowl.co.uk/robots.txt \$\endgroup\$ – Martin York Sep 8 '17 at 15:52

First of all, I think you are putting time.sleep() calls into the wrong places - you are putting them into the loops where you iterate over the extracted elements. Elements are already extracted and no requests are issued at that point - add delays between each requests - at the end of your functions.

I would also improve naming - Yp_link and Tuts_link can be renamed to more explicit YELLOW_PAGES_URL and WISEOWL_URL - note that I think these two need to be defined as proper constants - in upper case.

And, I would also switch to CSS selector locators for process_links() function as well.

As far as imports go, just don't put them on same lines - put each import on it's own line as per PEP8 importing guidelines.

  • \$\begingroup\$ Thanks sir alecxe, for the review. If i put time.sleep() just before the loop starts, will that be the right way to follow what you instructed? \$\endgroup\$ – MITHU Sep 8 '17 at 7:35
  • \$\begingroup\$ @Mithu right, just move it from out of the loop please. \$\endgroup\$ – alecxe Sep 8 '17 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.