A universal email crawler

Question

I've written a script in python using requests module in combination with selenium along with regex to parse email address (if any exists) from any website. I tried to create it in such a way so that it can traverse javascript enabled sites as well. My crawler is supposed to track any website link (given in it's list storage) then find contact us or about us etc keywords from that page and parsing the matching link it will go to the target page and using regular expression it will finally parse the email address from that page. It scrapes the email address along with the link address where it parses the email from. I tried with several links and most of the cases it succeeds. I know it's very hard to create a full-fledged one but I tried and it is not despairing at all. Any suggestion to improve this crawler will be vastly appreciated.

Here is what I have written:

from selenium import webdriver
import requests; import re
from lxml.html import fromstring
from urllib.parse import urljoin

main_url = [
'http://advanta.in/',
'http://warnaengineeringworks.com/',
'http://unifrostindia.com/',
'http://www.superrefrigerations.com/',
'http://www.westernequipments.com/',
]

def search_item(driver,url):
    driver.get(url)
    tree = fromstring(driver.page_source)
    for item in tree.xpath("//a/@href"):
        if "contact us" in item.lower() or "about us" in item.lower() or "contact" in item.lower() or "about" in item.lower():
            get_item(urljoin(url,item))
            break

def get_item(link):
    response = requests.get(link).text
    try:
        email = re.findall(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+',response)[0]
    except: pass
    if email:
        print(link,email)


if __name__ == '__main__':
    driver = webdriver.Chrome()
    try:
        for link in main_url:
            search_item(driver,link)
    finally:
        driver.quit()

Answer 1

There are several ways we can go to make the code cleaner and faster. Grouping by the scope.

search_item() function

• link would probably be a better variable name than a less descriptive item
• move the substrings you are looking for into a constant - defined as a list of keywords
• then, you can use any() instead of multiple or conditions
• fromstring() returns a root node of the tree and not the tree itself - would probably be more logical to name the variable root instead of tree
• do not re-do item.lower() - do it once

You may get something like this in the end:

KEYWORDS = ["contact us", "about us", "contact", "about"]

def search_item(driver,url):
    driver.get(url)
    root = fromstring(driver.page_source)

    for link in root.xpath("//a/@href"):
        link = link.lower()
        if any(keyword in link for keyword in KEYWORDS):
            get_item(urljoin(url, link))
            break

get_item() function

• you should pre-compile your regular expression
• use .search() instead of .findall(), since you are looking for a single match
• this would then allow you to avoid handling a bare exception and switch to LBYL approach instead of the EAFP

You may get something like:

EMAIL_PATTERN = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+')

def get_item(link):
    response = requests.get(link).text

    match = EMAIL_PATTERN.search(response)
    if match:
        print(link, match.group(0))

Few other general comments

• not sure if "contact us" and "about us" (with a space) would ever appear in an href value..
• re-group your imports as per PEP8 recommendations and avoid having multiple different imports on the same line

• main_url should probably named be something like start_urls (you need a multiple of "url" since you have a list of them). And, you need to indent the items since you are defining your list on multiple lines:

  start_urls = [
      'http://advanta.in/',
      'http://warnaengineeringworks.com/',
      'http://unifrostindia.com/',
      'http://www.superrefrigerations.com/',
      'http://www.westernequipments.com/',
  ]
  

• consider adding documentation strings to your functions