I've written a script in python using requests module in combination with selenium along with regex to parse email address (if any exists) from any website. I tried to create it in such a way so that it can traverse javascript enabled sites as well. My crawler is supposed to track any website link (given in it's list storage) then find contact us
or about us
etc keywords from that page and parsing the matching link it will go to the target page and using regular expression it will finally parse the email address from that page. It scrapes the email address along with the link address where it parses the email from. I tried with several links and most of the cases it succeeds. I know it's very hard to create a full-fledged one but I tried and it is not despairing at all. Any suggestion to improve this crawler will be vastly appreciated.
Here is what I have written:
from selenium import webdriver
import requests; import re
from lxml.html import fromstring
from urllib.parse import urljoin
main_url = [
'http://advanta.in/',
'http://warnaengineeringworks.com/',
'http://unifrostindia.com/',
'http://www.superrefrigerations.com/',
'http://www.westernequipments.com/',
]
def search_item(driver,url):
driver.get(url)
tree = fromstring(driver.page_source)
for item in tree.xpath("//a/@href"):
if "contact us" in item.lower() or "about us" in item.lower() or "contact" in item.lower() or "about" in item.lower():
get_item(urljoin(url,item))
break
def get_item(link):
response = requests.get(link).text
try:
email = re.findall(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+',response)[0]
except: pass
if email:
print(link,email)
if __name__ == '__main__':
driver = webdriver.Chrome()
try:
for link in main_url:
search_item(driver,link)
finally:
driver.quit()