I'm attempting to find companies who mention a particular service in on their homepage. To do this, I am iterating through a csv file with two columns - ID and URL. I'm using BeautifulSoup to get the html and regex to find the string.
At present, my code works, but it feels very clunky and takes forever. I'm also not writing my matching IDs to the new csv, which I haven't been able to figure out.
Since this is at least working, hopefully, it will help someone else who is spinning their wheels trying to figure it out.
How can it be improved?
import requests from bs4 import BeautifulSoup import re import csv with open('web1.csv', mode='r') as infile: reader = csv.reader(infile) with open('websites_new.csv', mode='w') as outfile: writer = csv.writer(outfile) mydict = dict((rows,rows) for rows in reader) newlist =  for v in mydict.itervalues(): try: page = requests.get('http://www.' + v) except: pass soup = BeautifulSoup(page.content, 'html.parser') soupString = str(soup) re1='.*?' re2='(secretword)' rg = re.compile(re1+re2,re.IGNORECASE|re.DOTALL) m = rg.search(soupString) if m is None: value = 'x' newlist.extend(value) else: newlist.extend(v) print newlist