Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general one. I would appreciate if you could comment do you see a more efficient way on how to perform this task. Here is my solution:
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from .lists_loop import * #used to import list of 'word_to_find'
import re
word_to_find = 'pharmacy'
class TestSpider(CrawlSpider):
name = 'test'
# these are lists of a lot of domains
allowed_domains = strip_url
start_urls = merch_url
rules = (
Rule(LinkExtractor(), callback='parse_item', follow=True),
)
def parse_item(self, response):
# Here I clean up the parsed text not to include /n or whitespace.
words = response.xpath("//a//text()").getall()
cleaned_words = [word.strip() for word in words]
cleaned_words = [word.lower() for word in cleaned_words if len(word) > 0]
# Then I loop through the cleaned_words in order to find a match
for single_word in cleaned_words:
re.search(r'\b%s\b' % word_to_find, single_word)
yield{
'Matching': 'Found the word {} in {}'.format(word_to_find, response.url)
}
else:
pass
I am stripping the found //a//text() and hence in the end searching each word separately if it matches any of my word_to_find
list of words (In this case I just added 1 word for simplicity). Maybe its better to search the whole text string but the problem I encountered is that a lot of times it includes a lot of whitespaces /n symbols etc.
Any advice would be appreciated.
allowed_domains = strip_url
,start_urls = merch_url
Where arestrip_url
andmerch_url
defined? I think the next two lines in that function should be further indented, unless you wantrules
to be defined in the global scope in which case it's in a poor location. \$\endgroup\$