Im new to Python and Scrapy and Im trying to write a code that would find if a website contains a specific word in it. I need to scrapy many websites with different layouts so the search is a general one. I would appreciate if you could comment do you see a more efficient way on how to perform this task. Here is my solution:

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from .lists_loop import * #used to import list of 'word_to_find'
import re

word_to_find = 'pharmacy'

class TestSpider(CrawlSpider):
    name = 'test'
    # these are lists of a lot of domains
    allowed_domains = strip_url
    start_urls = merch_url
rules = (
    Rule(LinkExtractor(), callback='parse_item', follow=True),

def parse_item(self, response):
    # Here I clean up the parsed text not to include /n or whitespace.
    words = response.xpath("//a//text()").getall()
    cleaned_words = [word.strip() for word in words]
    cleaned_words = [word.lower() for word in cleaned_words if len(word) > 0]

    # Then I loop through the cleaned_words in order to find a match
    for single_word in cleaned_words:
        re.search(r'\b%s\b' % word_to_find, single_word)
            'Matching': 'Found the word {} in {}'.format(word_to_find, response.url)

I am stripping the found //a//text() and hence in the end searching each word separately if it matches any of my word_to_find list of words (In this case I just added 1 word for simplicity). Maybe its better to search the whole text string but the problem I encountered is that a lot of times it includes a lot of whitespaces /n symbols etc.

Any advice would be appreciated.

  • \$\begingroup\$ allowed_domains = strip_url, start_urls = merch_url Where are strip_url and merch_url defined? I think the next two lines in that function should be further indented, unless you want rules to be defined in the global scope in which case it's in a poor location. \$\endgroup\$
    – Mast
    Dec 29, 2021 at 5:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.