I have a simple scrapy spider that crawls a page and returns H1 on the pages. Since, each page is unique, one cannot know how many h1's will be on the page. Since the scrapy spider returns a list, I need to convert the list of lists into variables that I can then insert.
The output of H1 could look like
['some text h1','second h1', 'third h1']
I have a working code that looks like the following
def _h1(self, page, response) : if isinstance(response, HtmlResponse): h1 = response.xpath("//h1/text()").getall() length_h1 = (len(h1)) page['h1_count'] = length_h1 if length_h1 >= 4: page["h1"] = h1 page["h11"] = h1 page["h12"] = h1 page["h13"] = h1 elif length_h1 == 3: page["h1"] = h1 page["h11"] = h1 page["h12"] = h1 elif length_h1 == 2: page["h1"] = h1 page["h11"] = h1 elif length_h1 == 1: page["h1"] = h1 else : page["h1"] = "---"
Now I am only accounting for 5 cases but sometimes the page may have as many as 15 h1's.
I have considered a for loop but not sure if that is a more memory efficient way or a better way exists in Python at all ? Please consider me a beginner and go gentle.