-1
\$\begingroup\$

I have a simple scrapy spider that crawls a page and returns H1 on the pages. Since, each page is unique, one cannot know how many h1's will be on the page. Since the scrapy spider returns a list, I need to convert the list of lists into variables that I can then insert.

The output of H1 could look like

['some text h1','second h1', 'third h1']

I have a working code that looks like the following

    def _h1(self, page, response) :
        if isinstance(response, HtmlResponse):
            h1 = response.xpath("//h1/text()").getall()
            length_h1 = (len(h1))
            page['h1_count'] = length_h1
            if length_h1 >= 4:
                page["h1"] = h1[0]
                page["h11"] = h1[1]
                page["h12"] = h1[2]
                page["h13"] = h1[3]
            elif length_h1 == 3:
                page["h1"] = h1[0]
                page["h11"] = h1[1]
                page["h12"] = h1[2]
            elif length_h1 == 2:
                page["h1"] = h1[0]
                page["h11"] = h1[1]
            elif length_h1 == 1:
                page["h1"] = h1[0]
            else :
                page["h1"] = "---"

Now I am only accounting for 5 cases but sometimes the page may have as many as 15 h1's.

I have considered a for loop but not sure if that is a more memory efficient way or a better way exists in Python at all ? Please consider me a beginner and go gentle.

\$\endgroup\$
  • 1
    \$\begingroup\$ The provided code doesn't work with 15 h1's, and so this code does not work the way you intend. \$\endgroup\$ – Peilonrayz May 19 at 11:48
  • \$\begingroup\$ @Peilonrayz - You are being pedantic and infact closing the question based on code that works and requires review for scalability. Good way to tel a new user that you are not welcome here ! Thank you. \$\endgroup\$ – Sam May 20 at 23:46
  • \$\begingroup\$ No. Everyone has to follow the rules. If you have a problem with how you've been treated you can raise it on Code Review Meta. \$\endgroup\$ – Peilonrayz May 20 at 23:51
1
\$\begingroup\$

Why don't use a 2-dimesion array? It's more simple and efficent, you don't need to waste time in condition statemento or loop.

def _h1(self, page, response) :
    if isinstance(response, HtmlResponse):
        page["h1"] = response.xpath("//h1/text()").getall()

If you need the number of H1 in page simply use len(page["h1"]) or if you need second result of your search use page["h1"][1] and so on.

| improve this answer | |
\$\endgroup\$
  • \$\begingroup\$ This makes much more sense. This creates another problem though that I am then inserting this into a db and a sqlite will usually not insert a list but that another problem altogether. Thank you for taking the time to respond. \$\endgroup\$ – Sam May 19 at 9:24
  • \$\begingroup\$ @Sam When you save in db use un loop, something like this for r in page["h1"]: db.insert(col=r) \$\endgroup\$ – n1k9 May 19 at 10:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.