The goal is to extract the the Features section from a Lego product page. In the Features section, usually there's a header (<h2> or <h3>) in the subsection and a paragraph made up of multiple <span> segments. The goal is to exact the Features section such that the output is an OrderedDict of h2/h3 as key and a list of text in the spans as values.

For example, https://www.lego.com/en-us/product/hokusai-the-great-wave-31208, the expected output is:

{'Recreate an iconic piece of Japanese art': ['Celebrate your passion for Japanese art when you build this incredible LEGO® version of Hokusai’s The Great Wave.'],
 'Build and relax': ['Art lovers can enjoy a relaxing and immersive building experience as they create this unique artwork from 1,810 pieces.'],
 'Hokusai – The Great Wave': ['Create your own artwork',
  'Build Hokusai’s The Great Wave with layers of LEGO® bricks.',
  'The Great Wave comes to life!',
  'The picture’s multiple layers create a stunning 3D effect.',
  'Finishing touch',
  'Add a decorative tile with Hokusai’s signature.'],
 'Immerse yourself in the world of art': ['Listen to the set’s Soundtrack, tailor-made with content to enhance the time you spend building this Japanese wall art.'],
 'A fantastic gift for art lovers': ['Designed for adults, this stunning piece of 3D art can be proudly displayed on a wall following a rewarding build experience.']}

I've tried chaining a couple of if-else and try-excepts to cover edge cases when parsing the HTML structure using BeautifulSoup. Though the code works, it looks a little messy and any suggestions/improvements would be very helpful!

import re
from collections import OrderedDict, defaultdict
from itertools import chain

import requests
from bs4 import BeautifulSoup

def slurp_features(bsoup):
    features = defaultdict(list)

    for div in bsoup.find("div", attrs={"data-test":"pdp-features-accordion-content-child"}).find_all('div'):
        if 'data-test' in div.attrs:
            if div.text:
                if len(div.find_all('h2')) == 1 or len(div.find_all('h3')) == 1:
                        header, *texts = div.find_all('span')
                    except ValueError:
                    section_text = []                
                    for tt in OrderedDict.fromkeys([t for t in texts]):
                        if isinstance(tt, tuple):
                            section_text += [ttt.text for tttt in chain(*tt)]
                    features[header.text] += section_text

    features = {k:list(OrderedDict.fromkeys(v)) for k,v in features.items()}
    to_delete = []
    for k,v in features.items():
        for vv in v:
            if vv in features:
    for d in to_delete:
        if d in features:
            del features[d]
    return features

response = requests.get("https://www.lego.com/en-us/product/hokusai-the-great-wave-31208")

bsoup = BeautifulSoup(response.content.decode('utf8'))

Here's some parts of the code that I think can be improved but not sure how to go about it,

  • The chain if clauses looks like unnecessary complication if 'data-test' in div.attrs: -> if div.text: -> if len(div.find_all('h2')) == 1 or len(div.find_all('h3')) == 1:
  • A for-loop to cast a list into an OrderedDict, for tt in OrderedDict.fromkeys([t for t in texts]):
  • This kinda check to see if the text is a tuple, if so flattening the nested list. Otherwise add the text to the section_text
  • The last part where we loop through the data twice, once to determine what should be removed and then another to actually remove them from the output features
  • \$\begingroup\$ OrderedDict is necessary for ancient python3 versions, but dict suffices with cPython interpreters since 3.7. A validator outline (scroll to bottom) suggests that simplying asking BS4 for H2's plus descendant .text would get you a long way toward your scraping goal. Write a (yield) generator for that raw datastream, then a consumer which cleans it to your liking. \$\endgroup\$
    – J_H
    Commented Apr 7, 2023 at 17:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.