I have a large HTML document where everything is inside a main div, structured like this:
<div class="main"> \\block 1 <div class="header"><span>content1a</span><span>content1b</span></div> <p>content1c</p> \\block 2 <div class="header"><span>content2a</span><span>content2b</span></div> <p>content2c</p> ... </div>
As you can see the contents of the header divs are related to the paragraphs, so my Python code separates these tag blocks into lists of a list so I can take out their contents later:
main_div = soup.find("div", class_="main") headers = main_div.find_all("div", class_="header") all_blocks =  current_block =  for tag in main_div.contents: if tag in headers: all_blocks.append(current_block) current_block = [tag] else: current_block.append(tag) all_blocks.append(current_block) # append final block all_blocks = all_blocks[1:] # take off first empty list
Problem is this seems to take forever on a ~12MB HTML file I'm feeding it (less than 1% done after 1h, while parsing the file with bs4 only took 25s). Is there something really inefficient in my code or is this going to be slow anyway?