3
\$\begingroup\$

This code outputs each tag followed by its attributes in the given HTML, both in lexicographic order. It must be possible to keep the spirit of the code without these many nested cycles and conditionals.

from bs4 import BeautifulSoup

htmlist = ['<div class="portal" role="navigation" id=\'p-coll-print_export\'>',
     '<h3>Print/export</h3>',
     '<div class="body">',
     '<ul>',
     '<li id="coll-create_a_book"><a href="/w/index.php?title=Special:Book&amp;bookcmd=book_creator&amp;referer=Main+Page">Create a book</a></li>',
     '<li id="coll-download-as-rl"><a href="/w/index.php?title=Special:Book&amp;bookcmd=render_article&amp;arttitle=Main+Page&amp;oldid=560327612',
     '<li id="t-print"><a href="/w/index.php?title=Main_Page&amp;printable=yes" title="Printable version of this page [p]" accesskey="p">Printable version</a></li>',
     '</ul>',
     '</div>',
     '</div>']

soup = BeautifulSoup("\n".join(htmlist), "html.parser")

tags = sorted(set([tag.name for tag in soup.find_all()]))

for tag in tags:
    tags_keys = list()
    for line in htmlist:
        aux=BeautifulSoup(line, "html.parser").find(tag)
        if aux:
            tags_keys.extend(list(aux.attrs))
    print(tag+":"+",".join(sorted(set(tags_keys))))

I tried something along the lines of what's below (in place of the outermost for), but couldn't even get it to work.

for tag in tags:
    print(tag+":"+",".join(sorted(soup.find(tag).attrs.keys())))

Edit: There was a typo in the code, htmlist was supposed to be as follows.

htmlist = ['<div class="portal" role="navigation" id="p-coll-print_export">',\
           '<h3>Print/export</h3>',\
           '<div class="body">',\
           '<ul>',\
           '<li id="coll-create_a_book"><a href="/w/index.php?title=Special:Book&amp;bookcmd=book_creator&amp;referer=Main+Page">Create a book</a></li>',\
           '<li id="coll-download-as-rl"><a href="/w/index.php?title=Special:Book&amp;bookcmd=render_article&amp;arttitle=Main+Page&amp;oldid=560327612&amp;writer=rl">Download as PDF</a></li>',\
           '<li id="t-print"><a href="/w/index.php?title=Main_Page&amp;printable=yes" title="Printable version of this page [p]" accesskey="p">Printable version</a></li>',\
           '</ul>',\
           '</div>',\
           '</div>',\
          ]
\$\endgroup\$
4
\$\begingroup\$

Notice in @alecxe's answer how he flipped the loops: instead of iterating over tags and then lines, he's iterating over lines and then tags. This performs much better, because only one BeautifulSoup is created per line, in contrast with your implementation where an instance is created for every tag and line pair, which is many more instances of BeautifulSoup, and wasted processing.

One more improvement over that could be using just one instance of BeautifulSoup, if you can join all the lines into a single document, instead of processing line by line:

soup = BeautifulSoup("\n".join(htmlist), "html.parser")

tags = defaultdict(set)
for tag in soup():
    tags[tag.name] |= set(tag.attrs)

for tag_name in sorted(tags):
    print("{name}:{attrs}".format(name=tag_name, attrs=",".join(sorted(tags[tag_name]))))

However this is not possible with the given lines, because this line is invalid HTML and will mess up the document:

 '<li id="coll-download-as-rl"><a href="/w/index.php?title=Special:Book&amp;bookcmd=render_article&amp;arttitle=Main+Page&amp;oldid=560327612',

If possible, I suggest to fix this line, and use the above approach.

 '<li id="coll-download-as-rl"><a href="/w/index.php?title=Special:Book&amp;bookcmd=render_article&amp;arttitle=Main+Page&amp;oldid=560327612"></a></li>',
\$\endgroup\$
  • \$\begingroup\$ Thank you for your insight. As I think you suspect, I actually pasted htmlist incorrectly somehow. I added the correct version to the question. \$\endgroup\$ – Git Gud Aug 27 '17 at 6:16
3
\$\begingroup\$

You can use a defaultdict data structure to map between tag names and tag attributes. Then, you can iterate over the sorted keys and print out tag names and attributes in the sorted order:

tags = defaultdict(set)
for line in htmlist:
    for tag in BeautifulSoup(line, "html.parser")():
        tags[tag.name] |= set(tag.attrs)

for tag_name in sorted(tags):
    print("{name}:{attrs}".format(name=tag_name, attrs=",".join(sorted(tags[tag_name]))))

Don't forget to import the defaultdict:

from collections import defaultdict
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.