3
\$\begingroup\$

I am trying to convert a BeautifulSoup4 HTML Table to a list of lists, iterating over each Tag elements and handling them accordingly.

I have an implementation of this that works at a surface level using BeautifulSoup4. However, the code is getting repetitive and complicated needlessly, but every time I try to improve it, I just end up breaking the functionality. I need some guidance on tidying this up.

Ultimately, I separate each type of HTML tags for any given row cell. The goal is to re-format the contents of the tables to an Excel spreadsheet and do partial cell formatting (still a work in progress, using xlwt).

Note I've left out as much as possible of the parsing, but just enough to give an idea.

from bs4 import BeautifulSoup
from bs4.element import Tag, NavigableString


def handle_bs4_element(element):
    if isinstance(element, Tag):
        if len(element.contents) > 1:
            # Handle each element separately and return a list? What if more elements are nested? Recursive call?
            _res = []
            for e_content in element.contents:
                _res.append(handle_bs4_element(e_content))
            if len(_res) == 1:
                return _res[0]
            else:
                return _res
        else:
            tag_name = element.name
            if tag_name == 'td':
                _res = []
                for td_content in element.contents:
                    _res.append(handle_bs4_element(td_content))
                if len(_res) == 1:
                    return _res[0]
                else:
                    return _res
            elif tag_name in ('div', 'span'):
                # This will probably contain more nested tags...
                _res = []
                for td_content in element.contents:
                    _res.append(handle_bs4_element(td_content))
                if len(_res) == 1:
                    return _res[0]
                else:
                    return _res
            elif tag_name in ('p', 'strong', 'em', 'h3'):  # Would handle each case separately, but just for the example
                return element.text
            elif tag_name == 'a':
                e_text = element.text
                e_link = element['href']
                if e_text != e_link:
                    return '{text} ({url})'.format(text=e_text, url=e_link)
                else:
                    return e_link
            else:
                print('Element HTML type not handled: {0}'.format(tag_name))
    elif isinstance(element, NavigableString):
        return element
    else:
        raise Exception('bs4 element of type {0} not handled...'.format(type(element)))


bs_table = BeautifulSoup(open('table_sample.html'), "html.parser")
headers = [h.text for h in bs_table.find_all('th')]
data = [headers]

rows = bs_table.find_all('tr')
for row in rows:
    row_cells = row.find_all('td')
    if row_cells:
        # Handle each row cell appropriately
        data.append([handle_bs4_element(rc) for rc in row_cells if handle_bs4_element(rc)])

print('\n'.join(map(str, data)))

table_sample.html:

        <table class="confluenceTable">
        <tbody>
        <tr>
            <th class="numberingColumn confluenceTh">&nbsp;</th>
            <th class="confluenceTh"><p><strong>Description</strong></p></th>
            <th colspan="1" class="confluenceTh"><p>Col 1</p></th>
            <th colspan="1" class="confluenceTh">Col 2</th>
            <th colspan="1" class="confluenceTh">Col 3</th>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">1</td>
            <td class="confluenceTd"><p>Some paragraph text</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">5</td>
            <td colspan="1" class="confluenceTd">2</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">2</td>
            <td colspan="4" class="confluenceTd"><h3 id="some-id1"><strong>HEADER 1</strong></h3></td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">3</td>
            <td colspan="1" class="confluenceTd"><div><p>Some text: </p><p>(1) Check out this <strong
                    style="line-height: 1.42857;">Figure 1.0.</strong></p></div></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">1</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">4</td>
            <td colspan="1" class="confluenceTd"><p>(2)&nbsp;&nbsp;&nbsp;Some more text</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">1</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">5</td>
            <td colspan="1" class="confluenceTd"><p>(3)&nbsp;&nbsp;&nbsp; Additional text</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">1</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">6</td>
            <td colspan="1" class="confluenceTd"><p>(4)&nbsp;&nbsp;&nbsp; A bit more text</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">1</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">7</td>
            <td colspan="1" class="confluenceTd"><span>(5)&nbsp;&nbsp;&nbsp; <span>A span <strong>Figure 1.0</strong> for
                edited text. At this point the </span>span starts again</span></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">1</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">8</td>
            <td colspan="4" class="confluenceTd"><h3 id="some-id2"><strong>HEADER 2</strong></h3></td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">9</td>
            <td colspan="1" class="confluenceTd"><p>Weird formatting, because Confluence</p><p>&nbsp;</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">4</td>
            <td colspan="1" class="confluenceTd">2</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">10</td>
            <td colspan="4" class="confluenceTd"><h3 id="some-id3"><strong>HEADER 3</strong></h3></td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">11</td>
            <td colspan="1" class="confluenceTd"><p>A paragraph about header 3.</p>
                <div class="confluence-information-macro confluence-information-macro-information">
                    <span class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
                    <div class="confluence-information-macro-body">This is just silly. <strong>Strong</strong> indeed.</div>
                </div>
            </td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">3</td>
            <td colspan="1" class="confluenceTd">3</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">12</td>
            <td colspan="1" class="confluenceTd"><span>Something about things or what not. Why is this in a span?</span></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">2</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">13</td>
            <td colspan="4" class="confluenceTd"><h3 id="some-id4">HEADER 4</h3></td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">14</td>
            <td colspan="1" class="confluenceTd"><p>Section 4 baby! Or header.</p>
                <div class="confluence-information-macro confluence-information-macro-information">
                    <span class="aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon"></span>
                    <div class="confluence-information-macro-body">Confluence formatting fun.</div>
                </div>
            </td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">3</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">15</td>
            <td colspan="1" class="confluenceTd"><span>Pretty boring span of text</span></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">2</td>
            <td colspan="1" class="confluenceTd">2</td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">16</td>
            <td colspan="4" class="confluenceTd"><h3 id="some-id5"><strong>HEADER 5</strong></h3></td>
        </tr>
        <tr>
            <td class="numberingColumn confluenceTd">17</td>
            <td colspan="1" class="confluenceTd"><p>A big paragraph describing more stuff. Super exciting.</p></td>
            <td colspan="1" class="confluenceTd">x</td>
            <td colspan="1" class="confluenceTd">4</td>
            <td colspan="1" class="confluenceTd">2</td>
        </tr>
        </tbody>
    </table>

Current output:

[u'\xa0', u'Description', u'Col 1', u'Col 2', u'Col 3']
[u'1', u'Some paragraph text', u'x', u'5', u'2']
[u'2', u'HEADER 1']
[u'3', [u'Some text: ', [u'(1) Check out this ', u'Figure 1.0.']], u'x', u'2', u'1']
[u'4', u'(2)\xa0\xa0\xa0Some more text', u'x', u'2', u'1']
[u'5', u'(3)\xa0\xa0\xa0 Additional text', u'x', u'2', u'1']
[u'6', u'(4)\xa0\xa0\xa0 A bit more text', u'x', u'2', u'1']
[u'7', [u'(5)\xa0\xa0\xa0 ', [u'A span ', u'Figure 1.0', u' for\n            edited text. At this point the '], u'span starts again'], u'x', u'2', u'1']
[u'8', u'HEADER 2']
[u'9', [u'Weird formatting, because Confluence', u'\xa0'], u'x', u'4', u'2']
[u'10', u'HEADER 3']
[u'11', [u'A paragraph about header 3.', u'\n', [u'\n', [], u'\n', [u'This is just silly. ', u'Strong', u' indeed.'], u'\n'], u'\n'], u'x', u'3', u'3']
[u'12', u'Something about things or what not. Why is this in a span?', u'x', u'2', u'2']
[u'13', u'HEADER 4']
[u'14', [u'Section 4 baby! Or header.', u'\n', [u'\n', [], u'\n', u'Confluence formatting fun.', u'\n'], u'\n'], u'x', u'2', u'3']
[u'15', u'Pretty boring span of text', u'x', u'2', u'2']
[u'16', u'HEADER 5']
[u'17', u'A big paragraph describing more stuff. Super exciting.', u'x', u'4', u'2']
\$\endgroup\$
2
  • 2
    \$\begingroup\$ As you have already encountered with the answer, leaving out parts of the code for "clearity" usually backfires on Code Review, because if we don't know the while story any recommendations might not actually make sense in your specific case. Always try to include the whole code. \$\endgroup\$
    – Graipher
    Commented Feb 7, 2017 at 6:36
  • 1
    \$\begingroup\$ @Graipher I was trying to stick to the MCVE principle applied in general StackOverflow questions (stackoverflow.com/help/mcve), but yea it did somewhat backfire. I am loving Code Review though! \$\endgroup\$
    – HEADLESS_0NE
    Commented Feb 7, 2017 at 14:00

1 Answer 1

Reset to default
4
\$\begingroup\$

Here is the list of things I would think about to improve:

  • you are doubling on calls to handle_bs4_element() here:

    data.append([handle_bs4_element(rc) for rc in row_cells if handle_bs4_element(rc)])

    Instead, you can either allow "falsy" values for the row cells and filter them afterwards, or expand the loop:

    result = []
for rc in row_cells:
    cell_text = handle_bs4_element(rc)
    if cell_text:
        result.append(cell_text)
data.append(result)

  • the DRY principle. There are several repeated blocks of code, like:

    if len(_res) == 1:
    return _res[0]
else:
    return _res

  • using list comprehensions is not only more Pythonic, but actually faster. E.g. you can replace:

    _res = []
for td_content in element.contents:
    _res.append(handle_bs4_element(td_content))

    with:

    _res = [handle_bs4_element(td_content) for td_content in element.contents]

  • you can use the short if/else one-liner, replacing:

    if len(_res) == 1:
    return _res[0]
else:
    return _res

    with:

    return _res[0] if len(_res) == 1 else _res

  • variable naming. _res should not be started with an underscore. You are confusing private class or instance attributes with regular variables. _res should probably be called result, or may be cell_data?

  • if you will have more of this kind of tag-specific processing logic, continuing to put it as an another elif would hurt readability and does not scale well. Consider using the "Extract Method" refactoring method and defining a separate functions for each of the cases.

  • instead of using the .contents list directly, look into using .get_text(), which completes an element's text including the children texts recursively. Not sure if applicable for your problem.

  • or, instead of .contents list, you can use the .children generator

As a side note, there is also a simpler way to parse HTML tables - pandas.read_html() which would load an HTML table into a DataFrame, you can then easily dump the dataframe into a list or into CSV, or into an Excel file directly. For example, the following code:

from pprint import pprint

import pandas as pd


df = pd.read_html('table_sample.html')[0]  # get the first parsed dataframe
pprint(df.values.tolist())

Would automagically produce:

[[nan, 'Description', 'Col 1', 'Col 2', 'Col 3'],
 [1.0, 'Some paragraph text', 'x', '5', '2'],
 [2.0, 'HEADER 1', nan, nan, nan],
 [3.0, 'Some text: (1) Check out this Figure 1.0.', 'x', '2', '1'],
 [4.0, '(2) Some more text', 'x', '2', '1'],
 [5.0, '(3) Additional text', 'x', '2', '1'],
 [6.0, '(4) A bit more text', 'x', '2', '1'],
 [7.0, '(5) A span Figure 1.0 for  edited text. At this point the span starts again', 'x', '2', '1'],
 [8.0, 'HEADER 2', nan, nan, nan],
 [9.0, 'Weird formatting, because Confluence', 'x', '4', '2'],
 [10.0, 'HEADER 3', nan, nan, nan],
 [11.0, 'A paragraph about header 3.  This is just silly. Strong indeed.', 'x', '3', '3'],
 [12.0, 'Something about things or what not. Why is this in a span?', 'x', '2', '2'],
 [13.0, 'HEADER 4', nan, nan, nan],
 [14.0, 'Section 4 baby! Or header.  Confluence formatting fun.', 'x', '2', '3'],
 [15.0, 'Pretty boring span of text', 'x', '2', '2'],
 [16.0, 'HEADER 5', nan, nan, nan],
 [17.0, 'A big paragraph describing more stuff. Super exciting.', 'x', '4', '2']]
\$\endgroup\$
6
  • 1
    \$\begingroup\$ The thing is that I need to format HTML tags. I know I can get just the text. I'm trying to handle each tag as nested they may be and do some specific stuff with it. For instance, I ignore certain span blocks because they correspond to labels I don't want to scrape. Similarly, I extract links href if they have the class external link. I need to do a lot of that case by case. \$\endgroup\$
    – HEADLESS_0NE
    Commented Feb 7, 2017 at 3:45
  • 2
    \$\begingroup\$ @HEADLESS_0NE okay, gotcha, I'll document my observations over your current code then, hope this'll help. Thanks for the feedback. \$\endgroup\$
    – alecxe
    Commented Feb 7, 2017 at 3:49
  • \$\begingroup\$ My main problem is code repetition and figuring out a way to recurse over nested Tag elements. \$\endgroup\$
    – HEADLESS_0NE
    Commented Feb 7, 2017 at 3:53
  • 1
    \$\begingroup\$ @HEADLESS_0NE okay, added some notes, check them out. Hope it helps.. \$\endgroup\$
    – alecxe
    Commented Feb 7, 2017 at 4:18
  • 1
    \$\begingroup\$ @HEADLESS_0NE yeah, 2 things here: extract the element specific processing logic into separate functions and, think about using a dictionary - map element names to functions - example: stackoverflow.com/a/17166144/771848. Should makes things more transparent and flat. \$\endgroup\$
    – alecxe
    Commented Feb 7, 2017 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.