Cleaning HTMLs to extract text

Question

I have written the following routine to clean an html raw code and extract the text.

It works, but the code is not very well written. Any ideas how to solve this faster and with fewer lines of code?

Thanks a lot for your help!

import re
def cleanhtml(raw_html):
    cleanr = re.compile('<.*?>')
    cleantext1 = re.sub(cleanr, ' ', raw_html)
    cleanr = re.compile('&amp;')
    cleantext2 = re.sub(cleanr, '&', cleantext1 )
    cleanr = re.compile('\n')
    cleantext3 = re.sub(cleanr, ' ', cleantext2)
    cleanr = re.compile('&nbsp;')
    cleantext4 = re.sub(cleanr, ' ', cleantext3)
    cleanr = re.compile('&#8217;')
    cleantext5 = re.sub(cleanr, '\'', cleantext4)
    cleanr = re.compile('&#8230;')
    cleantext6 = re.sub(cleanr, '...', cleantext5)
    cleanr = re.compile('&#8216;')
    cleantext7 = re.sub(cleanr, '\'', cleantext6)
    cleanr = re.compile('&#8221;')
    cleantext8 = re.sub(cleanr, '\'', cleantext7)
    cleanr = re.compile('&#8220;')
    cleantext9 = re.sub(cleanr, '\'', cleantext8)
    cleanr = re.compile('&#8211;')
    cleantext10 = re.sub(cleanr, '\'', cleantext9)
    cleanr = re.compile('&#8212;')
    cleantext11 = re.sub(cleanr, '\'', cleantext10)
    cleanr = re.compile('&#38;')
    cleantext12 = re.sub(cleanr, '&', cleantext11)
    cleanr = re.compile('&#2026;')
    cleantext13 = re.sub(cleanr, '...', cleantext12)
    cleanr = re.compile('&#160;')
    cleantext14 = re.sub(cleanr, ' ', cleantext13)
    cleanr = re.compile('&#038;')
    cleantext15 = re.sub(cleanr, '&', cleantext14)
    return cleantext15.strip()

Answer 1

Since you have basically the same statements over and over with different parameters, this can be turned into a loop. You can pair the pattern and the string you want to use to replace in a dict. This makes it clear what is paired with what. Take a look:

import re

def strip_html(raw_html: str) -> str:
    pairs = {
        '<.*?>': ' ',
        '&amp;': '&',
        '\n': ' ',
        '&nbsp;': ' ',
        '&#8217;': '\'',
        '&#8230;': '...',
        '&#8216;': '\'',
        '&#8221;': '\'',
        '&#8220;': '\'',
        '&#8211;': '\'',
        '&#8212;': '\'',
        '&#38;': '&',
        '&#2026;': '...',
        '&#160;': ' ',
        '&#038;': '&'
    }
    previous = raw_html
    for pattern in pairs:
        clean_text = re.sub(re.compile(pattern), pairs[pattern], previous)
        previous = clean_text
    return clean_text.strip()

Answer 2

The problem with this approach is that it won't scale well and if you want to extract text from diverse sources it will be impossible to handle all possible cases.

Regular expressions can be expensive, and if you have lots of text to process I would worry about performance. For many operations like changing '&' to '&', a regex is not even necessary, all you need is a simple search & replace.

Attempting to parse HTML or XML using regular expressions can easily turn into a coding nightmare so don't overdo it.

I suggest that you use a dedicated library instead, like BeautifulSoup

As a starting point:

from bs4 import BeautifulSoup

html = "<p>some HTML here</p>"
soup = BeautifulSoup(html, 'html.parser')
print(soup.get_text("\n"))

From the documentation: get_text (emphasis is mine):

If you only want the text part of a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string

Note the addition of "\n", that gets you multiline text where <p> or <br> tags are found. The function can also strip whitespace. There is a lot more to be said, so I will leave it to you to read the documentation and fine-tune the code according to your needs. This snippet is quite basic but may be sufficient for your purpose.

As you can see a lot can be done with just a few lines of code.