Sanitize user-supplied HTML with Python and Regular Expressions

Question

I have a product that needs to have users put content in a form that potentially contains HTML and display it back to other users. I'd like to mitigate the risk as much as possible, and I can limit users to a subset of html. My primary goal is to make sure that users can't compromise each other, and that the page itself is not compromised by poorly-formatted html (e.g. omitted closing tags).

My backend is Python. Unfortunately I've been asked to not include use a third party solution.

I'm structuring my approach based on this regular expression:

(?P<content>.*?)                # Content up to next tag
(?P<markup>                     # Entire tag
  <!\[CDATA\[(?P<cdata>.+?)]]>| # <![CDATA[ ... ]]>
  <!--(?P<comment>.+?)-->|      # <!-- Comment -->
  </\s*(?P<close_tag>\w+)\s*>|  # </tag>
  <(?P<tag>\w+)                 # <tag ...
    (?P<attributes>
      (?P<attribute>\s+
# <snip>: Use this part to get the attributes out of 'attributes' group.
        (?P<attribute_name>\w+)
        (?:\s*=\s*
          (?P<attribute_value>
            [\w:/.\-]+|         # Unquoted
            (?=(?P<_v>          # Quoted
              (?P<_q>['\"]).*?(?<!\\)(?P=_q)))
            (?P=_v)
          ))?
# </snip>
      )*
    )\s*
  (?P<is_self_closing>/?)   # Self-closing indicator
  >)                        # End of tag

Here is my full code:

import re

_html_regex = r"""
        (?P<content>.*?)                # Content up to next tag
        (?P<markup>                     # Entire tag
          $|                            # End of document
          <!\[CDATA\[(?P<cdata>.+?)]]>| # <![CDATA[ ... ]]>
          <!--(?P<comment>.+?)-->|      # <!-- Comment -->
          </\s*(?P<close_tag>\w+)\s*>|  # </tag>
          <(?P<tag>\w+)                 # <tag ...
            (?P<attributes>
              (?P<attribute>\s+
        # <snip>: Use this part to get the attributes out of 'attributes' group.
                (?P<attribute_name>[\w\-]+)
                (?:\s*=\s*
                  (?P<attribute_value>
                    [\w:/.\-]+|         # Unquoted
                    (?=(?P<_v>          # Quoted
                      (?P<_q>'|\"|\\\").*?(?<!\\)(?P=_q)))
                    (?P=_v)
                  ))?
        # </snip>
              )*
            )\s*
          (?P<is_self_closing>/?)   # Self-closing indicator
          >)                        # End of tag
        """

_html_attr_regex = r"""
        (?P<attribute_name>\w+)
        (?:\s*=\s*
          (?P<attribute_value>
            [\w:/.\-]+|         # Unquoted
            (?=(?P<_v>          # Quoted
              (?P<_q>['\"]).*?(?<!\\)(?P=_q)))
            (?P=_v)
          ))?
        """

def _check_href(href):
  href = href.strip()
  if href[0] == '"' or href[0] == "'":
    href = href[1:-1].strip()

  if href.startswith('javascript:'):
    return '"#"'

  return '"' + href + '"'

_no_attributes = {}

_allowed_tags = {
  'a': {'href': _check_href},
  'b': _no_attributes,
  'blockquote': _no_attributes,
  'div': _no_attributes,
  'h1': _no_attributes,
  'h2': _no_attributes,
  'h3': _no_attributes,
  'h4': _no_attributes,
  'h5': _no_attributes,
  'h6': _no_attributes,
  'hr': _no_attributes,
  'i': _no_attributes,
  'li': _no_attributes,
  'ol': _no_attributes,
  'p': _no_attributes,
  'pre': _no_attributes,
  'table': _no_attributes,
  'tbody': _no_attributes,
  'td': _no_attributes,
  'th': _no_attributes,
  'tr': _no_attributes,
  'u': _no_attributes,
  'ul': _no_attributes,
}

_self_closing_tags = [
  'hr',
]


def cleanup_html(html):
  if not html:
    return html

  tag_stack = []
  output = ""

  for match in re.finditer(_html_regex, html, re.DOTALL | re.VERBOSE):
    content = match.group('content')
    tag = match.group('tag')
    attributes = match.group('attributes')
    close_tag = match.group('close_tag')
    is_self_closing = match.group('is_self_closing')

    if content:
      output += content

    if close_tag and close_tag in tag_stack:
      while tag_stack[-1] != close_tag:
        output += '</%s>' % tag_stack.pop()

    elif tag and tag in _allowed_tags:
      output += '<' + tag
      for attr in re.finditer(_html_attr_regex, attributes, re.DOTALL | re.VERBOSE):
        attr_name = attr.group('attribute_name')
        attr_value = attr.group('attribute_value')
        if attr_name in _allowed_tags[tag]:
          output += ' ' + attr_name
          if attr_value:
            output += '=' + _allowed_tags[tag][attr_name](attr_value)

      if is_self_closing:
        output += '/>'
      else:
        if tag not in _self_closing_tags:
          tag_stack.append(tag)
        output += '>'

  while tag_stack:
    output += '</%s>' % tag_stack.pop()

  return output

The regex and some poorly formatted sample text are pre-loaded into https://regex101.com/r/vuhnKS/1/ for playing with.

The regex should allow fairly ugly HTML, but I think it does a fairly good job of cleaning it up. There's a whitelist of allowed tags in there, and <style> and <script> aren't allowed. My guess is that the greatest weakness is in _check_href and something slipping through there.

Are there any identifiable flaws in this though? Are there any other tags that are low-risk that I should allow?

Answer 1

It is generally thought that html parsing with regular-expressions is complex and difficult to maintain over time.

Python's standard library includes a basic html parser, html.parser, so it shouldn't be part of the 3rd-party exclusion set. You might try re-implementing your solution with it and then compare the two solutions.

Html.parser information:
https://docs.python.org/3/library/html.parser.html