I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.

The problem is that any alternatives are really too slow.

I am parsing this web page which is quite big (3MB). Somewhere in the html there is:

<form class="form-inline"  id="exportform" action="/references" class="hidden">
    <input type="hidden" name="format" value="" />

    <input type="hidden" name="item_id" value="678860" />

    <input type="hidden" name="item_id" value="656680" />

    <input type="hidden" name="item_id" value="678846" />

    <input type="hidden" name="item_id" value="650319" />

    <input type="hidden" name="item_id" value="651504" />

    <input type="hidden" name="item_id" value="678849" />

    <input type="hidden" name="item_id" value="649821" />

    <input type="hidden" name="item_id" value="649835" />

    <input type="hidden" name="item_id" value="651512" />

    <input type="hidden" name="item_id" value="651510" />


I need to extract the 10 numbers in the value field. I know they are exactly 10.

I tried different approaches:


from bs4 import BeautifulSoup

def soup(html):
    soup = BeautifulSoup(html, 'lxml')
    form = soup.find("form", {'id': 'exportform'})
    if not form:
        return []
    entries = form.find_all("input", {'name': 'item_id'})
    result = [entry['value'] for entry in entries]
    return result


import re
import itertools

def regex(html):
    r = re.compile(r'<input type="hidden" name="item_id" value="([0-9]+)">')

    s = r.finditer(html)
    result = []
    for ss in itertools.islice(s, 10):
    return result


import lxml.html
from lxml import etree # same performance with this

def xpath(html):
    root = lxml.html.fromstring(html)
    result = root.xpath('//form[@class="form-inline"]/*[@name="item_id"]')
    return [r.attrib['value'] for r in result]

I get the same results from the three, but very different timing:

  • BeautifulSoup: 3.3
  • regex: 0.00013
  • xpath: 0.57

In the regex case I am able to use the trick that I know there are exactly 10 numbers to find, gaining a factor 10. But also without that trick the regex approach is 400 times faster than xpath.

In addition asking xpath to give me the first 10:

(//form[@class="form-inline"]/*[@name="item_id"])[position() < 11]

is not improving


1 Answer 1


Well, your regex solution is obviously badly broken - it will fail if the attributes are in a different order, if they are separated by newlines, if they are delimited by single quotes, etc etc. If you try to replace it with a more correct regex (it will never be 100% correct of course) then you are quite likely to lose some of this speed - perhaps dramatically, if it starts backtracking.

The cost of the XPath solution is not actually in evaluating the XPath, it is (a) parsing the source document and building a tree representation, and (b) compiling the XPath expression. The key to getting a performance improvement is to make sure you are doing these operations only once. I don't know whether that's possible in your overall application scenario.

  • \$\begingroup\$ Just to elaborate on "more correct regex" -- this could mean for example case-insensitive <input, non-greedy match-anything, value, optional whitespace, equals sign, more optional whitespace, character class containing double and single quote in a capture group, digits in a capture group, backreference to the previous quote; as in egrep -i '<INPUT.*?VALUE\s*=\s*(["\'])([0-9]+)\1' ... or you could probably isolate just the form and let xpath parse that instead of the whole page. \$\endgroup\$ Jul 13, 2019 at 2:02
  • \$\begingroup\$ A fully-correct regex should also ignore <input when it appears in a comment or CDATA section, it should allow a digit 5 in the value attribute to be written as &#53;, it should allow attribute values to be defaulted from the DTD, etc etc. In practice you only need to worry about such things if you are defending against malicious users; but it's worth being aware of. \$\endgroup\$ Jul 13, 2019 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.