Beautifulsoup and lxml (xpath) too slow with respect to regex when parsing HTML

Question

I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.

The problem is that any alternatives are really too slow.

I am parsing this web page which is quite big (3MB). Somewhere in the html there is:

<form class="form-inline"  id="exportform" action="/references" class="hidden">
    <input type="hidden" name="format" value="" />

    <input type="hidden" name="item_id" value="678860" />

    <input type="hidden" name="item_id" value="656680" />

    <input type="hidden" name="item_id" value="678846" />

    <input type="hidden" name="item_id" value="650319" />

    <input type="hidden" name="item_id" value="651504" />

    <input type="hidden" name="item_id" value="678849" />

    <input type="hidden" name="item_id" value="649821" />

    <input type="hidden" name="item_id" value="649835" />

    <input type="hidden" name="item_id" value="651512" />

    <input type="hidden" name="item_id" value="651510" />

</form>

I need to extract the 10 numbers in the value field. I know they are exactly 10.

I tried different approaches:

BeautifulSoup

from bs4 import BeautifulSoup

def soup(html):
    soup = BeautifulSoup(html, 'lxml')
    form = soup.find("form", {'id': 'exportform'})
    if not form:
        return []
    entries = form.find_all("input", {'name': 'item_id'})
    result = [entry['value'] for entry in entries]
    return result

Regex

import re
import itertools

def regex(html):
    r = re.compile(r'<input type="hidden" name="item_id" value="([0-9]+)">')

    s = r.finditer(html)
    result = []
    for ss in itertools.islice(s, 10):
        result.append(ss.group(1))
    return result

xpath

import lxml.html
from lxml import etree # same performance with this

def xpath(html):
    root = lxml.html.fromstring(html)
    result = root.xpath('//form[@class="form-inline"]/*[@name="item_id"]')
    return [r.attrib['value'] for r in result]

I get the same results from the three, but very different timing:

• BeautifulSoup: 3.3
• regex: 0.00013
• xpath: 0.57

In the regex case I am able to use the trick that I know there are exactly 10 numbers to find, gaining a factor 10. But also without that trick the regex approach is 400 times faster than xpath.

In addition asking xpath to give me the first 10:

(//form[@class="form-inline"]/*[@name="item_id"])[position() < 11]

is not improving

Answer 1

Well, your regex solution is obviously badly broken - it will fail if the attributes are in a different order, if they are separated by newlines, if they are delimited by single quotes, etc etc. If you try to replace it with a more correct regex (it will never be 100% correct of course) then you are quite likely to lose some of this speed - perhaps dramatically, if it starts backtracking.

The cost of the XPath solution is not actually in evaluating the XPath, it is (a) parsing the source document and building a tree representation, and (b) compiling the XPath expression. The key to getting a performance improvement is to make sure you are doing these operations only once. I don't know whether that's possible in your overall application scenario.