I agree that using regex to parse HTML is not a good way, in particular I am worried about their fragility with respect to change in the HTML.
The problem is that any alternatives are really too slow.
I am parsing this web page which is quite big (3MB). Somewhere in the html there is:
<form class="form-inline" id="exportform" action="/references" class="hidden">
<input type="hidden" name="format" value="" />
<input type="hidden" name="item_id" value="678860" />
<input type="hidden" name="item_id" value="656680" />
<input type="hidden" name="item_id" value="678846" />
<input type="hidden" name="item_id" value="650319" />
<input type="hidden" name="item_id" value="651504" />
<input type="hidden" name="item_id" value="678849" />
<input type="hidden" name="item_id" value="649821" />
<input type="hidden" name="item_id" value="649835" />
<input type="hidden" name="item_id" value="651512" />
<input type="hidden" name="item_id" value="651510" />
</form>
I need to extract the 10 numbers in the value
field. I know they are exactly 10.
I tried different approaches:
BeautifulSoup
from bs4 import BeautifulSoup
def soup(html):
soup = BeautifulSoup(html, 'lxml')
form = soup.find("form", {'id': 'exportform'})
if not form:
return []
entries = form.find_all("input", {'name': 'item_id'})
result = [entry['value'] for entry in entries]
return result
Regex
import re
import itertools
def regex(html):
r = re.compile(r'<input type="hidden" name="item_id" value="([0-9]+)">')
s = r.finditer(html)
result = []
for ss in itertools.islice(s, 10):
result.append(ss.group(1))
return result
xpath
import lxml.html
from lxml import etree # same performance with this
def xpath(html):
root = lxml.html.fromstring(html)
result = root.xpath('//form[@class="form-inline"]/*[@name="item_id"]')
return [r.attrib['value'] for r in result]
I get the same results from the three, but very different timing:
- BeautifulSoup: 3.3
- regex: 0.00013
- xpath: 0.57
In the regex case I am able to use the trick that I know there are exactly 10 numbers to find, gaining a factor 10. But also without that trick the regex approach is 400 times faster than xpath.
In addition asking xpath to give me the first 10:
(//form[@class="form-inline"]/*[@name="item_id"])[position() < 11]
is not improving