I wrote this code to parse an HTML file which is basically a report generated by a software. This file is 40+ MB in size. The code is 100% perfect. The only problem that I am facing is the parsing process is extremely slow. It took nearly 5 minutes just parsing the first report. The script has a "break" statement in it and so it only works on 1st report but it is very slow.

import os
from lxml import html

def main():
    with open(os.getcwd()+'/dump.html', "r") as f:
        page = f.read()
    tree = html.fromstring(page)

    reportCount = len(tree.xpath('//div[contains(@class,"onereport")]'))

    sheet1, sheet2, sheet3=[], [], []
    for i in range(reportCount):
        print "Getting in "+str(i+1)+" report..."

        device     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Device")]/following-sibling::td/text()')[0].split("> ")
        deviceID   = device[len(device)-1]

        timespan   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Time Span")]/following-sibling::td/text()')[0]
        uptime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/text()')[0]
        uptimeDU   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/following-sibling::td/span/text()')[0]
        uptimeDo   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Down:"]/following-sibling::td/text()')[0]
        uptimeDoDU = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Down:"]/following-sibling::td/following-sibling::td/span/text()')[0]
        pingTime   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Ping Time"]/following-sibling::td/text()')[0]
        minimim    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Minimum"]/following-sibling::td/text()')[0]
        maximum    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Maximum"]/following-sibling::td/text()')[0]
        packetLoss = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Packet Loss"]/following-sibling::td/text()')[0]

        sheet1.append([deviceID, timespan, uptime, uptimeDU.replace("[", "").replace("]", ""), uptimeDo, uptimeDoDU.replace("[", "").replace("]", ""), pingTime, minimim, maximum, packetLoss])

        print "Getting in "+str(i+1)+" report's table1..."
        for j in range(len(tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr'))):
            dateTime  = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/nobr/text()')[0]
            pTime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[0]
            mini      = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[1]
            maxi      = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[2]
            pLoss     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[3]
            dTime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[4]
            coverage = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[5]

            sheet2.append([deviceID, dateTime, pTime, mini, maxi, pLoss, dTime, coverage])

        print "Getting in "+str(i+1)+" report's table2..."
        for k in range(len(tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr'))):
            status   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[1]/text()')[0]
            dtime    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[2]/nobr/text()')[0]
            duration = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[2]/nobr/span[2]/text()')[0]

            sheet3.append([deviceID, status, dtime, duration.replace("=", "").replace("(", "").replace(")", "")])

if __name__ == '__main__':
  • \$\begingroup\$ It takes 3 minutes on my sever with 1 core CPU and 1 GB ram, and on my own PC with 4 core CPU and 8 GB ram it takes 1.5 minutes. But the problem is if it will take 1+ minute for 1 report then it means that is at least 680 minutes for all the reports in html link i mentioned and that is like 11+ hours for just a 40mb html file. \$\endgroup\$
    – R0SENAM
    Commented Jun 11, 2017 at 15:58

1 Answer 1


Speeding up HTML parsing

One of the biggest problem the code has is that every single .xpath() call traverses the complete HTML tree from the very root node. This is very slow and you don't actually need to do that for every element.

Make your searches context-specific - loop over the "report" elements and look everything needed for this report directly inside the element - not from the root tree. Then, do similar things for the overview tables.

This is what lxml docs are also noting about:

A lot of time is usually spent in tree traversal to find the addressed elements in the tree. If you often work in subtrees, do what you would also do with deep Python objects: assign the parent of the subtree to a variable or pass it into functions instead of starting at the root. This allows accessing its descendants more directly.

Here is how I would go about the first part of parsing the report:

tree = html.parse('dump.html')

for index, report in enumerate(tree.xpath("//div[contains(@class, 'onereport')]"), start=1):
    print("Getting in {index} report...".format(index=index))

    overview_table1, overview_table2 = report.xpath(".//table[@class = 'overview']")[:2]

    device_id = overview_table1.xpath('.//td[contains(., "Device")]/following-sibling::td/text()')[0].split("> ")[-1]

    timespan = overview_table1.xpath('.//td[contains(., "Time Span")]/following-sibling::td/text()')[0]
    uptime = overview_table1.xpath('.//td[contains(., "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/text()')[0]

    uptime_stats = overview_table1.xpath('.//td[contains(., "Uptime Stats:")]')[0]
    uptime_du= uptime_stats.xpath('.//following-sibling::td[. = "Up:"]/following-sibling::td/following-sibling::td/span/text()')[0]
    uptime_do = uptime_stats.xpath('.//following-sibling::td[. = "Down:"]/following-sibling::td/text()')[0]

    uptime_do_du = uptime_stats.xpath('.//following-sibling::td[. = "Down:"]/following-sibling::td/following-sibling::td/span/text()')[0]

    ping_time = overview_table2.xpath('.//td[. = "Ping Time"]/following-sibling::td/text()')[0]
    minimim = overview_table2.xpath('.//td[. = "Minimum"]/following-sibling::td/text()')[0]
    maximum = overview_table2.xpath('.//td[. = "Maximum"]/following-sibling::td/text()')[0]
    packet_loss = overview_table2.xpath('.//td[. = "Packet Loss"]/following-sibling::td/text()')[0]

    print(device_id, timespan, uptime, uptime_do, uptime_du, uptime_do_du, ping_time, minimim, maximum, packet_loss)

Apply the same improvement to other code blocks as well.

Other Thoughts

I also don't really like these "by text" searches. If the table structures are consistent, see if you can improve on searching the nodes based on the previous sibling texts - in other words, stop looking for a td by text and using following-sibling. Instead, get, say, all tr elements knowing which field is at which position. E.g. for the second overview table:

ping_time, minimim, maximum, packet_loss = overview_table2.xpath('.//tr/td[2]/text()')

Should also have a positive impact on performance.

Also, what if you would move from XPath locators all together - look into using cssselect or BeautifulSoup (which would allow using lxml as an underlying parser).

Try iterative parsing with iterparse().

And, try the same code on Python3.6, if possible - it is generally a better, faster and more memory-efficient Python language implementation. You may get performance boosts "for free" just by upgrading your Python.

And, of course, profile and know your bottlenecks!

  • \$\begingroup\$ The modified version now takes only few minutes to parse everything, all thanks to you. \$\endgroup\$
    – R0SENAM
    Commented Jun 12, 2017 at 6:52
  • \$\begingroup\$ However you mentioned the use of bs4, is it more accurate because i have been working with lxml for a year now and the reason i chose it then was because it was fast and easier. Your thoughts? \$\endgroup\$
    – R0SENAM
    Commented Jun 12, 2017 at 7:30
  • \$\begingroup\$ lxml is faster than bs4 indeed. It takes advantage of xpath instead of using regex based searches. \$\endgroup\$ Commented Jun 12, 2017 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.