LXML parsing for an HTML file

Question

I wrote this code to parse an HTML file which is basically a report generated by a software. This file is 40+ MB in size. The code is 100% perfect. The only problem that I am facing is the parsing process is extremely slow. It took nearly 5 minutes just parsing the first report. The script has a "break" statement in it and so it only works on 1st report but it is very slow.

import os
from lxml import html


def main():
    with open(os.getcwd()+'/dump.html', "r") as f:
        page = f.read()
    tree = html.fromstring(page)

    reportCount = len(tree.xpath('//div[contains(@class,"onereport")]'))

    sheet1, sheet2, sheet3=[], [], []
    for i in range(reportCount):
        print "Getting in "+str(i+1)+" report..."

        device     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Device")]/following-sibling::td/text()')[0].split("> ")
        deviceID   = device[len(device)-1]

        timespan   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Time Span")]/following-sibling::td/text()')[0]
        uptime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/text()')[0]
        uptimeDU   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/following-sibling::td/span/text()')[0]
        uptimeDo   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Down:"]/following-sibling::td/text()')[0]
        uptimeDoDU = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[1]//tr/td[contains(text(), "Uptime Stats:")]/following-sibling::td[text()="Down:"]/following-sibling::td/following-sibling::td/span/text()')[0]
        pingTime   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Ping Time"]/following-sibling::td/text()')[0]
        minimim    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Minimum"]/following-sibling::td/text()')[0]
        maximum    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Maximum"]/following-sibling::td/text()')[0]
        packetLoss = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/table[2]//tr/td[text()="Packet Loss"]/following-sibling::td/text()')[0]

        sheet1.append([deviceID, timespan, uptime, uptimeDU.replace("[", "").replace("]", ""), uptimeDo, uptimeDoDU.replace("[", "").replace("]", ""), pingTime, minimim, maximum, packetLoss])

        print "Getting in "+str(i+1)+" report's table1..."
        for j in range(len(tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr'))):
            dateTime  = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/nobr/text()')[0]
            pTime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[0]
            mini      = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[1]
            maxi      = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[2]
            pLoss     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[3]
            dTime     = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[4]
            coverage = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[1]/form/table/thead[@class="headerswithlinks"]/following-sibling::tbody/tr['+str(j+1)+']/td/text()')[5]

            sheet2.append([deviceID, dateTime, pTime, mini, maxi, pLoss, dTime, coverage])

        print "Getting in "+str(i+1)+" report's table2..."
        for k in range(len(tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr'))):
            status   = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[1]/text()')[0]
            dtime    = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[2]/nobr/text()')[0]
            duration = tree.xpath('//div[@class="onereport report'+str(i+1)+'"]/div[2]/form/table/tbody/tr['+str(k+1)+']/td[2]/nobr/span[2]/text()')[0]

            sheet3.append([deviceID, status, dtime, duration.replace("=", "").replace("(", "").replace(")", "")])
        break

if __name__ == '__main__':
    main()

Answer 1

Speeding up HTML parsing

One of the biggest problem the code has is that every single .xpath() call traverses the complete HTML tree from the very root node. This is very slow and you don't actually need to do that for every element.

Make your searches context-specific - loop over the "report" elements and look everything needed for this report directly inside the element - not from the root tree. Then, do similar things for the overview tables.

This is what lxml docs are also noting about:

A lot of time is usually spent in tree traversal to find the addressed elements in the tree. If you often work in subtrees, do what you would also do with deep Python objects: assign the parent of the subtree to a variable or pass it into functions instead of starting at the root. This allows accessing its descendants more directly.

Here is how I would go about the first part of parsing the report:

tree = html.parse('dump.html')

for index, report in enumerate(tree.xpath("//div[contains(@class, 'onereport')]"), start=1):
    print("Getting in {index} report...".format(index=index))

    overview_table1, overview_table2 = report.xpath(".//table[@class = 'overview']")[:2]

    device_id = overview_table1.xpath('.//td[contains(., "Device")]/following-sibling::td/text()')[0].split("> ")[-1]

    timespan = overview_table1.xpath('.//td[contains(., "Time Span")]/following-sibling::td/text()')[0]
    uptime = overview_table1.xpath('.//td[contains(., "Uptime Stats:")]/following-sibling::td[text()="Up:"]/following-sibling::td/text()')[0]

    uptime_stats = overview_table1.xpath('.//td[contains(., "Uptime Stats:")]')[0]
    uptime_du= uptime_stats.xpath('.//following-sibling::td[. = "Up:"]/following-sibling::td/following-sibling::td/span/text()')[0]
    uptime_do = uptime_stats.xpath('.//following-sibling::td[. = "Down:"]/following-sibling::td/text()')[0]

    uptime_do_du = uptime_stats.xpath('.//following-sibling::td[. = "Down:"]/following-sibling::td/following-sibling::td/span/text()')[0]

    ping_time = overview_table2.xpath('.//td[. = "Ping Time"]/following-sibling::td/text()')[0]
    minimim = overview_table2.xpath('.//td[. = "Minimum"]/following-sibling::td/text()')[0]
    maximum = overview_table2.xpath('.//td[. = "Maximum"]/following-sibling::td/text()')[0]
    packet_loss = overview_table2.xpath('.//td[. = "Packet Loss"]/following-sibling::td/text()')[0]

    print(device_id, timespan, uptime, uptime_do, uptime_du, uptime_do_du, ping_time, minimim, maximum, packet_loss)
    break

Apply the same improvement to other code blocks as well.

Other Thoughts

I also don't really like these "by text" searches. If the table structures are consistent, see if you can improve on searching the nodes based on the previous sibling texts - in other words, stop looking for a td by text and using following-sibling. Instead, get, say, all tr elements knowing which field is at which position. E.g. for the second overview table:

ping_time, minimim, maximum, packet_loss = overview_table2.xpath('.//tr/td[2]/text()')

Should also have a positive impact on performance.

Also, what if you would move from XPath locators all together - look into using cssselect or BeautifulSoup (which would allow using lxml as an underlying parser).

Try iterative parsing with iterparse().

And, try the same code on Python3.6, if possible - it is generally a better, faster and more memory-efficient Python language implementation. You may get performance boosts "for free" just by upgrading your Python.

And, of course, profile and know your bottlenecks!