I have some zip files somewhere in the order of 2GB+ containing only html files. Each zip contains about 170,000 html files each.
My code reads the file without extracting them,
Passes the resultant html string into a custom HTMLParser object,
And then writes a summary of all the zip files into a CSV (for that particular zipfile).
Despite my code working, it takes longer than a few minutes to completely parse all the files. In order to save the files to a .csv, I've appended the parsed file contents to a list, and then went on to write rows for every entry in the list. I suspect this is what is drawing back performance.
I've also implemented some light multithreading, a new thread is spawned for each zip file encountered. However the magnitude of the files makes me wonder whether I should have implemented a Process for each file instead that spawned thread batches to parse the html files(i.e parse 4 files at a time).
My fairly naive attempts at timing the operation revealed the following results when processing 2 zip files at a time:
Accounts_Monthly_Data-June2017 has reached file 1500/188495
In: 0.6609588377177715 minutes
Accounts_Monthly_Data-July2017 has reached file 1500/176660
In: 0.7187837697565556 minutes
Which implies 12 seconds per 500 files, which is approximately 41 files per second; which is certainly much too slow.
You can find some example zip files at http://download.companieshouse.gov.uk/en_monthlyaccountsdata.html and an example CSV (for a single html file, the real csv would contain rows for every file) follows:
Company Number,Company Name,Cash at bank and in hand (current year),Cash at bank and in hand (previous year),Net current assets (current year),Net current assets (previous year),Total Assets Less Current Liabilities (current year),Total Assets Less Current Liabilities (previous year),Called up Share Capital (current year),Called up Share Capital (previous year),Profit and Loss Account (current year),Profit and Loss Account (previous year),Shareholder Funds (current year),Shareholder Funds (previous year)
07731243,INSPIRATIONAL TRAINING SOLUTIONS LIMITED,2,"3,228","65,257","49,687","65,257","49,687",1,1,"65,258","49,688","65,257","49,687"
I fairly new to implementing intermediate, highly-performant code in python so I can't see how I could further optimize what I've written, any suggestions are helpful.
I've provided a test zip of approximately 875 files: https://www.dropbox.com/s/xw3klspg1cipqzx/test.zip?dl=0
from html.parser import HTMLParser as HTMLParser
from multiprocessing.dummy import Pool as ThreadPool
import time
import codecs
import zipfile
import os
import csv
class MyHTMLParser(HTMLParser):
def __init__(self):
self.fileData = {} # all the data extracted from this file
self.extractable = False # flag to begin handler_data
self.dataTitle = None # column title to be put into the dictionary
self.yearCount = 0
HTMLParser.__init__(self)
def handle_starttag(self, tag, attrs):
yearCount = 0 # years are stored sequentially
for attrib in attrs:
if 'name' in attrib[0]:
if 'UKCompaniesHouseRegisteredNumber' in attrib[1]:
self.dataTitle = 'Company Number'
# all the parsed files in the directory
self.extractable = True
elif 'EntityCurrentLegalOrRegisteredName' in attrib[1]:
self.dataTitle = 'Company Name'
self.extractable = True
elif 'CashBankInHand' in attrib[1]:
self.handle_timeSeries_data('Cash at bank and in hand')
elif 'NetCurrentAssetsLiabilities' in attrib[1]:
self.handle_timeSeries_data('Net current assets')
elif 'ShareholderFunds' in attrib[1]:
self.handle_timeSeries_data('Shareholder Funds')
elif 'ProfitLossAccountReserve' in attrib[1]:
self.handle_timeSeries_data('Profit and Loss Account')
elif 'CalledUpShareCapital' in attrib[1]:
self.handle_timeSeries_data('Called up Share Capital')
elif 'TotalAssetsLessCurrentLiabilities' in attrib[1]:
self.handle_timeSeries_data('Total Assets Less Current Liabilities')
def handle_endtag(self, tag):
None
def handle_data(self, data):
if self.extractable == True:
self.fileData[self.dataTitle] = data
self.extractable = False
def handle_timeSeries_data(self, dataTitle):
if self.yearCount == 0:
self.yearCount += 1
self.dataTitle = dataTitle + ' (current year)'
else:
self.yearCount = 0
self.dataTitle = dataTitle + ' (previous year)'
self.extractable = True
def parseZips(fileName=str()):
print(fileName)
directoryName = fileName.split('.')[0]
zip_ref = zipfile.ZipFile(fileName, 'r')
zipFileNames = tuple(n.filename for n in zip_ref.infolist() if 'html' in n.filename or 'htm' in n.filename)
print('Finished reading ' + fileName+'!\n')
collectHTMLS(directoryName, zip_ref, zipFileNames)
def collectHTMLS(directoryName, zip_ref, zipFileNames):
print('Collection html data into a csv for '+ directoryName+'...')
parser = MyHTMLParser()
fileCollection = []
totalFiles = len(zipFileNames)
count = 0
startTime = time.time()/60
for f in zipFileNames:
parser.feed(str(zip_ref.read(f)))
fileCollection.append(parser.fileData)
if(count % 500 ==0):
print('%s has reached file %i/%i\nIn: {timing} minutes\n'.format(timing = ((time.time()/60)-startTime)) % (directoryName,count,totalFiles))
parser.fileData = {} #reset the dictionary
count += 1
print('Finished parsing files for ' + directoryName)
with open(directoryName+'.csv', 'w') as f:
w = csv.DictWriter(f, fileCollection[0].keys())
w.writeheader()
for parsedFile in fileCollection:
w.writerow(parsedFile)
f.close()
print('Finished writing to file from ' + directoryName)
def main():
zipCollection = [f for f in os.listdir('.') if os.path.isfile(f) and f.split('.')[1] == 'zip']
threadPool = ThreadPool(len(zipCollection))
threadPool.map_async(parseZips, zipCollection)
threadPool.close()
threadPool.join()
main()
