I have created a python system that runs Linux core files through the crash debugger with some python extensions. This all works fine but one bit of is problematic.
These files are send to the system in gzip format and consist of a single huge data file. The compressed file can often be as big as 20G. The unzipping works fine but is very slow and often uses huge amounts of memory. As an example, last night this processed a 14G gzip file and it took 9.2 hours to uncompress it (60G uncompressed) and the memory utilisation hovered between 30G, peaking at 60G.
Starting to think perhaps my code is the cause.
def chk_gzip_file(FILE): logger.info ("Will write uncompressed file to: "+ COREDIR) if os.path.isdir(FILE) == False: inF = gzip.open(FILE, 'rb') s = inF.read() inF.close() gzip_fname = os.path.basename(FILE) fname = gzip_fname[:-3] uncompressed_path = os.path.join(COREDIR, fname) open(uncompressed_path, 'w').write(s) uncompressedfile=COREDIR+"/"+fname return uncompressedfile else: logger.critical ("No gz file found : " + FILE) sys.exit()
I am not a programmer so I imagine this is fairly poor code. Can this be improved for huge files? I know that speed will be an issue as gzip uncompress is single threaded.