I have several huge 100MB text files that I need to scan through to pick out certain frame numbers which relate to a specific log packet of interest. My plan was to scan for these frame numbers and drop them into a list (often 6000+ frames per text file!). Ok so far, but there is second packet of interest which should accompany the first packet, these pairs of packets can only be matched by a frame number in my newly created list, so to avoid grabbing useless/blank data from non-matching frames. So I would then re-scan the text file to get the frame related packet data. It was suggested on Stack Overflow that I go down the regex route, and whilst this method worked it was extremely slow, taking up to 3 minutes to process a single text file.
I was wondering if there is a suitable optimization to my code, or even a completely different approach to grabbing this data?
for root, subFolders, files in os.walk(path): for filename in files: if filename.endswith('.txt'): with open(os.path.join(root, filename), 'r') as f: print '\tProcessing file: '+filename for line in f: #first find first key packet and grab frame number if 'KEY_FRAME details' in line: chunk = [next(f) for x in xrange(5)] FRAME = chunk.split() FRAME = FRAME #drop frame number into a list framelist.append(str(FRAME)) #return to the start of the file, and search for next packet f.seek(0) framed = re.compile('|'.join(framelist)) framed = framed.pattern #Look for any frame number in list based on 'FrameNumber = '+f and 'FN = '+f match sentences = f for s in sentences: if any(('FrameNumber = '+f) in s for f in framelist): print 'first found' #do stuff if any(('FN = '+f) in s for f in framelist): print 'second found' #do stuff
The data I scan through is multiple repetitions of various packets, some packets are blank of info, some have paired info related to the particular frame.
17:29:50.040 AFP 276 Second_Packet details FN = 54332 Tp = TDSAA Te = True St = Test 17:29:50.040 TWR 765 KEY_FRAME details TAPP = 1 FrameNumber = 54332 Cap = 2 Tee = NA 17:29:50.040 AKK 347 Second_Packet details FN = 50000 Tp = KLA Te = True St = NA 17:29:50.040 AFP 276 Second_Packet details FN = 54367 Tp = Ax56 Te = True St = Test 17:29:50.040 YYY 765 KEY_FRAME details TAPP = 1 FrameNumber = 54367 Cap = 2 Tee = NA 17:29:50.040 YYY 765 KEY_FRAME details TAPP = 1 etc......
Just to make it tougher the second packet often comes before the first key packet in the text files.
Finally, I am looking to present the data in packet groups like below, for each key frame number:
FrameNumber = 54367 TAPP = 1 Cap = 2 Tee = NA Tp = Ax56 Te = True St = Test