Streamlining series of try-except + if-statements for faster loop processing in Python

Question

I'm processing strings using regexes in a bunch of files in a directory. To each line in a file, I apply a series of try-statements to match a pattern and if they do, then I transform the input. After I have analyzed each line, I write it to a new file. I have a lot of these try-else followed by if-statements (I only included two here as an illustration). My issue here is that after processing a few files, the script slows down so much that it almost stalls the process completely. I don't know what in my code is causing the slowing down but I have a feeling it is the combination of try-else + if-statements. How can I streamline the transformations so that the data is processed at a reasonable speed?

Or is it that I need a more efficient iterator that does not tax memory to the same extent?

Any feedback would be much appreciated!

import re
import glob

fileCounter = 0 

for infile in glob.iglob(r'\input-files\*.txt'):

    fileCounter += 1
    outfile = r'\output-files\output_%s.txt' % fileCounter

    with open(infile, "rb") as inList, open(outfile, "wb") as outlist:

        for inline in inlist:

            inword = inline.strip('\r\n')

            #apply some text transformations
            #Transformation #1
            try: result = re.match('^[AEIOUYaeiouy]([bcćdfghjklłmnńprsśtwzżź]|rz|sz|cz|dz|dż|dź|ch)[aąeęioóuy](.*\[=\].*)*', inword).group()
            except: result = None

            if result == inword:
                inword = re.sub('(?<=^[AEIOUYaeiouy])(?=([bcćdfghjklłmnńprsśtwzżź]|rz|sz|cz|dz|dż|dź|ch)[aąeęioóuy])', '[=]', wbWord)

            #Transformation #2 etc.
            try: result = re.match('(.*\[=\].*)*(\w?\w?)[AEIOUYaąeęioóuy]\[=\][ćsśz][ptkbdg][aąeęioóuyrfw](.*\[=\].*)*', inword).group()
            except: result = None

            if result == inword:   
                inword =  re.sub('(?<=[AEIOUYaąeęioóuy])\[=\](?=[ćsśz][ptkbdg][aąeęioóuyrfw])', '', inword)
                inword =  re.sub('(?<=[AEIOUYaąeęioóuy][ćsśz])(?=[ptkbdg][aąeęioóuyrfw])', '[=]', inword)

            outline = inword + "\n"
            outlist.write(outline)

    print "Processed file number %s" % fileCounter          
print "*** Processing completed ***" 

Answer 1

• It's not 100 % clear what exceptions you expect, but I presume you are trying to handle the case when the regex does not match. I suggest handling it this way:

      #Transformation #1
      match = re.match(pattern, inword)
      result = match.group() if match else None
  

• "after processing a few files, the script slows down" Have you considered the possibility that a particular file, or even a particular line is slow to process? A possible explanation to that would be that regexes can suffer from catastrophic backtracking.