I am preprocessing a set of files where one file is included in another file with an include tag like shown below:

file A

include file B
include file C
 contents of file A
include file D
 contents of file A

Here I need to replace the include tags with contents of each of the respective files, just like what a compiler does. I have two functions for that:

def parseContent(self, rdbFilePath, content):
    finalContent = self.removeComments(content)
    includeFileSearch = re.compile(r'(?P<tag>(\s)*include+(\s*)\"(\s*)(?P<file>[a-zA-Z0-9\.\_/]*)\")')
    for includes in includeFileSearch.finditer(finalContent):
        finalContent = re.sub(includes.group('tag'),self.parseIncludes(rdbFilePath, includes.group('file')), finalContent)
    return finalContent

def parseIncludes(self, rdbFilePath, file):
    path = rdbFilePath + "/" + file
    f = open(path)
    pathDir = os.path.dirname(path)
    includedFileContent = self.parseContent(pathDir, f.read())
    return includedFileContent

As you can see, functions parseContent and parseIncludes call each other recursively to replace all the include tags in every file. The logic works fine. But it takes a bit long time to execute. Is there any better way to do the same with lesser execution time?

  • \$\begingroup\$ no it is not xml \$\endgroup\$ – Hari Krishnan Apr 19 '19 at 5:03
  • 1
    \$\begingroup\$ What do you mean by slow? How big are your files? \$\endgroup\$ – Roland Illig Apr 19 '19 at 5:06
  • \$\begingroup\$ Have you tried what this code does when the included file contains the text $0.00 plus $1.23? In other words, using re.sub looks wrong to me. \$\endgroup\$ – Roland Illig Apr 19 '19 at 5:10
  • \$\begingroup\$ Is your file format line-based? In that case it might be easier to parse the input into a list of strings, instead of working with the finalContent (which by the way is not so final at all, since you are modifying it constantly). \$\endgroup\$ – Roland Illig Apr 19 '19 at 5:13
  • \$\begingroup\$ @RolandIllig I see no advantage to working line by line. \$\endgroup\$ – 200_success Apr 19 '19 at 5:39

parseContent() calls both re.finditer() and re.sub(), which is inefficient, because the re.sub() has to find the insertion point that re.finditer() had already found. What you want is just re.sub():

repl can be a string or a function… If repl is a function, it is called for every non-overlapping occurrence of pattern. The function takes a single match object argument, and returns the replacement string.

Going further, you should make just one substitution pass that handles both comments and includes. (You didn't show us the removeComments() code, so you'll have to adapt the regex yourself.)

The double-quotes in the regex have no special significance, and do not need to be escaped with backslashes. Also, the dot and the underscore characters do not need to be escaped within a character class.

DIRECTIVE_RE = re.compile(r'(?P<comment>#.*)|(?P<include>(\s)*include+(\s*)"(\s*)(?P<file>[a-zA-Z0-9._/]*)")')

def process_content(self, rdb_file_path, content):
    def directive_handler(match):
        if match.group('comment'):
            return ''
        elif match.group('include'):
            path = os.path.join(rdb_file_path, match.group('file'))
            with open(path) as f:
                return self.process_content(rdb_file_path, f.read())
    return DIRECTIVE_RE.sub(directive_handler, content)

open() should almost always be called using a with block to ensure that the filehandle gets closed automatically.

Note that I have used os.path.join() as a portable alternative to + "/" +.

| improve this answer | |
  • \$\begingroup\$ @Hari furthermore, you are not doing anything with the captured optional whitespaces, so you can remove their parentheses. \w is thr concise way to express [A-Za-z0-9_] and this may be used inside or outside of a character class. In your case, you can use [\w.]. Out of curiosity, do you have input that contains includeeeeeeeeeeeeeeeeeeeeeee that you want to match? \$\endgroup\$ – mickmackusa Apr 22 '19 at 20:57
  • \$\begingroup\$ Errm... [\w./] (I overlooked the forward slash) \$\endgroup\$ – mickmackusa Apr 22 '19 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.