Regular Expression to Remove Comments

Question

This was developed in response to a question on Stack Overflow, Removing comments using regex.

What the OP needed was a regular expression to use in C# to remove comnents from lines of a file. All comments were specified as beginning with a double slash // and everything after that is ignored. Comments may also begin at the start of a line, resulting in the line being blank. Quotes, double or single, when properly closed, will shield any double slashes from triggering a comment. There was no indication that the target file was a C# source file, only that the processing file was C#.

Taking advantage of the C# environment given by the problem enabled the use of conditional matching. These constructs are, to my knowledge, only supported in the .NET and PCRE engines, so were acceptable to the OP's use case, but not in many other cases.

When used the pattern will return the non-comment portion of the line(s). It will return a newline for lines of code that are either blank, or have a comment beginning in the first column. This keeps the line numbers the same in the original version and the stipped version if the results are written to a new file. Makes comparison easier.

The patterns given here are as they need to be seen by the regex engine. I am not a C# programmer, so I don't know all the nuances of escaping quoted strings. Getting the pattern into the code, such that all the backslashes and quotes are seen properly by the regex engine was explicitly left to the OP.

The pattern was built and tested under the ignore white-space adn multi-line options, rather than the other way around, so the commented version is the original and the single line version was created from it. The comments were intended to aid the OP in understanding the pattern, so they are rather verbose.

In my testing it passed every pathological test I could throw at it, and I believe it is sound.

The test file I created is:

variable = "5 // \ \\"not // here " variable = 6 / /3 + '7 // \"\' not // here ' and then some //real comment
FullLine here
;Second Line here
;third line here
";fourth line here"
Fifth Line here /* comment */
Sixth line here /* comment */ syntax error?
variable = "5 // \\ \"not // here " variable = 6 //3 + '7 // \"\' not // here ' and then some //real comment
variable = "5 // \\ \"not // here " variable = 6 / /3 + '7 // \"\' not // here ' and then some //real comment
variable = "some//thing" and "some\" thing" ////actual comment
a    //// some comment
d /// other comment
/*** one-liner, c-style block comment  ***/
variable = 'some//thing' and "some\" thing" ////actual comment
c #// later comment
/******
* c-style block comment two
* Multi-line -1 //extra non-comment
* Multi-line 2  # extra non-comment
*******/
variable = "some thing" and "some \\ thing else" ////actual comment
    /// some comment
/******
* c-style block comment two
* Multi-line -3 //extra non-comment
* Multi-line 4  # extra non-comment
*******/ more syntax errors?
variable = "some \"thing" ////actual comment
// other comment
variable = 'some//thing' ////actual comment
// later comment
variable = 'some thing' ////actual comment
//last comment
"qwe\" qwe" \\asd
variable = 'some \'thing' ////actual comment
Variable = 4 no comment
Variable = 45 no comment /* is this a c-legal comment */
Variable = 46 no comment /* is this a c-legal comment */
variable = "5 // not here "
variable = "5 // \\ \"not // here " variable = 6 / /3 + '7 // \"\' not // here ' and then some //real comment //nestes

The single-line version is:

^((?:(?:(?:[^"'/\n]|/(?!/))*)(?("(?=(?:\\\\|\\"|[^"])*"))(?:"(?:\\\\|\\"|[^"])*")|(?('(?=(?:\\\\|\\'|[^'])*'))(?:'(?:\\\\|\\'|[^'])*')|(?(/)|.))))*)

The commented version is:

^( # Start the only capturing group
    (?: # A non-capturing group to allow for repeating the logic
        (?: # Capture either of the two options below
            [^"'/\n] # Capture everything not a single quote, double quote, a slash, or a newline
            | # OR
            /(?!/) # Capture a slash not followed by a slash [slash an negative look-ahead slash]
        )* # As many times as possible, even if none
        (?(" # Start a conditional match for double-quoted strings
                (?=(?:\\\\|\\"|[^"])*") # Followed by a properly closed double-quoted string
            ) # Then
            (?:"(?:\\\\|\\"|[^"])*") # Capture the whole double-quoted string
            | # Otherwise
            (?(' # Start a conditional match for single-quoted strings
                (?=(?:\\\\|\\'|[^'])*') # Followed by a properly closed single-quoted string
                ) # Then
                (?:'(?:\\\\|\\'|[^'])*') # Capture the whole double-quoted string
                | # Otherwise
                (?([^/]) # If next character is not a slash
                .) # Capture that character, it is either a single quote, or a double quote not part of a properly closed
            ) # end the conditional match for single-quoted strings
        ) # End the conditional match for double-quoted strings
    )* # Close the repeating non-capturing group, capturing as many times as possible, even if none
) # Close the only capturing group

I'm looking to see if there is some optimisation possible, of some trap I've fallen into that will create either a false postive, or miss a match that should be made. Not that my test file includes comments in the /* ... */ style, and they are not stripped by the pattern, nor did I want them to be, as the OP made no mention of doing so.

Answer 1

Maintainability

No matter the possibilities of a regex, it's not intended for parsing complex languages. Nobody (no sane person) is able to make changes to this regex (even if provided with comments):

^((?:(?:(?:[^"'/\n]|/(?!/))*)(?("(?=(?:\\\\|\\"|[^"])*"))(?:"(?:\\\\|\\"|[^"])*")|(?('(?=(?:\\\\|\\'|[^'])*'))(?:'(?:\\\\|\\'|[^'])*')|(?(/)|.))))*)

In addition, even if you think you are covering it all, some people might still find edge cases, rendering the regex incomplete, requiring even more subtle changes to the regex. At some point, you might (and probably will) get in a situation where you realise the problem is harder than you imagined, and that the problem might even require complex context, that just isn't solvable through regex. Back to drawing boards and think about an alternative solution the your problem.

As P.Roe so eloquently states in the comments:

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

Missing Edge Case(s)

• There is a case I noticed you didn't cover. C# allows unicode escape sequences. Literals can have \xnnnn escape sequences, while all C# code can handle \unnnn and \Unnnnnnnn escape sequences. The preprocessor of the compiler transforms these latter two into characters, but only in the following cases:
  • identifiers
  • character literals
  • regular string literals This means \u2215 which represents a slash / is not found by your regex.
• I'm also not sure whether you can track inline comments in interpolated strings. Should be verified.

Alternative Approach

Scanning text to parse to some intermediate language, in this case C# comments grammar, requires a process called lexing or tokenizing. In a Lexer, you would like to scan the text and use an internal state machine to determine what you are currently looking at. A state might be QuoteState in which state a comment delimiter is a literal and not a comment. The end result of lexing provides you tokens (comments in this case) and their position in the input text.