Regex match comments and quotes

Question

UPDATE

New regex - covers escaped slashes and uses lazy qualifiers:

/(?>(')(?>(?>\\\\)|\\'|.)*')|(?>(")(?>(\\\\)|(?>\\")|.)*")|(?>((?>\/\/|#)).*^)|(?>(\/\*).*\*\/)/msU

Here are the different parts:

Single quotes: (?>(')(?>(?>\\\\)|\\'|.)*')

Double quotes: (?>(")(?>(\\\\)|(?>\\")|.)*")

Single line comments: (?>((?>\/\/|#)).*^)

Multiline comments: (?>(\/\*).*\*\/)

I moved the html comments to a different regex because html will be handled differently.

Here's an example of what it does: (The matched parts are bold)

---

//single line comment

random text ... #another comment

Multiline comments:

/* this is a multiline comment

matches the whole thing */

Quotes:

"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "

'matches the end quote because it it not escaped \\'

---

This is the original post:

I'm working on making syntax highlighting on my blog, and I created a regex to match strings and comments (I'm going to make a callback function to format each differently)

Here's my regex:

(?>(")(?>[^"]|(?>(?<=\\)"))*")|(?>(')(?>[^']|(?<=\\)')*')|(?>((?>\/\/|#))[^\n\r]*)|(?>(\/\*)(?>.(?!\*\\))*\*\/)|(?>(<!--)(?>.(?!-->))*.-->)

I tested it and it seemed to work, but is there anything I'm missing or an easier way to do it? It currently matches strings ("...", '...') and takes into account escaped chars and strings within other strings. It also matches comments (//..., #..., /*...*/, <!--...-->).

Here's what I'm using each part of the regex for:

Part 1: (?>(")(?>[^"]|(?>(?<=\\)"))*"): Match strings quoted with "

Part 2: (?>(')(?>[^']|(?<=\\)')*'): Match strings quoted with '

Part 3: (?>((?>\/\/|#))[^\n\r]*): Match single line comments # or //

Part 4: (?>(\/\*)(?>.(?!\*\\))*\*\/): Match multiline comments /*...*/

Part 5: (?>(<!--)(?>.(?!-->))*.-->): Match html comments <!--...-->

Answer 1

I think I may have thrown you slightly when I mentioned lazy quantifiers. The truth is, using greedy quantifiers improves pattern efficiency.

Regarding your atomic grouping, I don't see any benefit (in my pattern anyhow) because I am actively avoiding the need to backtrack in each "alternative" (the expressions between the pipes).

For the single and double quote and multiline comment alternatives, I want the dot to match "any character AND new lines". For the inline comment alternative, I want the dot to match "any non-newline character". For this reason, I am using "inline modifiers", specifically (?s) where needed.

I try to use pattern delimiting characters that do not occur in my pattern. This helps to make the pattern more readable and prevents having to do unnecessary escaping.

~                        # pattern delimiter
(?|                      # branch reset
(")(?:[^"\\]|\\(?s).)*"  # capture double quote, match any non-double-quote/non-backslash unless backslash is followed by any character (used as an escaping character)
|                        # OR
(')(?:[^'\\]|\\(?s).)*'  # capture single quote, match any non-single-quote/non-backslash unless backslash is followed by any character (used as an escaping character)
|                        # OR
(#|//).*                 # capture hash or two-slashes, match the rest of the line                         # OR
(/\*)(?s).*?\*/          # capture \*, match zero or more of any character (including newlines), then */ (lazy quantifier)
|                        # OR
(<!--)(?s).*?-->)        # capture <!--, match zero or more of any character (including newlines) (lazy quantifier)
)                        # end branch reset
~                        # pattern delimiter

(Pattern Demo)

Implementation: (PHP Demo)

const PATTERN = <<<'PATTERN'
~(?|(")(?:[^"\\]|\\(?s).)*"|(')(?:[^'\\]|\\(?s).)*'|(#|//).*|(/\*)(?s).*?\*/|(<!--)(?s).*?-->)~
PATTERN;

const LOOKUP = [
    '#'    => 'gainsboro',
    '//'   => 'lightgrey',
    '/*'   => 'silver',
    '<!--' => 'darkgrey',
    "'"    => 'mint',
    '"'    => 'aqua'
];

echo preg_replace_callback(PATTERN, function($m) {
        return "<span style=\"color:" . LOOKUP[$m[1]] . ";\">{$m[0]}</span>";
    }, $string);

By declaring the lookup array as a constant, scoping issues are avoided inside of the preg_replace_callback() callback function. In other words, you don't have to pass in the lookup array with use().

I am using nowdoc syntax when declaring the pattern constant so that I don't need to escape any quotes. I am declaring a constant for no other reason than the fact that the pattern will not change in the script.

By wrapping the whole expression in a "branch reset" ((?|...)), you can avoid calling end($m) inside the custom function to access the captured group. The branch reset ensures that each capture group is always the second element in the matches output (at [1]). If you removed the branch reset in the Regex101 demo above, you will see that the captured "marker" matches will have differing indexes.

Hmm... If your code is relying on the different indexes as part of a lookup-based replacement, then don't use the branch reset -- it's a good chance to break out one of the new weapons: array_key_last($m).

Test Input:

$string = <<<'STRING'
//single line comment

random text ... #another comment

Multiline comments:

/* this is a multiline comment

with 'squote and "dquote"

matches the whole thing */

// single line 'squoted' "dquoted" comment w/ extra " for no reason

More comments <!-- yatta yatta
yatta
yatta -->

Quotes:

"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "

a "nested 'squote with nested \"dquote\"'" assuming only outermost quoting matters for formatting

'matches the end quote because it it not escaped \\'
STRING;

Output (unrendered):

<span style="color:lightgrey;">//single line comment</span>

random text ... <span style="color:gainsboro;">#another comment</span>

Multiline comments:

<span style="color:silver;">/* this is a multiline comment

with 'squote and "dquote"

matches the whole thing */</span>

<span style="color:lightgrey;">// single line 'squoted' "dquoted" comment w/ extra " for no reason</span>

More comments <span style="color:darkgrey;"><!-- yatta yatta
yatta
yatta --></span>

Quotes:

<span style="color:aqua;">"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "</span>

a <span style="color:aqua;">"nested 'squote with nested \"dquote\"'"</span> assuming only outermost quoting matters for formatting

<span style="color:mint;">'matches the end quote because it it not escaped \\'</span>

Reasons to re-invent the wheel. The wheel that you need is very narrow and employing/loading a complete library may be overkill in terms of performance or memory consumption for your task.

In terms of step count, my pattern is slightly more efficient than yours. Though to be honest, I have been informed (by Stackoverflow regex gurus who I respect) that step count is not a reliable metric to gauge pattern efficiency. I generally use it, though, as a rough indicator of pattern efficiency.

I reckon taking these opportunities to sharpen regular expression knowledge is a healthy exercise for programmers. The more you work with regular expressions, the less scary they become.

I am purposely not bothering to acknowledge any tinfoil-hat fringe cases regarding premature-terminating substrings like:

• /* arithmetic symbols include: +-*/ */ and
• <!-- This is a long --> arrow -->

because these deliberate monkeywrenches are not commonly escaped by slashes and they can be sensibly overcome by recrafting the comment. Such as:

• /* re-ordered arithmetic symbols include: +-/* */ and
• <!-- I only use short -> arrows in comments-->

Answer 2

If it is PHP source code you want to highlight, why not use the tokenizer that's build into PHP itself? I use that and it seems to work fine:

$tokens = token_get_all($sourceCode);

Then all you have to do is walk through all the tokens and give them a color.

See: https://www.php.net/manual/en/function.token-get-all.php

No extra library, no faffing around with endless regular expressions.

Answer 3

Instead of using a single regular expression, you should keep a separate expression for each kind of token you want to match.

What about these string literals?

""
"\n"
"\\"
"\\\\\\\\"
"{$arr['key']->["value"]}"

Some further questions:

• Does PHP have literals for regular expressions?
• Are there other tokens that look like strings?
• Are you the first person in the world who wants to split PHP code into separate tokens?
• What about multiline string literals?
• What about ?> these <?php string literals?

Since the answer to the third question is a clear No, just use an existing library.

My advice is to keep the regular expressions as small as possible. Otherwise your code will become unreadable in the future.