5
\$\begingroup\$

UPDATE

New regex - covers escaped slashes and uses lazy qualifiers:

/(?>(')(?>(?>\\\\)|\\'|.)*')|(?>(")(?>(\\\\)|(?>\\")|.)*")|(?>((?>\/\/|#)).*^)|(?>(\/\*).*\*\/)/msU

Here are the different parts:

Single quotes: (?>(')(?>(?>\\\\)|\\'|.)*')

Double quotes: (?>(")(?>(\\\\)|(?>\\")|.)*")

Single line comments: (?>((?>\/\/|#)).*^)

Multiline comments: (?>(\/\*).*\*\/)

I moved the html comments to a different regex because html will be handled differently.

Here's an example of what it does: (The matched parts are bold)


//single line comment

random text ... #another comment

Multiline comments:

/* this is a multiline comment

matches the whole thing */

Quotes:

"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "

'matches the end quote because it it not escaped \\'


This is the original post:

I'm working on making syntax highlighting on my blog, and I created a regex to match strings and comments (I'm going to make a callback function to format each differently)

Here's my regex:

(?>(")(?>[^"]|(?>(?<=\\)"))*")|(?>(')(?>[^']|(?<=\\)')*')|(?>((?>\/\/|#))[^\n\r]*)|(?>(\/\*)(?>.(?!\*\\))*\*\/)|(?>(<!--)(?>.(?!-->))*.-->)

I tested it and it seemed to work, but is there anything I'm missing or an easier way to do it? It currently matches strings ("...", '...') and takes into account escaped chars and strings within other strings. It also matches comments (//..., #..., /*...*/, <!--...-->).

Here's what I'm using each part of the regex for:

Part 1: (?>(")(?>[^"]|(?>(?<=\\)"))*"): Match strings quoted with "

Part 2: (?>(')(?>[^']|(?<=\\)')*'): Match strings quoted with '

Part 3: (?>((?>\/\/|#))[^\n\r]*): Match single line comments # or //

Part 4: (?>(\/\*)(?>.(?!\*\\))*\*\/): Match multiline comments /*...*/

Part 5: (?>(<!--)(?>.(?!-->))*.-->): Match html comments <!--...-->

\$\endgroup\$
  • \$\begingroup\$ What if there is an escaped backslash before a quote (or any other known ending delimiter)? Are you deliberately avoiding non-greedy (lazy) quantifiers? Do you actually need to separately capture the leading characters for your process? I will assume, yes. \$\endgroup\$ – mickmackusa Apr 11 at 3:06
  • \$\begingroup\$ shoot I didn't think of an escaped backslash! I'm don't remember why I didn't use lazy qualifiers but I'll check, and yes I have that capture group so the callback function can easily identify if it's a quote or a comment. thanks for the feedback! \$\endgroup\$ – maxpelic Apr 11 at 12:24
  • \$\begingroup\$ So theoretically, you'd need to continue matching if a single or double quote is preceded by an odd number of slashes. \$\endgroup\$ – mickmackusa Apr 11 at 12:28
  • \$\begingroup\$ Please add context to your question. Specifically add a handful of realistic/tricky input strings and your exact expected output. \$\endgroup\$ – mickmackusa Apr 11 at 12:39
  • \$\begingroup\$ Yeah I'll do that \$\endgroup\$ – maxpelic Apr 11 at 12:40
2
\$\begingroup\$

I think I may have thrown you slightly when I mentioned lazy quantifiers. The truth is, using greedy quantifiers improves pattern efficiency.

Regarding your atomic grouping, I don't see any benefit (in my pattern anyhow) because I am actively avoiding the need to backtrack in each "alternative" (the expressions between the pipes).

For the single and double quote and multiline comment alternatives, I want the dot to match "any character AND new lines". For the inline comment alternative, I want the dot to match "any non-newline character". For this reason, I am using "inline modifiers", specifically (?s) where needed.

I try to use pattern delimiting characters that do not occur in my pattern. This helps to make the pattern more readable and prevents having to do unnecessary escaping.

~                        # pattern delimiter
(?|                      # branch reset
(")(?:[^"\\]|\\(?s).)*"  # capture double quote, match any non-double-quote/non-backslash unless backslash is followed by any character (used as an escaping character)
|                        # OR
(')(?:[^'\\]|\\(?s).)*'  # capture single quote, match any non-single-quote/non-backslash unless backslash is followed by any character (used as an escaping character)
|                        # OR
(#|//).*                 # capture hash or two-slashes, match the rest of the line                         # OR
(/\*)(?s).*?\*/          # capture \*, match zero or more of any character (including newlines), then */ (lazy quantifier)
|                        # OR
(<!--)(?s).*?-->)        # capture <!--, match zero or more of any character (including newlines) (lazy quantifier)
)                        # end branch reset
~                        # pattern delimiter

(Pattern Demo)

Implementation: (PHP Demo)

const PATTERN = <<<'PATTERN'
~(?|(")(?:[^"\\]|\\(?s).)*"|(')(?:[^'\\]|\\(?s).)*'|(#|//).*|(/\*)(?s).*?\*/|(<!--)(?s).*?-->)~
PATTERN;

const LOOKUP = [
    '#'    => 'gainsboro',
    '//'   => 'lightgrey',
    '/*'   => 'silver',
    '<!--' => 'darkgrey',
    "'"    => 'mint',
    '"'    => 'aqua'
];

echo preg_replace_callback(PATTERN, function($m) {
        return "<span style=\"color:" . LOOKUP[$m[1]] . ";\">{$m[0]}</span>";
    }, $string);

By declaring the lookup array as a constant, scoping issues are avoided inside of the preg_replace_callback() callback function. In other words, you don't have to pass in the lookup array with use().

I am using nowdoc syntax when declaring the pattern constant so that I don't need to escape any quotes. I am declaring a constant for no other reason than the fact that the pattern will not change in the script.

By wrapping the whole expression in a "branch reset" ((?|...)), you can avoid calling end($m) inside the custom function to access the captured group. The branch reset ensures that each capture group is always the second element in the matches output (at [1]). If you removed the branch reset in the Regex101 demo above, you will see that the captured "marker" matches will have differing indexes.

Hmm... If your code is relying on the different indexes as part of a lookup-based replacement, then don't use the branch reset -- it's a good chance to break out one of the new weapons: array_key_last($m).

Test Input:

$string = <<<'STRING'
//single line comment

random text ... #another comment

Multiline comments:

/* this is a multiline comment

with 'squote and "dquote"

matches the whole thing */

// single line 'squoted' "dquoted" comment w/ extra " for no reason

More comments <!-- yatta yatta
yatta
yatta -->

Quotes:

"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "

a "nested 'squote with nested \"dquote\"'" assuming only outermost quoting matters for formatting

'matches the end quote because it it not escaped \\'
STRING;

Output (unrendered):

<span style="color:lightgrey;">//single line comment</span>

random text ... <span style="color:gainsboro;">#another comment</span>

Multiline comments:

<span style="color:silver;">/* this is a multiline comment

with 'squote and "dquote"

matches the whole thing */</span>

<span style="color:lightgrey;">// single line 'squoted' "dquoted" comment w/ extra " for no reason</span>

More comments <span style="color:darkgrey;"><!-- yatta yatta
yatta
yatta --></span>

Quotes:

<span style="color:aqua;">"also matches strings with \" escaped quotes or 'the other kind of quotation marks in it' "</span>

a <span style="color:aqua;">"nested 'squote with nested \"dquote\"'"</span> assuming only outermost quoting matters for formatting

<span style="color:mint;">'matches the end quote because it it not escaped \\'</span>

Reasons to re-invent the wheel. The wheel that you need is very narrow and employing/loading a complete library may be overkill in terms of performance or memory consumption for your task.

In terms of step count, my pattern is slightly more efficient than yours. Though to be honest, I have been informed (by Stackoverflow regex gurus who I respect) that step count is not a reliable metric to gauge pattern efficiency. I generally use it, though, as a rough indicator of pattern efficiency.

I reckon taking these opportunities to sharpen regular expression knowledge is a healthy exercise for programmers. The more you work with regular expressions, the less scary they become.

I am purposely not bothering to acknowledge any tinfoil-hat fringe cases regarding premature-terminating substrings like:

  • /* arithmetic symbols include: +-*/ */ and
  • <!-- This is a long --> arrow -->

because these deliberate monkeywrenches are not commonly escaped by slashes and they can be sensibly overcome by recrafting the comment. Such as:

  • /* re-ordered arithmetic symbols include: +-/* */ and
  • <!-- I only use short -> arrows in comments-->
\$\endgroup\$
  • \$\begingroup\$ On second thought, I think I'll recommend simply using lazy matching with the multiline comment blocks. regex101.com/r/27LzpN/2 I'll adjust my answer when I get to my computer again. \$\endgroup\$ – mickmackusa Apr 12 at 23:42
  • \$\begingroup\$ that's for all the time you spent on this! I'll take a look in detail when I get a chance. \$\endgroup\$ – maxpelic Apr 13 at 2:19
  • \$\begingroup\$ As I spend my Saturday in the garden, my mind keeps playing out refinements. I'll update and let you know. \$\endgroup\$ – mickmackusa Apr 13 at 2:21
3
\$\begingroup\$

If it is PHP source code you want to highlight, why not use the tokenizer that's build into PHP itself? I use that and it seems to work fine:

$tokens = token_get_all($sourceCode);

Then all you have to do is walk through all the tokens and give them a color.

See: https://www.php.net/manual/en/function.token-get-all.php

No extra library, no faffing around with endless regular expressions.

\$\endgroup\$
  • \$\begingroup\$ This function is blowing my mind, never saw it before. However, unless I misunderstand its usage and/or the OP's requirements, I don't think it can be relied upon. 3v4l.org/CffGq correct me if I am wrong. \$\endgroup\$ – mickmackusa Apr 11 at 12:26
  • \$\begingroup\$ @mickmackusa Can you clarify what problem you see? I cannot correct you if you don't tell. ;-) Oh, wait there's a weird link. I'll have a look. \$\endgroup\$ – KIKO Software Apr 11 at 12:27
  • \$\begingroup\$ I kind of expected the double quoted substring to be a single chunk for the OP's requirements, but that wasn't the case. \$\endgroup\$ – mickmackusa Apr 11 at 12:30
  • \$\begingroup\$ @mickmackusa I need to have a think about this... \$\endgroup\$ – KIKO Software Apr 11 at 12:31
  • \$\begingroup\$ It is not meant to be valid php code, just content in a blog I guess. There is no sample data with the question, so I don't know what kind of input strings to expect. \$\endgroup\$ – mickmackusa Apr 11 at 12:34
1
\$\begingroup\$

Instead of using a single regular expression, you should keep a separate expression for each kind of token you want to match.

What about these string literals?

""
"\n"
"\\"
"\\\\\\\\"
"{$arr['key']->["value"]}"

Some further questions:

  • Does PHP have literals for regular expressions?
  • Are there other tokens that look like strings?
  • Are you the first person in the world who wants to split PHP code into separate tokens?
  • What about multiline string literals?
  • What about ?> these <?php string literals?

Since the answer to the third question is a clear No, just use an existing library.

My advice is to keep the regular expressions as small as possible. Otherwise your code will become unreadable in the future.

\$\endgroup\$
  • \$\begingroup\$ I didn't want to use separate expressions because a string in a comment should not be matched, and vice versa \$\endgroup\$ – maxpelic Apr 10 at 14:29
  • \$\begingroup\$ Good point. In that case, you should still keep them separate and test them separately, and in the one place where you use the complete regular expression, put it together from the individual parts. \$\endgroup\$ – Roland Illig Apr 10 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.