6
\$\begingroup\$

I recently came to write a regex to parse URLs. Now I wonder, did I miss something? Did I make a mistake or could I have written it cleaner? That's why I'm here.

In order to write the regex, I took this paper as a reference (together with some Wikipedia articles about URIs/URLs).

//According to http://www.ietf.org/rfc/rfc3986.txt
private static final String URL_UNRESERVED_CHARS = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
private static final String URL_UNRESERVED_SPECIAL_CHARS = "-._~";
private static final String URL_UNRESERVED = URL_UNRESERVED_CHARS + URL_UNRESERVED_SPECIAL_CHARS;
private static final String URL_RESERVED_GEN_DELIMS = ":/?#[]@";
private static final String URL_RESERVED_SUB_DELIMS = "!$&'()*+,;=";
private static final String URL_CHAR_ENCODING_SIGN = "%";

public static final String URL_ALLOWED_CHARS = URL_UNRESERVED + URL_RESERVED_GEN_DELIMS + URL_RESERVED_SUB_DELIMS + URL_CHAR_ENCODING_SIGN;

private static final String REGEX_SCHEME = "[A-Za-z][A-Za-z0-9+.-]*:"; //Also called 'protocol'
private static final String REGEX_AUTHORATIVE_DECLARATION = "/{2}";
private static final String REGEX_USERINFO = "(?:[A-Za-z0-9-._~]|%[A-Fa-f0-9]{2})+(?::(?:[A-Za-z0-9-._~]|%[A-Fa-f0-9]{2})+)?@";
private static final String REGEX_HOST = "(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\\.){1,126}[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?";
private static final String REGEX_PORT = ":[0-9]+";
private static final String REGEX_PATH = "/(?:[A-Za-z0-9-._~]|%[A-Fa-f0-9]{2})*";
private static final String REGEX_QUERY = "\\?(?:[A-Za-z0-9-._~]+(?:=(?:[A-Za-z0-9-._~+]|%[A-Fa-f0-9]{2})+)?)(?:[&|;][A-Za-z0-9-._~]+(?:=(?:[A-Za-z0-9-._~+]|%[A-Fa-f0-9]{2})+)?)*";
//FRAGMENTs don't need to be parsed as they won't be sent to the server anyways

public static final String REGEX_URL = "(?:" + REGEX_SCHEME + REGEX_AUTHORATIVE_DECLARATION + ")?(?:" + REGEX_USERINFO + ")?" + REGEX_HOST + "(?:" + REGEX_PORT + ")?(?:" + REGEX_PATH + ")*(?:" + REGEX_QUERY + ")?";

The above code snippet is written in Java (which explains for the \\ as they need to be escaped in Strings in order to be taken as literals). I split the regex up into multiple parts in order to increase readability and maintainability (although it technically shouldn't require any).

That would be the regex in its true form as when encountered in the wild:

(?:[A-Za-z][A-Za-z0-9+.-]*:/{2})?(?:(?:[A-Za-z0-9-._~]|%[A-Fa-f0-9]{2})+(?::(?:[A-Za-z0-9-._~]|[%][A-Fa-f0-9]{2})+)?@)?(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\.){1,126}[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?(?::[0-9]+)?(?:/(?:[A-Za-z0-9-._~]|%[A-Fa-f0-9]{2})*)*(?:\?(?:[A-Za-z0-9-._~]+(?:=(?:[A-Za-z0-9-._~+]|%[A-Fa-f0-9]{2})+)?)(?:&|;[A-Za-z0-9-._~]+(?:=(?:[A-Za-z0-9-._~+]|%[A-Fa-f0-9]{2})+)?)*)?

TL;DR Regex above does validate a URL against the defined rules here. I couldn't break it yet, but is it breakable? Can some of it be rewritten in a cleaner fashion?

\$\endgroup\$
7
\$\begingroup\$

Holy regex Batman!

Without testing, you should be able to replace all occurrences of:

  • [0-9] with \d
  • [A-Za-z0-9] with [\w^_]
  • [A-Fa-f0-9] with \p{XDigit}
  • [A-Za-z0-9-._~] with [\w.~-]

and so forth. See documentation for Pattern.

As for your capturing groups I'm quite sure you can do some clever back referencing there but I will save my sanity and not try to parse the regexp. :)

\$\endgroup\$
  • 1
    \$\begingroup\$ Patterns - totally forgot about these little beauties \$\endgroup\$ – dot_Sp0T Jan 27 '15 at 18:14
1
\$\begingroup\$

The above regex will break for the following example

http://www.pierobon.org/iis/review1.htm.html#one

I got this example from here.

Update: This is failing for even

ldap://[2001:db8::7]/c=GB?objectClass?one

The example is from RFC 3986.

\$\endgroup\$
  • \$\begingroup\$ Downvoted because the example given in the formatting contains a <i>fragment</i>. Fragments must never reach the webserver, thus they are not part of the regex! \$\endgroup\$ – dot_Sp0T Jun 13 '16 at 7:38
-1
\$\begingroup\$

(Some how the update is not showing in the above answer) Update: This is failing for even

ldap://[2001:db8::7]/c=GB?objectClass?one

The example is from RFC 3986.

Even I need a foolproof regex to validate URI entered by users.

\$\endgroup\$
  • \$\begingroup\$ Indeed. The regex in my question and the whole question is actually about URLs (which are a form of absolute URIs). Back when I wrote the question the difference didn't seem to be there, but I also learned lots of things in the meantime. I'll update the phrasing in the question to point that out.|| On the other hand to solve your issue, just take the subregex validating the schema from above (slightly modify it) and append .*. As follows: [A-Za-z][A-Za-z0-9+.-]+:.+ <- this should match the simplest form of URI possible and thus every more complex form as well - hth \$\endgroup\$ – dot_Sp0T May 13 '16 at 13:43
  • \$\begingroup\$ The above comment refers to the phrasing: Even I need a foolproof regex to validate URI entered by users. found in your answer. I interpreted the phrasing as you needing a regex to parse URIs, thus I took the time to write the extensive comment \$\endgroup\$ – dot_Sp0T May 13 '16 at 13:47
  • \$\begingroup\$ Please follow these instructions to merge your multiple accounts: codereview.stackexchange.com/help/merging-accounts \$\endgroup\$ – BCdotWEB May 13 '16 at 14:32
  • \$\begingroup\$ Finally, this is the regex. Tested against all the examples in RFC \$\endgroup\$ – Krishna Chaitanya May 16 '16 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.