URL pattern matching for request blocking

Question

I'm trying to write a URL pattern matching utility that, given a pattern and a URL it'll tell whether the URL matches the pattern or not. I want to use this for users to provide a list of URL patterns that will be use to block requests to URLs that match the given pattern. I don't want to use straight Regex because it'd be painful for the user to write, especially since all dots would need to be escaped.

I found this set of pattern matching rules which seem simple yet powerful enough for what I need. So I want to write a class that matches the specs given on that page. In the class below I attempt to do this, although I leave the "file" scheme out since I won't be needing it. Also, I'm letting the user provide no path, in which case I'll consider it equivalent to providing an empty path. Finally, I'm letting the user use extra asterisks on the host, which would be treated as literal asterisks, which I don't think should be much of a problem. I've tested it with the cases listed on the page and a few others and worked fine, but I wonder if there's some edge cases I'm not considering that break Google specifications.

public class UrlPattern
{
    public readonly Regex SchemeMatch;
    public readonly Regex HostMatch;
    public readonly Regex PathMatch;

    public UrlPattern(string pattern)
    {
        if (string.IsNullOrWhiteSpace(pattern))
            throw new ArgumentException("Pattern cannot be null or whitespace.");

        pattern = pattern.Trim();

        var index = pattern.IndexOf("://");

        if (index < 0)
            throw new ArgumentException("Pattern must have a scheme-host separator.");

        var scheme = pattern.Substring(0, index);
        var hostAndPath = pattern.Substring(index + 3);

        index = hostAndPath.IndexOf("/");

        if (index < 0)
        {
            hostAndPath = $"{hostAndPath}/";
            index = hostAndPath.IndexOf("/");
        }

        var host = hostAndPath.Substring(0, index);
        var path = hostAndPath.Substring(index);

        var schemePattern = MakeSchemePattern(scheme);
        var hostPattern = MakeHostPattern(host);
        var pathPattern = MakePathPattern(path);

        SchemeMatch = new Regex(schemePattern, RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase);
        HostMatch = new Regex(hostPattern, RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase);
        PathMatch = new Regex(pathPattern, RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase);
    }

    private static string MakePathPattern(string path)
    {
        return $"^{Regex.Escape(path).Replace("\\*", ".*")}$";
    }

    private static string MakeHostPattern(string host)
    {
        if (host == "*")
            return ".+";

        if (host.StartsWith("*."))
            return $"^(.+\\.)?{Regex.Escape(host.Substring(2))}$";

        return $"^{Regex.Escape(host)}$";
    }

    private static string MakeSchemePattern(string scheme)
    {
        switch (scheme)
        {
            case "*":
                return "^https?$";
            case "http":
                return "^http$";
            case "https":
                return "^https$";
            case "ftp":
                return "^ftp$";
            default:
                throw new ArgumentException("Invalid scheme pattern.");
        }
    }

    public bool IsMatch(Uri uri)
    {
        if (!SchemeMatch.IsMatch(uri.Scheme))
            return false;

        if (!HostMatch.IsMatch(uri.Host))
            return false;

        if (!PathMatch.IsMatch(uri.PathAndQuery))
            return false;

        return true;
    }

    public bool IsMatch(string url)
    {
        return IsMatch(new Uri(url));
    }
}

Answer 1

General considerations

You are writing a "denylist". There's some clean and simple denylist rules that can help rule out a lot of things already. One of these rules is denying URLs based on their scheme alone.

A few examples for schemes that you may not have considered:
steam://, file://, sftp://, chrome://, about://

Most of these are URLs you most likely want to block. In that case "inverted matching" behaviour is really useful.

---

Furthermore denylists usually tend to be used for "overblocking out of caution". I'm not sure how that's for your exact use-case, but if you want that to happen, making your patterns as general as possible is a great idea :)

---

URI vs. URL:
Something to additionally keep in mind here: URI stands for "Uniform Resource Identifier" and URL for "Uniform Resource Locator". Your code specifically exposes a Uri, not a Url overload for matching. This implies that one should be able to blacklist things like the following URIs:

[email protected]:username/repository.git
mailto:[email protected]
irc://#c-sharp@freenode

If your class is only intended to deal with URLs, you should only expose an overload for Url :)

Personal preferences

I personally very much prefer to use factory methods over constructors to validate arguments. YMMV, but I expect constructors to not throw exceptions.

I also prefer to add braces wherever they delimit a block. I can understand the omission for the argument checking though.

Implementation considerations

I like that you're using a lot of intermediate variables. Some minor naming nitpicks could be had there, but I think this code will get the most benefit from using tuples:

var (scheme, hostAndPath) = SplitSchemeHost(pattern);
var (host, path) = SplitHostPath(hostAndPath);

This allows you to push the index matching behaviour into these methods and keep it out of the constructor. (Minor nitpick: 3 is a magic number in the hostAndPath creation)

Aside from that this code is very clean already. I'd maybe not create the patterns within the interpolated strings, but given the simplicity here it seems appropriate to do so.