Simple, functional URL parsing library in Clojure

Question

I recently wrote some code to handle parsing of URLs, even if those URLs might not be one hundred percent perfectly well-formed. (This constraint made using Java's URL class impossible.)

The resulting, relatively short library is presented below.

(ns fuzzy-urls.url
  (:require [clojure.string :as string]
            [clojure.walk :refer (keywordize-keys)]))

(defn- maybe
  "Helper function for creating predicates that might be nil."
  [pred]
  (some-fn nil? pred))

; A Url is composed of seven fields:
;
; http://sky@www:801/cgi-bin/finger?name=shriram&host=nw#top
; {1-}   {2} {3} {4}{---5---------} {----6-------------} {7}
;
; 1 = scheme, 2 = user, 3 = host, 4 = port,
; 5 = path (two elements), 6 = query, 7 = fragment
;
; The types of these fields are as follows:
; - scheme: string / nil
; - user: string / nil
; - host: string / nil
; - port: int / nil
; - path: [string]
; - query: {keyword, string / nil}
; - fragment: string / nil

(defrecord Url [scheme user host port path query fragment])
(defn url? [x] (instance? Url x))

(defn make-url
  "Convenience constructor for Url."
  [& {:keys [scheme user host port path query fragment]
      :or {path []
           query {}}}]
  (->Url scheme user host port path query fragment))

; This regex is taken mostly verbatim from RFC 3986, Appendix B.
; It has been modified slightly to disable capturing of uninteresting groups.
; The resulting capture groups correspond to the following fields:
; (1) scheme (without trailing colon)
; (2) user+host+port (no leading or trailing slashes)
; (3) path (with leading slash)
; (4) query string (without leading question mark)
; (5) fragment (without leading hash mark)
(def ^:private url-pattern
  #"^(?:([^:/?#]+):)?(?://([^/?#]*))?([^?#]*)(?:\?([^#]*))?(?:#(.*))?")

(def ^:private host-pattern
  #"^(?:([^@]*)@)?(.+?)(?::(\d+))?$")


(defn- path-string->list
  "Converts a path string (which may be nil) to a vector of path elements."
  [path]
  {:pre  [((maybe string?) path)]
   :post [(vector? %) (every? string? %)]}
  ; drop the first element of the path because it contains a leading slash
  (into [] (and path (rest (string/split path #"/")))))

(defn- query-map?
  "A predicate for determining if a map is a valid representation of a query string."
  [query]
  {:pre [(map? query)]}
  (every?
    (fn [[k v]]
      (and (keyword? k)
           ((maybe string?) v)))
    query))

(defn- query-string->map
  "Converts a query string (which may be nil) to a map representation."
  [query]
  {:pre  [((maybe string?) query)]
   :post [(map? %) (query-map? %)]}
  (if-not query
    {}
    (let [elements (string/split query #"&")
          pairs (map #(string/split % #"=" 2) elements)]
      (keywordize-keys  (into {} ; this is necessary when v is nil
                              (for [[k v] pairs] [k v]))))))

(defn- query-map->string
  "Converts a map representation of a query string to a string."
  [query]
  {:pre  [(query-map? query)]
   :post [(string? %)]}
  (string/join
    "&"
    (for [[k v] query]
      (str (name k) "=" v))))

(defn string->url
  "Parses a string into a url. Malformed or incomplete urls are supported,
  and the relevant fields will be left nil."
  [string]
  {:pre  [(string? string)]
   :post [(url? %)]}
  (let [[_ scheme user+host+port path query fragment]
        (re-matches url-pattern string)
        [_ user host port]
        (if user+host+port (re-matches host-pattern user+host+port) [])]
    (->Url (some-> scheme string/lower-case)
           user
           (some-> host string/lower-case)
           (some-> port Integer/parseInt)
           (path-string->list path)
           (query-string->map query) fragment)))

(defn url->string
  "Gets the string representation of a url. Missing portions are not included
  in the result."
  [url]
  {:pre  [(url? url)]
   :post [(string? %)]}
  (let [{:keys [scheme user host port path query fragment]} url]
    (str
      (some-> scheme (str "://"))
      (some-> user (str "@"))
      host
      (some->> port (str ":"))
      (when-not (empty? path) (str "/" (string/join "/" path)))
      (when-not (empty? query) (str "?" (query-map->string query)))
      (some->> fragment (str "#")))))

I built this with a set of design goals in mind:

• Concise and readable. This is really my main priority here. I'm trying to write pretty code that's easy to work with.
• Functional. This follows somewhat from the first point. I wanted my library to be purely functional (not like that's a terribly difficult goal for a URL parser).
• Safe. Most of the functions have :pre and :post assertions to ensure type safety as long as assertions are enabled.
• Idiomatic. I'm a fairly new Clojure programmer, and my primary experience in writing functional code has been in Scheme and Racket. Clojure has its own set of constructs and paradigms, and I wanted to make my code Clojure-y, not simply Scheme written in Clojure.

All of the code is available on GitHub, which includes some additional functionality not included in this request for review. It could be helpful to look at the example usage to get a feel for how this code is intended to be used.

Any suggestions on how I could improve the code while maintaining the above set of goals would be appreciated, with an emphasis on making the code idiomatic. I'd like to ensure I'm using the language correctly.

Answer 1

When you create a Url record, you get a map->Url function that does what your make-url does, but in a more idiomatic way.

In path-string->list, instead of using and, consider using when.

(when path            
  (vec (rest (string/split path #"/"))))

I'm assuming you really, really, really wanted a vector there. (move vec outside the when to be more like the original code, if that's important, but it shouldn't be)

In query-map?, what does that anonymous function do? Since it's not obvious, I suggest lifting it out and giving it a name so that query-map? becomes better.

(defn is-keyword-string-pair [[k v]]
  (and (keyword? k)
       ((maybe string?) v)))

Most people would find ((maybe string?) v) a bit odd. Maybe (def maybe-string (maybe string?)) and use it there.

Here's some ideas for query-string->map. I'd probably also switch this to use zipmap for clarity:

(defn query-pairs [query]
  (when (not-empty query)
    (let [elements (string/split query #"&")]
      (map #(string/split % #"=" 2) elements))))

(defn query-string->map [query]
  (into {}
        (for [[k v] (query-pairs query)]
          [(keyword k) v])))

Instead of (if user+host+port (re-matches host-pattern user+host+port) []) just use (when user+host+port (re-matches host-pattern user+host+port)).

(when-not (empty? path) ...) could be (when (not-empty path) ...).

I personally prefer (when scheme (str scheme "://)) to (some-> scheme (str "://"), but many Clojurists seem to prefer the latter.