3
\$\begingroup\$

I'm trying to learn Clojure recently and I thought writing a simple web app would be a good way to dive in.

This function gets the list of alive threads from the API and reduces, filters and maps until I'm left with a list of image names (which I can append to "http://i.4cdn.org/wg/" to get the URL's.).

This is the first time I'm using defer/@ so I'm not that sure if I used them correctly.

Here is the code:

(defn scrape []
  (let
    [{:keys [status hey body error] :as resp}
      @(http/get "http://a.4cdn.org/wg/threads.json")]
    (->>
      (json/read-str (resp :body))
      (map #(%1 "threads"))
      (reduce concat [])
      (map #(%1 "no"))
      (map #(str "http://a.4cdn.org/wg/thread/" %1 ".json"))
      (map http/get)
      (map deref)
      (map #(%1 :body))
      (map json/read-str)
      (map #(%1 "posts"))
      (reduce concat [])
      (filter #(not (nil? (%1 "tim"))))
      (map #(str (%1 "tim") (%1 "ext")))
      (print))))
\$\endgroup\$
1
\$\begingroup\$
  • Don't need :as resp if only body is ever used.
  • I'd also drop the :keys destructuring if it's not reused.
  • While couple of those maps make sense, this sequence is also pretty repetitive, I'd suggest condensing repeated invocations into a single call at least.
  • The (not (nil? ...)) is probably the same as the value itself, so just (filter #(%1 "tim")) should suffice.
  • The function should most likely return the list instead of printing it so the files can be examine further in another part of the code.
  • Maybe use mapcat instead of the map/reduce/concat combination?

The deref is fine I think, it also depends on how the calls should be batched together (or not).

All in all:

(ns fourchan.core
  (:require [clojure.data.json :as json])
  (:require [org.httpkit.client :as http]))

(defn scrape []
  (->>
   (json/read-str (@(http/get "http://a.4cdn.org/wg/threads.json") :body))
   (mapcat #(%1 "threads"))
   (map #(http/get (str "http://a.4cdn.org/wg/thread/" (%1 "no") ".json")))
   (mapcat #((json/read-str ((deref %1) :body)) "posts"))
   (filter #(%1 "tim"))
   (map #(str (%1 "tim") (%1 "ext")))))
\$\endgroup\$
  • \$\begingroup\$ thanks, is there any way to make this code non-blocking without modifying the ->> sturcture? \$\endgroup\$ – ulucs Nov 24 '16 at 11:51
  • \$\begingroup\$ What exactly do you want to not block? As the simplest case just wrap the whole block in future maybe? \$\endgroup\$ – ferada Nov 24 '16 at 12:14
1
\$\begingroup\$

Some suggestion:

  1. Use :key-fn in json/parse to simplify some of the mappings
  2. For single argument anonymous function, use % instead of %1
  3. Ideally, you should use transducer to compose your mapping functions. But in this case, you are actually taking advantage of the implementation detail of lazy sequence to parallelize your async http requests. Where each realisation of a lazy sequence chunk allows 32 concurrent async requests to be opened (when you pipe the mapping of http/get to deref)

E.g. this takes ~4 seconds in my machine:

(defn scrape []
  (let [{:keys [body]} @(http/get "http://a.4cdn.org/wg/threads.json")]
    (->> (json/read-str body :key-fn keyword)
         (mapcat :threads)
         (map :no)
         (map #(str "http://a.4cdn.org/wg/thread/" % ".json"))
         (map http/get)
         (map deref)
         (map :body)
         (map #(json/read-str % :key-fn keyword))
         (mapcat :posts)
         (filter :tim)
         (map #(str (% :tim) (% :ext))))))

Turning into transducer (which avoids realisation of intermediary lazy sequences, i.e. no batching) takes ~24 seconds:

(defn scrape []
  (let
      [{:keys [body]} @(http/get "http://a.4cdn.org/wg/threads.json")
       xf             (comp
                       (mapcat :threads)
                       (map :no)
                       (map #(str "http://a.4cdn.org/wg/thread/" % ".json"))
                       (map http/get)
                       (map deref)
                       (map :body)
                       (map #(json/read-str % :key-fn keyword))
                       (mapcat :posts)
                       (filter :tim)
                       (map #(str (% :tim) (% :ext))))]
    (sequence xf (json/read-str body :key-fn keyword))))

If you really need to remove the blocking of http requests without relying on the implementation details of lazy sequence, you should consider core.async. A quick implementation using core.async achieves ~1.2 second in my case.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.