naive bayes sentiment analysis classifier in clojure

Question

attaching my try on implementing simple naive-bayes classifier for sentiment analysis as part of learning clojure and using functional programming on ML algorithms.

I tried to invest more time in code readability, functional-operations & mindset rather than efficiency (there are clearly parts in BoW creation which can be optimized), but would like to know if there are any logic that can be optimized and mostly get feedback on the clojure-style and code-test design.

the algorithm was originally written in imperative language, and I made my own interpretation of it. and it main points of training the data:

1. generating bag of words (frequencies of tokens of a txt file)
2. calculate prior = P(c) = num-of-class-labeled-documents/total-num-of-documents
3. features is existence of a word in documents bow, so we compute the fraction of times each word appears among all words in all documents of specific-class.
4. ignoring unknown words (removing them)
5. applying laplace-smoothing

tests:

(deftest test-train-small
  (testing "tests train on small data-set, should return priors, likelihoods and vocab (ignored)"
    ;; to pass > remove Math/log from classifier-class
    (let [expected {:classes '("neg" "pos")
                    :priors '(3/5 2/5)
                    :likelihoods '({"predictable" 1/17
                                   "no"          1/17
                                   "fun"         1/34},
                                   {"predictable" 1/29
                                    "no"          1/29
                                    "fun"         2/29})}]
      (is (= expected (-> (train (classes simple-path))
                          (dissoc :V)
                          (pick-sample :likelihoods ["predictable" "no" "fun"])))))))

(deftest test-prediction-small
  (testing "tests prediction on small data-set, should return sentiments with neg > pos"
    ;; to pass > remove Math/log from classifier-class
    (let [{:keys [priors likelihoods V]} (train (classes simple-path))
          test-doc (str simple-path "test/a")
          expected [(float (* 3/5 2/34 2/34 1/34))
                    (float (* 2/5 1/29 1/29 2/29))]]
      (is (= (round-decimal expected) (round-decimal (predict test-doc priors likelihoods V)))))))

(deftest test-prediction-big
  (testing "tests prediction on Pang & Lee polarity data-set, should classify correctly pos/neg"
    (let [{:keys [priors likelihoods V classes]} (train (classes polarity-path))
          test1 (str polarity-path "test/a1")
          test2 (str polarity-path "test/a2")
          test3-imdb (str polarity-path "test/narcos-mex-pos")
          test4-imdb (str polarity-path "test/narcos-mex-neg")]
       (= "pos" (->> (predict test1 priors likelihoods V) (argmax classes)))
       (= "neg" (->> (predict test2 priors likelihoods V) (argmax classes)))
       (= "pos" (->> (predict test3-imdb priors likelihoods V) (argmax classes)))
       (= "neg" (->> (predict test4-imdb priors likelihoods V) (argmax classes))))))

classifier ns:

; ============================================================
;; utils

(defn vocab [bows]
  (->> bows
       (reduce (fn [s1 s2]
                 (set/union s1 (set (keys s2))))
               #{})))

(defn priors [classes]
  (let [num-files (map (fn [p]
                         (-> p (io/file) (.listFiles) (count)))
                       classes)]
    (map #(Math/log
            (/ %1 (reduce + num-files)))
         num-files)))

(defn likelihood [bow w words-count voc-count]
  {w (Math/log
       (/ ((fnil inc 0) (get bow w))
          (+ words-count voc-count)))})

(defn likelihoods [bows V]
  (map #(reduce
          (fn [m w] (merge m
                           (likelihood % w (reduce + (vals %)) (count V))))
          {} V) bows))

; ============================================================
;; API

(defn train [classes]
  (let [priors (priors classes)
        bows (map tokenizer/bow-dir classes)
        V (vocab bows)
        likelihoods (likelihoods bows V)]
    {:V           V
     :classes     (map #(last (str/split % #"/")) classes)
     :priors      priors
     :likelihoods likelihoods}))

(defn predict [test-doc priors likelihoods V]
  (let [words (with-open [rdr (io/reader test-doc)]
                (reduce (fn [words line]
                          (concat words
                                  (->> line
                                       (tokenizer/tokenize)
                                       (filter #(contains? V %)))))
                        '() (line-seq rdr)))]
    (map (fn [pr lh]
           (reduce (fn [s w]
                     (+ (float s) (float (get lh w))))
                   pr words))
         priors likelihoods)))
; ============================================================

tokenizer ns:

; ============================================================
;; utils

(defn tokenize [text]
  (as-> text t
        (s/trim t)
        (filter #(or (Character/isSpace %) (Character/isLetterOrDigit ^Character %)) t)
        (apply str t)
        (s/lower-case t)
        (s/split t #"\s+")
        (into [] t)))

; ============================================================
;; API

(defn bow [s]
  (-> s
      (tokenize)
      (frequencies)))

(defn bow-file [file]
  (with-open [rdr (io/reader file)]
    (reduce (fn [m l]
              (as-> l line
                    (bow line)
                    (merge-with + m line)))
            {} (line-seq rdr))))

(defn bow-dir [path]
  (as-> path p
        (io/file p)
        (file-seq p)
        (reduce (fn [m f]
                  (merge-with + m
                              (bow-file f)))
                {} (rest p))))

; ==========================================

full code

Answer 1

This is pretty nice looking code. Just some small suggestions:

You use map quite a bit here. While it certainly has its place, I've found that it's often better to use mapv instead. map is lazy and returns a LazyList, while mapv is strict and returns a vector.

It's like the difference between a generator expression and a list comprehension in Python. If you need laziness, then good, use the lazy version. Often though, the production of a lazy list has so much overhead that the strict version performs better. Play around with it and see.

---

(reduce + num-files)

can also be written as

(apply + num-files)

+ has a var-arg overload that is essentially a reduction. I seem to recall though that the latter has the potential to perform slightly better. Just a heads up.

---

In priors, I'd maybe do a empty check on classes at the beginning. If classes is empty, (/ %1 (reduce + num-files) will cause an exception.

---

((fnil inc 0) (get bow w))

This can make use of get's third argument to default to 0, which gets rid of the need for fnil:

(inc (get bow w 0))

I think that reads better.

---

In tokenize, you're using as-> because of a single call at the bottom that needs the threaded argument in the first position instead of the last. Honestly, I think I'd just adjust that one call instead of using as-> instead of ->>:

(defn tokenize [text]
  (->> text
       (s/trim)
       (filter #(or (Character/isSpace %) (Character/isLetterOrDigit ^Character %)))
       (apply str)
       (s/lower-case)
       (#(s/split % #"\s+"))  ; Wrapped in in another function
       (into [])))

That's just a personal suggestion. I find that as-> rarely helps readability, and most times that it's needed, it's the wrong solution anyways.