attaching my try on implementing simple naive-bayes classifier for sentiment analysis as part of learning clojure and using functional programming on ML algorithms.
I tried to invest more time in code readability, functional-operations & mindset rather than efficiency (there are clearly parts in BoW creation which can be optimized), but would like to know if there are any logic that can be optimized and mostly get feedback on the clojure-style and code-test design.
the algorithm was originally written in imperative language, and I made my own interpretation of it. and it main points of training the data:
- generating bag of words (frequencies of tokens of a txt file)
- calculate prior = P(c) = num-of-class-labeled-documents/total-num-of-documents
- features is existence of a word in documents bow, so we compute the fraction of times each word appears among all words in all documents of specific-class.
- ignoring unknown words (removing them)
- applying laplace-smoothing
tests:
(deftest test-train-small
(testing "tests train on small data-set, should return priors, likelihoods and vocab (ignored)"
;; to pass > remove Math/log from classifier-class
(let [expected {:classes '("neg" "pos")
:priors '(3/5 2/5)
:likelihoods '({"predictable" 1/17
"no" 1/17
"fun" 1/34},
{"predictable" 1/29
"no" 1/29
"fun" 2/29})}]
(is (= expected (-> (train (classes simple-path))
(dissoc :V)
(pick-sample :likelihoods ["predictable" "no" "fun"])))))))
(deftest test-prediction-small
(testing "tests prediction on small data-set, should return sentiments with neg > pos"
;; to pass > remove Math/log from classifier-class
(let [{:keys [priors likelihoods V]} (train (classes simple-path))
test-doc (str simple-path "test/a")
expected [(float (* 3/5 2/34 2/34 1/34))
(float (* 2/5 1/29 1/29 2/29))]]
(is (= (round-decimal expected) (round-decimal (predict test-doc priors likelihoods V)))))))
(deftest test-prediction-big
(testing "tests prediction on Pang & Lee polarity data-set, should classify correctly pos/neg"
(let [{:keys [priors likelihoods V classes]} (train (classes polarity-path))
test1 (str polarity-path "test/a1")
test2 (str polarity-path "test/a2")
test3-imdb (str polarity-path "test/narcos-mex-pos")
test4-imdb (str polarity-path "test/narcos-mex-neg")]
(= "pos" (->> (predict test1 priors likelihoods V) (argmax classes)))
(= "neg" (->> (predict test2 priors likelihoods V) (argmax classes)))
(= "pos" (->> (predict test3-imdb priors likelihoods V) (argmax classes)))
(= "neg" (->> (predict test4-imdb priors likelihoods V) (argmax classes))))))
classifier ns:
; ============================================================
;; utils
(defn vocab [bows]
(->> bows
(reduce (fn [s1 s2]
(set/union s1 (set (keys s2))))
#{})))
(defn priors [classes]
(let [num-files (map (fn [p]
(-> p (io/file) (.listFiles) (count)))
classes)]
(map #(Math/log
(/ %1 (reduce + num-files)))
num-files)))
(defn likelihood [bow w words-count voc-count]
{w (Math/log
(/ ((fnil inc 0) (get bow w))
(+ words-count voc-count)))})
(defn likelihoods [bows V]
(map #(reduce
(fn [m w] (merge m
(likelihood % w (reduce + (vals %)) (count V))))
{} V) bows))
; ============================================================
;; API
(defn train [classes]
(let [priors (priors classes)
bows (map tokenizer/bow-dir classes)
V (vocab bows)
likelihoods (likelihoods bows V)]
{:V V
:classes (map #(last (str/split % #"/")) classes)
:priors priors
:likelihoods likelihoods}))
(defn predict [test-doc priors likelihoods V]
(let [words (with-open [rdr (io/reader test-doc)]
(reduce (fn [words line]
(concat words
(->> line
(tokenizer/tokenize)
(filter #(contains? V %)))))
'() (line-seq rdr)))]
(map (fn [pr lh]
(reduce (fn [s w]
(+ (float s) (float (get lh w))))
pr words))
priors likelihoods)))
; ============================================================
tokenizer ns:
; ============================================================
;; utils
(defn tokenize [text]
(as-> text t
(s/trim t)
(filter #(or (Character/isSpace %) (Character/isLetterOrDigit ^Character %)) t)
(apply str t)
(s/lower-case t)
(s/split t #"\s+")
(into [] t)))
; ============================================================
;; API
(defn bow [s]
(-> s
(tokenize)
(frequencies)))
(defn bow-file [file]
(with-open [rdr (io/reader file)]
(reduce (fn [m l]
(as-> l line
(bow line)
(merge-with + m line)))
{} (line-seq rdr))))
(defn bow-dir [path]
(as-> path p
(io/file p)
(file-seq p)
(reduce (fn [m f]
(merge-with + m
(bow-file f)))
{} (rest p))))
; ==========================================