This is my very first foray into Clojure (I'm normally a Python-pushing data-type). I'm trying to create a simple term-document matrix as a vector of vectors, out of a vector of strings.
For those who aren't into textmining, the term-document matrix is a dataset in matrix form where the column names represent every word in a set of documents, each row is a document, and each cell is the number of times a given word appears in a given document.
This is the very first step in what I hope to be a useful text data-cleaning library, as well as a clojure learning project. After the basics are nailed down, I want to add functionality like n-grams, stemming, removing sparse terms, etc. etc. My ultimate goals involve performance, so I want to optimize this beginning part within an inch of its life right from the start in order to build on it later.
I'm also trying to minimize dependencies (right now there are none), though I'm willing to use incanter
or clojure.core.matrix
if there are big performance gains to be gotten thereby.
So obviously I have a long way to go, but here are some questions on this initial step:
Is this "good clojure?" I tried to stick to sort of basic functional programming practice, composing lots of short functions with discrete behavior and such. But I'm not yet sure what the norms are otherwise.
How do I optimize this? The particular parts of the existing code that smell funny to me, performance-wise, are:
terdocmmap
: there's gotta be a more efficient way of handling the sorting here than building a bunch of sparse maps then sorting them all. Ideally I'd like to build them in sorted form from the start somehow.termdocmatrix
: the maps -> sequences -> vectors conversion seems really wasteful; I'd like to come up with a more efficient way.
I'm not worrying about namespace and project structure at this stage.
(require '[clojure.string :as str])
(require '[clojure.walk :as walk])
(defn whitesplit
"split a vector of string into vector of vectors of strings on whitespace"
[docs]
(map #(str/split % #" ") docs))
(defn stringcounts
"count frequencies of strings in vector of vectors of strings"
[stringvecs]
(map frequencies stringvecs))
(defn liststrings
"list all strings in doc set"
[stringvecs]
(distinct
(apply concat stringvecs)))
(defn makezeroes
[stringlist]
(zipmap stringlist (repeat 0)))
(defn expandcounts
"based on strings in all docs, fill counts with 0 for unused strings in each single doc"
[zeroes counts]
(map #(merge-with + % zeroes) counts))
(defn bigmap
"split vector of docs by spaces then make zero-filled map of counts"
[docs]
(let [stringvecs (whitesplit docs)]
(expandcounts
(-> stringvecs liststrings makezeroes)
(-> stringvecs stringcounts))))
(defn terdocmmap
"make a sorted document-term-map of vector of docs with keywords"
[docs]
(walk/keywordize-keys ; this is mainly for later flexibility
(map #(into (sorted-map) %) (bigmap docs))))
(defn tdseqs
"convert document-term-map into sequence of sequences"
[tdmap]
(cons
(keys (first tdmap))
(map vals tdmap)))
(defn nestvecify
"sequence of sequences --> vector of vectors"
[seqofseq]
(into [] (map #(into [] %) seqofseq)))
(defn termdocmatrix
"make document term matrix as vector of vectors from vector of docs"
[docs]
(-> docs terdocmmap tdseqs nestvecify))
Example input and output:
(termdocmatrix ["this is a cat" "this is a dog" "woof and a meow" "woof woof woof meow meow words"])
Produces:
[[:cat :is :this :words :dog :and :meow :woof :a] [1 1 1 0 0 0 0 0 1] [0 1 1 0 1 0 0 0 1] [0 0 0 0 0 1 1 1 1] [0 0 0 1 0 0 2 3 0]]