I have made a small script which grabs data into a map from 80 CSV files and calculates some statistics like average, standard deviation etc. It's also adding some additional data to map from filename.
Can you please check if it is a correct way to do so? The idea is to get a list of files in the folder using file-seq
, map this to function which reads file lazy, while skipping the first line, convert all text to decimals using read-string
, multiply all the data by 1e9 and then calculates all the necessary statistics.
The script works perfectly. I am just concerned about the right style to write programs in Clojure since I am pretty new to it.
Data in CSV files is just numbers in scientific format like "1.721e-9" written in one column.
(def data
(map (fn [fsc]
(->> (io/reader fsc)
(line-seq)
(rest)
(map read-string)
(map #(* % 1e9))
((fn [se]
(let [x se
x2 (map #(* % %) se)
n (count se)
sum-x (reduce + x)
sum-x2 (reduce + x2)
average (/ sum-x n)
variance (-
(/ sum-x2 n)
(math/expt average 2))]
(merge {:n n
:average average
:variance variance
:st-dev (math/sqrt variance)
:st-dev-sample (math/sqrt
(/ (* n variance)
(- n 1)))}
(-> (.getName fsc)
(clojure.string/split #"\s")
((partial zipmap [:type :color :voltage :temperature]))
(#(assoc % :voltage (read-string (re-find #"[+-]?\d+" (% :voltage)))))
(#(assoc % :temperature (read-string (% :temperature))))
)))))))
(->> (clojure.java.io/file "data")
(file-seq)
(rest))))