Counting words from stored .md files

Question

The following searches recursively for all the mark down files - i. ending with the extension .md - inside a folder. It then stores the text of the files in an array. Finally, it sums their word count using a wordCount() function (from the compromise package).

import nlp from 'compromise'
import fs from 'fs'
import path from 'path'

const filePaths = searchRecursive(`${__dirname}/test`, '.md')

const fileTexts = filePaths.map((filePath: string) => {
    return fs.readFileSync(filePath, 'utf-8')
}) // -> ['one two', 'one two three']

let totalWordCount = 0

fileTexts.forEach((fileText: string) => {
    totalWordCount += nlp(fileText).wordCount()
})

console.log(totalWordCount) // -> 5

function searchRecursive (dir: string, pattern: string) {
    ...
}

I've heard that forEach shouldn't be used in functional programming. How can I modify this code so it doesn't use forEach?

Answer 1

How to modify this code, so it doesn't use forEach?

One way to achieve this is using Array.prototype.reduce():

const totalWordCount = fileTexts.reduce((big: sum, fileText: string) => {
    return sum + nlp(fileText).wordCount()
}, 0)

Notice that totalWordCount is assigned once so const can be used which helps avoid accidental re-assignment and other bugs.

If fileTexts is only used to calculate totalWordCount then the two loops could be combined into a single loop (e.g. with reduce()).

To learn more about functional techniques try these functional exercises.

Answer 2

Another suggestion: with TypeScript, you only need to note the type of a parameter when TypeScript can't infer it itself. You might find it easier to read and write code when you avoid explicitly typing functions except when necessary for the TS to compile, or when the type of a return value isn't clear at a glance. In other words, if I were you, I'd switch out:

.map((filePath: string) => {

with

.map((filePath) => {

and, if you did have to use forEach somewhere else, switch

fileTexts.forEach((fileText: string) => {

with

fileTexts.forEach((fileText) => {

It cuts down on a bit of syntax noise.

---

On another note, you're currently waiting for each file to be read in serial due to the readFileSync. If you have 100 files, and it takes 0.1 seconds to read each file, your script will have to run for at least 10 seconds. It would be more elegant to switch out readFileSync with a non-blocking version, and then use Promise.all. Luckily, modern versions of node have fs.promises, which are Promise-based version of the non-blocking functions:

Promise.all(
  filePaths.map(
    filePath => fs.promises.readFile(filePath, 'utf-8')
      // if nlp takes more than an instant to run,
      // make sure to chain it onto the Promise here so it can run ASAP
      // rather than below, which could result in a lot of CPU load at once
      .then(fileText => nlp(fileText).wordCount())
  )
)
  .then((wordCounts) => {
    const totalWordCount = wordCounts.reduce((a, b) => a + b, 0);
    // done
  })
  .catch((err) => {
    // handle errors
  });

Answer 3

I'll leave the alternatives of the code to the other answers, but just to make a comment about functional programming:

Functional programming has three core tenets as I see it:

• Functions are first class citizens (you can pass them as arguments, put them in lists)
• Functions are deterministic - (for a function call of the same arguments, it always returns the same thing).
• Functions don't cause side effects.

Now with the second two points it's apparent that in a purely functional context, some things simply are not possible:

• A function that generates a random number
• A function that gets the current time
• A function that fetches some data from a database or API
• A function that writes data to a file.

So the point here is - even if you're adopting a functional paradigm, at some point, unless your program is just a simple 'sort this list' type program, it likely needs to have side effects and encounter nondeterministic behavior.

In your case, reading from a file is nondeterministic behavior.

So the question is, in a functional paradigm, how do you deal with this?

Some resources I recommend looking at:

https://stackoverflow.com/questions/330371/are-databases-and-functional-programming-at-odds

Basically the suggestion I make is that you isolate your nondeterministic code into functions that just do that non-deterministic behavior. You then pass those functions into the logic functions don't need to know if the function is deterministic or not.

From your example, lets say you have this:

function getFileTexts(filePaths) {
    const fileTexts = filePaths.map((filePath: string) => {
        return fs.readFileSync(filePath, 'utf-8')
    })
    return fileTexts; 
}

So the problem with this is that the fs.readFileSync is nondeterministic.

What you can do pass the 'read file' function in, as an argument.

function getFileTexts(filePaths, readFileFn) {
    const fileTexts = filePaths.map((filePath: string) => {
        return readFileFn(filePath); 
    })
    return fileTexts; 
}

And you would call this with:

const fileTexts = getFileTexts(filePaths, (filePath) =>  fs.readFileSync(filePath, 'utf-8')); 

Now, even though in the real world that function isn't going to deterministic, at least now it's possible for it to be.

That is, in your tests for example, you pass an alternative, purely deterministic function like:

const fileTexts = getFileTexts(filePaths, () => 'foo bar biz');