High CPU and memory usage iterating over 300K objects

Question

I am trying to run a series of keywords against a series of categories and then within those categories there are some options. So I have ended up doing a forEach over for of over a every method and when dealing with a lot of entries node consumes way too much memory.

When dealing with 300K objects from a 29MB csv converted to JSON file and processed, pm2 monitor says node peaks to 4GB RAM usage and 200% CPU is this normal?

Here is an example of the code with a minial data sample

const keywords = [
  {
    Keyword: 'foo',
    URL: 'https://www.facebook.co.uk'
  },
  {
    Keyword: 'foo',
    URL: 'https://www.twitter.co.uk/blue'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.facebook.co.uk/twitter'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.apple.co.uk/green'
  }
]

const categories = [
  {
    name: 'Tech',
    options: [
      {
        method: 'include',
        regex: 'facebook'
      },
      {
        method: 'exclude',
        regex: 'twitter'
      }
    ]
  },
  {
    name: 'Green',
    options: [
      {
        method: 'include',
        regex: 'green'
      }
    ]
  }
]

keywords.forEach((obj) =>
  categories.forEach(({name, options}) => 
    obj[name] = options.every(({method, regex}) => method === 'include' ? obj.URL.includes(regex) : !obj.URL.includes(regex))))

console.log(keywords)

.as-console-wrapper { max-height: 100% !important; top: 0; }

Answer 1

Performance

Whenever possible avoid strings. Why ?

JS strings are immutable

Passing a string to a function or assigning it to a variable requires the string to be copied^[*1]. This adds a lot of processing overhead (memory processing, (assignment, and GC), and string iteration) which can be avoided.

^{[*1] See update at bottom of answer.}

For example using your function

function process(keywords, categories) {
  keywords.forEach(obj =>
    categories.forEach(({name, options} => 
      obj[name] = options.every(({method, regex}) => 
        method === 'include' ? obj.URL.includes(regex) : !obj.URL.includes(regex))
    )
  );
}

The two inner loops create new strings for each iteration. name in the outer loop categories.forEach(( and method and regex in the loop options.every(({

As these string are all stored as references in objects there is no need to copy the strings to new variables. Just use the references directly as follows...

 function process(keywords, categories) {
   keywords.forEach(obj =>
     categories.forEach(cat => 
       obj[cat.name] = cat.options.every(opt => 
         opt.include ? obj.URL.includes(opt.regex) : !obj.URL.includes(opt.regex))
     )
   );
 }

Avoid state strings

Using string to store simple states is much slower than using simpler types like boolean or number.

For example you use the expression opt.method === "include" to check the type of test to do on URL. The negative (false) for opt.method === "include" (method = "exclude") is quicker as the compare fails on the first character "e" !== "i" . However the match needs to iterate each of the 7 characters to find true.

JS has no clue to help check the match (the strings include and exclude are the same length).

As there are only two states, include or exclude, you can use a boolean state.

Example the option objects can be

options: [
  { include: true, regex: 'facebook' }, // includes
  { include: false, regex: 'twitter' }  // excludes
]

And then the inner test can be a constant (and fast) complexity

opt.include ? obj.URL.includes(opt.regex) : !obj.URL.includes(opt.regex))

If you can not create the boolean for categories as stored (in JSON). Process the options once outside the function process

function optimzeCats(categories) {
    categories.forEach(cat => cat.options.forEach(opt => 
         opt.include = opt.method === 'include'
    ));
}

Having to do so will of course reduce the gain gained.

Reduce scope searches

Using node (V8) means that there is an additional overhead each (scope step) you need to use a variable outside the current scope.

In your code the outer loop keywords.forEach(obj puts the variable obj 2 scope steps above its use in the inner loop. obj.URL.includes(

As (I assume) the number of keywords greatly outnumbers the number of cats changing the scope distance to obj will give another worthwhile performance gain. This can be done by swapping the order of the first two outer loops.

 function process(keywords, categories) {
   categories.forEach(cat => 
     keywords.forEach(obj =>
       obj[cat.name] = cat.options.every(opt => 
         opt.method === 'include' ? obj.URL.includes(opt.regex) : !obj.URL.includes(opt.regex))
     )
   );
 }

Further optimizations

All of the above should give up to 15% performance gain and a unknown but worthwhile reduction in memory use. (Note only in regard to processing, as I don't know how you handle the JSON string)

There are likely many more optimizations however these will depend very much on what is being stored in both data structures and how the results are expected to be used.

Update

Correction...

After comments and then some research. Strings are not copied (duplicated) when assigned but rather a map reference (hash) to the unique string within the global context represents the string.

Answer 2

29MB csv converted to JSON

Had to do something similar and process almost 30MB worth of JSON (in PHP, but same problem). While 30MB doesn't sound much, it is A LOT when it's parsed and loaded entirely as an object in memory for your app. It's not a 1:1 size of JSON = size in memory, as the object in memory contains metadata to describe object relationships, types, etc. We experienced a lot of OOM errors, long-running processes, random crashing, etc.

What we ended up doing is preprocess that JSON into smaller files using jq (it's is like sed for JSON data). We ran the huge JSON through jq and picked out only parts of the file we actually needed for the operation and simplified the resulting structure. The result is JSON which is now just kilobytes compared to the megabytes it originally was.

An alternate route would be to load up this CSV in something like a database (e.g. MariaDB) or a search index (e.g. Solr), and use its capability of structuring the data optimally for really complex but fast and memory-efficient searches.

If you really have to operate the search in Node, then I would suggest flattening the data (i.e. just an array of objects, no nested objects in properties). That's because the main culprit here is that three-level loop. Every time you have to add a level, you're adding a loop which multiplies the amount of steps done for one item. Just flattening it reduces the operation to just how many items there are in that array.

Answer 3

This is a half-baked answer, but you might find it useful.

As far as I can tell, each keyword is processed independently. For processing one keyword you need all the categories but you don't need any other keywords. (If that's not true, then my suggestion won't be of any use).

Instead of processing your keywords into a complete JSON object, you could pre-process it into a JSON lines format so each line of the file is a valid JSON object (one line per keyword). Then, you can read the keywords one line at a time, process one line, and write that line to disk; then proceed to reading the next line. In this way, the total memory needed by the application is the memory needed to store the categories plus the memory of one keyword.

I don't know offhand if the JavaScript file IO streams API handles memory this way -- that would be the simplest. However, if it doesn't, I'm sure you could find a lower level API to read one line at a time.

Answer 4

In his answer to the OP's question, Blindman67 said:

Passing a string to a function or assigning it to a variable requires the string to be copied.

In a comment, radarbob said:

Passing a string to a function or assigning it to a variable requires the string to be copied. The reference - the string's memory address - is copied is my understanding.

Responding in a comment, Blindman67 said:

@radarbob In JS strings CAN NOT be assigned by reference. Strings are always assigned as a copy (duplicated).

---

If the law is against you, talk about the evidence. If the evidence is against you, talk about the law, and, if the law and the evidence are both against you, then pound on the table and yell like hell.

The People, Yes by Carl Sandburg

In this case, the law is the ECMAScript® 2021 Language Specification, case law is V8 source code, and the trial court is Node.js v17.3.1 with V8 v9.6.180.15.

Since Blindman67 is yelling at us, "strings CAN NOT be assigned by reference", it's reasonable to assume that they are also pounding the table. :-(

Sticking to evidence, we can run a benchmark using Node.js v17.3.1 with V8 v9.6.180.15. The benchmark assigns the same text string with length 508,674,733 UTF-16 code units to each of 1,048,576 string variables, the elements of an array. If the string was copied on assignment then this would require at least 533,384,116,830,208 UTF-16 code units or about 1,066,768,233,660,416 (one quadrillion) bytes. The process.memoryUsage().heapUsed is 983 MB. The single text string uses about 970.22 MB. Text string code units are then read to ensure that they are not garbage collected. The benchmark runs successfully on computers with much less than a one quadrillion (peta) bytes of memory.