4
\$\begingroup\$

Brief Description:

In my NodeJS Application, I have a model called User.

User.js

module.exports = {

  attributes: {
    name: {
      type: 'string'
    }
  }
};

My application consists users searching functionality using name attribute. For that I used find users using exact name as below(I am using MongoDB as my Database). My User Controller as below.

UserController.js

 var name = req.param('name');

 User.find({ name: { 'contains': name } }).exec(function (err, results) {
     if (err) {
         return res.serverError(err);
     }
     return res.send(results);
 });

Now I need to enhance my searching functionality.

As an example if my DB contains record as Michael and if I searched as micel, Michael should be given as a search result.

For that I used Fuzzy Logic Algorithm and I used fuzzy npm package. Now my implementation is as below.

UserController.js

var fuzzy = require('fuzzy');

module.exports = {

    search: function (req, res) {

        var term = req.param('name');
        var thisCtrl = this;

        User.find().exec(function (err, users) {

            if (err) {
                return res.serverError(err);
            } 
            thisCtrl.createSearchQuery(term, users, function (orQuery) {
                User.find({or: orQuery}).exec( function (err, finalResulst) {
                    if (err) {
                        return res.serverError(err);
                    } 
                    return res.send(finalResulst)
                })
            })
        });
    },

    createSearchQuery: function (term, users, cb) {

        var userNamesArr = [];
        var or = [];

        for (var i = 0; i < users.length; i++) {
            userNamesArr.push(users[i].name)
        }

        var results = fuzzy.filter(term, userNamesArr);
        var matches = results.map(function(el) { return el.string; });

        for (var i = 0; i < matches.length; i++) {

            var tempTerm = { name: { 'contains' : matches[i] }};
            or.push(tempTerm)
        }
        cb(or);
    }
};

This is working as I needed.

But My Question is,

I am getting all the users in my DB and create search term using Fuzzy Logic and finding again in User collection. Now this is ok if I have only about 100 user records. But the time when User collection contains Millions of user records , Will this causes to performance issue to my application or is there a Better Approach than this I should follow.

Sorry for my bad English and Suggestions are highly appreciated.

\$\endgroup\$
1
\$\begingroup\$

Question

"Will this causes to performance issue to my application or is there a Better Approach than this I should follow."

Well that will depend on a lot of factors.

  • 1million clients.
  • fuzzy.js and its performance.
  • A single medium level server.

State of development

First a quick look at the link you provided to fuzzy and at the bottom of the page you find the ToDo list which contains 2 points of interest

  1. Async batch updates so the UI doesn't block for huge sets. Or maybe Web Workers?
  2. Performance performance performance!

Following through to the gitHub repository and we find that the last activity was 3 years ago. This suggests (not implies) that development has ended. That a key issue recognized by the developers (performance) will not be worked on in the near future.

Performance

To try and determine the performance per search entry I went to their demo and timed the search over several differently size lists of names, and search strings.

The performance test shows a slightly below linear complexity (in the range of 628 - 1,250,000 names), however its too close to call and the safe bet is its linear. It searched 1.25million names in ~1 +/- 0.2seconds

Tests method

The test was performed in the console running the following snippet many times

t = performance.now();
res = fuzzy.filter(searchStr, data);
totalTime += performance.now() - t;
count ++;

A complex regular expression, on the same data required 1/10th the time.

t = performance.now();
res = data.filter(name => regExpSearch.test(name));
totalTime += performance.now() - t;
count ++;

Conclusion

What Node.js does well is IO, and what is does not do well is processing.

Assuming one medium level server.

  • 1 second for a search is a very long time.

  • Fuzzy.js performance is unlikely to change.

  • A response time over a second puts your site in the bottom 20% of services, and is unacceptable.

  • I estimate that at 30 thousand users performance will become a serious problem (see below)

If you are serving 1million active users, (using the 20/80 rule) with 20% of users visiting your site one day a year 20% of them once a month and 20% of them once a week (~600,000 searches a year) with users geographically clustered resulting in significant peak times.

It is highly likely that you will regularly have 30+ concurrent queries and a few times a year you can expect 300, in which case your server will be blocked for over 5 minutes.

(NOTE) these are very rough estimates based on one search per user per visit.

The answers

"Will this causes to performance issue to my application or..."

Yes for a very heavy trafficked site this will present a serious performance issue.

No for a very small traffic loads nobody will notice.

"...is there a Better Approach than this I should follow."

Yes of course there is, use a regExp, and there are many more. But I can not hand out advice on speculation. The information given above is already dubious as it is based on a single request rather than the site/service as a whole.

Refection

Well gee... any nerd knows this...

1+ million regular users is cash in the bank.

A service must start somewhere and that is with a few users, with growth being gradual and predictable (once moving). If you are monetizing the service then growth is great and not a problem (if managed well).

Start with the basics and work your way up. Replacing infrastructure (the search methods, upgrading the server/s) is part of the business. Performance is the least of your initial problems.

If you plan for viral growth then you are in the realm of highly speculative investment capital. Outsource a solution to people that have the experience or you will miss a lucrative opportunity.

If you plan to provide a free service for 1+ million users, good on you :), but you will need to have a fairly deep pocket.

\$\endgroup\$
  • \$\begingroup\$ Great answer. Thanks a lot. \$\endgroup\$ – Sudarshana Dayananda May 29 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.