2
\$\begingroup\$

I've created a Views Counter in Python, Gevent and MongoDB (Flask is also included in the full stack as you can see from the context issue in the code).

My gut still tells me that it can be still somehow improved though.

What the code does is initializing a dict "buffer" (just a shelve persisted in memory [no writeback]), define a function where Mongo's bulk_op is initialized, a list comprehension that iterates over the view_count buffer and sets individual find-updates for them with their respective key values that is then ended by an execute with write concern disabled.

Then there's the run_buffer_op function that runs a loop infinitely, which checks if the buffer has over 5000 items, and in that case, flushes its content to the database (by executing the former function), otherwise it just waits 15 minutes before flushing.

This function is finally run (or better, spawned) by Gevent.

Do you see some possible further improvements?

'''
Views counter buffer
====================
'''
# Initialize vc buffer (ObjectId + Views_count pair). We don't need **writeback** as it should just be persisted in memory
view_count=shelve.open('view_count', writeback=False)

def flush_to_db():
  # Apparently you have to set the context to make the db bulk operation work, otherwise it'll return a *Working outside of context* error :(
  with app.app_context():
    bulk = mongo.db.test.initialize_unordered_bulk_op()
    # What this list comprehension does is iterating over the *view_count* buffer
    # and set individual find-updates for them with their respective key value.
    # Like: bulk.find({'_id': '7rhf3d32dh23jd78988ej8'}).update({'$set':{'count':2}})
    #       bulk.find({'_id': '7rhf3d32dh23dg48988ej8'}).update({'$set':{'count':10}})
    #       bulk.find({'_id': '7rhf3d32th23dg48988ek9'}).update({'$set':{'count':7}})
    #       ...

    [bulk.find({ '_id': k }).update({'$set':{'count': v }}) for k,v in view_count.iteritems()]

    # Execute the bulk update with no **write concern**. We can afford to lose some views count if that ever happens in change of better performance.
    bulk.execute(write_concern=None)

def run_buffer_op():
  # Run the loop infinitely
  while True:
    # If buffer has more than 5000 items flush it now to the db
    if len(view_count) > 5000:
      flush_to_db()
    # Else just wait 15 minutes
    gevent.sleep(900)
    flush_to_db()

# Spawn the loop
gevent.spawn(run_buffer_op)
\$\endgroup\$

1 Answer 1

1
\$\begingroup\$

Instead of using a list comprehension, I would simply use a for-loop. It will do the exact same thing and save having to allocate a temporary list.

Also, in run_buffer_op, you first check the number of views. If that value isn't >5000 (almost a DBZ joke) then you sleep for 15min. Once it wakes up, you IMMEDIATELY flush_to_db no matter what. Based on your comments, it seems you only want to flush_to_db if there are >5000 views. Thus, the flush_to_db call immediately after the gevent.sleep(900) seems a little redundant.

\$\endgroup\$
2
  • \$\begingroup\$ Thank you for the answer Darin, it confirmed some of my concerns. The flush_to_db call immediately after the gevent.sleep(900) is because the buffer could be filled before 15 minutes and thus denote an increased site activity, so it seems reasonable to update the views earlier in that case. Otherwise, if the buffer still isn't filled (because of lower activity) we wait 15 minutes to flush it to the db (even if there are just 2-3 hundred views made in that lapse of time). \$\endgroup\$ Commented May 24, 2014 at 18:53
  • \$\begingroup\$ Anyway I switched to a pure Redis + Gevent implementation removing MongoDB from the equation. Other than the obvious benefits there is also zero overhead on syncing the buffer in a distributed environment. Hope this helps someone who wants to implement a views counter with the least overhead possible, but I really suggest to look at Redis. Most web apps/websites can afford to lose the last few minutes of data in case of a pure Redis implementation, something that you can solve if you really want (such as putting a "dumb" Redis node as fallback). \$\endgroup\$ Commented May 25, 2014 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.