5
\$\begingroup\$

I building a very simple price tracker web app. I am using MongoDB with pymongo. The user will enter the URL of the product he wishes to track and the desired amount, when the price goes below this amount, he should get an alert. It doesn't matter how many products he is tracking, I will send the alert when price changes in one product. I will sending generalized alert like 'Some of the product/s available at cheaper price' or something like that.

There are three collections in the database, one to store product details, one for user details and one to map products to the users who are tracking it.

A typical document in products collection will be:

{
  _id: ObjectId("53a2bfcfa7603606c2765342"),
  name: "Nexus 7 from Google (7-Inch, 16 GB, Black)",
  url: "http://rads.stackoverflow.com/amzn/click/B00DVFLJDS",
  base_price: 222,
  current_price: 213.96,
  img_url: "http://i.imgur.com/something.jpg",
  history: [
    [
      ISODate("2014-06-19T12:08:13.354Z"),
      293
    ],
    [
      ISODate("2014-06-24T14:31:38.216Z"),
      424
    ],
    [
      ISODate("2014-06-24T14:32:06.992Z"),
      424
    ]
  ]
}

in users, the tracked_products contain the Object Id of the product document and the desired price:

{
  _id: ObjectId("539c4adea760360886d7ef02"),
  email_id: "johnappleseed@apple.com",
  channels: {
    'twitter': 'usertwitterid',
    'ios_push_id': '45462722672762576422'
    'android_push_id': '2572652754762474'
    'chrome_push_id': '456456454646464'
  },
  tracked_products: [
    [
        ObjectId("53a2bfcfa7603606c2765342"), 
        200
    ],
    [
        ObjectId("53a2d2ada7603606c2765344"),
        345
    ],
    [
        ObjectId("53a2d294a7603606c2765343"),
        120
    ]
  ]
}

and lastly trackers, here subscribers contain the Object Id of user document who are tracking this product:

{
    _id: ObjectId("77a2bfcfa7603606c2765399"),
    product_id: ObjectId("53a2bfcfa7603606c2765342"),
    subscribers: [
        [
            ObjectId("539c4adea760360886d7ef02"),
            200
        ],
        [
            ObjectId("12a2bfcfa7603606c2765399"),
            234
        ],
        [   
            ObjectId("14a2bfcfa7603606c2765399"),
            345
        ],
}

Following is the code, which checks for the current_price and alerts the user if there is change in the price:

subscribers = set()

for product in product.collection:
    # price changed? if yes, then update
    current_price = get_current_price(product['url'])

    if current_price == product['current_price']:
        # no price change. move on
        continue

    product['history'].append((datetime.datetime.utcnow(), current_price))

    # find the subscribers:
    tracker_document = db.trackers.find_one({'product_id': product['_id']})

    for tracker in tracker_document['subscribers']:
        user_id, desired_price = tracker
        if current_price <= desired_price:
            subscribers.add(user_id)

    product.collection.save(product)

if not subscribers:
    # no price changes in any products!
    sys.exit()
email_queue = list()
twitter_queue = list()
ios_push_queue = list()
droid_push_queue = list()
chrome_push_queue = list()

for user_id in subscribers:
    user_document = db.users.find_one({'_id': user_id})
    email_queue.append(user_document['email_id'])
    twitter_queue.append(user_document['channels']['twitter'])
    ios_push_queue.append(user_document['channels']['ios_push_id'])
    droid_push_queue.append(user_document['channels']['droid_push_id'])
    chrome_push_queue.append(user_document['channels']['chrome_push_id'])

alert_users(email_queue, twitter_queue, ios_push_queue, droid_push_queue, chrome_push_queue)

I am looking for any suggestions, code improvements/readability, performance improvement, schema change etc. Anything!

PS: I am aware this would be lot easier with RDBMS and join operations, but I am sticking with MongoDB as of now.

\$\endgroup\$

1 Answer 1

2
\$\begingroup\$

You should create a function that returns all the subscribers for a product:

def get_subscribers(product, current_price):
    tracker_document = db.trackers.find_one({'product_id': product['_id']})

    subscribers = set()
    for user_id, desired_price in tracker_document['subscribers']:
        if current_price <= desired_price:
            subscribers.add(user_id)

    return subscribers

This could even be condensed even more using a list comprehension:

def get_subscribers(product, current_price):
    tracker_document = db.trackers.find_one({'product_id': product['_id']})

    return set([user_id for user_id, desired_price in tracker_document['subscribers']
                if current_price <= desired_price])

This change makes your code look like this:

subscribers = set()
for product in product.collection:
    current_price = get_current_price(product['url'])
    if current_price == product['current_price']:
        continue

    product['history'].append((datetime.datetime.utcnow(), current_price))
    product.collection.save(product)

    subscribers |= get_subscribers(product, current_price)

The next change I would suggest is: instead of using several lists for your queues, wrap them into a single defaultdict. This simplifies this code and would require only minimal changes in your alert_users code.

It may actually be better, depending on your alert_users function, to to alert a single user at a time instead of building queues of the users and calling alert_users with a batch of users.

Here is the batch example:

from collections import defaultdict

queues = defaultdict(list)
for user_id in subscribers:
    user_document = db.users.find_one({'_id': user_id})

    queues['email_id'].append(user_document['email_id'])
    for queue_type in ['twitter', 'ios_push_id', 'droid_push_id', 'chrome_push_id']:
        queues[queue_type].append(user_document['channels'][queue_type])

alert_users(queues)

Here is the user-at-a-time example:

for user_id in subscribers:
    user_document = db.users.find_one({'_id': user_id})

    alert_user(user_document['email_id'], *user_document['channels'])
\$\endgroup\$
4
  • \$\begingroup\$ amazing! Thanks a lot. Btw I am using smtplib which connects to Mandrill to send emails. So I do not have my own email server. Since it has to connect to Mandrill everytime, I thought batch would be better compared to user-at-a-time basis. Similarly for iOS and Android push notifications I am using external services. Any thoughts? and also do you have any suggestions making this overall process faster? \$\endgroup\$
    – avi
    Jun 30, 2014 at 14:58
  • \$\begingroup\$ in get_subscribers function, why send product dictionary, just sending product['_id'] will also do right? \$\endgroup\$
    – avi
    Jul 4, 2014 at 3:15
  • \$\begingroup\$ Either is fine. Since a user subscribed to a product, it made sense to send the entire product to the function. However, since the ID defines a product, just the ID would work as well. \$\endgroup\$ Jul 7, 2014 at 11:51
  • \$\begingroup\$ i have a similar use case except i need to ping every minute instead of every day, the prices in my case are not stored in the database but on redis as i have 250k prices to be updated every minute, it takes a lot of time to send prices every minute to the database or send a query to the database with the product and current price, do you think it would be better simply to fetch all alerts from the db and check which ones are triggerable in code? \$\endgroup\$
    – PirateApp
    Feb 24, 2018 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.