E-commerce product price tracker

I building a very simple price tracker web app. I am using MongoDB with pymongo. The user will enter the URL of the product he wishes to track and the desired amount, when the price goes below this amount, he should get an alert. It doesn't matter how many products he is tracking, I will send the alert when price changes in one product. I will sending generalized alert like 'Some of the product/s available at cheaper price' or something like that.

There are three collections in the database, one to store product details, one for user details and one to map products to the users who are tracking it.

A typical document in products collection will be:

{
_id: ObjectId("53a2bfcfa7603606c2765342"),
name: "Nexus 7 from Google (7-Inch, 16 GB, Black)",
base_price: 222,
current_price: 213.96,
img_url: "http://i.imgur.com/something.jpg",
history: [
[
ISODate("2014-06-19T12:08:13.354Z"),
293
],
[
ISODate("2014-06-24T14:31:38.216Z"),
424
],
[
ISODate("2014-06-24T14:32:06.992Z"),
424
]
]
}


in users, the tracked_products contain the Object Id of the product document and the desired price:

{
email_id: "johnappleseed@apple.com",
channels: {
'ios_push_id': '45462722672762576422'
'android_push_id': '2572652754762474'
'chrome_push_id': '456456454646464'
},
tracked_products: [
[
ObjectId("53a2bfcfa7603606c2765342"),
200
],
[
345
],
[
ObjectId("53a2d294a7603606c2765343"),
120
]
]
}


and lastly trackers, here subscribers contain the Object Id of user document who are tracking this product:

{
_id: ObjectId("77a2bfcfa7603606c2765399"),
product_id: ObjectId("53a2bfcfa7603606c2765342"),
subscribers: [
[
200
],
[
ObjectId("12a2bfcfa7603606c2765399"),
234
],
[
ObjectId("14a2bfcfa7603606c2765399"),
345
],
}


Following is the code, which checks for the current_price and alerts the user if there is change in the price:

subscribers = set()

for product in product.collection:
# price changed? if yes, then update
current_price = get_current_price(product['url'])

if current_price == product['current_price']:
# no price change. move on
continue

product['history'].append((datetime.datetime.utcnow(), current_price))

# find the subscribers:
tracker_document = db.trackers.find_one({'product_id': product['_id']})

for tracker in tracker_document['subscribers']:
user_id, desired_price = tracker
if current_price <= desired_price:

product.collection.save(product)

if not subscribers:
# no price changes in any products!
sys.exit()
email_queue = list()
ios_push_queue = list()
droid_push_queue = list()
chrome_push_queue = list()

for user_id in subscribers:
user_document = db.users.find_one({'_id': user_id})
email_queue.append(user_document['email_id'])
ios_push_queue.append(user_document['channels']['ios_push_id'])
droid_push_queue.append(user_document['channels']['droid_push_id'])
chrome_push_queue.append(user_document['channels']['chrome_push_id'])



I am looking for any suggestions, code improvements/readability, performance improvement, schema change etc. Anything!

PS: I am aware this would be lot easier with RDBMS and join operations, but I am sticking with MongoDB as of now.

You should create a function that returns all the subscribers for a product:

def get_subscribers(product, current_price):
tracker_document = db.trackers.find_one({'product_id': product['_id']})

subscribers = set()
for user_id, desired_price in tracker_document['subscribers']:
if current_price <= desired_price:

return subscribers


This could even be condensed even more using a list comprehension:

def get_subscribers(product, current_price):
tracker_document = db.trackers.find_one({'product_id': product['_id']})

return set([user_id for user_id, desired_price in tracker_document['subscribers']
if current_price <= desired_price])


This change makes your code look like this:

subscribers = set()
for product in product.collection:
current_price = get_current_price(product['url'])
if current_price == product['current_price']:
continue

product['history'].append((datetime.datetime.utcnow(), current_price))
product.collection.save(product)

subscribers |= get_subscribers(product, current_price)


The next change I would suggest is: instead of using several lists for your queues, wrap them into a single defaultdict. This simplifies this code and would require only minimal changes in your alert_users code.

It may actually be better, depending on your alert_users function, to to alert a single user at a time instead of building queues of the users and calling alert_users with a batch of users.

Here is the batch example:

from collections import defaultdict

queues = defaultdict(list)
for user_id in subscribers:
user_document = db.users.find_one({'_id': user_id})

queues['email_id'].append(user_document['email_id'])
for queue_type in ['twitter', 'ios_push_id', 'droid_push_id', 'chrome_push_id']:
queues[queue_type].append(user_document['channels'][queue_type])



Here is the user-at-a-time example:

for user_id in subscribers:
user_document = db.users.find_one({'_id': user_id})


• amazing! Thanks a lot. Btw I am using smtplib which connects to Mandrill to send emails. So I do not have my own email server. Since it has to connect to Mandrill everytime, I thought batch would be better compared to user-at-a-time basis. Similarly for iOS and Android push notifications I am using external services. Any thoughts? and also do you have any suggestions making this overall process faster?
• in get_subscribers function, why send product dictionary, just sending product['_id'] will also do right?