Check duplicate items in Mongodb

Question

I need to query MongoDB and check if an item exists or not:

class MongoDBPipeline(object):

    def __init__(self):
        connection = pymongo.MongoClient(
            settings['MONGODB_SERVER'],
            settings['MONGODB_PORT']
        )
        db = connection[settings['MONGODB_DB']]
        self.collection = db[settings['MONGODB_COLLECTION']]

    def process_item(self, item, spider):
        dup_check = self.collection.find({'image':item['image']}).count()
        if dup_check == 0 :     
            self.collection.insert(dict(item))
            log.msg("Image added to MongoDB database!",
                     level=log.DEBUG, spider=spider)
            print "image Added!"
        else:
           print "Image Exist"      
        return item

For now it works, but I want to know if I am doing it the right way or if there is a better solution.

Answer 1

Method

The fastest way to check if an item into a MongoDB is unique (and if it isn't, not insert it) is to create a unique index on the related columns and catch the error upon insertion time.

Watch more here.

MongoDB - Features

Maybe this would be OT, but I would like to let you know all this:

MongoDB is inconsistent by default. The documentation claims "strong consistency," but the default implementation considers an operation "complete" as soon as it is queued in the send buffer of a client, even before it has been seen by any node. There is a big discrepancy between the discussion in Mongo blogs and the default implementation in the code. You have to take extra measures to ensure that an update has propagated to all replicas.

I'll list you the advantages and the disadvantages, just to let you make an idea, I don't like so much MongoDB.

---

Advantages

• Lightning fast.

• You can perform rich queries, can create on the fly indexes with a single command.

• Lightning fast.

• Replication is very easy.

Disadvantages

• Indexes take up a lot of RAM. They are B-tree indexes and if you have many, you can run out of system resources really fast.

• Very unreliable. No single server durability. If something crashes while it's updating 'table-contents' - you lose all you data. Repair takes a lot of time, but usually ends up in 50-90% data loss if you aren't lucky. So only way to be fully secure is to have 2 replicas in different datacentres.

---

Obviously, there could be some alternative to MongoDB.

Answer 2

You can create an index on that field:

db.collection.createIndex({"field":"",{unique:true})

Mongo will throw an error if the same value is being inserted.