I have 271 million records, line by line in a text file that I need to add to MongoDB, and I'm using Python and Pymongo in order to do this.
I first split the single file containing the 271 million records into multiple files containing each 1 million lines, and have written the current code to add it to the database:
import os import threading from pymongo import MongoClient class UserDb: def __init__(self): self.client = MongoClient('localhost', 27017) self.data = self.client.large_data.data threadlist =  def start_add(listname): db = UserDb() with open(listname, "r") as r: for line in r: if line is None: return a = dict() a['no'] = line.strip() db.data.insert_one(a) print(listname, "Done!") for x in os.listdir(os.getcwd()): if x.startswith('li'): t = threading.Thread(target=start_add, args=(x,)) t.start() threadlist.append(t) print("All threads started!") for thread in threadlist: thread.join()
This starts as many threads as there are files, and adds every line to the db as it goes through it. The bad thing is that after 3 hours it had only added 8.630.623.
What can I do to add the records faster?
There are 261 threads currently running.
Example of one row of data: 12345678 (8 digits)