I have a .csv file containing 100 millions records, I would to create multiple files from this file, after certain condition satisfied in each line, I came to this code below, but it is slow, I think, it is due the fact the a file is opened and closed in each iteration. The result is many files renamed after countries, each file has inside the correspondant line from the large file. Is there any approach or method that can get rid of this problem and increase the speed?
import csv with open('stats.csv','r', encoding = 'utf-8') as inf, open('stats_failed.txt', 'a', encoding = 'utf-8', newline = '') as failed_ouf: csr = csv.reader(inf) for i, record in enumerate(csr): try: country = record #because some country have these characters, invalid for file renaming if '/' in country: country = country.replace('/','_SLASH_') elif '\\' in country: country = country.replace('\\','_BACKSLASH_') #pop the country because irrelevant since the file has its name record.pop(3) with open(country, 'a', encoding = 'utf-8', newline = '') as output: csw = csv.writer(output) csw.writerow(record) #just for tracking if i % 100_000 == 0: print(i) output.flush() except: failed_csw = csv.writer(failed_ouf) failed_csw.writerow(record)
EDITED To those who asked: It looks like this:
col1,col2,country,mod1,code,mod2 \N,\N,Germany,C510,\N,C510 \N,\N,"China, Taipei",C25C,\N,C25C \N,\N,Italy,GLF5,\N,GLF5 \N,\N,France,C25B,\N,C25B \N,\N,Germany,C525,\N,C525 \N,\N,Turkey,PC12,\N,PC12 \N,\N,Germany,C680,\N,C680 \N,\N,Germany,GL5T,\N,GL5T \N,\N,Germany,C25B,\N,C25B \N,\N,Italy,GLF4,\N,GLF4 \N,\N,France,CRJ2,\N,CRJ2 \N,\N,Palestine,G280,\N,G280 \N,\N,Slovakia,C525,\N,C525 .... \N,\N,United States,E55P,\N,E55P \N,\N,Russia,C56X,\N,C56X
EDITED2 If it is relevant, the size is about 4 GB