I have a CSV file that I would like to split up by row, do some whitespace processing on the string data in each row of the document column, and then output the processed data into separate TXT files in a new directory. The whitespace processing is to standardize the documents by removing newlines, carriage returns, and replacing all empty strings with " "
.
The thing is, I will be doing this on corpora of 3+ million documents, so was wondering if there is a better way to do this instead of iterating through each one.
with open(csv_file) as file:
reader = csv.reader(file, delimiter=',')
count = -1
for row in reader:
row[1] = str(row[1]).replace(r'\n', '')
row[1] = str(row[1]).replace(r'\r', '')
if not row[1]:
row[1] = ' '
with open('corpus' + str(count) + '.txt', 'w') as output:
output.write(str(row[1]))
count += 1
Input CSV data:
document_id,document
0,"Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
1,"Beef ribs jerky biltong fatback."
2,"Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey."
3,"Cow ham strip steak pastrami venison."
4,"Landjaeger fatback pork loin pig sausage."
Output individual TXT files with contents of processed document
column:
corpus_0 "Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
corpus_1 "Beef ribs jerky biltong fatback."
corpus_2 "Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey."
corpus_3 "Cow ham strip steak pastrami venison."
corpus_4 "Landjaeger fatback pork loin pig sausage."