Split CSV file into a text file per row, with whitespace normalization

Question

I have a CSV file that I would like to split up by row, do some whitespace processing on the string data in each row of the document column, and then output the processed data into separate TXT files in a new directory. The whitespace processing is to standardize the documents by removing newlines, carriage returns, and replacing all empty strings with " ".

The thing is, I will be doing this on corpora of 3+ million documents, so was wondering if there is a better way to do this instead of iterating through each one.

with open(csv_file) as file:
    reader = csv.reader(file, delimiter=',')
    count = -1
    for row in reader:
        row[1] = str(row[1]).replace(r'\n', '')
        row[1] = str(row[1]).replace(r'\r', '')
        if not row[1]:
            row[1] = ' '
        with open('corpus' + str(count) + '.txt', 'w') as output:
            output.write(str(row[1]))
        count += 1

Input CSV data:

document_id,document
0,"Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
1,"Beef ribs jerky biltong fatback."
2,"Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey." 
3,"Cow ham strip steak pastrami venison."
4,"Landjaeger fatback pork loin pig sausage." 

Output individual TXT files with contents of processed document column:

corpus_0 "Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
corpus_1 "Beef ribs jerky biltong fatback."
corpus_2 "Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey."
corpus_3 "Cow ham strip steak pastrami venison."
corpus_4 "Landjaeger fatback pork loin pig sausage." 

Answer 1

It seems odd, right now, that you throw away the document_id, and that's at least worth an explanatory comment.

The code that transforms each document — the whitespace processing — should be moved into a separate function. This because it is a distinct operation from the CSV reading, and allows it to be separately unit tested.

The code needs to handle the header row from the input.

If by "better" you are asking for faster, well for that you could split up the input file and run multiple copies of your program (and changing how output filenames are generated, since the line numbers wouldn't be the same).

Answer 2

Since you don't actually use the first column or row you should preprocess the files to remove them: sed -i -e '1d;s/[0-9]\+,//' *. You can then use split --lines=1 --suffix-length=7 to put each line in its own file. This should be faster than your Python script - these tools are very optimized for fast text processing, even of large files.

You don't need to convert cell values to str. From the documentation:

Each row read from the csv file is returned as a list of strings.