2
\$\begingroup\$

I have a CSV file that I would like to split up by row, do some whitespace processing on the string data in each row of the document column, and then output the processed data into separate TXT files in a new directory. The whitespace processing is to standardize the documents by removing newlines, carriage returns, and replacing all empty strings with " ".

The thing is, I will be doing this on corpora of 3+ million documents, so was wondering if there is a better way to do this instead of iterating through each one.

with open(csv_file) as file:
    reader = csv.reader(file, delimiter=',')
    count = -1
    for row in reader:
        row[1] = str(row[1]).replace(r'\n', '')
        row[1] = str(row[1]).replace(r'\r', '')
        if not row[1]:
            row[1] = ' '
        with open('corpus' + str(count) + '.txt', 'w') as output:
            output.write(str(row[1]))
        count += 1

Input CSV data:

document_id,document
0,"Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
1,"Beef ribs jerky biltong fatback."
2,"Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey." 
3,"Cow ham strip steak pastrami venison."
4,"Landjaeger fatback pork loin pig sausage." 

Output individual TXT files with contents of processed document column:

corpus_0 "Bacon ipsum dolor amet kevin jerky sausage filet mignon landjaeger, turducken drumstick burgdoggen kielbasa frankfurter doner tongue meatloaf."
corpus_1 "Beef ribs jerky biltong fatback."
corpus_2 "Short loin capicola pastrami meatball. Brisket meatloaf jowl salami porchetta jerky hamburger t-bone meatball turkey."
corpus_3 "Cow ham strip steak pastrami venison."
corpus_4 "Landjaeger fatback pork loin pig sausage." 
\$\endgroup\$
  • \$\begingroup\$ I don't see any newlines in the sample data. Since this has a tendency to complicate things, can you include sample data with newlines and the expected outcome, if they are really present? \$\endgroup\$ – Maarten Fabré Dec 7 '18 at 9:38
  • 1
    \$\begingroup\$ Do you have 3M CSVs, each with lots of documents, or do you have one CSV with 3M document rows? \$\endgroup\$ – Colin Phipps Jan 6 at 8:49
1
\$\begingroup\$

It seems odd, right now, that you throw away the document_id, and that's at least worth an explanatory comment.

The code that transforms each document — the whitespace processing — should be moved into a separate function. This because it is a distinct operation from the CSV reading, and allows it to be separately unit tested.

The code needs to handle the header row from the input.

If by "better" you are asking for faster, well for that you could split up the input file and run multiple copies of your program (and changing how output filenames are generated, since the line numbers wouldn't be the same).

\$\endgroup\$
0
\$\begingroup\$

Since you don't actually use the first column or row you should preprocess the files to remove them: sed -i -e '1d;s/[0-9]\+,//' *. You can then use split --lines=1 --suffix-length=7 to put each line in its own file. This should be faster than your Python script - these tools are very optimized for fast text processing, even of large files.

You don't need to convert cell values to str. From the documentation:

Each row read from the csv file is returned as a list of strings.

\$\endgroup\$
  • 1
    \$\begingroup\$ The fact that the code calls .replace() to eliminate carriage returns and newlines suggests that naive line processing is insufficient. You actually need to respect the CSV quoting rules. \$\endgroup\$ – 200_success Nov 6 '18 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.