3
\$\begingroup\$

I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like it to be as performant as possible.

This is basically a mockup of a more complex CSV transformation I would be working on for work (I don't yet know the nature of the transformation I would be performing). To clarify, I was directed to write this at work, and it would be tested/scrutinised to see if its performant enough, but it's also not the final script we would eventually run.

CSV-Transform.py

import pandas as pd
import csv

chunksize = 10 ** 6 # Or whatever value the memory permits
source_file = ""
# Change to desired source file
destination_file = ""
# Change to desired destination file


def process(chunk, headers, dest):
   df = pd.DataFrame(chunk, columns=headers)
   df.to_csv(dest, header=False,  index=False)


def transform_csv(source_file, destination_file):
   with open(source_file) as infile:
      reader = csv.DictReader(infile)
      new_headers = reader.fieldnames[::-1]
   
   with open(destination_file, "w+") as outfile:
      outfile.write(",".join(new_headers))
      outfile.write("\n")

   with open(destination_file, 'a') as outfile:
      for chunk in pd.read_csv(source_file, chunksize=chunksize):
         process(chunk, new_headers, outfile)


transform_csv(source_file, destination_file)
\$\endgroup\$
6
  • 1
    \$\begingroup\$ Welcome to CodeReview@SE. mockup of a more complex CSV transformation I would be working on does not comply with actual code from a project rather than … hypothetical code. \$\endgroup\$ – greybeard Aug 31 '20 at 10:19
  • 2
    \$\begingroup\$ I’m voting to close this question because a mockup of would-be code is not the concrete code from a project, with enough code and / or context required here. \$\endgroup\$ – greybeard Aug 31 '20 at 10:22
  • \$\begingroup\$ @greybeard: to clarify, I was directed by my boss to write this mockup first ahead of the task. So it's not just hypothetical code. It is actual code I wrote at work. The script would also be scrutinised to see if its performant enough. We just don't yet have the schema from our client. \$\endgroup\$ – Cinera Verinia Aug 31 '20 at 10:24
  • \$\begingroup\$ I have added the elaboration. I don't actually know if this script would be tested against 80 million records, but when my boss mentioned the task of transforming a CSV file of 80 million records, I pitched the approach of using a Pandas dataframe and he asked me to write this mockup first so he could check my approach and see if it would scale. \$\endgroup\$ – Cinera Verinia Aug 31 '20 at 10:32
  • \$\begingroup\$ (I'm mildly interested in benchmark results of using the 10 ** 6 chunksize vs. max. memory permits. Plan you measurement, and put your best guess in writing before getting first figures.) \$\endgroup\$ – greybeard Aug 31 '20 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.