I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like it to be as performant as possible.
This is basically a mockup of a more complex CSV transformation I would be working on for work (I don't yet know the nature of the transformation I would be performing). To clarify, I was directed to write this at work, and it would be tested/scrutinised to see if its performant enough, but it's also not the final script we would eventually run.
import pandas as pd import csv chunksize = 10 ** 6 # Or whatever value the memory permits source_file = "" # Change to desired source file destination_file = "" # Change to desired destination file def process(chunk, headers, dest): df = pd.DataFrame(chunk, columns=headers) df.to_csv(dest, header=False, index=False) def transform_csv(source_file, destination_file): with open(source_file) as infile: reader = csv.DictReader(infile) new_headers = reader.fieldnames[::-1] with open(destination_file, "w+") as outfile: outfile.write(",".join(new_headers)) outfile.write("\n") with open(destination_file, 'a') as outfile: for chunk in pd.read_csv(source_file, chunksize=chunksize): process(chunk, new_headers, outfile) transform_csv(source_file, destination_file)