Reorder the Columns in a CSV File in Descending Order

Question

I wrote a script to reorder the columns in a CSV file in descending order and then write to another CSV file. My script needs to be able to handle several tens of millions of records, and I would like it to be as performant as possible.

This is basically a mockup of a more complex CSV transformation I would be working on for work (I don't yet know the nature of the transformation I would be performing). To clarify, I was directed to write this at work, and it would be tested/scrutinised to see if its performant enough, but it's also not the final script we would eventually run.

CSV-Transform.py

import pandas as pd
import csv

chunksize = 10 ** 6 # Or whatever value the memory permits
source_file = ""
# Change to desired source file
destination_file = ""
# Change to desired destination file


def process(chunk, headers, dest):
   df = pd.DataFrame(chunk, columns=headers)
   df.to_csv(dest, header=False,  index=False)


def transform_csv(source_file, destination_file):
   with open(source_file) as infile:
      reader = csv.DictReader(infile)
      new_headers = reader.fieldnames[::-1]
   
   with open(destination_file, "w+") as outfile:
      outfile.write(",".join(new_headers))
      outfile.write("\n")

   with open(destination_file, 'a') as outfile:
      for chunk in pd.read_csv(source_file, chunksize=chunksize):
         process(chunk, new_headers, outfile)


transform_csv(source_file, destination_file)