I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now:

df = pd.DataFrame(columns=col_names)
for filename in os.listdir(path):
    with gzip.open(path+"/"+filename, 'rb') as f:
        temp = pd.read_csv(f, names=col_names)
    df = df.append(temp)

I have noticed that the above code runs quite fast initially, but it keeps on getting slower and slower as it reads more and more files. How can I improve this?

  • \$\begingroup\$ what's your files extension .tar.gz or .gz ? \$\endgroup\$ Jan 15, 2020 at 13:36
  • \$\begingroup\$ @RomanPerekhrest file extension is .gz \$\endgroup\$
    – Ank
    Jan 15, 2020 at 13:40

1 Answer 1


Ultimate optimization

  • avoid calling pd.DataFrame.append function within a loop as it'll create a copy of accumulated dataframe on each loop iteration. Apply pandas.concat to concatenate pandas objects at once.

  • no need to gzip.open as pandas.read_csv already allows on-the-fly decompression of on-disk data.

    compression : {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}, default ‘infer’

  • avoid hardcoding filepathes with path+"/"+filename. Instead use suitable os.path.join feature: os.path.join(dirpath, fname)

The final optimized approach:

import os
import pandas as pd

dirpath = 'path_to_gz_files'   # your directory path
df = pd.concat([pd.read_csv(os.path.join(dirpath, fname))
                for fname in os.listdir(dirpath)], ignore_index=True)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.