4
\$\begingroup\$

I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now:

df = pd.DataFrame(columns=col_names)
for filename in os.listdir(path):
    with gzip.open(path+"/"+filename, 'rb') as f:
        temp = pd.read_csv(f, names=col_names)
    df = df.append(temp)

I have noticed that the above code runs quite fast initially, but it keeps on getting slower and slower as it reads more and more files. How can I improve this?

\$\endgroup\$
  • \$\begingroup\$ what's your files extension .tar.gz or .gz ? \$\endgroup\$ – RomanPerekhrest Jan 15 at 13:36
  • \$\begingroup\$ @RomanPerekhrest file extension is .gz \$\endgroup\$ – Ank Jan 15 at 13:40
5
\$\begingroup\$

Ultimate optimization

  • avoid calling pd.DataFrame.append function within a loop as it'll create a copy of accumulated dataframe on each loop iteration. Apply pandas.concat to concatenate pandas objects at once.

  • no need to gzip.open as pandas.read_csv already allows on-the-fly decompression of on-disk data.

    compression : {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}, default ‘infer’

  • avoid hardcoding filepathes with path+"/"+filename. Instead use suitable os.path.join feature: os.path.join(dirpath, fname)


The final optimized approach:

import os
import pandas as pd

dirpath = 'path_to_gz_files'   # your directory path
df = pd.concat([pd.read_csv(os.path.join(dirpath, fname))
                for fname in os.listdir(dirpath)], ignore_index=True)
|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.