I have a lot of compressed csv files in a directory. I want to read all those files in a single dataframe. This is what I have done till now:
df = pd.DataFrame(columns=col_names) for filename in os.listdir(path): with gzip.open(path+"/"+filename, 'rb') as f: temp = pd.read_csv(f, names=col_names) df = df.append(temp)
I have noticed that the above code runs quite fast initially, but it keeps on getting slower and slower as it reads more and more files. How can I improve this?