Memory efficient way to merge multiple Pandas DataFrames [closed]

Question

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote:

def merge_files(args):

    list_df = {}
    with args.input as data_in:
        for line in data_in:
            line = line.strip()
            df = pd.read_csv(line, sep="\s+")
            df.drop_duplicates(inplace = True)
            # change second column name (from the file)
            df.columns = ["sample", line.split("/")[-4]]
            list_df[line.split("/")[-4]] = df
    # Merging all the files into one and fill the missing ratio values with 0
    df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['sample'], how='outer'), list(list_df.values())).fillna(0)
    df_merged.to_csv(args.output, sep='\t', encoding='utf-8', index=False)

But this approach consumes too much memory, Is there a better solution?

Update

• The function is called in a script and given args contains input file (contain locations of the csv files one per line) and output file.

• The input file contains locations of 20 files each is ~2M in size and contains ~59133 lines.

• Using Python 3, Pandas version '0.25.2'

Answer 1

The issue was resolved, there was a duplication in keys in more than two DF, which was leading to memory usage increase. Thanks.