0
\$\begingroup\$

I am trying to merge multiple pandas data-frames that I read from a file. Here is the function I wrote:

def merge_files(args):

    list_df = {}
    with args.input as data_in:
        for line in data_in:
            line = line.strip()
            df = pd.read_csv(line, sep="\s+")
            df.drop_duplicates(inplace = True)
            # change second column name (from the file)
            df.columns = ["sample", line.split("/")[-4]]
            list_df[line.split("/")[-4]] = df
    # Merging all the files into one and fill the missing ratio values with 0
    df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['sample'], how='outer'), list(list_df.values())).fillna(0)
    df_merged.to_csv(args.output, sep='\t', encoding='utf-8', index=False)

But this approach consumes too much memory, Is there a better solution?

Update

  • The function is called in a script and given args contains input file (contain locations of the csv files one per line) and output file.
  • The input file contains locations of 20 files each is ~2M in size and contains ~59133 lines.

  • Using Python 3, Pandas version '0.25.2'

\$\endgroup\$
5
  • 1
    \$\begingroup\$ Please If you will downvote the question mention the reason, so I can learn something. \$\endgroup\$
    – Medhat
    Feb 6, 2020 at 23:16
  • 2
    \$\begingroup\$ You're receiving downvotes/close votes because you haven't provided enough information to warrant this question as on topic. How is this function used? How big is the file you're reading in? What version of python are you using? You should try to provide as much information about your code as possible. \$\endgroup\$
    – Linny
    Feb 6, 2020 at 23:20
  • \$\begingroup\$ Thank you, now at least I can put this data. \$\endgroup\$
    – Medhat
    Feb 6, 2020 at 23:22
  • \$\begingroup\$ 20 files each is ~2M in size Do you mean 20 MB? But this approach consumes too much memory How much is too much ? What is the line df.columns = ["sample", line.split("/")[-4]] for? What about list_df[line.split("/")[-4]] = df? You're repeating the same operation twice in two lines. Can you share what the files look like? \$\endgroup\$
    – AMC
    Feb 8, 2020 at 22:24
  • \$\begingroup\$ I mean 2 MB and the ` ["sample", line.split("/")[-4]]` to identify the sample name from the directory path. \$\endgroup\$
    – Medhat
    Feb 9, 2020 at 23:56

1 Answer 1

1
\$\begingroup\$

The issue was resolved, there was a duplication in keys in more than two DF, which was leading to memory usage increase. Thanks.

\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.