11
\$\begingroup\$

I have multiple (1000+) JSON files each of which contain a JSON array. I want to merge all these files into a single file.

I came up with the following, which reads each of those files and creates a new object with all the contents. I then write this new object into a new file.

Is this approach efficient? Is there a better way to do so?

head = []
with open("result.json", "w") as outfile:
    for f in file_list:
        with open(f, 'rb') as infile:
            file_data = json.load(infile)
            head += file_data
    json.dump(head, outfile)
\$\endgroup\$
  • \$\begingroup\$ do you need multithread solution? \$\endgroup\$ – Nikita Apr 18 '15 at 0:16
  • \$\begingroup\$ My concern with this approach is that you load the entire array into memory in the head variable. If you have a lot of data in those 1000+ files, you might start running out of memory. \$\endgroup\$ – alexwlchan Apr 18 '15 at 14:00
  • 6
    \$\begingroup\$ Cheap-and-nasty approach: read each file, then immediately append the contents onto the result file, minus the opening and closing array markers ([]). Add commas and array markers as appropriate between items. Then you only have one file in memory at a time. \$\endgroup\$ – alexwlchan Apr 18 '15 at 14:01
4
\$\begingroup\$
  1. First off, if you want reusability, turn this into a function. The function should have it's respective arguments.
  2. Secondly, instead of allocating a variable to store all of the JSON data to write, I'd recommend directly writing the contents of each of the files directly to the merged file. This will help prevent issues with memory.
  3. Finally, I just have a few nitpicky tips on your variable naming. Preferably, head should have a name more along the lines of merged_files, and you shouldn't be using f as an iterator variable. Something like json_file would be better.
\$\endgroup\$
1
\$\begingroup\$

This is essentially alexwlchan's comment spelled out:

Parsing and serializing JSON doesn't come for free, so you may want to avoid it. I think you can just output "[", the first file, ",", the second file etc., "]" and call it a day. If all inputs are valid JSON, unless I'm terribly mistaken, this should also be valid JSON.

In code, version 1:

def cat_json(outfile, infiles):
    file(outfile, "w")\
        .write("[%s]" % (",".join([mangle(file(f).read()) for f in infiles])))

def mangle(s):
    return s.strip()[1:-1]

Version 2:

def cat_json(output_filename, input_filenames):
    with file(output_filename, "w") as outfile:
        first = True
        for infile_name in input_filenames:
            with file(infile_name) as infile:
                if first:
                    outfile.write('[')
                    first = False
                else:
                    outfile.write(',')
                outfile.write(mangle(infile.read()))
        outfile.write(']')

The second version has a few advantages: its memory requirements should be something like the size of the longest input file, whereas the first requires twice the sum of all file sizes. The number of simultaneously open file handles is also smaller, so it should work for any number of files.

By using with, it also does deterministic (and immediate!) deallocation of file handles upon leaving each with block, even in python implementations with non-immediate garbage collection (such as pypy and jython etc.).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.