5
\$\begingroup\$

I have a bunch of JSON records, each record with a varying degree of completeness. In other words, record A may contain keys not in record B, and vice-versa. To get a better understanding of the data within, I have created a function to take N number of records and merge them together, creating one single frankenstein record containing all keys and a single value for each key.

import sys
import json
def frankenstein(out, in_dict, key=None):
    if isinstance(in_dict, dict):
        for k, v in in_dict.items():
            if isinstance(in_dict[k], list) and v:
                out.setdefault(k, [])
                frankenstein(out[k], v, k)
            elif isinstance(in_dict[k], dict) and v:
                out.setdefault(k, {})
                frankenstein(out[k], v, k)
            elif v:
                out[k] = v
    elif isinstance(in_dict, list):
        s = {}
        for item in in_dict:
            if isinstance(item, dict):
                frankenstein(s, item)
            elif not out:
                out.append(item)
            if s:
                if not out:
                    out.append(s)
                else:
                    frankenstein(s, out[0])
                    out[0] = s
if __name__ == '__main__':
    l = [
        {
            "name": "foo bar",
            "experience": [
                {
                    "company": {
                        "name": "oracle",
                        "hq": "123 main st",
                        "size": 100
                    },
                    "function": [
                        {
                            "name": "go getter"
                        }
                    ],
                    "location": {
                        "doubleday": "publisher"
                    },
                    "animal": "horse"
                }
            ],
            "skills": ["programming", "eating"]
        },
        {
            "name": "poo dar",
            "experience": [
                {
                    "company": {
                        "name": "microsoft",
                        "url": "foo.bar/com"
                    },
                    "function": [
                        {
                            "name": "bread",
                            "level": "really high"
                        }
                    ],
                    "solitary": {
                        "fat": "cat"
                    },
                    "health": "no good"
                }
            ],
            "skills": ["igz"]
        },
        {
            "name": "poo mar",
            "experience": [
                {
                    "function": [
                        {
                            "zoo": "creature"
                        }
                    ],
                    "location": {
                        "taste": "food"
                    },
                    "ping": {
                        "pong": "bong"
                    }
                }
            ],
            "skills": ["woots own"]
        }
    ]
    out = {}
    for item in l:
        frankenstein(out, item)
    print(json.dumps(out, indent=4))

This is the output from the code:

{
    "name": "poo mar",
    "experience": [
        {
            "function": [
                {
                    "name": "bread",
                    "level": "really high",
                    "zoo": "creature"
                }
            ],
            "location": {
                "taste": "food",
                "doubleday": "publisher"
            },
            "ping": {
                "pong": "bong"
            },
            "company": {
                "name": "oracle",
                "url": "foo.bar/com",
                "hq": "123 main st",
                "size": 100
            },
            "solitary": {
                "fat": "cat"
            },
            "health": "no good",
            "animal": "horse"
        }
    ],
    "skills": [
        "programming"
    ]
}

I've tested the function and it does work. What I'd like is some feedback with regards to the code. Am I doing this in the most efficient way possible? Are there better ways to do it?

\$\endgroup\$
2
  • \$\begingroup\$ This question is probably answered by this stackoverflow.com/questions/38987/… \$\endgroup\$ – user985366 Aug 28 '20 at 11:04
  • 5
    \$\begingroup\$ @user985366 None of the answers in the linked question on SO cover a deep merge, and none of them deal with nested lists of dicts like this code. Beyond that, I don't believe this question is asking how to do it, since they obviously achieved their desired goal; but rather, it's asking for a code review on their code, which seems to be the obvious intention of this community. \$\endgroup\$ – Franz Kafka Aug 28 '20 at 11:13
5
\$\begingroup\$

The key parameter to frankenstein() does not appear to be used anywhere and can be eliminated.

Use blank lines to split the code up into smaller logical chunks. It aides reading and understanding the code.

in_dict is a misleading name, because it can be a list, dict, or something else altogether.

In the for k, v in in_dict.items(): loop, in_dict[k] and v are the same thing, v is tested 3 times, and the return value of setdefault() is discarded then looked up on the next line. It can be rewritten it like so:

    for k, v in in_dict.items():
        if not v:
            continue

        if isinstance(v, (list, dict)):
            out_k = out.setdefault(k, v.__class__())
            frankenstein(out_k, v)

        else:
            out[k] = v

The logic for handling lists seems convoluted. For example, it looks like s can get updated each time through the loop and also appended to out[k]. The question doesn't clearly state the rules for combining things, so maybe it's correct. A comment or doc-string explaining the purpose of the function and the rules for merging values would be helpful.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.