turn a dict that may contain a pandas dataframe to several dicts

Question

I have a dict that may be 'infinitely' nested and contain several pandas DataFrame's (all the DataFrame's have the same amount of rows).

I want to create a new dict for each row in the DataFrame's, with the row being transformed to a dict (the key's are the column names) and the rest of the dictionary staying the same.

Note: I am not making a cartesian product between the rows of the different DataFrame's.

I have an example in my SO question

here is what i came up with:

import pandas as pd
from copy import deepcopy
from functools import partial

def map_keys_by_type(d, typ, path=None):
    for k,v in d.items():
        p = path.copy() if path else []
        p.append(k)
        if isinstance(v, typ):
            yield p
        if isinstance(v, dict):
            yield from map_keys_by_type(v, typ, p)

def nested_get(nested_key, input_dict):
    internal_dict_value = input_dict
    for k in nested_key:
        internal_dict_value = internal_dict_value.get(k, None)
        if internal_dict_value is None:
            return None
    return internal_dict_value

def nested_set(dic, keys, value):
    for key in keys[:-1]:
        dic = dic.setdefault(key, {})
    dic[keys[-1]] = value

def dup_dicts(keys, iter_of_values, init_dict):
    for values in iter_of_values:
        init_dict = deepcopy(init_dict)
        [nested_set(init_dict, key, value) for key, value in zip(keys, values)]
        yield init_dict

if __name__ == '__main__':
    keys = list(map_keys_by_type(d, pd.DataFrame))
    dfs = map(partial(nested_get, input_dict=d), keys)
    dfs_as_dicts = map(partial(pd.DataFrame.to_dict, orient='records'), dfs)
    iter_of_dicts = dup_dicts(keys,zip(*dfs_as_dicts), d)

any improvements?

Answer 1

I'm assuming that the code works as intended.

Firstly, let me tell you that your code looks very pythonic - this is a huge plus.

My comments:

Introduction

You state that

I have a dict that may be 'infinitely' nested

Keep in mind that since you seem to be doing recursive calls your stack is limited hence is your ability to retrive nested dict.

map_keys_by_type

Argument d is not informative, try using dic (if you want to follow your change type->typ) or dictionary.

Although, this is not extreamly important you might consider changing p to something more informative. When I've read the code first time I almost immediacy forgot what this variable stands for. However, as I said this is not very important because the function is small.

nested_get

This bit

for k in nested_key:
    internal_dict_value = internal_dict_value.get(k, None)
    if internal_dict_value is None:
        return None

in my opinion would be clearer if you did:

if k not in internal_dict_value:
    return None
else:
    internal_dict_value = internal_dict_value[k]

not in shows your intention directly.

Everything else looks okay for me.