Convert a dictionary of iterables into an iterator of dictionaries

I have a user-editable dictionary of content used to fill in a string.Template and I wanted to allow the user to fill the same template with several values at once.

Using only values, itertools.product would be enough for my needs but I needed dictionaries to fill in a template. I also wanted users to be able to enter a single string or numerical value as values for the dictionary without having anything blowing up and requiring them to encapsulate their single value in a 1-length tuple.

I used itertools.product(*map(pair_iter_safe, <the_dict>.items())) to generate iterables of key-value pairs suitable for the dict() constructor or the dict.update() method. With pair_iter_safe being:

def pair_iter_safe(kv_pair):
key, value = kv_pair
if isinstance(value, str):
value = (value,)
else:
try:
value = iter(value)
except TypeError:
value = (value,)

for val in value:
yield key, val


For a broader picture here is a simplified example of calling code:

import sys
import string
import itertools

PARAMETERS = {
'name': 'John Doe',
'age': range(40,45),
'activity': ('foo', 'bar', 'baz'),
}

SCENARIO = """\
[@campaign] @name:
Hi! I'm @age and I like to @activity.
"""

class AtTemplate(string.Template):
delimiter = '@'

def run_scenario(parameters, stdout):
scenario = AtTemplate(SCENARIO).substitute(parameters)
print(scenario, file=stdout)

def run_campaign(campaign_name='test', stdout=sys.stdout):
parameters = {'campaign': campaign_name}
for setup in itertools.product(*map(pair_iter_safe, PARAMETERS.items())):
parameters.update(setup)
run_scenario(parameters, stdout)

def pair_iter_safe(kv_pair):
key, value = kv_pair
if isinstance(value, str):
value = (value,)
else:
try:
value = iter(value)
except TypeError:
value = (value,)

for val in value:
yield key, val

if __name__ == '__main__':
run_campaign()


Is there anything wrong with the approach or the pair_iter_safe function altogether?

Interesting question.

I decided to look only at itertools.product(*map(pair_iter_safe, <the_dict>.items())) and pair_iter_safe.

The way I look at it is from a theoretic memory usage perspective. Due to how you have defined it, all (key, value) pairs need to be in memory at the same time, which could take a bit of memory.

I would suggest the following:

def iter_safe(value):
if isinstance(value, str):
value = (value,)
try:
iter(value)
except TypeError:
return (value,)
else:
return value


And using it as follows:

def dict_combinations(d):
keys, values_list = zip(*d.items())
for values in itertools.product(*map(iter_safe, values_list)):
yield dict(zip(keys, values))


To me this is cleaner, having separated out the iter_safe logic from the dicts.

Though merely theoretical, I think this should save on memory usage due to there not being as much key-value pairs in memory all the time.

Of course, if you still want pair_iter_safe, you should be able to do something like

def pair_iter_safe(kv_pair):
key, value = kv_pair
for v in iter_safe(value):
yield key, v

• I might end up only yielding zip(keys, values) to feed into dict.update but other than that, great answer. It is much closer to what I had in mind in the first place, I just forgot that zip is its own reflection. May 12 '16 at 7:15
• Changed basestring to str. Maybe (str, bytes) would be appropriate as well? May 12 '16 at 9:50
• In the initial version I wrote I also used a loop instead of zip(*...). It's so easy to forget that I'd rather have it be called unzip instead. May 12 '16 at 9:52
• bytes won't be an issue in my use cases, but its worth noting if anyone have similar needs. May 12 '16 at 10:05