5
\$\begingroup\$

I have a user-editable dictionary of content used to fill in a string.Template and I wanted to allow the user to fill the same template with several values at once.

Using only values, itertools.product would be enough for my needs but I needed dictionaries to fill in a template. I also wanted users to be able to enter a single string or numerical value as values for the dictionary without having anything blowing up and requiring them to encapsulate their single value in a 1-length tuple.

I used itertools.product(*map(pair_iter_safe, <the_dict>.items())) to generate iterables of key-value pairs suitable for the dict() constructor or the dict.update() method. With pair_iter_safe being:

def pair_iter_safe(kv_pair):
    key, value = kv_pair
    if isinstance(value, str):
        value = (value,)
    else:
        try:
            value = iter(value)
        except TypeError:
            value = (value,)

    for val in value:
        yield key, val

For a broader picture here is a simplified example of calling code:

import sys
import string
import itertools


PARAMETERS = {
    'name': 'John Doe',
    'age': range(40,45),
    'activity': ('foo', 'bar', 'baz'),
}


SCENARIO = """\
[@campaign] @name:
    Hi! I'm @age and I like to @activity.
"""


class AtTemplate(string.Template):
    delimiter = '@'


def run_scenario(parameters, stdout):
    scenario = AtTemplate(SCENARIO).substitute(parameters)
    print(scenario, file=stdout)


def run_campaign(campaign_name='test', stdout=sys.stdout):
    parameters = {'campaign': campaign_name}
    for setup in itertools.product(*map(pair_iter_safe, PARAMETERS.items())):
        parameters.update(setup)
        run_scenario(parameters, stdout)


def pair_iter_safe(kv_pair):
    key, value = kv_pair
    if isinstance(value, str):
        value = (value,)
    else:
        try:
            value = iter(value)
        except TypeError:
            value = (value,)

    for val in value:
        yield key, val


if __name__ == '__main__':
    run_campaign()

Is there anything wrong with the approach or the pair_iter_safe function altogether?

\$\endgroup\$
3
\$\begingroup\$

Interesting question.

I decided to look only at itertools.product(*map(pair_iter_safe, <the_dict>.items())) and pair_iter_safe.

The way I look at it is from a theoretic memory usage perspective. Due to how you have defined it, all (key, value) pairs need to be in memory at the same time, which could take a bit of memory.

I would suggest the following:

def iter_safe(value):
    if isinstance(value, str):
        value = (value,)
    try:
        iter(value)
    except TypeError:
        return (value,)
    else:
        return value

And using it as follows:

def dict_combinations(d):
    keys, values_list = zip(*d.items())
    for values in itertools.product(*map(iter_safe, values_list)):
        yield dict(zip(keys, values))

To me this is cleaner, having separated out the iter_safe logic from the dicts.

Though merely theoretical, I think this should save on memory usage due to there not being as much key-value pairs in memory all the time.

Of course, if you still want pair_iter_safe, you should be able to do something like

def pair_iter_safe(kv_pair):
    key, value = kv_pair
    for v in iter_safe(value):
        yield key, v
\$\endgroup\$
4
  • 1
    \$\begingroup\$ I might end up only yielding zip(keys, values) to feed into dict.update but other than that, great answer. It is much closer to what I had in mind in the first place, I just forgot that zip is its own reflection. \$\endgroup\$ May 12 '16 at 7:15
  • \$\begingroup\$ Changed basestring to str. Maybe (str, bytes) would be appropriate as well? \$\endgroup\$ May 12 '16 at 9:50
  • 1
    \$\begingroup\$ In the initial version I wrote I also used a loop instead of zip(*...). It's so easy to forget that I'd rather have it be called unzip instead. \$\endgroup\$ May 12 '16 at 9:52
  • \$\begingroup\$ bytes won't be an issue in my use cases, but its worth noting if anyone have similar needs. \$\endgroup\$ May 12 '16 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.