List all possible permutations from a python dictionary of lists

Question

The following code takes a dictionary of lists (indexed by keyword) and converts it into a list of list of dictionaries that contains all possible permutations of those lists. However, my solution seems hackish, and is certainly space inefficient.. I feel like I'm missing something obvious.

The use case is for enumerating all the possible hyperparameters for a neural network (think sklearn grid search). So I specify the options in a compact dictionary containing the name of the parameter, and a list of values to test, and then convert that into a list of single-entry-per-key dictionaries that contain all possible permutations of those configuration parameters.

It's important to note I'm trying to make this generic, so that there can be an arbitrary number of keys specified in the initial dictionary, with an arbitrary number of options for each key.

Here's the code:

import copy

config_overrides = {
    'nodetype': ['srnn', 'lstm', 'gru'],
    'nhidden': [1,2,3,4,5,6,8,10,12,16,20],
    'opt': ['sgd', 'rmsprop', 'adagrad', 'adamax', 'adam', 'nadam', 'adadelta'],
    'actfunc': ['softplus', 'softsign', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear', 'relu'],
    'loss': ['mse', 'msle']
}

experiments = [{},]

for k,v in config_overrides.items():
    new_values = len(v)
    current_exp_len = len(experiments)
    for _ in range(new_values-1):
       experiments.extend(copy.deepcopy(experiments[:current_exp_len]))
    for validx in range(len(v)):
       for exp in experiments[validx*current_exp_len:(validx+1)*current_exp_len]:
            exp[k] = v[validx]

print(len(experiments))
print([x for x in experiments[1034:1039]])

The output of the above program, which includes random outputs from the middle of the list, and shows it appears tobe working correctly (the above example should come out to 3234 permutations):

3234
[{'loss': 'mse', 'opt': 'adadelta', 'actfunc': 'sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'msle', 'opt': 'adadelta', 'actfunc': 'sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'mse', 'opt': 'sgd', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'msle', 'opt': 'sgd', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'mse', 'opt': 'rmsprop', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}]

A few immediate questions:

• Is list.extend() the best way to expand the list?
• The deepcopy() is necessary, or else you end up with copies of the original dictionary. But that also means the strings are copied, redundantly. Not a huge deal, but doesn't feel right to me.

Other than that, it just seems like there should be a cleaner, less obtuse way of doing this..and maybe I'm just too tired to think of it. Any ideas?

Answer 1

The tool you need here is itertools.product:

>>> import itertools
>>> keys, values = zip(*config_overrides.items())
>>> experiments = [dict(zip(keys, v)) for v in itertools.product(*values)]
>>> len(experiments)
3234
>>> experiments[1034]
{'nhidden': 4, 'nodetype': 'lstm', 'loss': 'msle', 'opt': 'nadam', 'actfunc': 'sigmoid'}

Of course, if you are going to be iterating over the experiments one by one, then there is no point in allocating a list, and you could do this instead:

keys, values = zip(*config_overrides.items())
for v in itertools.product(*values):
    experiment = dict(zip(keys, v))
    # etc.

And depending on what you are doing with the experiments, you may not even need to build the dictionary:

keys, values = zip(*config_overrides.items())
for experiment in itertools.product(*values):
    for key, value in zip(keys, experiment):
        # etc.

Answer 2

To expand the dictionary into a list, you need to build the product from all possible params.

You can use itertools.product to build the product with all values from the dictionary config_overrides.

The resulting list of lists has all combinations of values from config_overrides and each items length is equal to len(config_overrides.keys())

Therefore we can use zip to attach each key to the position of a param in your product.

With the list of pairs, we can now easily create a dictionary.

For the sake of one liners here my version:

from itertools import product
experiments = [dict(zip(config_overrides.keys(), value)) for value in product(*config_overrides.values())]