8
\$\begingroup\$

The following code takes a dictionary of lists (indexed by keyword) and converts it into a list of list of dictionaries that contains all possible permutations of those lists. However, my solution seems hackish, and is certainly space inefficient.. I feel like I'm missing something obvious.

The use case is for enumerating all the possible hyperparameters for a neural network (think sklearn grid search). So I specify the options in a compact dictionary containing the name of the parameter, and a list of values to test, and then convert that into a list of single-entry-per-key dictionaries that contain all possible permutations of those configuration parameters.

It's important to note I'm trying to make this generic, so that there can be an arbitrary number of keys specified in the initial dictionary, with an arbitrary number of options for each key.

Here's the code:

import copy

config_overrides = {
    'nodetype': ['srnn', 'lstm', 'gru'],
    'nhidden': [1,2,3,4,5,6,8,10,12,16,20],
    'opt': ['sgd', 'rmsprop', 'adagrad', 'adamax', 'adam', 'nadam', 'adadelta'],
    'actfunc': ['softplus', 'softsign', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear', 'relu'],
    'loss': ['mse', 'msle']
}

experiments = [{},]

for k,v in config_overrides.items():
    new_values = len(v)
    current_exp_len = len(experiments)
    for _ in range(new_values-1):
       experiments.extend(copy.deepcopy(experiments[:current_exp_len]))
    for validx in range(len(v)):
       for exp in experiments[validx*current_exp_len:(validx+1)*current_exp_len]:
            exp[k] = v[validx]

print(len(experiments))
print([x for x in experiments[1034:1039]])

The output of the above program, which includes random outputs from the middle of the list, and shows it appears tobe working correctly (the above example should come out to 3234 permutations):

3234
[{'loss': 'mse', 'opt': 'adadelta', 'actfunc': 'sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'msle', 'opt': 'adadelta', 'actfunc': 'sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'mse', 'opt': 'sgd', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'msle', 'opt': 'sgd', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}, {'loss': 'mse', 'opt': 'rmsprop', 'actfunc': 'hard_sigmoid', 'nodetype': 'lstm', 'nhidden': 4}]

A few immediate questions:

  • Is list.extend() the best way to expand the list?
  • The deepcopy() is necessary, or else you end up with copies of the original dictionary. But that also means the strings are copied, redundantly. Not a huge deal, but doesn't feel right to me.

Other than that, it just seems like there should be a cleaner, less obtuse way of doing this..and maybe I'm just too tired to think of it. Any ideas?

\$\endgroup\$
4
  • \$\begingroup\$ Note I quite possibly have an off-by-one error in the existing code, but you get the idea. \$\endgroup\$
    – clemej
    Jul 26, 2017 at 4:32
  • 3
    \$\begingroup\$ When I try to run this script I get an error in line 19: for exp in experiments[validx*current_exp_len(validx+1)*current_exp_len]: TypeError: 'int' object is not callable \$\endgroup\$
    – Ludisposed
    Jul 26, 2017 at 7:25
  • 3
    \$\begingroup\$ you get the idea. yes, we do, it's off-topic because it's broken ;-) \$\endgroup\$
    – t3chb0t
    Jul 26, 2017 at 10:50
  • \$\begingroup\$ Sorry,, a : got eaten while I was formatting my code cut-n-paste I'll be sure to re-test the cut-n-paste code in the future. \$\endgroup\$
    – clemej
    Jul 26, 2017 at 14:40

2 Answers 2

16
\$\begingroup\$

The tool you need here is itertools.product:

>>> import itertools
>>> keys, values = zip(*config_overrides.items())
>>> experiments = [dict(zip(keys, v)) for v in itertools.product(*values)]
>>> len(experiments)
3234
>>> experiments[1034]
{'nhidden': 4, 'nodetype': 'lstm', 'loss': 'msle', 'opt': 'nadam', 'actfunc': 'sigmoid'}

Of course, if you are going to be iterating over the experiments one by one, then there is no point in allocating a list, and you could do this instead:

keys, values = zip(*config_overrides.items())
for v in itertools.product(*values):
    experiment = dict(zip(keys, v))
    # etc.

And depending on what you are doing with the experiments, you may not even need to build the dictionary:

keys, values = zip(*config_overrides.items())
for experiment in itertools.product(*values):
    for key, value in zip(keys, experiment):
        # etc.
\$\endgroup\$
2
  • \$\begingroup\$ Sigh.. why does it seem the answer is always itertools? ;). Thanks, this is exactly what I was looking for. \$\endgroup\$
    – clemej
    Jul 26, 2017 at 14:46
  • \$\begingroup\$ Wow this is gold! \$\endgroup\$
    – gosuto
    Dec 2, 2020 at 17:32
0
\$\begingroup\$

To expand the dictionary into a list, you need to build the product from all possible params.

You can use itertools.product to build the product with all values from the dictionary config_overrides.

The resulting list of lists has all combinations of values from config_overrides and each items length is equal to len(config_overrides.keys())

Therefore we can use zip to attach each key to the position of a param in your product.

With the list of pairs, we can now easily create a dictionary.

For the sake of one liners here my version:

from itertools import product
experiments = [dict(zip(config_overrides.keys(), value)) for value in product(*config_overrides.values())]
\$\endgroup\$
3
  • 2
    \$\begingroup\$ Welcome to Code review! This answer doesn't really offer much for the OP. Please (re-) read The help center page How do I write a good answer?. Note it states: "Every answer must make at least one insightful observation about the code in the question. Answers that merely provide an alternate solution with no explanation or justification do not constitute valid CR answers and may be deleted." \$\endgroup\$ Jul 13, 2019 at 2:29
  • 1
    \$\begingroup\$ It might be better if you commented on the code in the question rather than the code in the answers. \$\endgroup\$
    – pacmaninbw
    Jul 13, 2019 at 3:42
  • \$\begingroup\$ I've edited your answer to make your point be the main part of your answer, rather than seem like an after-thought at the end of your answer. Whilst I think this follows a literal interpretation of our rules as it now clearly has "one insightful observation", I think it still lacks an explanation about what the needed steps are and how they're beneficial. Please can you flesh out these aspects of your answer, thank you. \$\endgroup\$
    – Peilonrayz
    Jul 19, 2019 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.