Shuffling multiple iterators in order

Question

I have a Python function w/ specified functionality, and two proposed approaches - which do you prefer, and why? (performance, simplicity, clarity, etc) (both use import random)

---

PROBLEM: Need a plain Python function that accepts an arbitrary number of iterables (tuples, lists, dictionaries), and returns them shuffled in the same order:

a = (1, 2, {3: 4}, 5)
b = [(5,6), [7,8], [9,0], [1,2]]
c = {'arrow': 5, 'knee': 'guard', 0: ('x',2)}

x, y, z = magic(a, b, c)
print(x, y, z, sep='\n')
# ({3: 4}, 1, 2)
# [[9, 0], (5, 6), [7, 8]]
# {0: ('x', 2), 'arrow': 5, 'knee': 'guard'}

The function must:

1. Return iterables shuffled in the same order (see above)
2. Accept any number of iterables
3. Preserve iterables types
4. Support nested iterables of any depth and type
5. Not shuffle nested elements themselves (eg. [7,8] above doesn't become [8,7])
6. Return iterables with length of shortest iterable's length w/o raising error (see above)
7. Return shuffled iterables in the same order they're passed in (see above)

NOTE: requires Python 3.7. Can work with Python 3.6- if excluding dicts, as dicts aren't ordered.

---

SOLUTION 1:

def ordered_shuffle(*args):
    args_types = [type(arg) for arg in args]                               # [1]
    _args      = [arg if type(arg)!=dict else arg.items() for arg in args] # [2]
    args_split = [arg for arg in zip(*_args)]                              # [3]
    args_shuffled = random.sample(args_split, len(args_split))             # [4]
    args_shuffled = map(tuple, zip(*args_shuffled))                        # [5]
    return [args_types[i](arg) for i, arg in enumerate(args_shuffled)]     # [6]

Explanation: img

---

SOLUTION 2:

def shuffle_containers(*args):
    min_length = min(map(len, args))
    idx = list(range(min_length))
    random.shuffle(idx)
    results = []
    for arg in args:
        if isinstance(arg, list):
            results.append([arg[i] for i in idx])
        elif isinstance(arg, tuple):
            results.append(tuple(arg[i] for i in idx))
        elif isinstance(arg, dict):
            items = list(arg.items())
            results.append(dict(items[i] for i in idx))
        else:
            raise ValueError(
                "Encountered", type(arg),
                "expecting only list, dict, or tuple"
            )
    return results

Answer 1

While I do agree with others that Solution 2 is more readable with some improvements, there are also a few improvements that can be done on Solution 1.

• It is unnecessary to construct lists from iterables (e.g., generator expressions) when all that is needed is an iterable. For example,

  _args      = [arg if type(arg)!=dict else arg.items() for arg in args]
  args_split = [arg for arg in zip(*_args)]
  

  Here, the unpacking operator * works on arbitrary iterables. So one can just do

  _args      = (arg if type(arg)!=dict else arg.items() for arg in args)
  args_split = [arg for arg in zip(*_args)]
  

  The parantheses keep the generator expressions without actually materializing them into lists.

• It is better to use isinstance(arg, cls) rather than type(arg) == cls

• Unpacking an iterable into a list can be done using list(iterable), which is more efficient than a list comprehension [arg for arg in iterable] that uses an explicit for-loop.

• This expression

  [args_types[i](arg) for i, arg in enumerate(args_shuffled)]
  

  can be rewritten using zip to avoid the need of indices:

  [cls(arg) for cls, arg in zip(args_types, args_shuffled)]
  

Following is an improved version of Solution 1

def ordered_shuffle(*args):
    arg_types = map(type, args)
    arg_elements = (arg.items() if isinstance(arg, dict) else arg for arg in args)
    zipped_args = list(zip(*arg_elements))
    random.shuffle(zipped_args)
    return [cls(elements) for cls, elements in zip(arg_types, zip(*zipped_args))]

Answer 2

functools.singledispatch

functools library includes the singledispatch() decorator. It lets you provide a generic function, but provide special cases based on the type of the first argument.

import functools
import random

@functools.singledispatch
def shuffle(arg, order):
    """this is the generic shuffle function"""

    lst = list(arg)
    return type(arg)(lst[i] for i in order)


@shuffle.register(dict)
def _(arg, order):
    """this is shuffle() specialized to handle dicts"""

    item = list(arg.items())
    return dict(item[i] for i in order)


def ordered_shuffle(*args):
    min_length = min(map(len, args))

    indices = random.sample(range(min_length), min_length)

    return [shuffle(arg, indices) for arg in args]

Usage:

a = (1, 2, {3: 4}, 5)
b = [(5,6), [7,8], [9,0], [1,2]]
c = {'arrow': 5, 'knee': 'guard', 0: ('x',2)}

ordered_shuffle(a, b, c)

Output:

[({3: 4}, 1, 2),
 [[9, 0], (5, 6), [7, 8]],
 {0: ('x', 2), 'arrow': 5, 'knee': 'guard'}]

Answer 3

I think your solution 2 is heading the right direction here. What I consider it's advantages over solution 1:

1. It is much more readable
2. It clearly shows that every member of *args is treated the same.

You might want to generic-ify it a bit more to handle more types. For example, the following has a good chance of also handling custom container types:

import collections
# -- Other code ---
        if isinstance(arg, collections.abc.Mapping):
            items = list(arg.items())
            results.append(type(arg)(items[i] for i in idx))
        else:
            results.append(type(arg)(arg[i] for i in idx))

Which will inspect the type of the iterable and feed it an iterator in an attempt to create a new one. This version here will handle lists and tuples the same as your does. If any custom container type supports a __init__(self, *args) constructor to fill itself, it'll also work.

To be honest, those will probably be very rare. I've never seen one. But this is a very easy generic to support, and it's already worth it because you have the same code handling tuples and lists, in my opinion.

Keep in mind you need at least python 3.6 for dictionaries to have a stable insertion ordering.

Readability > Shortness.

Unless you're planning on visiting Codegolf, don't do this. But if you do, you can shorten your functions a lot more, like this:

f = lambda *a:[type(y[i])(b) for i,bin enumerate(map(list, zip(*random.sample([y for y in 
    zip(*[x if type(x)!=dict else x.items() for x in a])], min(len(z) for z in a)))))]

I hope this example makes it abundantly clear why readability is important. If not, try and figure out if there's a bug in here.

Answer 4

Below is an answer highlight and performance comparison of various solutions. Note: the differences should be negligible in practice - if otherwise, timing variance is substantial, and all are ~on par (per benchmark)

---

SOLUTION 1 GZ0-Mod+:

def ordered_shuffle(*args):
   zipped_args = list(zip(*(a.items() if isinstance(a, dict) else a for a in args)))
   random.shuffle(zipped_args)
   return [cls(elems) for cls, elems in zip(map(type, args), zip(*zipped_args))] 

---

SOLUTION 1 GZ0-Mod++: (latest)

def ordered_shuffle(*args):
    zipped_args = list(zip(*(a.items() if isinstance(a, dict) else a for a in args)))
    np.random.shuffle(zipped_args)
    return [(_type(data) if _type != np.ndarray else np.asarray(data)) 
            for _type, data in zip(map(type, args), zip(*zipped_args))]

This solution is simply the first but accounting for numpy arrays (also _type should be more intuitive, and spares random import since numpy is already imported).

---

UPDATE: Ran a performance benchmark - results for iterations = 1e7, and a, b, c = tuple, list, dict of length 10, nested, respectively:

• #1: Solution 1 GZ0-Mod+   114.2 sec, 71.2% faster than slowest
• #2: Solution 2   ---------- 131.9 sec, 48.0% faster than slowest
• #3: Solution 1   ---------- 155.7 sec, 25.4% faster than slowest
• #4: Gloweye's lambda  --- 160.5 sec, 21.7% faster than slowest
• #5: RootTwo's functools -- 195.3 sec

---

Selected answer rationale:

• Best minimalism, acceptable readability
• Uses plain Python - no high-level libraries; easier to debug & understand
• Well-explained answer, describing pitfalls of original approach & how it's remedied
• Best performance (a bonus, not major factor)