Regrouping list by nested groups

Question

I have a list of units (as in entities in a game):

workers = [engineer1, engineer2, scientist1, scientist2]

These workers have a couple of properties I want to group them by: city and specialization. This grouping is nested, first we group by specializations and then by cities. Then we throw the list of workers that match the filter. Note that for each level we also include relevant information.

Rearranging the initial array into the desired grouped collection, we'd end up with the following (example included):

{
  "workers": [
    {
      "name": "Engineer",
      "cities": [
        {
          "name": "Paris",
          "workers": [
            {
              "name": "Homer Simpson",
              "building_points": 3
            }
          ],
          "population": 2244000
        },
        {
          "name": "Berlin",
          "workers": [
            {
              "name": "Donald Duck",
              "building_points": 4
            }
          ],
          "population": 3500000
        }
      ],
      "description": "Boosts production"
    },
    {
      "name": "Scientist",
      "cities": [
        {
          "name": "Paris",
          "workers": [
            {
              "name": "Marie Curie",
              "building_points": 7
            },
            {
              "name": "Tyrion Lannister",
              "building_points": 7
            }
          ],
          "population": 2244000
        }
      ],
      "description": "Boosts research"
    }
  ]
}

Note that this structure is strictly a requirement. It could probably be rearranged into a more proper way, but that's just how it must be.

Notice we first have a parent "worker" key under which everything is placed. Then we group by specialization including the name and description attributes, then we group by city including the name and population attributes, and finally we have list of workers that satisfy the given specialization and city, also including specific information like name and building_points.

I am relatively new to Python so I am inexperienced in using its full functional potential. I tried using some of its grouping and one-lining tools like map, groupby, lambdas and list comprehensions, but I couldn't simplify the operations past the first level of nesting.

Here is my actual attempt:

grouped_workers = defaultdict(list)

workers_by_spec = {spec: list(workers) for spec, workers in groupby(workers, lambda w: w.specialization)}

for spec, spec_workers in workers_by_spec.items():
    workers_by_city = {city: list(workers) for city, workers in groupby(spec_workers, lambda w: w.city)}

    cities = []
    for city, city_workers in workers_by_city.items():
        cities.append({
            "name": city.name,
            "population": city.population,
            "workers": [{"name": worker.name, "building_points": worker.building_points} for worker in city_workers]
        })

    grouped_workers["workers"].append({"name": spec.name, "description": spec.description, "cities": cities})

Something tells me Python is the right language to perform these instructions in a more elegant, smooth, clear and compact way. Here's where I would appreciate tips that'd help me use the potential this language has to do the same I am already doing but the right way.

---

In case any of you want to try out the actual code I am going to include the classes (and the dummy objects), but note that these are not what I am trying to improve, but only the way I restructure the initial worker list.

class Unit:
    def __init__(self, name):
        self.name = name;


class City:
    def __init__(self, name, population):
        self.name = name
        self.population = population


class Specialization:
    def __init__(self, name, description):
        self.name = name
        self.description = description


class Worker(Unit):
    def __init__(self, name, building_points, city, specialization):
        super().__init__(name)
        self.building_points = building_points
        self.city = city
        self.specialization = specialization

    def __str__(self, *args, **kwargs):
        return "Worker [name=%s, bp=%d, city=%s, spec=%s]" % (
            self.name, self.building_points, self.city, self.specialization)

paris = City("Paris", 2244000)
berlin = City("Berlin", 3500000)
engineer = Specialization("Engineer", "Boosts production")
scientist = Specialization("Scientist", "Boosts research")

engineer1 = Worker("Homer Simpson", 3, paris, engineer)
engineer2 = Worker("Donald Duck", 4, berlin, engineer)
scientist1 = Worker("Marie Curie", 7, paris, scientist)
scientist2 = Worker("Tyrion Lannister", 7, paris, scientist)

Note: The code works in both Python 2 and Python 3.

Answer 1

This is my attempt at trying to convert it to one large list comprehension, I got pretty far but things just stopped going my way... to many nested loops (and not enough pre-planning) and hard to keep track of conceptually.

grouped_workers = defaultdict(list)
specializations = list(x[0] for x in groupby(list_of_workers, lambda x :x.specialization.name))
cities = list(set(x[0] for x in groupby(list_of_workers, lambda x :x.city.name)))
for i in list_of_workers:
    grouped_workers['workers'] = [{'name':x,'cities':[{'name':y,'workers':[{'name':p.name,'building_points':p.building_points} for p in list_of_workers if p.specialization.name == x and p.city.name ==y]} for y in cities ]} for x in specializations]

output:

{'workers': [{'cities': [{'name': 'Berlin',
                      'workers': [{'building_points': 4,
                                   'name': 'Donald Duck'}]},
                     {'name': 'Paris',
                      'workers': [{'building_points': 3,
                                   'name': 'Homer Simpson'}]}],
          'name': 'Engineer'},
         {'cities': [{'name': 'Berlin', 'workers': []},
                     {'name': 'Paris',
                      'workers': [{'building_points': 7,
                                   'name': 'Marie Curie'},
                                  {'building_points': 7,
                                   'name': 'Tyrion Lannister'}]}],
          'name': 'Scientist'}]}

if you were open to trying a different data structure, then maybe there would be a simpler answer, but your data structure needs to be specific, and so your code is specific as well so you already have the best answer that matches the structure you need. And alternative and more "pythonic" approach would be to use a smaller list comprehension in a function to filter by common searches for what you need when you need it, eg. Another thing to note is that you already have the data stored in objects, why do you need to extract the attributes into an dictionary, firstly its using more memory to store it in two places, and secondly its no longer in object-oriented programming and so you loose all the advantages of having classes in the first place.

def group_workers_by_city(list_of_workers,cityname = None):
    return [x for x in list_of_workers if x.city.name == cityname or cityname == None]

def group_workers_by_class(list_of_workers,specialization):
    return [x for x in list_of_workers if x.specialization == specialization]

engineer = Specialization("Engineer", "Boosts production")

print(group_workers_by_city(list_of_workers,'Paris'))
print(group_workers_by_class(list_of_workers,engineer))
print(group_workers_by_city(group_workers_by_class(list_of_workers,engineer),'Paris'))

This implies that you have your Specialization class has an __eq__ overload.