Gathering statistics on a list of dicts

Question

Imagine a large collection of Python dictionaries (a list of dicts just to make it simple). Each element of this list has hundreds of keys (aka attributes). More importantly not all elements share the same attributes. There is considerable overlap though. So this is a collection of semi-structured data.

I want to get a better sense of how this data looks like, so I wrote a small helper class to do what I want. More specifically, some attributes only take a small set of values. I want to know what kind of values a these keys/attributes take, and the count of each unique value. For example, for an attribute called 'color' I want to know that there are 20 reds, 5 greens, 8 blues, and 1 purple. I also want to know how many dictionaries do not have this attribute (i.e., this attribute is not defined for them).

If you are familiar with R, the information I am after is very similar to what you get when calling the built-in function summary() on a dataframe.

Here's my Python class:

class AttributeValueCount:
    '''
    A helper class to collect statistics about the different values that attributes 
    (keys in dicts) take. Initialised with a list of attributes we'd like monitored.
    We can then call the update_counts method repeadedly with different dictionaries,
    to update the number of times each attribute takes specific values. 
    '''
    def __init__(self, keylist):
        self._keylist = keylist
        self._counts = {key: defaultdict(int) for key in keylist}

    def update_counts(self, single_dict):
        for key in self._keylist:
            if key in single_dict: 
                self._counts[key][single_dict[key]] += 1
            else:
                self._counts[key]['Non existent'] += 1

    def __repr__(self):
        s = ''
        for key in self._keylist:
            s += '-- {} --\n'.format(key)
            for value, count in self._counts[key].items():
                s += '\t {} : {}\n'.format(value, count)
        return s

Example Input

data = [
    {'name': 'apple', 'type': 'farm', 'color': 'red', 'volume': 2000},
    {'name': 'orange', 'type': 'farm', 'color': 'orange', 'volume': 5000},
    {'name': 'elderberry', 'type': 'wild', 'color': None},
    {'name': 'pythonfruit', 'type': 'extinct'} 
]

value_counts = AttributeValueCount(['type', 'color'])

for fruit in data:
    value_counts.update_counts(fruit)

print(value_counts)

Example Output

-- type --
     farm : 2
     wild : 1
     extinct : 1
-- color --
     red : 1
     orange : 1
     None : 1
     Non existent : 1

Does this implementation look pythonic to you? What other ways could I achieve my result? I am mostly interested about major design decisions and not so much about style or minor enhancements. For example, a minor enhancement would be to allow update_counts to take lists of dicts, or any kind of iterable. A major decision would be: I should not be defining a class at all, maybe a function factory would be better (idk, I am just throwing around terms). But I do not want to deter you, any feedback/observation is useful. I just realised for instance that I don't do much argument validation.

Update

The answer by Mathias provided some good points, and I also realised I should have included a bit more information: As mentioned the dictionaries contain hundreds of attributes, and I am only interested in some of them. To put the "some" in perspective, I only need to know the counts for at most 10-15 of the attributes. I also have thousands of dictionaries. For this reason I believe that flowA: specifying the attributes you want and then feed in the data, is preferable to flowB: giving all the data to the class and then ask questions on specific attributes. To my mind FlowB should just be a function definition count_values(data, key) not a class definition. Moreover, the added advantage of flowA is that you do not have to have all data present, you can increment the count incrementally.

Answer 1

Minor improvements

I think defining value_counts = AttributeValueCount('type', 'color') would be more user-friendly than having to pass in a list, so you could just define the constructor as

class AttributeValueCount:
    def __init__(self, *keylist):
        self._keylist = keylist
        self._counts = {key: defaultdict(int) for key in keylist}

and remove that need for an iterable. And if a user still has an iterable to pass to the constructor, they can still unpack it:

keys = ['type', 'color']
value_counts = AttributeValueCount(*keys)

I would also use add and update methods as in a dictionary or a set: add is for a single element and update is for several:

    def update(self, iterable):
        for element in iterable:
            self.add(element)

    def add(self, element):
        for key in self._keylist:
            if key in element: 
                self._counts[key][element[key]] += 1
            else:
                self._counts[key]['Non existent'] += 1

You can also simplify this add method using dict.get():

    def add(self, element):
        for key in self._keylist:
            category = element.get(key, 'Non existent')
            self._counts[key][category] += 1

Lastly, the __repr__ method is meant to return an "official" representation of the object, something that could potentialy be passed to eval to reconstruct it. I’d rather use __str__ here, or even something completely different, like the summary you’re talking about:

    def summary(self):
        return '\n'.join(
                '-- {} --\n{}'.format(
                    key,
                    '\n'.join(
                        '\t {}: {}'.format(value, count)
                        for value, count in self._counts[key].items()
                    ))
                for key in self._keylist
        )

Note that I used str.join instead of string concatenation as it is usually faster and more memory efficient.

Major improvements

The first thing I would add is the ability to specify the name of the "missing" category. I have a tendency to consider None in such cases, rather than any arbitrary string, but seeing at your data, it seems like None could be a valid entry. So I'd say: let the user decide based on their data.

There is still a question as where to specify this default: in __init__ or at each update/add call? Both might be valid.

An other change I would make is to completely invert the flow of things. Instead of specifying the keys you're interested in beforehand, you just store the data. And you provide methods to retrieve the counts associated to each key latter, this will allow data exploration to be more dynamic. This will give a session like:

>>> counts = AttributeValueCount(data, missing='Non existent')
>>> print(counts['type'])
farm: 2
wild: 1
extinct: 1
>>> print(counts['color'])
red: 1
orange: 1
None: 1
Non existent: 1
>>> print(counts['volume'])
2000: 1
5000: 1
Non existent: 2

Which seems easier than having to recreate an AttributeValueCount('volume') afterwards.

There, also, you have 2 possibilities:

1. only store values using __init__, add and update and compute the counts later (\$\mathcal{O}(n)\$ at each __getitem__);
2. compute the counts upfront (\$\mathcal{O}(n\times{}m)\$) but have a constant time __getitem__.

I would go for the second solution as it is interesting how to deal with missing keys when you first encounter a new category:

from collections import Counter


class AttributeValueCount:
    def __init__(self, iterable, *, missing=None):
        self._missing = missing
        self.length = 0
        self._counts = {}
        self.update(iterable)

    def update(self, iterable):
        categories = set(self._counts)
        for length, element in enumerate(iterable, self.length):
            categories.update(element)
            for category in categories:
                try:
                    counter = self._counts[category]
                except KeyError:
                    self._counts[category] = counter = Counter({self._missing: length})
                counter[element.get(category, self._missing)] += 1
        self.length = length + 1

    def add(self, element):
        self.update([element])

    def __getitem__(self, key):
        return self._counts[key]

    def summary(self, key=None):
        if key is None:
            return '\n'.join(self.summary(key) for key in self._counts)

        return '-- {} --\n{}'.format(key, '\n'.join(
                '\t {}: {}'.format(value, count)
                for value, count in self._counts[key].items()
        ))

Usage being:

>>> from count_attributes import AttributeValueCount
>>> data = [
...     {'name': 'apple', 'type': 'farm', 'color': 'red', 'volume': 2000},
...     {'name': 'orange', 'type': 'farm', 'color': 'orange', 'volume': 5000},
...     {'name': 'elderberry', 'type': 'wild', 'color': None},
...     {'name': 'pythonfruit', 'type': 'extinct'} 
... ]
>>> c = AttributeValueCount(data, missing='Non existent')
>>> print(c.summary())
-- type --
     Non existent: 0
     farm: 2
     wild: 1
     extinct: 1
-- name --
     Non existent: 0
     apple: 1
     orange: 1
     elderberry: 1
     pythonfruit: 1
-- volume --
     Non existent: 2
     2000: 1
     5000: 1
-- color --
     Non existent: 1
     red: 1
     orange: 1
     None: 1
>>> c['volume']
Counter({'Non existent': 2, 2000: 1, 5000: 1})
>>> c.add({'foo': 'bar'})
>>> print(c.summary())
-- name --
     Non existent: 1
     apple: 1
     orange: 1
     elderberry: 1
     pythonfruit: 1
-- type --
     Non existent: 1
     farm: 2
     wild: 1
     extinct: 1
-- volume --
     Non existent: 3
     2000: 1
     5000: 1
-- color --
     Non existent: 2
     red: 1
     orange: 1
     None: 1
-- foo --
     Non existent: 4
     bar: 1