Imagine a large collection of Python dictionaries (a list of dicts just to make it simple). Each element of this list has hundreds of keys (aka attributes). More importantly not all elements share the same attributes. There is considerable overlap though. So this is a collection of semi-structured data.
I want to get a better sense of how this data looks like, so I wrote a small helper class to do what I want. More specifically, some attributes only take a small set of values. I want to know what kind of values a these keys/attributes take, and the count of each unique value. For example, for an attribute called 'color' I want to know that there are 20 reds, 5 greens, 8 blues, and 1 purple. I also want to know how many dictionaries do not have this attribute (i.e., this attribute is not defined for them).
If you are familiar with R, the information I am after is very similar to what you get when calling the built-in function summary() on a dataframe.
Here's my Python class:
class AttributeValueCount:
'''
A helper class to collect statistics about the different values that attributes
(keys in dicts) take. Initialised with a list of attributes we'd like monitored.
We can then call the update_counts method repeadedly with different dictionaries,
to update the number of times each attribute takes specific values.
'''
def __init__(self, keylist):
self._keylist = keylist
self._counts = {key: defaultdict(int) for key in keylist}
def update_counts(self, single_dict):
for key in self._keylist:
if key in single_dict:
self._counts[key][single_dict[key]] += 1
else:
self._counts[key]['Non existent'] += 1
def __repr__(self):
s = ''
for key in self._keylist:
s += '-- {} --\n'.format(key)
for value, count in self._counts[key].items():
s += '\t {} : {}\n'.format(value, count)
return s
Example Input
data = [
{'name': 'apple', 'type': 'farm', 'color': 'red', 'volume': 2000},
{'name': 'orange', 'type': 'farm', 'color': 'orange', 'volume': 5000},
{'name': 'elderberry', 'type': 'wild', 'color': None},
{'name': 'pythonfruit', 'type': 'extinct'}
]
value_counts = AttributeValueCount(['type', 'color'])
for fruit in data:
value_counts.update_counts(fruit)
print(value_counts)
Example Output
-- type --
farm : 2
wild : 1
extinct : 1
-- color --
red : 1
orange : 1
None : 1
Non existent : 1
Does this implementation look pythonic to you? What other ways could I achieve my result? I am mostly interested about major design decisions and not so much about style or minor enhancements. For example, a minor enhancement would be to allow update_counts to take lists of dicts, or any kind of iterable. A major decision would be: I should not be defining a class at all, maybe a function factory would be better (idk, I am just throwing around terms). But I do not want to deter you, any feedback/observation is useful. I just realised for instance that I don't do much argument validation.
Update
The answer by Mathias provided some good points, and I also realised I should have included a bit more information: As mentioned the dictionaries contain hundreds of attributes, and I am only interested in some of them. To put the "some" in perspective, I only need to know the counts for at most 10-15 of the attributes. I also have thousands of dictionaries. For this reason I believe that flowA: specifying the attributes you want and then feed in the data, is preferable to flowB: giving all the data to the class and then ask questions on specific attributes. To my mind FlowB should just be a function definition count_values(data, key) not a class definition. Moreover, the added advantage of flowA is that you do not have to have all data present, you can increment the count incrementally.