Parsing a string of key-value pairs as a dictionary

Question

I always use nested list and dictionary comprehension for unstructured data and this is a common way I use it.

In [14]: data = """
41:n
43:n
44:n
46:n
47:n
49:n
50:n
51:n
52:n
53:n
54:n
55:cm
56:n
57:n
58:n"""
In [15]: {int(line.split(":")[0]):line.split(":")[1] for line in data.split("\n") if len(line.split(":"))==2}
Out [15]:
{41: 'n',
 43: 'n',
 44: 'n',
 46: 'n',
 47: 'n',
 49: 'n',
 50: 'n',
 51: 'n',
 52: 'n',
 53: 'n',
 54: 'n',
 55: 'cm',
 56: 'n',
 57: 'n',
 58: 'n'}

Here I am doing line.split(":")[0] three times. Is there any better way to do this?

Answer 1

Your string looks very similar to the YAML syntax. Indeed it is almost valid syntax for an associative list, there are only spaces missing after the :. So, why not use a YAML parser?

import yaml

data = """
41:n
43:n
44:n
46:n
47:n
49:n
50:n
51:n
52:n
53:n
54:n
55:cm
56:n
57:n
58:n"""

print(yaml.load(data.replace(":", ": ")))
# {41: 'n',
#  43: 'n',
#  44: 'n',
#  46: 'n',
#  47: 'n',
#  49: 'n',
#  50: 'n',
#  51: 'n',
#  52: 'n',
#  53: 'n',
#  54: 'n',
#  55: 'cm',
#  56: 'n',
#  57: 'n',
#  58: 'n'}

You might have to install it first, which you can do via pip install yaml.

Answer 2

You have too much logic in the dict comprehension:

{int(line.split(":")[0]):line.split(":")[1] for line in data.split("\n") if len(line.split(":"))==2}

First of all, let's expand it to a normal for-loop:

>>> result = {}
>>> for line in data.split("\n"):
...     if len(line.split(":"))==2:
...         result[int(line.split(":")[0])] = line.split(":")[1]
>>> result

I can see that you use the following check if len(line.split(":"))==2: to eliminate the first blank space from the data.split("\n"):

>>> data.split("\n")
['',
 '41:n',
 '43:n',
 ...
 '58:n']

But the docs for str.split advice to use str.split() without specifying a sep parameter if you wanna discard the empty string at the beginning:

>>> data.split()
['41:n',
 '43:n',
 ...
 '58:n']

So, now we can remove unnecessary check from your code:

>>> result = {}
>>> for line in data.split():
...     result[int(line.split(":")[0])] = line.split(":")[1]
>>> result

Here you calculate line.split(":") twice. Take it out:

>>> result = {}
>>> for line in data.split():
...    key, value = line.split(":")
...    result[int(key)] = value
>>> result

This is the most basic version. Don't put it back to a dict comprehension as it will still look quite complex. But you could make a function out of it. For example, something like this:

>>> def to_key_value(line, sep=':'):
...     key, value = line.split(sep)
...     return int(key), value

>>> dict(map(to_key_value, data.split()))
{41: 'n',
 43: 'n',
 ...
 58: 'n'}

Another option that I came up with:

>>> from functools import partial
>>> lines = data.split()
>>> split_by_colon = partial(str.split, sep=':')
>>> key_value_pairs = map(split_by_colon, lines)
>>> {int(key): value for key, value in key_value_pairs}
{41: 'n',
 43: 'n',
 ...
 58: 'n'}

Also, if you don't want to keep in memory a list of results from data.split, you might find this helpful: Is there a generator version of string.split() in Python?

Answer 3

Note it is much easier to read if you chop up the comprehension into blocks, instead of having them all on one line

You could use unpacking to remove some usages of line.split

>>> {
...    int(k): v
...    for line in data.split() 
...    for k, v in (line.split(':'),)
... }
{41: 'n', 43: 'n', 44: 'n', 46: 'n', 47: 'n', 49: 'n', 50: 'n', 51: 'n', 52: 'n', 53: 'n', 54: 'n', 55: 'cm', 56: 'n', 57: 'n', 58: 'n'}

Or if the first argument can be of str type you could use dict().

This will unpack the line.split and convert them into a key, value pair for you

>>> dict(
...    line.split(':') 
...    for line in data.split() 
... )
{'41': 'n', '43': 'n', '44': 'n', '46': 'n', '47': 'n', '49': 'n', '50': 'n', '51': 'n', '52': 'n', '53': 'n', '54': 'n', '55': 'cm', '56': 'n', '57': 'n', '58': 'n'}

Answer 4

There's nothing wrong with the solution you have come with, but if you want an alternative, regex might come in handy here:

In [10]: import re
In [11]: data = """ 
    ...: 41:n 
    ...: 43:n 
    ...: 44:n 
    ...: 46:n 
    ...: 47:n 
    ...: 49:n 
    ...: 50:n 
    ...: 51:n 
    ...: 52:n 
    ...: 53:n 
    ...: 54:n 
    ...: 55:cm 
    ...: 56:n 
    ...: 57:n 
    ...: 58:n"""                                                                                                                                                                                                                                                         

In [12]: dict(re.findall(r'(\d+):(.*)', data))                                                                                                                                                                                                                           
Out[12]: 
{'41': 'n',
 '43': 'n',
 '44': 'n',
 '46': 'n',
 '47': 'n',
 '49': 'n',
 '50': 'n',
 '51': 'n',
 '52': 'n',
 '53': 'n',
 '54': 'n',
 '55': 'cm',
 '56': 'n',
 '57': 'n',
 '58': 'n'}

Explanation:

1st Capturing Group (\d+):

\d+ - matches a digit (equal to [0-9])
+ Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy)
: matches the character : literally (case sensitive)

2nd Capturing Group (.*):

.* matches any character (except for line terminators)
* Quantifier — Matches between zero and unlimited times, as many times as possible, giving back as needed (greedy)

If there might be letters in the first matching group (though I doubt it since your casting that to an int), you might want to use:

dict(re.findall(r'(.*):(.*)', data))

I usually prefer using split()s over regexes because I feel like I have more control over the functionality of the code.

You might ask, why would you want to use the more complicated and verbose syntax of regular expressions rather than the more intuitive and simple string methods? Sometimes, the advantage is that regular expressions offer far more flexibility.

---

Regarding the comment of @Rahul regarding speed I'd say it depends:

Although string manipulation will usually be somewhat faster, the actual performance heavily depends on a number of factors, including:

• How many times you parse the regex
• How cleverly you write your string code
• Whether the regex is precompiled

As the regex gets more complicated, it will take much more effort and complexity to write equivlent string manipulation code that performs well.

As far as I can tell, string operations will almost always beat regular expressions. But the more complex it gets, the harder it will be that string operations can keep up not only in performance matters but also regarding maintenance.