2
\$\begingroup\$

I'm working on a parser library for substituting strings in dictionaries, with things like globals etc. Wanting to get opinions on the direction I'm taking and where I can improve the script. This currently works

class DictParser(object):
    def __init__(self, dictionary):
        self.working_dict = dictionary

    def __traverse(self, obj, callback=None):
        if isinstance(obj, dict):
            return {k: self.__traverse(v, callback) for k, v in obj.items()}
        elif isinstance(obj, list):
            return [self.__traverse(elem, callback) for elem in obj]
        else:
            if not callback:
                return obj
            else:
                return callback(obj)

    def replace_with_global_var(self, pattern):
        pattern_variable = re.compile(pattern)

        def substitute(x):
            if x.group(1) in globals():
                return globals()[x.group(1)]
            else:
                raise IndexError('Missing Global: %s' % x.group(1))

        def transformer(item):
            if isinstance(item, str):
                return pattern_variable.sub(substitute, item)
            else:
                return item

        self.working_dict = self.__traverse(self.working_dict, callback=transformer)

    def retrieve(self):
        return self.working_dict

Usage:

a = 'blah'

my_dict = {
    'test': '((a))',
    'another': 'my ((a)) test',
    'nested': {
        'child': 'here is ((a)) ((a))'
    },
    'boo': 1,
    'test': False
}

d = DictParser(my_dict)
d.replace_with_global_var(r'\(\((.+?)\)\)')
print(d.retrieve())

This was a reasonably quick hack together, tried using repl so I could print the d variable directly but didn't quite work out.

\$\endgroup\$
2
\$\begingroup\$

I would have used str.format together with a transparent dictionary to achieve this result. It gives you the familiarity of the format syntax and not having to come up with the regex on your own. This comes at the cost of not being able to specify the regex anymore at all.

The transparent dict is taken from this answer and is needed because otherwise format raises a KeyError when trying to do a partial replace.

from collections import defaultdict

class SafeDict(dict):
    def __missing__(self, key):
        return '{' + key + '}'


def parse(val, vars, callback=None):
    if callback:
        val = callback(val)
    if isinstance(val, dict):
        return parse_dict(val, vars, callback)
    elif isinstance(val, list):
        return parse_list(val, vars, callback)
    try:
        return val.format(**SafeDict(vars))
    except Exception:
        return val


def parse_list(l, vars, callback=None):
    parsed = type(l)()
    for val in l:
        parsed.append(parse(val, vars, callback))
    return parsed


def parse_dict(d, vars, callback=None):
    parsed = type(d)()
    for key, val in d.items():
        parsed[key] = parse(val, vars, callback)
    return parsed

parse_dict and parse_list could be replaced by the one-liners you use. This way, however, they return the same data type as was originally there (this is relevant for sub-classes of list or dict). If you don't need this, just replace them with what you've got.

usage:

a = 'blah' 
my_dict = {'test': '{a}', 
           'another': 'my {a} test',
           'nested': {'child': 'here is {a} {a}'},
           'boo': 1,
           'test2': False}
print(parse(my_dict, globals()))

When adding more parsers for other data types than list or dict (their sub-classes are already included), you would probably define a mapping from type to parsers to avoid repetition in parse.

As an alternative you should have a look at f-strings, which will be included in Python 3.6, scheduled to be released soon.

With this you could do:

a = 'blah' 
table = "Users"
row = 42
parsed_dict = {'test': f'{a}', 
               'another': f'my {a} test',
               'nested': {'child': f'here is {a} {a}'},
               'boo': 1,
               'test2': False,
               'complex': f'result: {db_lookup(table, row))}'}
\$\endgroup\$
  • \$\begingroup\$ Unfortunately that'd be quite limiting, I need to use this with multiple replace functions, not just pthe variable one. I plan on extending it to do things like looking up the value of a DB table etc (i.e. $tablename:column:filter$) so the regex' will be vital I think. \$\endgroup\$ – Allan M Oct 20 '16 at 6:58
  • \$\begingroup\$ @AllanM You should have a look at f-strings, which will be included in Python 3.6. They allow general expressions inside the {}. \$\endgroup\$ – Graipher Oct 20 '16 at 7:04
  • \$\begingroup\$ Unfortunately I need to support Python 2 so that's not gonna be an option. I'll have a play with your suggestion. \$\endgroup\$ – Allan M Oct 20 '16 at 7:07
  • \$\begingroup\$ @Graipher Except f-strings need that the expression within the curly braces is valid at the time the f-string is defined. This can be a limitation if you want to build a query based on the variables you want to replace. Instead of fetching the whole DB beforehand. \$\endgroup\$ – Mathias Ettinger Oct 20 '16 at 7:11
  • \$\begingroup\$ @AllanM Yes, it definitely is. And there is actually already a solution to this, i-strings. They are just like f-strings but delay the evaluation by allowing you to specify your own __format__ function. But that won't come till a later Python version and is irrelevant if you need to support Python 2.x. \$\endgroup\$ – Graipher Oct 20 '16 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.