10
\$\begingroup\$

The Merriam-Webster Dictionary API at some point discontinued the use of XML in favor of exclusively returning JSON. This had the unfortunate effect of making finding a definition listed by sense number (i.e. definition 1 a (2)) much more complicated because they're trying to force a semi-arbitrary document based model into nested json.

After reading through the documents I was able to figure out the object nesting hierarchy they return but never directly explain. The relevant nestings I believe go:

(arrows show what an object can be nested in i.e. dt is in a sense or sdsense 
 which can in turn be inside another sense. Sorry for the unclear names, thats
 just what the api calls everything)

dt ============> sense ===========> sseq ========> def
  \==> sdsense ==/   \ \           /  ^  \==> vd ==/
                      \ \==> bs ==    |
                       \           \  |
                        \========+> pseq

Most fields can be either arbitrarily long arrays like [['sense', {...}]] or objects {'sense': [{...}]} depending on context which is a pain especially because sometimes they're nested, sometimes they're flat. Objects can also contain any combination of nested objects repeated any number of times. For example, an sseq can contain a pseq which contains a bs, a bs of it's own, another pseq, and then a few sense objects. Not every one is that complex but there are no restriction stopping them.

To make matters more complicated the sense number (a property of either sense or sdsense) can be explicit: 1 a (2) or implicit: a or (3) which means it depends on a previous sense or sdsense that could be nested in a completely different object. They can also change between implicit and explicit as you go up and down the hierarchy.

And lastly that vd field means that instead of def being an array of sseq it is of the form def: [{'vd': "[verb type]", "sseq": [...]}, {'vd': "[verb type]", "sseq": [...]}] and that sense numbers can suddenly either explicitly or implicitly reference the verb type as well which is never actually nested inside anything relevant.

Check the offial API at the Merriam Webster website for all the quirks and exceptions.

TLDR: json an api returns is a hot mess and there's no easy way to reference the definition 1 a (2) of a dictionary entry. Since a project I'm working on depends on referencing definitions directly by that number I wrote the python parser below to reform the json into a well nested object that can be accessed like parse_resp('word')['1']['a']['(2)']['def'].

My approach writing the code was to start with parsing the dt object and then work my back through each layer, creating a better structured object as I go. lists and dicts are still mixed together inside the object because the relationship between implicit sense numbers needs to be preserved.

Finally, once that object is completely built, there's an admittedly gross method unpack_defs that uses a couple global variables to track the last well-formed sense number while it recursively pulls the entire object inside-out into one well-nested dict.

I'm sure I could rewrite at least that piece better but given this was supposed to be a simple API request in a larger project that the lack of xml support greatly complicated, once it was working for all the exceptions of the API I called it a day and moved on.

Code:

import urllib.request
import json
import os
import re


# TO-DO: GET RID OF GLOBAL STATE
curr_num = ''
curr_let = ''


def reset():
    global curr_let
    global curr_num

    curr_num = ''
    curr_let = ''


# parse_resp is the main function called from other packages and accepts a Merriam Webster api response
# parse_resp defaults to returning only the definitions referenced in the first-use 'date' field because that's what the overall project is interested in
def parse_resp(resp, all_defs=False):
    reset()
    entries = [parse_entry(entry) for entry in resp]
    
    if all_defs == True:
        return [non_empty for non_empty in entries if non_empty]
    
    for entry in entries:
        entry['def'] = get_sense_by_sn(entry['sn'], entry['def'])
    
    entries = [e for e in entries if e['date'] and e['def']]

    return entries


def get_sense_by_sn(sense_number, entry):
    def loop_entry(entry):
        x = entry
        for sn in sense_number:
            x = x[sn]

        return x['def']

    try: return loop_entry(entry)  
    except: pass

    # check for implicit references to transitive / intransitive verb if explicit path fails
    try: return loop_entry(entry['t'])
    except: pass

    try: return loop_entry(entry['i'])
    except: pass

    return ''


def parse_entry(defs):
    (date, sn) = parse_date(defs)
    entry = unpack_defs(parse_defs(defs))

    if entry is None:
        return

    return {
        'date': date,
        'sn': sn,
        'def': entry
    }


def unpack_defs(defs):
    global curr_num
    global curr_let

    if isinstance(defs, dict):
        (key, value) = next(iter(defs.items()))
        key = str(key)

        if is_verb(key):
            return {key: unpack_defs(value)}
        else:
            return fmt_def(key, value)

    if isinstance(defs, list):
        def_dict = dict()

        for d in defs:
            if isinstance(d, list):
                def_dict.update(unpack_defs(d))
            if isinstance(d, dict):
                (key, value) = next(iter(d.items()))
                
                if is_number(key):
                    curr_num = key
                    def_dict.update(fmt_def(key, value))
                elif is_paren(key):
                    
                    def_dict[curr_num][curr_let].update(fmt_def(key, value))
                else:
                    keys = key.split()
                    
                    def nest_keys(keys):
                        global curr_num
                        global curr_let

                        key = keys[0]

                        if len(keys) > 1:

                            if is_number(key):
                                curr_num = key

                            if is_letter(key):
                                curr_let = key

                            return {key: nest_keys(keys[1:])}
                        else:
                            return (fmt_def(key, value))

                    if not curr_num or (is_number(keys[0]) and curr_num != keys[0]):
                        def_dict.update(nest_keys(keys))

                    elif curr_num:
                        def_dict[curr_num].update(fmt_def(key, value))

        return def_dict


def fmt_def(key, value):
    key = str(key)

    return {key: {'def': value}}


def is_number(string):
    try:
        int(string)
        return True
    except:
        return False


def is_paren(string):
    return '(' in string[0]


def is_letter(string):
    return string.isalpha()


def is_verb(string):
    return string == 't' or string == 'i'
        

# First Known Use: date
# Hierarchical Context
#   Top-level member of dictionary entry
# Data Model
#   "date" : string
def parse_date(entry):
    if isinstance(entry, dict):
        if 'date' in entry.keys():
            return clean_date(entry['date'])

    return ('', '')


def clean_date(date_string):
    date = int(re.search(r'[0-9]+', date_string).group())
    sn = re.findall(r'(?<=\|)[\w()]*', date_string) or ['1']
    sn = [non_empty for non_empty in sn if non_empty]

    # quick and dirty conversion of centuries 
    if date < 100:
        date = (date-1) * 100

    return (date, sn)


# Definition: def
# Hierarchical Context
#   Occurs as top-level member of dictionary entry and in dros.
# Data Model
#   array of one or more objects
def parse_defs(defs):
    if isinstance(defs, dict):
        defs = defs['def']

    for entry in defs:
        if is_vd(entry):
            return parse_vd(entry)

        elif is_sseq(entry):
            return parse_sseq(entry)


# Verb Divider: vd
# Hierarchical Context
#   Occurs in def, preceding an sls (optional) and sseq (required)
# Data Model
#   "vd" : string
def is_vd(vd):
    if isinstance(vd, dict):
        return 'vd' in vd.keys()
    
    return False


# Verbs can be transititive or intransitive and referenced as 'i' or 'v' in date
def parse_vd(vd):
    verb_type = 't'
    if 'intransitive' in vd['vd']:
        verb_type = 'i'

    return {verb_type: parse_sseq(vd['sseq'])}


#
# Shared logic for sseq and pseg
#
def parse_array(array):
    sense_list = []

    for sense in array:
        if is_sense(sense):
            sense_list.append(parse_sense(sense))
        elif is_bs(sense):
            sense_list.append(parse_bs(sense))
        elif is_pseq(sense):
            sense_list.append(parse_pseq(sense))
        elif isinstance(sense, list):
            sense_list.append(parse_array(sense))

    if len(sense_list) == 1:
        return sense_list[0]

    return sense_list


# Sense Sequence: sseq
# Hierarchical Context:
#   Occurs in def
# Data Model:
#   "sseq" : array
def is_sseq(sseq):
    if isinstance(sseq, dict):
        return 'sseq' in sseq.keys()

    return False


def parse_sseq(sseq):
    if isinstance(sseq, dict):
        sseq = sseq['sseq']

    sense_list=[]

    for array in sseq:
        sense_list.append(parse_array(array))

    if len(sense_list) == 1:
        return sense_list[0]

    return sense_list


# Parenthesised Sense Sequence:
# Hierarchical Context:
#   Occurs as an element in an sseq array.
# Data Model:
#   array consisting of one or more sense elements and an optional bs element.
def is_pseq(pseq):
    if isinstance(pseq, list):
        return pseq[0] == 'pseq'
    
    return False


def parse_pseq(pseq):
    sense_list = []

    for sense in pseq[1:]:
        sense_list.append(parse_array(sense))

    return sense_list


# Binding Substitution: bs
# Hierarchical Context
#   Occurs as an element in an sseq or pseq array, where it is followed by one or more sense elements.
# Data Model:
#   array of the form ["bs", {sense}]
def is_bs(bs):
    if isinstance(bs, list):
        return 'bs' == bs[0]
    
    return False


def parse_bs(bs):
    return parse_sense(bs[1])


# Sense: sense
# Hierarchical Context
#   Occurs as an element in an sseq array.
# Data Model:
#    object or array consisting of one dt (required) and zero or more et, ins, lbs, prs, sdsense, sgram, sls, sn, or vrs
def is_sense(sense):
    if isinstance(sense, dict):
        return 'sense' in sense.keys()
    elif isinstance(sense, list):
        return 'sense' == sense[0]
    
    return False


def parse_sense(sense):
    if isinstance(sense, dict):
        sense = sense['sense']
    elif isinstance(sense, list):
        sense = sense[1]
    else: return

    sn = 1
    if 'sn' in sense.keys():
        sn = sense['sn']

    sn = str(sn)

    dt = parse_dt(sense['dt'])
    if 'sdsense' in sense.keys():
        dt += parse_sdsense(sense['sdsense'])

    return {sn: dt}


# Divided Sense: sd
# Hierarchical Context
#   Occurs within a sense, where it is always preceded by dt.
# Data Model:
#   "sdsense" : object with the following members:
#   "sd" : string   sense divider (required)
#   et, ins, lbs, prs, sgram, sls, vrs  (optional)
#   dt  definition text (required)
def parse_sdsense(sdsense):
    return parse_dt(sdsense['dt'])


# Defining Text: dt
#Hierarchical Context
#   Occurs as an element in an sseq array.
# Data Model:
#   "dt" : array consisting of one or more elements:
#   ["text", string] where string contains the definition content (required)
#   optional bnw, ca, ri, snote, uns, or vis elements
def parse_dt(dt):
    for elem in dt:
        if elem[0] == 'text':
            ### Implement text cleaning
            return re.sub(r'\{[\w]*\}', '', elem[1]).strip()
\$\endgroup\$
16
\$\begingroup\$

Globals

# TO-DO: GET RID OF GLOBAL STATE

Yep. You seem to already understand this one. Having a reset betrays a place in your design that would be well-suited to an object that can be thrown away and re-constructed, which is usually a more useful model than existing state that can be reset.

Moving on.

Docstrings

Rather than

# parse_resp is the main function called from other packages and accepts a Merriam Webster api response
# parse_resp defaults to returning only the definitions referenced in the first-use 'date' field because that's what the overall project is interested in
def parse_resp(resp, all_defs=False):

the standard asks for

def parse_resp(resp, all_defs=False):
    """
    parse_resp is the main function called from other packages and accepts a Merriam Webster api response
    parse_resp defaults to returning only the definitions referenced in the first-use 'date' field because that's what the overall project is interested in
    """

There are bits of Python that actually parse the latter style for official documentation; so you really should follow that style.

Booleans

if all_defs == True:

should just be

if all_defs:

If you're worried that that isn't explicit enough, the answer is not a comparison to a boolean literal; the answer is to mark your all_defs: bool = False with a type hint.

Exceptions

It doesn't matter how poorly-formed the input data are - this:

try: return loop_entry(entry)  
except: pass

is not appropriate. If you know of a specific data condition that will throw e.g. KeyError, then catch KeyError - or better yet, do a .get() and write logic-by-if instead of logic-by-exception. The current strategy of "swallow all exceptions no matter what" can turn debugging edge cases from an inconvenience into a nightmare.

Tuple unpacking

Outer parens are not needed here:

(date, sn) = parse_date(defs)

Spooky utility methods

Since this is undocumented and with no type hints, I can only guess that this:

def is_paren(string):
    return '(' in string[0]

actually does accept a string. Since that's the case, string[0] will only ever be a one-character string, and so == is more appropriate than in. Better yet, simply call startswith.

The opposite is true here:

def is_verb(string):
    return string == 't' or string == 'i'

can be

def is_verb(string: str):
    return string in {'t', 'i'}

Date parsing

clean_date is undocumented so I don't know what date format it expects, but it's likely that you should be using a "real" date-parsing method out of datetime instead of what you have now.

Overall

You need unit tests. They are not optional for a use case like this. You need examples, pulled from the wild, of JSON bodies representing all of the edge cases you know about.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.