Optimising function to strip honorifics from names

Question

Problem

I have a list of around ~1000 honorifics, see below for a sample.

Given an input string of a name, for example "her majesty queen elizabeth windsor", the function should return "elizabeth windsor". If there is no honorific present at the start of the name (to simplify the problem), the function should simple return the name itself (e.g. elizabeth windsor -> elizabeth windsor).

I have pretty intense latency constraints, so need to optimise this code as much as possible.

Working solution

Here is my working solution, there are some additional constraints to reduce false positives (for example lance is both an honorific and a first name), see the unit tests:

def strip_honorific(source: str, honorifics: List[str]) -> str:
    source_tokens = source.split()
    if len(source_tokens) > 2:
        for honorific in honorifics:
            if source.startswith(f"{honorific} "):
                stripped_source = source[len(honorific) + 1 :]
                if len(stripped_source.split()) > 1:
                    return stripped_source
    return source

Unit tests

def test_honorifics():
    assert strip_honorific(source="her majesty queen elizabeth windsor", honorifics = honorifics) == "elizabeth windsor"
    assert strip_honorific(source="elizabeth windsor", honorifics = honorifics) == "elizabeth windsor"
    assert strip_honorific(source="mrs elizabeth windsor", honorifics = honorifics) == "elizabeth windsor"
    assert strip_honorific(source="mrselizabeth windsor", honorifics = honorifics) == "mrselizabeth windsor"
    assert strip_honorific(source="mrselizabeth windsor", honorifics = honorifics) == "mrselizabeth windsor"
    assert strip_honorific(source="her majesty queen", honorifics = honorifics) == "her majesty queen"
    assert strip_honorific(source="her majesty queen elizabeth", honorifics = honorifics) == "her majesty queen elizabeth"
    assert strip_honorific(source="kapitan fred", honorifics = honorifics) == "kapitan fred"
    
    
test_honorifics()

Benchmark

For a basic benchmark, I've used the below list of honorifics (minus the ellipses).

source_lst = [
    "her majesty queen elizabeth windsor",
    "mr fred wilson",
    "the rt hon nolan borak",
    "his most eminent highness simon smithson", 
    "kapteinis jurijs jakovļevs", 
    "miss nancy garland",
    "missnancy garland",
]


times = []
for _ in range(1000):
    for source in source_lst:
        t0 = time.time()
        strip_honorific(source=source, honorifics = honorifics)
        times.append(time.time() - t0)

print(f"Mean time: {sum(times)/ len(times)}s") # Mean time: 5.11584963117327e-06s

Honorifics list

honorifics = [
    "mr", 
    "mrs", 
    "the hon",
    "the hon dr",
    "the hon lady",
    "the hon lord",
    "the hon mrs",
    "the hon sir",
    "the honourable",
    "the rt hon",
    "her majesty queen",
    "his majesty king",
    "vina",
    "flottiljamiral",
    "superintendent",
    "rabbi",
    "diraja",
    "domnul",
    "kindralleitnant",
    "countess",
    "pan",
    "khatib",
    "zur",
    "vice",
    "don",
    "flotiles",
    "dipl",
    "his most eminent highness", 
    ...
    "the reverend", 
    "archbishop", 
    "sheik", 
    "shaikh", 
    "the rt hon lord", 
    "la tres honorable"
    "ekselence", 
    "kapteinis", 
    "kapitan", 
    "excellenza"
    "mr", 
    "mrs", 
    "miss"
] 

Answer 1

Combine all the honorifics into a regular expression so they can all be tested at once rather that testing them one-by-one. Sort the honorifics longest to shortest

import re

honorifics.sort(key=len, reverse=True)
all_honorifics = '|'.join(f'{h} ' for h in honorifics)

honorifics_regex = re.compile(rf"\A(?:{all_honorifics})")

def strip_honorific(source, honorifics_regex):
    match = honorifics_regex.search(source)
    if match:
        stripped_source = source[match.end():]
        if len(stripped_source.split()) > 1:
            return stripped_source
        
    return source

Use the timeit module from the standard library:

import timeit

code = "for source in source_lst: name = strip_honorific(source, regex)"

timeit.timeit(code, globals=globals())

The regex version runs in about 5ms, compared to 20ms for the original code.