8
\$\begingroup\$

In English, "Sh" is two letters. In other languages it's considered a single letter. I'm trying to calculate the length of a string in a Unicode aware way, with this in mind. I'm imagining a function like

def count_letters(my_string, lang="en")

But here's how I would do that for just one language, using a library that has had no new versions since 2015, uniseg:

from uniseg.graphemecluster import grapheme_clusters


def albanian_digraphs(s, breakables):
    digraphs = ["Dh", "Gj", "Ll", "Nj", "Rr", "Sh", "Th", "Xh", "Zh"]
    digraphs += [d.lower() for d in digraphs]
    for i, breakable in enumerate(breakables):
        for first, second in digraphs:
            if s.endswith(first, 0, i) and s.startswith(second, i):
                yield 0
                break
        else:
            yield breakable


# from https://sq.wiktionary.org/wiki/Speciale:PrefixIndex?prefix=dh
for text in ('dhallanik', 'dhelpëror', 'dhembshurisht', 'dhevështrues', 'dhimbshëm', 'dhjamosje', 'dhjetëballësh', 'dhjetëminutësh', 'dhogaç', 'dhogiç', 'dhomë-muze', 'dhuratë', 'dhëmbinxhi', 'dhëmbçoj', 'dhëmbëkatarosh'):
    print(list(grapheme_clusters(text, albanian_digraphs)))

#['dh', 'a', 'll', 'a', 'n', 'i', 'k']
#['dh', 'e', 'l', 'p', 'ë', 'r', 'o', 'r']
#['dh', 'e', 'm', 'b', 'sh', 'u', 'r', 'i', 'sh', 't']
#['dh', 'e', 'v', 'ë', 'sh', 't', 'r', 'u', 'e', 's']
#['dh', 'i', 'm', 'b', 'sh', 'ë', 'm']
#['dh', 'j', 'a', 'm', 'o', 's', 'j', 'e']
#['dh', 'j', 'e', 't', 'ë', 'b', 'a', 'll', 'ë', 'sh']
#['dh', 'j', 'e', 't', 'ë', 'm', 'i', 'n', 'u', 't', 'ë', 'sh']
#['dh', 'o', 'g', 'a', 'ç']
#['dh', 'o', 'g', 'i', 'ç']
#['dh', 'o', 'm', 'ë', '-', 'm', 'u', 'z', 'e']
#['dh', 'u', 'r', 'a', 't', 'ë']
#['dh', 'ë', 'm', 'b', 'i', 'n', 'xh', 'i']
#['dh', 'ë', 'm', 'b', 'ç', 'o', 'j']
#['dh', 'ë', 'm', 'b', 'ë', 'k', 'a', 't', 'a', 'r', 'o', 'sh']
\$\endgroup\$
  • 1
    \$\begingroup\$ @Graipher According to the code, s is the original string and breakables is a list of similar length containing zeroes or ones. Ones mean break before the corresponding letter, zeroes mean keep the corresponding letter in the same grapheme than the previous one. The function's purpose is to modify breakables before breaks are applied to s. \$\endgroup\$ – 409_Conflict Sep 13 at 8:43
  • \$\begingroup\$ @MathiasEttinger Ah, I did not see that the function was used by another function...thanks! \$\endgroup\$ – Graipher Sep 13 at 8:47
3
\$\begingroup\$

For starter, you do not need to create you digraphs list each time the function is called: they won't change, so better create them once as a global constant. You also forgot to add capitalized versions in the list so that 'HELLO' is split into ['H', 'E', 'LL', 'O'] instead of the current ['H', 'E', 'L', 'L', 'O'].

Second, your linear research in the digraphs list can be time consuming when all you want to know is if characters at position i-1 (if any) and i forms a digraph present in your list. I’d rather write it as yield 0 if s[i-1:i+1] in digraphs else breakable. Of course, for this to work efficiently, you will need digraphs to be a set instead of a list where lookups are \$\mathcal{O}(1)\$ instead of \$\mathcal{O}(n)\$.

Lastly, I would only consider lowercase variants and turn the selected two characters lowercase before checking:

from uniseg.graphemecluster import grapheme_clusters


DIGRAPHS = {"dh", "gj", "ll", "nj", "rr", "sh", "th", "xh", "zh"}


def albanian_digraphs(s, breakables):
    for i, breakable in enumerate(breakables):
        yield 0 if s[i-1:i+1].lower() in DIGRAPHS else breakable


if __name__ == '__main__':    
    # from https://sq.wiktionary.org/wiki/Speciale:PrefixIndex?prefix=dh
    for text in ('dhallanik', 'dhelpëror', 'dhembshurisht', 'dhevështrues', 'dhimbshëm', 'dhjamosje', 'dhjetëballësh', 'dhjetëminutësh', 'dhogaç', 'dhogiç', 'dhomë-muze', 'dhuratë', 'dhëmbinxhi', 'dhëmbçoj', 'dhëmbëkatarosh'):
        print(list(grapheme_clusters(text, albanian_digraphs)))

Also note the use of if __name__ == '__main__' to separate the actual code from the tests.

\$\endgroup\$
  • \$\begingroup\$ According to Wikipedia the uppercase version of the Albanian letter "sh" is "Sh", not "SH". So not clustering "SH" was deliberate. But I guess it doesn't matter, it's worth the simplification of the code. \$\endgroup\$ – Boris Sep 13 at 11:52
  • 2
    \$\begingroup\$ @Boris Well, you can always drop the .lower() call and restore your original collection of digraphs if you think it fits your intended usage better. \$\endgroup\$ – 409_Conflict Sep 13 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.