Python 3 script to phonetically transliterate written English to Cyrillic and/or Greek

Question

This is a Python 3 script I have written over three days that transliterates (not translating) English orthography written in Latin alphabet to the Cyrillic alphabet, the Greek alphabet or a mixed of those two above.

The script transliterates the text, meaning it doesn't translate the meaning of the text from one language to another language, it doesn't translate English into русский or Ελληνικά, rather, it attempts to analyze the pronunciation of the word to find the phonemes in it, and represent the phonemes by letters from another alphabet (either Greek or Cyrillic) or combinations thereof, phoneme for phoneme.

I wrote this because this in my opinion is a truly cool thing to do, something that is rarely done and I highly doubt Python has a built-in function to do this; Cyrillic, Greek and Latin scripts are closely related, they are so similar yet also so fundamentally different, yet Cyrillic and Latin scripts both stem from (ancient) Greek script. English is always written in Latin alphabet, writing it in another script is really fascinating.

I won't lie, I was inspired by 1337, however I found it too basic, contenting just at merely substitute letters with similar glyph, with some using letters from Greek and Cyrillic scripts, and these usages aren't phonetically correct at all. So I wrote this transliterating script, because if it merely translates the input, less knowledgeable people will just use Google Translate to get the original text, while in this case, you have to actually know how to read Cyrillic and Greek scripts to understand the transliterated output.

I researched extensively on grapheme-to-phoneme conversion methods, ARPABET, modern Greek phonology and Russian phonology, and English phonology, and tried to stick to the spelling conventions and the pronunciations as close as possible;

I tried to write a script to evaluate the pronunciation of written English, with a set of established rules to help determine pronunciation of each grapheme, having written 100+ lines of code with 30+ conditions, but ultimately gave up because there are simply too many variables at play here, instead I use g2p here, as g2p isn't in the standard library you will need to download it, g2p converts English into ARPABET, I have considered using epitran, however I am using Windows and I don't have flite installed, I tried the example from the official site and it just throws errors. g2p can convert words not found in dictionaries but may not be very accurate for such words.

The output of the script, is, technically still English, however represented in scripts other than Latin, as the output is English and not the language the script and spelling suggest, Google Translate won't reliably "translate" the output back to the original input. The transliteration is a unidirectional process, transliterating the output back to the input is impossible without a dictionary and an artificial intelligence to evaluate the possibilities, there are three modes available: 'cyrillic' (only using letters from Cyrillic script), 'greek' (only using letters from Greek script) and 'random' (randomly select letters from Greek and Cyrillic scripts), and I personally prefer the random mode as its outputs will confuse Google Translate making it unable to correctly pronounce the output, meaning you can't guess it by using Google Translate...

Currently it is impossible to preserve cases because I am using g2p, but I have succeeded in preserving punctuations, and the loading time might be a bit too long, it is caused by g2p importing.

The code:

import random
import re
import sys
from g2p_en import G2p
from string import punctuation

g2p = G2p()

ARPA2ALPHA = {
    '@': 'a', 'A': 'a', 'AA': 'a', 'AE': 'a',
    'AH': 'a', 'AO': 'o', 'AW': 'au', 'AX': 'e',
    'AXR': 'er', 'AY': 'ai', 'B': 'b', 'C': 'ch',
    'CH': 'ch', 'D': 'd', 'DH': 'dh', 'DX': 't',
    'E': 'e', 'EH': 'e', 'EL': 'l', 'EM': 'm',
    'EN': 'n', 'ER': 'er', 'EY': 'ei', 'F': 'f',
    'G': 'g', 'H': 'h', 'HH': 'h', 'I': 'i',
    'IH': 'i', 'IX': 'i', 'IY': 'ii', 'J': 'j',
    'JH': 'j', 'K': 'k', 'L': 'l', 'M': 'm',
    'N': 'n', 'NG': 'ng', 'NX': 'nn', 'O': 'oi',
    'OW': 'ou', 'OY': 'oi', 'P': 'p', 'Q': "'",
    'R': 'r', 'S': 's', 'SH': 'sh', 'T': 't',
    'TH': 'th', 'U': 'u', 'UH': 'u', 'UW': 'u',
    'UX': 'u', 'V': 'v', 'W': 'w', 'WH': 'w',
    'X': 'i', 'Y': 'y', 'Z': 'z', 'ZH': 'zh',
    'a': 'a', 'b': 'b', 'c': 'o', 'd': 'd',
    'e': 'ei', 'f': 'f', 'g': 'g', 'h': 'h',
    'i': 'ii', 'k': 'k', 'l': 'l', 'm': 'm',
    'n': 'n', 'o': 'ou', 'p': 'p', 'r': 'r',
    's': 's', 't': 't', 'u': 'u', 'v': 'v',
    'w': 'w', 'x': 'e', 'y': 'y', 'z': 'z'
}

TRANSLITERATION = {
    'a': {0: 'а', 1: 'α'},     
    'b': {0: 'б', 1: 'μπ'},    
    'd': {0: 'д', 1: 'ντ'},    
    'e': {0: 'э', 1: 'ε'},     
    'f': {0: 'ф', 1: 'φ'},     
    'g': {0: 'г', 1: 'γκ'},    
    'h': {0: 'х', 1: 'χ'},     
    'i': {0: 'и', 1: 'ι'},     
    'j': {0: 'дж', 1: 'τζ'},   
    'k': {0: 'к', 1: 'κ'},     
    'l': {0: 'л', 1: 'λ'},     
    'm': {0: 'м', 1: 'μ'},     
    'n': {0: 'н', 1: 'ν'},     
    'o': {0: 'о', 1: 'ο'},     
    'p': {0: 'п', 1: 'π'},     
    'r': {0: 'р', 1: 'ρ'},     
    's': {0: 'с', 1: 'σ'},     
    't': {0: 'т', 1: 'τ'},     
    'u': {0: 'у', 1: 'ου'},    
    'v': {0: 'в', 1: 'β'},     
    'y': {0: 'й', 1: 'γ'},     
    'z': {0: 'з', 1: 'ζ'},     
    'ai': {0: 'ай', 1: 'αϊ'},  
    'au': {0: 'ао', 1: 'αου'}, 
    'ch': {0: 'ч', 1: 'τσ'},   
    'dh': {0: 'з', 1: 'δ'},    
    'dz': {0: 'дз', 1: 'τζ'},  
    'ei': {0: 'эй', 1: 'εϋ'},  
    'er': {0: 'эр', 1: 'ερ'},  
    'ii': {0: 'ий', 1: 'ει'},  
    'ks': {0: 'кс', 1: 'ξ'},   
    'ng': {0: 'нг', 1: 'γγ'},  
    'oi': {0: 'ой', 1: 'οϊ'},  
    'ou': {0: 'оу', 1: 'ω'},   
    'ps': {0: 'пс', 1: 'ψ'},   
    'sh': {0: 'ш', 1: 'σ'},    
    'th': {0: 'т', 1: 'θ'},    
    'ts': {0: 'ц', 1: 'τσ'},   
    'ui': {0: 'уй', 1: 'ουι'}, 
    'ya': {0: 'я', 1: 'γα'},   
    'ye': {0: 'е', 1: 'γε'},   
    'yo': {0: 'ё', 1: 'γο'},   
    'yu': {0: 'ю', 1: 'γου'},  
    'zh': {0: 'ж', 1: 'ζ'},    
    'shch': {0: 'щ', 1: 'στσ'},
    "'": {0: 'ъ', 1: '΄'}
}

PHONEMES = list(TRANSLITERATION.keys()) + ['ia', 'ie', 'io', 'w']
PHONEME_REGEX = re.compile('(' + r'|'.join(sorted(PHONEMES, key=len, reverse=True)) + ')')

def parser(s: str) -> list:
    switch = {'ia': 'ya', 'ie': 'ye', 'io': 'yo', 'w': 'v'}
    reduct = lambda x: ''.join(i for i in x if i.isalpha()) if x[0].isalpha() else x
    phonemes = [reduct(p) for p in g2p(s)]
    parsed = ''.join([ARPA2ALPHA.get(p, p) for p in phonemes])
    parsed = [i for i in PHONEME_REGEX.split(parsed) if i]
    phonemes = [switch.get(p, p) for p in parsed]
    return phonemes

def transliterate(s: str, mode='random'):
    phonemes = parser(s)
    modes = {'cyrillic': 0, 'greek': 1}
    def convert(p, mode):
        if TRANSLITERATION.get(p, None):
            return TRANSLITERATION[p].get(modes.get(mode.lower(), random.randint(0, 1)))
        return p
    result = ''.join([convert(p, mode) for p in phonemes])
    if result.endswith('σ'):
        result = result[:-1] + 'ς'
    for j in [' ' + i for i in punctuation]:
        result = result.replace(j, j[1])
    return result

if __name__ == '__main__':
    print(transliterate(*sys.argv[1:]))

Example outputs:

In [17]: transliterate('aphrodite', 'cyrillic')
Out[17]: 'афрадайтий'

In [18]: transliterate('ionization')
Out[18]: 'αϊαναζэйσан'

In [19]: transliterate('imagination')
Out[19]: 'иματζαнэйшаν'

In [20]: transliterate('esperanza')
Out[20]: 'эσпэранζа'

In [21]: transliterate('esperanza', 'Greek')
Out[21]: 'εσπερανζα'

In [22]: transliterate('esperanza', 'Cyrillic')
Out[22]: 'эспэранза'

In [23]: transliterate('liberty', 'Cyrillic')
Out[23]: 'либэртий'

In [24]: transliterate('liberty', 'Greek')
Out[24]: 'λιμπερτει'

In [25]: transliterate('fraternity', 'Greek')
Out[25]: 'φρατερνατει'

In [26]: transliterate('fraternity', 'Cyrillic')
Out[26]: 'фратэрнатий'

In [27]: transliterate('equality', 'Cyrillic')
Out[27]: 'иквалатий'

In [28]: transliterate('equality', 'greek')
Out[28]: 'ικβαλατει'

In [29]: transliterate('diversity', 'greek')
Out[29]: 'ντιβερσατει'

In [30]: transliterate('diversity', 'cyrillic')
Out[30]: 'дивэрсатий'

In [31]: transliterate('Hello, World!', 'cyrillic')
Out[31]: 'халоу, вэрлд!'

In [32]: transliterate('Hello, World!', 'greek')
Out[32]: 'χαλω, βερλντ!'

In [33]: transliterate('Hello, World!')
Out[33]: 'χαлоу, вερлд!'

In [34]: transliterate("To be, or not to be, that is the question")
Out[34]: 'ту μπий, ορ νаτ ту бий, зат ιз δα квэσчан'

In [35]: transliterate("He who fights with monsters should look to it that he himself does not become a monster. And when you gaze long into an abyss the abyss also gazes into you.")
Out[35]: 'χει ху фαϊτσ βιδ μαнστερζ σуντ лук ту ιτ зαт χий хιμσэλф νταζ νατ бιкαм α маνστερ. αнντ βэν γου гεϋζ лοнг иντου αν аμπισ δа абισ ολсоу γκэйзαζ интου ю.'

How can my script be improved?

Answer 1

This is cool! I wonder how pronounce-able the results are by a native Greek speaker. Is that your language?

Things that could be improved:

• It's not all that useful to represent the values of TRANSLITERATION as dicts. Given that they're all two items long, it's safe to just represent them as 2-tuples
• Whenever compiling a regex with dynamic substrings, ensure that those substrings go through re.escape for safety's sake
• Generally I find that there's a lot of reliance on comprehensions. This isn't the end of the world, but converting them to generator functions improves legibility, and the resulting bytecode will be very similar to a comprehension anyway
• Represent your mode values as enums rather than free strings.
• get(p, None) is redundant and can just be get(p)
• For the non-random examples you've shown, those are perfect cases for unit tests. For the random ones, you could expand coverage to run them and compare them against regexes with character classes whenever there's variation permitted.

Suggested

import random
import re
import sys
from enum import Enum
from typing import Iterator

from g2p_en import G2p
from string import punctuation

g2p = G2p()

ARPA2ALPHA = {
    '@': 'a', 'A': 'a', 'AA': 'a', 'AE': 'a',
    'AH': 'a', 'AO': 'o', 'AW': 'au', 'AX': 'e',
    'AXR': 'er', 'AY': 'ai', 'B': 'b', 'C': 'ch',
    'CH': 'ch', 'D': 'd', 'DH': 'dh', 'DX': 't',
    'E': 'e', 'EH': 'e', 'EL': 'l', 'EM': 'm',
    'EN': 'n', 'ER': 'er', 'EY': 'ei', 'F': 'f',
    'G': 'g', 'H': 'h', 'HH': 'h', 'I': 'i',
    'IH': 'i', 'IX': 'i', 'IY': 'ii', 'J': 'j',
    'JH': 'j', 'K': 'k', 'L': 'l', 'M': 'm',
    'N': 'n', 'NG': 'ng', 'NX': 'nn', 'O': 'oi',
    'OW': 'ou', 'OY': 'oi', 'P': 'p', 'Q': "'",
    'R': 'r', 'S': 's', 'SH': 'sh', 'T': 't',
    'TH': 'th', 'U': 'u', 'UH': 'u', 'UW': 'u',
    'UX': 'u', 'V': 'v', 'W': 'w', 'WH': 'w',
    'X': 'i', 'Y': 'y', 'Z': 'z', 'ZH': 'zh',
    'a': 'a', 'b': 'b', 'c': 'o', 'd': 'd',
    'e': 'ei', 'f': 'f', 'g': 'g', 'h': 'h',
    'i': 'ii', 'k': 'k', 'l': 'l', 'm': 'm',
    'n': 'n', 'o': 'ou', 'p': 'p', 'r': 'r',
    's': 's', 't': 't', 'u': 'u', 'v': 'v',
    'w': 'w', 'x': 'e', 'y': 'y', 'z': 'z'
}

TRANSLITERATION = {
    'a': ('а', 'α'),
    'b': ('б', 'μπ'),
    'd': ('д', 'ντ'),
    'e': ('э', 'ε'),
    'f': ('ф', 'φ'),
    'g': ('г', 'γκ'),
    'h': ('х', 'χ'),
    'i': ('и', 'ι'),
    'j': ('дж', 'τζ'),
    'k': ('к', 'κ'),
    'l': ('л', 'λ'),
    'm': ('м', 'μ'),
    'n': ('н', 'ν'),
    'o': ('о', 'ο'),
    'p': ('п', 'π'),
    'r': ('р', 'ρ'),
    's': ('с', 'σ'),
    't': ('т', 'τ'),
    'u': ('у', 'ου'),
    'v': ('в', 'β'),
    'y': ('й', 'γ'),
    'z': ('з', 'ζ'),
    'ai': ('ай', 'αϊ'),
    'au': ('ао', 'αου'),
    'ch': ('ч', 'τσ'),
    'dh': ('з', 'δ'),
    'dz': ('дз', 'τζ'),
    'ei': ('эй', 'εϋ'),
    'er': ('эр', 'ερ'),
    'ii': ('ий', 'ει'),
    'ks': ('кс', 'ξ'),
    'ng': ('нг', 'γγ'),
    'oi': ('ой', 'οϊ'),
    'ou': ('оу', 'ω'),
    'ps': ('пс', 'ψ'),
    'sh': ('ш', 'σ'),
    'th': ('т', 'θ'),
    'ts': ('ц', 'τσ'),
    'ui': ('уй', 'ουι'),
    'ya': ('я', 'γα'),
    'ye': ('е', 'γε'),
    'yo': ('ё', 'γο'),
    'yu': ('ю', 'γου'),
    'zh': ('ж', 'ζ'),
    'shch': ('щ', 'στσ'),
    "'": ('ъ', '΄'),
}

PHONEMES = sorted(
    (
        *TRANSLITERATION.keys(),
        'ia', 'ie', 'io', 'w',
    ),
    key=len, reverse=True,
)

PHONEME_REGEX = re.compile(
    '('
    + '|'.join(
        re.escape(p) for p in PHONEMES
    )
    + ')'
)
PARSE_SWITCH = {'ia': 'ya', 'ie': 'ye', 'io': 'yo', 'w': 'v'}


def reduce(x: str) -> str:
    if x[0].isalpha():
        return ''.join(i for i in x if i.isalpha())
    return x


def parse_phonemes(s: str) -> Iterator[str]:
    for p in g2p(s):
        phoneme = reduce(p)
        yield ARPA2ALPHA.get(phoneme, phoneme)


def parser(s: str) -> Iterator[str]:
    parsed = ''.join(parse_phonemes(s))
    for i in PHONEME_REGEX.split(parsed):
        if i:
            yield PARSE_SWITCH.get(i, i)


class Mode(Enum):
    CYRILLIC = 0
    GREEK = 1
    RANDOM = 2


def convert(p: str, mode: Mode) -> str:
    trans = TRANSLITERATION.get(p)
    if trans:
        if mode == Mode.RANDOM:
            index = random.randrange(2)
        else:
            index = mode.value
        return trans[index]
    return p


def transliterate(s: str, mode: Mode = Mode.RANDOM) -> str:
    phonemes = parser(s)
    result = ''.join(convert(p, mode) for p in phonemes)
    if result.endswith('σ'):
        result = result[:-1] + 'ς'
    for i in punctuation:
        j = ' ' + i
        result = result.replace(j, i)
    return result


def test() -> None:
    assert transliterate('aphrodite', Mode.CYRILLIC) == 'афрадайтий'
    assert transliterate('esperanza', Mode.CYRILLIC) == 'эспэранза'
    assert transliterate('liberty', Mode.CYRILLIC) == 'либэртий'
    assert transliterate('fraternity', Mode.CYRILLIC) == 'фратэрнатий'
    assert transliterate('equality', Mode.CYRILLIC) == 'иквалатий'
    assert transliterate('diversity', Mode.CYRILLIC) == 'дивэрсатий'
    assert transliterate('Hello, World!', Mode.CYRILLIC) == 'халоу, вэрлд!'

    assert transliterate('esperanza', Mode.GREEK) == 'εσπερανζα'
    assert transliterate('liberty', Mode.GREEK) == 'λιμπερτει'
    assert transliterate('fraternity', Mode.GREEK) == 'φρατερνατει'
    assert transliterate('equality', Mode.GREEK) == 'ικβαλατει'
    assert transliterate('diversity', Mode.GREEK) == 'ντιβερσατει'
    assert transliterate('Hello, World!', Mode.GREEK) == 'χαλω, βερλντ!'


if __name__ == '__main__':
    test()