2
\$\begingroup\$

This is a Python 3 module that generates random pronounceable word-like strings based on Markov Chains. It has many modes, each mode conforms to the structures of dictionary words to a degree, the two highest conforming modes use Markov Chain trees, with the output of THE highest conforming mode practically indistinguishable from real words (except the fact the result is very likely not found in dictionaries, but sometimes it does return real words...).

Disclaimer: The Markov Chain trees are very time and memory consuming to generate, and practically impossible to generate without memoization, therefore they need to be serialized and of course they are pregenerated and serialized for later use.

Now I have tested a bunch of serialization libraries and in short I determined that pickle suits the module best, because I need a pure Python (binary code) serialization format that isn't a data interchange serialization format therefore it doesn't need to worry about cross-language compatibility and doesn't need to convert the data types at all. I need the deserialized object exactly as is when it was serialized. Also I have tested the performances of all those serialization libraries I have found rigorously, and determined that pickle is by far the fastest.

But pickle has a bad reputation about security and compatibility, so if you don't trust me don't unpickle the pickles, just run preprocess.py to regenerate the objects (and repickle them). I assure you I have not a single idea on how to manipulate pickle bytecode, I can't even read them. And the objects are pickled using Python 3.9.7 in case that's important.

You will need this folder: Google Drive, the script to generate pseudowords is letterbased_generator.py.

The module is feature incomplete, many features aren't implemented yet, for example the chunks modes aren't implemented, and I planned add argparse to it to make it a fully functional command line utility, and add an infinite while loop to make it a shell, and add a GUI to it...

But those currently implemented are fully functional.

In this module there are three conditions, alternate, singlecon and letter, in alternate condition, a vowel can only be followed by a consonant, and a consonant can only be followed by a vowel, but because the specialness of the letter y, I observed it being used both as a consonant and a vowel, I consider it a semi-vowel and therefore it can be followed by any letter except itself.

In singlecon condition, a vowel can be followed by all letters, but a consonant can only be followed by a vowel, singlecon here means in every three letters there must be exactly two vowels and a single consonant, every consonant letter must be separated from another consonant by one or two vowel letters. Y can be followed by all other 25 letters.

In letter condition, letter y cannot follow itself. And there can't be four consecutive vowels or consonants.

The first two conditions ensure the string cannot be unpronounceable, basically it can be completely syllabified without leaving anything behind. I have observed a bunch of syllable structures, and I have chosen the easiest definitions. A syllable in the first two conditions must contain exactly one vowel and at most two consonants, the vowel can have two letters (diphthongs) and the consonants must not be next to each other, in other words, a syllable can be V, VC, CV, and CVC.

The first two conditions ensure the balance between consonants and vowels, the last one is less strict and allows the string to have more English resembleness.

I have currently written three functions: random_alternate, random_singlecon and weighted_pseudoword.

All functions have a bunch of default parameters, all functions allows you to specify an exact length (pass an int between 4 and 18), or randomly choose a length from a narrow range (pass a LENGTH Enum), or randomly choose from all available length.

control_length parameter determines, if no exact length is given, whether or not the random sampling of the length is weighted or uniform, the biased_length parameter determines whether the length sampling is biased towards the lower end or not.

All three functions allows you to specify a starting string to be the beginning of the generated word. The start must be 1 to 3 letters long and meet the respective conditions.

The first two functions generates pseudowords without using Markov chains, that is, a letter's probability of being chosen is not determined by the preceding letter, other than what the condition dictates. The letters are choosing completely at random, you can set biased_first to True to make the starting letter distribution follow that of English words (if start isn't specified). The outputs of these functions are highly random and bear little resembleness to English words. The longer they are the more gibberish they become, but the shorter ones somehow do seem like valid words in an unknown natural language.

The third function do use Markov chains, and there are three levels, which level is used is determined by two booleans: control_state and high_conformity.

To get the lowest level, both must be False The second level is reached by setting the first to True. The third (highest) level is reached by setting both to True.

The first level keeps track of only letter preceding letter and a letter's probability is only affected by one letter before it. The second level keeps track of three letters and a letter's probability is determined by two letters before it.

The third level is similar to the second level, the difference is that in the second level the next state doesn't actually have to occur in dictionary words after the previous state. In the third level they have to.

The third level outputs are very English like.

Code

tools.py

import random
from bisect import bisect
from itertools import accumulate
from typing import Any

cache = dict()
def weighted_choice(dic: dict) -> Any:
    marker = id(dic)
    if marker not in cache:
        if not isinstance(dic, dict):
            raise TypeError('The argument of the function should be a dictionary')
        
        choices, weights = zip(*dic.items())
        if set(map(type, weights)) != {int}:
            raise TypeError('The values of the argument must be integers')
        if 0 in weights:
            raise ValueError('The values of the argument shouldn\'t contain 0')
        
        accreted = list(accumulate(weights))
        cache[marker] = (choices, accreted)
    
    else:
        choices, accreted = cache[marker]
    
    chosen = random.random() * accreted[-1]
    return choices[bisect(accreted, chosen)]

traversed = dict()
def traverse_tree(tree: dict, length: int, initial_tree: dict, start=None) -> Any:
    def walker(tree: dict, chosen=list()) -> Any:
        marker = id(tree)
        if marker not in traversed:
            if not isinstance(tree, dict):
                raise TypeError('The argument of the function should be a dictionary')
            
            choices = list(tree)
            weights = [b for a, b in choices]
            
            if set(map(type, weights)) != {int}:
                raise TypeError('The values of the argument must be integers')
            if 0 in weights:
                raise ValueError('The values of the argument shouldn\'t contain 0')
            
            accreted = list(accumulate(weights))
            traversed[marker] = (choices, accreted)
        
        else:
            choices, accreted = traversed[marker]
        
        selected = random.random() * accreted[-1]
        key = choices[bisect(accreted, selected)]
        next_branch = tree[key]
        chosen = chosen + [key]
        
        if next_branch != 0:
            return walker(next_branch, chosen)
        else:
            return chosen[0][0] + ''.join(i[0][2:] for i in chosen[1:])
    if start:
        origin = initial_tree[length]
        if start not in origin:
            raise LookupError('The initial state cannot be found in the corresponding initial state tree')
        obj = origin[start]
        if isinstance(obj, dict):
            start = weighted_choice(obj)
            key = (start, obj[start])
        else:
            key = (start, obj)
        chosen = [key]
        return walker(tree[length][key], chosen)
    return walker(tree[length])

conditions.py

from pathlib import Path
import json

INCLUDIR = Path(__file__).parent / 'include'
IS_VOWEL = json.loads((INCLUDIR / 'is_vowel.json').read_text())
IS_CONSONANT = json.loads((INCLUDIR / 'is_consonant.json').read_text())

def alternate(x, y): return IS_VOWEL[x] != IS_VOWEL[y]
def singlecon(x, y): return IS_CONSONANT[x] + IS_CONSONANT[y] < 2

alternate_triplets = {
    (0, 0.5, 0), (0, 0.5, 1),
    (0, 1, 0), (0, 1, 0.5),
    (0.5, 0, 0.5), (0.5, 0, 1),
    (0.5, 1, 0), (0.5, 1, 0.5),
    (1, 0, 0.5), (1, 0, 1),
    (1, 0.5, 0), (1, 0.5, 1)
}

singlecon_triplets = {(0, 0, 0), (0, 0, 1), (0, 1, 0), (1, 0, 0), (1, 0, 1)}

def alternate_triplet(tri):
    return tuple(IS_VOWEL[i] for i in tri) in alternate_triplets

def singlecon_triplet(tri):
    return tuple(IS_CONSONANT[i] for i in tri) in singlecon_triplets

def alternate_start(s):
    if not isinstance(s, str):
        raise TypeError('Argument should be a str')
    if len(s) not in range(1, 4):
        raise ValueError('Argument should have a length between one to three')
    l = len(s)
    if l == 1:
        return True
    elif l == 2:
        return alternate(s[0], s[1])
    elif l == 3:
        return alternate_triplet(s)

def singlecon_start(s):
    if not isinstance(s, str):
        raise TypeError('Argument should be a str')
    if len(s) not in range(1, 4):
        raise ValueError('Argument should have a length between one to three')
    l = len(s)
    if l == 1:
        return True
    elif l == 2:
        return singlecon(s[0], s[1])
    elif l == 3:
        return singlecon_triplet(s)

def letter_start(s):
    if not isinstance(s, str):
        raise TypeError('Argument should be a str')
    if len(s) not in range(1, 4):
        raise ValueError('Argument should have a length between one to three')
    return True

letterbased_generator.py

import json
import pickle
import random
from ast import literal_eval
from collections import deque, namedtuple
from conditions import *
from enum import Enum
from math import pow
from pathlib import Path
from tools import weighted_choice, traverse_tree

DIRECTORY = Path(__file__).parent
DATADIR = DIRECTORY / 'data'


def load_dict(path):
    text = (DATADIR / path).read_text()
    if path.endswith('.json'):
        return json.loads(text)
    elif path.endswith('.repr'):
        return literal_eval(text)


def load_pickle(pkl):
    return pickle.loads(Path(DATADIR / pkl).read_bytes())


LETTER_FREQUENCY = load_dict('letter_frequency.json')

VOWELS = 'aeiouy'
CONSONANTS = 'bcdfghjklmnpqrstvwxyz'

VOWEL_FREQUENCY = {k: v for k, v in LETTER_FREQUENCY.items() if k in VOWELS}
CONSONANT_FREQUENCY = {k: v for k,
                        v in LETTER_FREQUENCY.items() if k in CONSONANTS}

LETTER_FIRST = load_dict('letter_first.json')
ALTERNATE_INITIAL_LOW = load_pickle('alternate_tree_low_initial_states.pickle')
ALTERNATE_INITIAL_HIGH = load_pickle(
    'alternate_tree_high_initial_states.pickle')
ALTERNATE_SECOND = load_dict('alternate_second.json')
ALTERNATE_SEQUENCE = load_dict('alternate_sequence.json')
ALTERNATE_TERMINAL = load_dict('alternate_terminal.json')
ALTERNATE_TREE_HIGH = load_pickle('alternate_tree_high.pickle')
ALTERNATE_TREE_LOW = load_pickle('alternate_tree_low.pickle')
ALTERNATE_PREEND = set(ALTERNATE_SEQUENCE) - set(ALTERNATE_TERMINAL)

SINGLECON_INITIAL_LOW = load_pickle('singlecon_tree_low_initial_states.pickle')
SINGLECON_INITIAL_HIGH = load_pickle(
    'singlecon_tree_high_initial_states.pickle')
SINGLECON_SECOND = load_dict('singlecon_second.json')
SINGLECON_SEQUENCE = load_dict('singlecon_sequence.json')
SINGLECON_TERMINAL = load_dict('singlecon_terminal.json')
SINGLECON_TREE_HIGH = load_pickle('singlecon_tree_high.pickle')
SINGLECON_TREE_LOW = load_pickle('singlecon_tree_low.pickle')
SINGLECON_PREEND = set(SINGLECON_SEQUENCE) - set(SINGLECON_TERMINAL)

LETTER_INITIAL_LOW = load_pickle('letter_tree_low_initial_states.pickle')
LETTER_INITIAL_HIGH = load_pickle('letter_tree_high_initial_states.pickle')
LETTER_SECOND = load_dict('letter_second.json')
LETTER_SEQUENCE = load_dict('letter_sequence.json')
LETTER_TERMINAL = load_dict('letter_terminal.json')
LETTER_TREE_HIGH = load_pickle('letter_tree_high.pickle')
LETTER_TREE_LOW = load_pickle('letter_tree_low.pickle')
LETTER_PREEND = set(LETTER_SEQUENCE) - set(LETTER_TERMINAL)

Structure = namedtuple('Structure', ['Initial_Low', 'Initial_High', 'Second',
                        'Sequence', 'Terminal', 'Tree_High', 'Tree_Low', 'Pre_End', 'Condition'])


class Mode(Enum):
    Alternate = Structure(ALTERNATE_INITIAL_LOW, ALTERNATE_INITIAL_HIGH, ALTERNATE_SECOND, ALTERNATE_SEQUENCE,
                            ALTERNATE_TERMINAL, ALTERNATE_TREE_HIGH, ALTERNATE_TREE_LOW, ALTERNATE_PREEND, alternate_start)
    Letter = Structure(LETTER_INITIAL_LOW, LETTER_INITIAL_HIGH, LETTER_SECOND, LETTER_SEQUENCE,
                        LETTER_TERMINAL, LETTER_TREE_HIGH, LETTER_TREE_LOW, LETTER_PREEND, letter_start)
    SingleCon = Structure(SINGLECON_INITIAL_LOW, SINGLECON_INITIAL_HIGH, SINGLECON_SECOND, SINGLECON_SEQUENCE,
                        SINGLECON_TERMINAL, SINGLECON_TREE_HIGH, SINGLECON_TREE_LOW, SINGLECON_PREEND, singlecon_start)


WORD_LENGTH = load_dict('word_length.repr')
LENGTH_RANGE = sorted(WORD_LENGTH.keys())
TOTAL = sum(WORD_LENGTH.values())
LENGTH_DISTRIBUTION = {k: 1 - (1 - v / TOTAL) for k, v in WORD_LENGTH.items()}
LMIN = LENGTH_RANGE[0]
LMAX = LENGTH_RANGE[-1]
LDIFF = LMAX - LMIN
LNUM = LDIFF + 1
LSTEP = int(LNUM / 5)
VERYSHORT, SHORT, MEDIUM, LONG, VERYLONG = ({k: v for k, v in WORD_LENGTH.items(
) if k in range(LMIN+i*LSTEP, LMIN+(i+1)*LSTEP)} for i in range(5))


def random_vowel(): return random.choice(VOWELS)
def random_consonant(): return random.choice(CONSONANTS)
def weighted_vowel(): return weighted_choice(VOWEL_FREQUENCY)
def weighted_consonant(): return weighted_choice(CONSONANT_FREQUENCY)


class RANDOM(Enum):
    Random = [random_vowel, random_consonant]
    Weighted = [weighted_vowel, weighted_consonant]


class LENGTH(Enum):
    VeryShort = VERYSHORT
    Short = SHORT
    Medium = MEDIUM
    Long = LONG
    VeryLong = VERYLONG


LALL = set(LENGTH_RANGE) | {None, *LENGTH.__members__.values()}

def get_length(length, weighted_length, biased_length):
    if length not in LALL:
        raise ValueError(
        f'Argument `length` should be an `int` between {LMIN} and {LMAX} or `None` or `LENGTH`')
    if isinstance(length, LENGTH):
        if weighted_length:
            length = weighted_choice(length.value)
        else:
            length = random.choice(list(length.value))
    if not length:
        if weighted_length:
            length = weighted_choice(WORD_LENGTH)
            if length > 10 and biased_length:
                chance = random.random()
                if chance >= LENGTH_DISTRIBUTION[length]:
                    length = int(LMIN + LDIFF * pow(random.random(), 4))
        else:
            length = random.choice(LENGTH_RANGE)
            if length > 10 and biased_length:
                length = int(LMIN + LDIFF * pow(random.random(), 4))
    return length

def random_alternate(length=None, mode: RANDOM = RANDOM.Weighted, weighted_length=True, biased_first=False, biased_length=True, start=None) -> str:
    length = get_length(length, weighted_length, biased_length)
    m = mode.value
    if not start:
        if biased_first:
            word = weighted_choice(LETTER_FIRST)
            swing = word in CONSONANTS
        else:
            swing = random.randrange(2)
            word = m[swing]()
        current_length = 1
    else:
        if not alternate_start(start):
            raise ValueError('Specified string is not valid in this mode')
        word = start
        current_length = len(word)
        swing = IS_CONSONANT[word[-1]]
    while current_length < length:
        swing = 1 - swing
        letter = m[swing]()
        if word[-1] == 'y':
            while letter == 'y':
                letter = m[swing]()
        word += letter
        current_length += 1
    return word

def random_singlecon(length=None, mode: RANDOM = RANDOM.Weighted, weighted_length=True, biased_first=False, biased_length=True, start=None) -> str:
    length = get_length(length, weighted_length, biased_length)
    m = mode.value
    if not start:
        state = deque(maxlen=2)
        if biased_first:
            word = weighted_choice(LETTER_FIRST)
        else:
            word = m[random.randrange(2)]()
        state.append(IS_VOWEL[word])
        current_length = 1
    else:
        if not singlecon_start(start):
            raise ValueError('Specified string is not a valid starting string in this mode')
        word = start
        current_length = len(word)
        state = deque([IS_VOWEL[i] for i in word[-2:]], maxlen=2)
    two_vowels = deque([1, 1])
    while current_length < length:
        if state == two_vowels:
            letter = m[1]()
        elif state[-1] == 0:
            letter = m[0]()
        else:
            letter = m[random.randrange(2)]()
            if state[-1] == 0.5:
                while letter == 'y':
                    letter = m[random.randrange(2)]()
        word += letter
        state.append(IS_VOWEL[letter])
        current_length += 1
    return word

def weighted_pseudoword(mode: Mode, length=None, weighted_length=True, biased_length=False, control_terminal=True, control_states=False, high_conformity=False, start=None):
    length = get_length(length, weighted_length, biased_length)
    initial_low, initial_high, second, sequence, terminal, treehigh, treelow, preend, condition = mode.value
    three_vowels = deque([1,1,1])
    three_consonants = deque([0,0,0])
    if not start:
        state = deque(maxlen=3)
        first = weighted_choice(LETTER_FIRST)
        state.append(IS_VOWEL[first])
        letter = weighted_choice(second[first])
        state.append(IS_VOWEL[letter])
        word = first + letter
        current_length = 2
    else:
        if not condition(start):
            raise ValueError('Specified string is not a valid starting string in this mode')
        word = start
        current_length = len(word)
        letter = word[-1]
        state = deque([IS_VOWEL[i] for i in word], maxlen=3)
    def control_letter(letter, last, obj):
        if mode == Mode.Letter:
            if state == three_vowels:
                while IS_VOWEL[letter]:
                    letter = weighted_choice(obj[last])
            elif state == three_consonants:
                while not IS_VOWEL[letter]:
                    letter = weighted_choice(obj[last])
            elif state[-1] == 0.5:
                while letter == 'y':
                    letter = weighted_choice(obj[last])
            state.append(IS_VOWEL[letter])
        return letter
    
    if not control_states:
        while current_length < length - 1:
            last = letter
            letter = weighted_choice(sequence[last])
            letter = control_letter(letter, last, sequence)
            word += letter
            current_length += 1
        if control_terminal:
            if letter in preend:
                last = word[-2]
                word = word[:-1]
                while letter in preend:
                    letter = weighted_choice(sequence[last])
                    letter = control_letter(letter, last, sequence)
                word += letter
            end = weighted_choice(terminal[letter])
            end = control_letter(end, letter, terminal)
        else:
            end = weighted_choice(sequence[letter])
            end = control_letter(end, letter, sequence)
        word += end
        return word
    else:
        if high_conformity:
            return traverse_tree(treehigh, length, initial_high, start)
        return traverse_tree(treelow, length, initial_low, start)


if __name__ == '__main__':
    print(weighted_pseudoword(Mode.Alternate, control_states=True))

Example output

# Alternate
menisitype
conatala
wagulinal
caponetal
penumus
sexagasoles
myelatory
defecidevenape
tonetar
filitace
delacesal
bicatiticic
salanine
disarananife
everiver
gasem
unalayere
oceturaman
rateryino
tasisaverege
unexiset
hanit
fucalibe
deconyme
caranisure
recesibe


# Letter
suborn
jacker
sesque
header
plagrest
gorgange
menthial
intemple
euphoned
shortial
backener
stillier
inapperm
sheepend
antipate
armophic
shordery
candiate
swardent
helicate
lactogener
probationidatiouse
windersionicaneral
comminaristeroused
calatraisensingeal
antisorturialissed
phospecutterreness
monoundrenoticated
epitationanthurian
disgradulouriodont
activaleshinesings
portunifiant
sporthostery
biograperted
millanomised
archeterlock
sapophallide
mounderate
parch
illucianic
messiblene
microlitic
reprecarry
noncellage
solabritor
periscoper
ingraphyte
quadrithen
nitractick
offeropous
unaceoused
imposteron
septablent
pasticator
housnesian
intogratic
tubularate
instropoda
undustrily
morillinal
seminoman
bricology
anaphillo
genitento
plenothes
interhead
imperfect
advisable
clamarchy
plumblass
folianist
shinelion
irretical
cabackler
aristilic
hydrocome
rabberone
bombasely
regenesce
diatorial
anishinal
inconsory
annescene
hypocrama
pinnotone
castrutic
accessily
rectionia
atteritan
recorphic
microcent
mangeropy
annulough
packerina
slipsidic
effectose
ideadstor
explorant
pectivate
fouritude
gelassive
pasticule
perimidae
monomator
handecule
petistify
bordenose
palessate
filingeal
undiately
pillatess
survescus
astration
merculary
carenogen
bluentler
recensent
triciatic
ballisary
notiallow
dragmator
prophoric
stageless
misolvert
fossioned
deniticky
preverser
gutterter
congolden
disatiate
recollure
telectron
neurophic
freedicer
amarrower
pher
sting
retics
motosis
histilic
evenette
dischand
melanity
gallouse
succento
bilithes
convolar
tribular
dropoger
ventoria
haemical
sensular
tricalin
trisonal
comphane
blassion
stranose
buttinge
blowerer
saliania
wherball
composis
mediffle
perforth
pilleter
enterded
gentiver
misenism
persatic
corrulin
bedentic
alcohere
billiary
cottless
paragged
torthorn
bombiner
helluric
complian
homophic
armouser
supplier
apoteral
siphough
finiscus
spletate
fuminant
lustaled
collabic
formeant
bonderse
prection
lactiole
operasse
incology
misapple
unconder
illegate
centence
colocula
shineous
mediaria
colleral
simicule
numergic
multifer
convened
mistolet
simillus
incinion
acannery
winderal
succular
replated
isothala
minimate
copporal
stoneral
valering
parotory
metarded
martival
monotomy
hardiate
arboarse
lactiver
unresign
flaginal
unseased
somnific
nonautor
constate
granchia
collaria
needless
chilaria
labilish
zinclast
cornison
conjunce
underion
creanced
parabler
resentic
indulate
mesolate
misonison
breverrow
marmounce
thoroused
ophiloquy
seriarily
fishmento
metatious
decontery
retragant
gamentary
averlando
pharporic
substacer
sapochone
freederge
gentlesce
steentral
rubinatic
stipuline
cringness
pasticate
unfortial
unveraphy
ascilland
retisania
deparific
nephagony
octoratic
stoneuric
sulpathen
consecule
pharmatic
bisecting
pentisant
circulata
pasticate
puriouser
squantian
intection
wastiblet
oversento
philosite
forthropy
spareopal
landantal
lambolity
adulattle
affrancer
cursenary
animprely
suspector
ballegite
prackeric
alphilite
nonmentry
favorough
maltisate
intenular
aborabill
graptilly
survenate
inderachy
ordinesis

# 8 letters, initial sel
seleopal
selegram
seleness
selenize
selesian

# 8 letters, initial mel
melopher
melately
melighty
mellatic
melongal
melocate
mellined
melastic
melodily
melation
mellular
melantly
melinion
melicant
melively
melairer
mellarge
mellifer
melaired
meligate
melained
meliania
melotone
melastor
melience
melastic
melapsis
meligate
melastal
meliacer
melatern
mellopic
melastry
mellarge
mellarin
melocule
meliento
mellabic

How can my code be improved?

Update

Posted all the code in the relevant scripts required by the generation process, I have strictly separated code and data because the volume of the data is way too large to fit into the scripts, you still need the files if you want to test the scripts.

Important: The data is pickled using Python 3.9.7 x64 on Windows 10, and I have comfirmed they can be unpickled by Python 3.10 x64 on Windows 10, but I don't know if it will also be able to be unpickled on Unix-like systems or 32 bit x86 CPUs (of course my computer's CPU is x86-64), you might as well run the preprocess.py, it takes around 100 seconds on my computer with Intel i5-4430 @3.00GHz CPU and 16 GiB DDR3 (8GiB x 2) Kingston RAM, give or take.

Also important, as the commandline argument parser isn't implemented yet, you have to paste all the necessary code (minus the import local scripts part) into a running IPython or PtIPython shell to run the code, I use PtIPython and the examples are generated using the shell. DO NOT PASTE THE SCRIPTS INTO VANILLA PYTHON INTERPRETER, it will very likely raise parsing errors as many blank lines that semantically signify end of indented blocks are missing, I omitted blank lines to save space, they will be added later, I suppose.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.