This is a Python 3 script that statistically analyzes 105,230 words with a total of 913,935 characters, its results will be used for pseudoword generation purposes.

You will need this file for it to run.

This script analyzes the following things:

  • 1, How many words contains each of the 26 letters.

  • 2, How many times each of the letters has occurred in the total characters of all the words.

  • 3, How many words are of x length for all possible frequent lengths.

  • 4, Given an integer from 1 to 5, for all possible combinations of the letters of the integer length, how many words start with each of these combinations (minus the infrequent ones).

  • 5, Like the above, but count how many words ends with the combination instead.

  • 6, In all the words, how many letters b has followed letter a for each b in the 26 letters for each a in the 26 letters.

The outputs are in .json format.

Here is the script:

import json
from collections import Counter
from itertools import product
from pathlib import Path
from string import ascii_lowercase

def process(obj):
    obj = obj.most_common()
    obj = [(k, v) for k, v in obj if v >= 20]
    return dict(sorted(obj, key=lambda x: (-x[1], x[0])))

def writer(var):
    Path(f'D:/corpus/{var}.json').write_text(json.dumps(eval(var), indent=4))

words = set(Path('D:/corpus/gcide_dict.txt').read_text().splitlines())

word_length = Counter(len(i) for i in words)
word_length = process(word_length)


letter_in_word = dict(Counter({i: sum(i in j for j in words) for i in ascii_lowercase}).most_common())

all_words = Counter(''.join(words))
len_all_words = len(''.join(words))
letter_frequency = all_words.most_common()
letter_frequency = {i[0]: {'count': i[1], 'frequency': i[1] / len_all_words} for i in letter_frequency}


for i in range(5):
    prefix = Counter(j[:i + 1] for j in words if len(j) >= i + 1)
    prefix = process(prefix)
    suffix = Counter(j[::-1][:i + 1][::-1] for j in words if len(j) >= i + 1)
    suffix = process(suffix)
    vars().update({f'prefix_{i + 1}': prefix, f'suffix_{i + 1}': suffix})

for i in range(5):
    writer(f'prefix_{i + 1}')
    writer(f'suffix_{i + 1}')

letter_follow = {i: {j[1]: sum(k.count(''.join(j)) for k in words) for j in product(i, ascii_lowercase)} for i in ascii_lowercase}
letter_follow = {key: process(Counter(letter_follow[key])) for key in ascii_lowercase}


As usual, I want to ask about performance, are my implementations efficient?

What libraries I can use to help speed up the execution?

How can my code be optimized?

Then, I want to ask about data storage and retrieval, I can use SQL but I don't know how should the data be stored;

How many tables do I need?

How should I store nested dictionaries in a table?

How can I store the data most integrally?

I am looking for a sqlite3 way to store the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.