1
\$\begingroup\$

This is a Python 3 script that statistically analyzes 105,230 words with a total of 913,935 characters, its results will be used for pseudoword generation purposes.

You will need this file for it to run.

This script analyzes the following things:

  • 1, How many words contains each of the 26 letters.

  • 2, How many times each of the letters has occurred in the total characters of all the words.

  • 3, How many words are of x length for all possible frequent lengths.

  • 4, Given an integer from 1 to 5, for all possible combinations of the letters of the integer length, how many words start with each of these combinations (minus the infrequent ones).

  • 5, Like the above, but count how many words ends with the combination instead.

  • 6, In all the words, how many letters b has followed letter a for each b in the 26 letters for each a in the 26 letters.

The outputs are in .json format.

Here is the script:

import json
from collections import Counter
from itertools import product
from pathlib import Path
from string import ascii_lowercase

def process(obj):
    obj = obj.most_common()
    obj = [(k, v) for k, v in obj if v >= 20]
    return dict(sorted(obj, key=lambda x: (-x[1], x[0])))

def writer(var):
    Path(f'D:/corpus/{var}.json').write_text(json.dumps(eval(var), indent=4))

words = set(Path('D:/corpus/gcide_dict.txt').read_text().splitlines())

word_length = Counter(len(i) for i in words)
word_length = process(word_length)

writer('word_length')

letter_in_word = dict(Counter({i: sum(i in j for j in words) for i in ascii_lowercase}).most_common())

all_words = Counter(''.join(words))
len_all_words = len(''.join(words))
letter_frequency = all_words.most_common()
letter_frequency = {i[0]: {'count': i[1], 'frequency': i[1] / len_all_words} for i in letter_frequency}

writer('letter_in_word')
writer('letter_frequency')

for i in range(5):
    prefix = Counter(j[:i + 1] for j in words if len(j) >= i + 1)
    prefix = process(prefix)
    suffix = Counter(j[::-1][:i + 1][::-1] for j in words if len(j) >= i + 1)
    suffix = process(suffix)
    vars().update({f'prefix_{i + 1}': prefix, f'suffix_{i + 1}': suffix})

for i in range(5):
    writer(f'prefix_{i + 1}')
    writer(f'suffix_{i + 1}')

letter_follow = {i: {j[1]: sum(k.count(''.join(j)) for k in words) for j in product(i, ascii_lowercase)} for i in ascii_lowercase}
letter_follow = {key: process(Counter(letter_follow[key])) for key in ascii_lowercase}

writer('letter_follow')

As usual, I want to ask about performance, are my implementations efficient?

What libraries I can use to help speed up the execution?

How can my code be optimized?

Then, I want to ask about data storage and retrieval, I can use SQL but I don't know how should the data be stored;

How many tables do I need?

How should I store nested dictionaries in a table?

How can I store the data most integrally?

I am looking for a sqlite3 way to store the data.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.