These are the scripts I wrote to generate valid ngrams, the output will be filtered manually then used in Markov chain style pseudoword generation.
You will need PowerShell 7 and gcide_xml to make these scripts work.
I wanted to find an English corpus/dictionary library for Python, and didn't find one with enough number of words, but I found gcide instead.
The xml files in are broken, at least I can't view them in Firefox, though I found the entries can be extracted using this regex: (?<=<ent>)(\w*)(?=</ent>)
.
The entries are in 26 xml files with names match gcide_\w\.xml
, I don't like the idea of using os.walk()
with for
loop to get the needed files, then opening reading processing and closing these 26 files in another for loop, so I used a PowerShell one-liner to get the job done, though if there is a Python library that can do this in one-loop I'd happily use that instead of PowerShell.
The entries may contain duplicates (homographs), invalid words (acronyms, especially ones with only consonants, words starting with numbers, roman numerals and other nonsense), and most entries are in Titlecase, while some are in lowercase, and the roman numerals are always in lowercase, so I preprocessed the entries.
I then used two other scripts to generate a list of potential candidates and pre-eliminated some.
Then I used another script to count the occurrences of each candidates in the entries (how many words in the list of words contain the candidates) to further eliminate invalid ngrams.
I then used another script to pull the ngrams directly from the words
By order of execution and importation, the scripts are:
getdict.py
import re
from itertools import chain
from pathlib import Path
from string import ascii_lowercase
folder = Path(r'D:\corpus\gcide_xml-0.51').rglob('*')
xml_files = [i for i in folder if re.search('gcide_\w.xml', str(i))]
words = [re.findall("(?<=<ent>)(\w*)(?=</ent>)", i.read_text()) for i in xml_files]
words = sorted(set(chain(*words)))
LETTERS = set(ascii_lowercase)
VOWELS = set('aeiouy')
CONSONANTS = LETTERS - VOWELS
words = [i for i in words if not re.match('^[cdilmvx]+$', i) and not re.search('\d', i)]
words = set(i.lower() for i in words)
words = [i for i in words if not set(i).issubset(CONSONANTS)]
words = sorted(words)
Path('D:/corpus/gcide_dict.txt').write_text('\n'.join(words))
validngrams.py
import json
import re
from collections import Counter
from collections import defaultdict
from pathlib import Path
def process(obj):
obj = obj.most_common()
obj = [(k, v) for k, v in obj if v >= 20]
return dict(sorted(obj, key=lambda x: (-x[1], x[0])))
words = Path('D:/corpus/gcide_dict.txt').read_text().splitlines()
cngrams = Counter()
for w in words:
slices = set()
for i in range(1, len(w) + 1):
for j in range(i):
slices.update(re.findall('\w{%s}' % i, w[j:]))
for k in slices:
cngrams[k] += 1
cngrams = process(cngrams)
validngrams = defaultdict(lambda: defaultdict(dict))
for k, v in cngrams.items():
validngrams[k[0]]['%sgrams' % len(k)].update({k: v})
validngrams = {k: v for k, v in sorted(validngrams.items())}
Path('D:/corpus/validngrams.json').write_text(json.dumps(validngrams, indent=4))
I want to ask about possible improvements on code style, structure, and performance, especially performance of the last script, it took about 20 minutes on my machine with Intel Core i5-4430 on PtIpython to get the count of occurrences of each candidate, I have tried list comprehension and of course nested for loops to achieve this, but none of them are much more performant than the map()
method, and of course I have Google searched for a better method, and found nothing better than list comprehension.
How can the formatting, code style, structure, readability and most importantly, performance of my scripts be improved?
The last script desperately needs boost in performance. What are some libraries that can make the process of finding how many strings in list A contain a string from list B for each string in list B significantly faster?
Update:
Uploaded gcide_dict.txt to Google Drive as per request.
And I have found a faster way to get the count of ngrams, and updated the code.
In my testing, the map()
way:
validngrams = list(map(lambda x: (sum(x in i for i in words), x), ngrams))
Takes a grand total of 1395 seconds on my machine:
In [82]: %time validngrams = list(map(lambda x: (sum(x in i for i in words), x), ngrams))
Wall time: 23min 15s
Whereas the Counter()
way takes only 1040 seconds:
In [48]: %time validngrams = Counter(i for i in ngrams for j in words if i in j)
Wall time: 17min 20s
It saves 355 seconds, and is 25.45 percent faster.
Though I am still looking for a way to make it down to about 12 minutes.
Update:
Found a Python 3 way to get the absolute paths of all files inside a directory and its subdirectories, without using os.walk()
, the dinosaur from Python 2.
Using pathlib
, listing files recursively can be done in simply by the .rglob()
method of Path()
class, instead of using a for loop like os.walk()
.
And with pathlib
, reading the contents of multiple files can be simply done by loop through their Path
s and call the read_text()
method, instead of multiple with open()
blocks.
The updated code eliminated the need of PowerShell and is tremendously faster than PowerShell, I edited the corresponding code above.
Update
I was thinking, instead of generating possible ngrams and verifying them, I could just directly get the ngrams from the words themselves by splitting the words using regex, and since re.findall()
does not find combinations that whose index of first letter modulo the length of the combination is not zero, I used steps to find all combinations.
And then, because there might be duplicates, I used a set to eliminate the duplicates.
And of course, the results contain combinations that are unpronounceable by themselves and require linguistical analysis to filter out invalid ones, but the job of finding them is done.
And this approach is tremendously faster than the last one, it only takes about 9 seconds on PtIpython.
gcide_dict.txt
might be useful to the reviewers \$\endgroup\$validngrams.py
using a Suffix Tree. Store the strings in a tree and then search the tree for the ngrams. \$\endgroup\$