This is a continuation of my earlier question.

I have a plain text file with the content of a dictionary (Webster's Unabridged Dictionary) in this format:

A (named a in the English, and most commonly ä in other languages).

Defn: The first letter of the English and of many other alphabets.
The capital A of the alphabets of Middle and Western Europe, as also
the small letter (a), besides the forms in Italic, black letter,
etc., are all descended from the old Latin A, which was borrowed from
the Greek Alpha, of the same form; and this was made from the first
letter (Aleph, and itself from the Egyptian origin. The Aleph was a
consonant letter, with a guttural breath sound that was not an
element of Greek articulation; and the Greeks took it to represent
their vowel Alpha with the ä sound, the Phoenician alphabet having no
vowel symbols. This letter, in English, is used for several different
vowel sounds. See Guide to pronunciation, §§ 43-74. The regular long
a, as in fate, etc., is a comparatively modern sound, and has taken
the place of what, till about the early part of the 17th century, was
a sound of the quality of ä (as in far).

2. (Mus.)

Defn: The name of the sixth tone in the model major scale (that in
C), or the first tone of the minor scale, which is named after it the
scale in A minor. The second string of the violin is tuned to the A
in the treble staff.
 -- A sharp (A#) is the name of a musical tone intermediate between A
and B.
 -- A flat (A) is the name of a tone intermediate between A and G.

A per se Etym: (L. per se by itself), one preëminent; a nonesuch.
O fair Creseide, the flower and A per se Of Troy and Greece. Chaucer.

A, prep. Etym: [Abbreviated form of an (AS. on). See On.]

1. In; on; at; by. [Obs.] "A God's name." "Torn a pieces." "Stand a
tiptoe." "A Sundays" Shak. "Wit that men have now a days." Chaucer.
"Set them a work." Robynson (More's Utopia)

2. In process of; in the act of; into; to; -- used with verbal
substantives in -ing which begin with a consonant. This is a
shortened form of the preposition an (which was used before the vowel
sound); as in a hunting, a building, a begging. "Jacob, when he was a
dying" Heb. xi. 21. "We'll a birding together." " It was a doing."
Shak. "He burst out a laughing." Macaulay. The hyphen may be used to
connect a with the verbal substantive (as, a-hunting, a-building) or
the words may be written separately. This form of expression is now
for the most part obsolete, the a being omitted and the verbal
substantive treated as a participle.

Ma*lay", n.

Defn: One of a race of a brown or copper complexion in the Malay
Peninsula and the western islands of the Indian Archipelago.

Ma*lay", Ma*lay"an, a.

Defn: Of or pertaining to the Malays or their country.
 -- n.

Defn: The Malay language. Malay apple (Bot.), a myrtaceous tree
(Eugenia Malaccensis) common in India; also, its applelike fruit.

Ma"la*ya"lam, n.

Defn: The name given to one the cultivated Dravidian languages,
closely related to the Tamil. Yule.

Mal"brouck, n. Etym: [F.] (Zoöl.)

Defn: A West African arboreal monkey (Cercopithecus cynosurus).

I want to convert this file to a different format to make it easier and more efficient to search in it. This is the idea:

  1. Split the file into a collection of entries
  2. Save each entry in its own file:
    • Not all in the same directory (100k+ files), split to multiple subdirs
  3. Create an index file
    • One line per entry, in the format: FILENAME:WORD

I implemented the suggestions in @jcollado's excellent previous answer, and made some other improvements:

  • Refactored the functions, use parse_content generator to return term, content pairs
  • Added a count for terms that appear multiple times, in the form of -1, -2, -3, ... appended (to display later by GUI apps as subscripts)
  • Changed the directory splitting logic, because there were still hundreds of files in most output directories. For example now the word greeting will go in g/gr/gre/greeting-NUM.txt instead of gr/greeting-NUM.txt
  • Replaced OptionParser with ArgumentParser
  • Made it easier to debug, with the --dry-run and --max-count options
  • Fixed some minor bugs

This is the script I'm using now:

#!/usr/bin/env python

import re
import os
import logging

from argparse import ArgumentParser

DATA_DIR = 'data'
INDEX_PATH = os.path.join(DATA_DIR, 'index.dat')

re_entry_start = re.compile(r'[A-Z][A-Z0-9 ;\'-.,]*$')
re_nonalpha = re.compile(r'[^a-z]')

def write_entry_file(dirname, filename, content):
    basedir = os.path.join(DATA_DIR, dirname)
    if not os.path.isdir(basedir):

    path = os.path.join(basedir, filename)
    with open(path, 'w') as fh:
        fh.write('\n'.join(content) + '\n')

def is_new_term(line, prev_line_blank):
    return re_entry_start.match(line) and prev_line_blank and '  ' not in line

def parse_content(arg):
    prev_line_blank = True
    term = None
    term_count = 0
    content = []
    with open(arg) as fh:
        for line0 in fh:
            line = line0.strip()
            if is_new_term(line, prev_line_blank):
                if term:
                    for term in term.split('; '):
                        yield term, content
                prev_term = term
                term = line.lower()
                if term == prev_term:
                    term_count += 1
                    subscript = '-' + str(term_count)
                    term_count = 1
                    subscript = ''
                content = [term + subscript]
            prev_line_blank = not line

def get_split_path(term, count):
    slug = re_nonalpha.sub('_', term.lower()).ljust(3, '_')
    dirname = os.path.join(slug[0], slug[:2], slug[:3])
    filename = '{}-{}.txt'.format(slug, count)
    return dirname, filename

def parse_file(arg, dry_run=False, max_count=0):
    def rebuild_index():
        count = 0
        for term, content in parse_content(arg):
            count += 1
            if max_count and count > max_count:
            dirname, filename = get_split_path(term, count)
            entry = '{}/{}:{}'.format(dirname, filename, term)
            if not dry_run:
                fh.write(entry + '\n')
                write_entry_file(dirname, filename, content)

    if dry_run:
        if not os.path.isdir(DATA_DIR):
        with open(INDEX_PATH, 'w') as fh:

def main():
    parser = ArgumentParser(description='Generate index and entry files '
                                        'from cleaned plain text file')
    parser.add_argument('--dry-run', '-d', '-n', action='store_true',
                        help="Dry run, don't write to files")
    parser.add_argument('--max-count', '-c', type=int,
                        help="Exit after processing N records")
    parser.add_argument('files', help="File(s) to parse", nargs='+')
    args = parser.parse_args()

                        format='%(levelname)s: %(message)s')

    for arg in args.files:
        parse_file(arg, dry_run=args.dry_run, max_count=args.max_count)

if __name__ == '__main__':

It creates an index file like this:

a/a_/a__/a__-8.txt:a 1
a/aa/aar/aaron_s_rod-14.txt:aaron's rod

Although I like it much better than the previous version, but as I've made significant changes, I'm wondering:

  • Any new mistakes I added?
  • Anything else I can still do better?

The full input file (cleaned data) is here (10 MB download, 27 MB unzipped).

The open-source project is on GitHub.


The code looks good and it's been improved in multiple ways (argparse, index file opened just once, dry-run flag instead of debug one, ...). Hence, this review is going to focus in small details:

  • Please add docstrings to document the code. This will make the code more readable and make easier to potential collaborators in github to join the project.

  • Try to order the imports alphabetically. Not really needed, but is a nice touch.

  • For a better separation of concerns, write_entry_file shouldn't take care of joining content. This should happen earlier in parse_file.

  • The count variable in rebuild_index seems to be a good candidate for enumerate:

    for count, (term, content) in enumerate(parse_content(arg)):
  • The dry_run check is in a couple of places in parse_file and might be confusing at first. What about something like:

    def parse_file(arg, dry_run=False, max_count=0):
        def rebuild_index():
            count = 0
            for term, content in parse_content(arg):
                count += 1
                if max_count and count > max_count:
                dirname, filename = get_split_path(term, count)
                entry = '{}/{}:{}'.format(dirname, filename, term)
                yield entry, dirname, filename, content
        if dry_run:
            for _index_data in rebuild_index():
            if not os.path.isdir(DATA_DIR):
            with open(INDEX_PATH, 'w') as fh:
                for entry, dirname, filename, content in rebuild_index():
                    fh.write(entry + '\n')
                    write_entry_file(dirname, filename, content)
  • The argument parsing code should be in a separate function and main should take the arguments. This way, main is testable in case you want to write test cases in the future:

    if __name__ == '__main__':
        args = parse_arguments()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.