6
\$\begingroup\$

I am currently studying Latin from Wheelock's Latin. Given that a core component of language learning is memorizing vocabulary I needed somewhere I could memorize vocabulary, online, namely Quizlet (which allows you to import character-separated data sets, .csv, etc.). Thankfully, I was able to find a site, The Bridge, which provides a large selection of Latin (and Greek) vocabulary sets from a variety of sources which can be imported into Quizlet. Unfortunately, when these vocabulary sets are exported, the terms in the exported .tsv file look like this:

"term without long marks" "term with long marks" "simple definition" "expanded definition" "number" "number" "source link"

And in order to cleanly export the data to Quizlet, I needed it to look like this (with the vertical bar | being used as a separator instead of a comma):

term with macrons|expanded definition

To solve this problem, I made the following simple Python script:

"""
latin_vocab_reformatter.py

This Python program is used to reformat Latin .tsv vocab files exported from the website
bridge.haverford.edu. The data exported from The Bridge is formatted in the following 
style:

    "entry without macrons" "entry with macrons" "simple definition" "expanded definition" etc. \n
    ...

The program will first strip each vocab term of its entry without macrons, its simple 
definition, and any extraneous items on the end. It will also format each entry in
.csv-esque style (with the vertical bar "|" used instead due to the definitions themselves
containing commas) like so:

    entry with macrons|expanded definition \n
    ...
"""
import os
import sys
import codecs


def write_terms_to_file(original_path, file_name, term_string):
    """
    Write the final term string to a .txt file.

    original_path - the original path of the input file
    file_path - the name of the file to write.
    term_string - the final term string
    """
    with codecs.open(original_path + "\\" + file_name + ".txt", encoding="utf-8", mode="w+") as new_file:
        new_file.write("\ufeff" + term_string);


def remove_extraneous_items(split_tsv_string):
    """
    Remove extraneous items (described in file docstring) from a split .tsv string.

    split_tsv_string - the split tsv string.
    """
    terms = []
    for term in split_tsv_string:
        term_items = []
        reading_term = ""
        reading_term_item = False

        for character in term:
            if character == "\"" and (not reading_term_item):
                reading_term_item = True
                continue

            if character == "\"" and reading_term_item:
                term_items.append(reading_term);
                reading_term = ""
                reading_term_item = False
                continue

            if reading_term_item:
                reading_term += character

        del term_items[0]
        del term_items[1]
        del term_items[2:]
        terms.append(term_items)

    new_terms = []
    term_index = 0
    for term in terms:
        new_terms.append("|".join(term))
        term_index += 1


    return "\r\n".join(new_terms)


def split_file_into_list(tsv_file_string):
    """
    Read an input .tsv file string into a list, split on newlines.

    tsv_file_string - the input file string.
    """
    return tsv_file_string.split("\n");


def read_file_into_text(tsv_file_path):
    """
    Read an input .tsv file into text and return it.

    tsv_file_path - the input .tsv file path.
    """
    with codecs.open(tsv_file_path, encoding="utf-8", mode="r") as tsv_file:
        return tsv_file.read()


def main():
    input_file_path = sys.argv[1];
    output_file_name = sys.argv[2];

    tsv_file_string = read_file_into_text(input_file_path);
    tsv_string_list = split_file_into_list(tsv_file_string);
    final_term_string = remove_extraneous_items(tsv_string_list);

    write_terms_to_file(
        os.path.dirname(os.path.abspath(input_file_path)), 
        output_file_name, 
        final_term_string
    )


if __name__ == '__main__':
    main()

The program is run from the command line, like so:

python latin_vocab_reformatter.py "Wheelock's Latin Export.tsv" wheelock_output
                                  [path to .tsv file]           [output file name without extension]
\$\endgroup\$
1
\$\begingroup\$

This script is much more complicated than it needs to be.

To read tabular data that is double-quoted and delimited with tabs, use the csv module with the 'excel-tab' dialect. For symmetry, I would write the output using the csv module with '|' as the delimiter.

Your program reads the entire input into memory, transforms it, and then writes it all out. The code would scale better and be easier to understand if it worked a row at a time.

To specify the encoding of the input and output files, you don't need to use the codecs module; you can just use open(…, encoding='utf-8'). Furthermore, the U+FEFF byte-order mark is meaningless in UTF-8; it would be better not to emit it — especially since you might be appending to a file that already contains some text.

I'm not a fan of constructing the output path using original_path + "\\" + file_name + ".txt" — I think it's too mysterious and magical. If the input is specified using a full path, why shouldn't the same be true for the output? If you really did want to construct the path, you should use os.path.join() to make the path separator portable.

The most confusing part of the code is in remove_extraneous_items():

del term_items[0]
del term_items[1]
del term_items[2:]

It's not obvious at all which items are being kept, especially since each deletion affects the indices of the subsequent items. Deleting items from the end would make the code easier to understand:

del term_items[4:]
del term_items[2]
del term_items[0]

Better yet, just construct a tuple of the items you want to keep: (term_items[1], term_items[3]).

Suggested solution

import csv
import sys

def convert_terms(reader, writer):
    for row in reader:
        writer.writerow((row[1], row[3]))

def main(input_file_path, output_file_path):
    with open(input_file_path, encoding='utf-8') as input_file, \
         open(output_file_path, 'w+', encoding='utf-8') as output_file:
            convert_terms(
                csv.reader(input_file, dialect='excel-tab'),
                csv.writer(output_file, delimiter='|')
            )

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.