0
\$\begingroup\$

I fully admit that the code below is long and ugly. What's happening is that in the validation section of this machine learning model (used for text recognition), I would like to obtain the character error rate (obtained using the editdistance function) and the sentence accuracy (ignoring whitespace) across these 4 types of texts:

  • Everything included
  • Sentences longer and shorter than a specified threshold
  • Sentences longer than 1 character

As you can see, the way I've done it is by initializing 16 variables, sequentially updating them in each batch, and then printing it out in the end in an f-string. Is there a better way to do this? I wanted to do list comprehension, but I'm unsure if this is possible since it has to be computed across batches, and the batches are currently iterated through via a while loop (Or should I convert it to a for loop and do a double list comprehension?)

num_char_err = 0
num_char_err_short = 0
num_char_err_long = 0
num_char_err_not1 = 0
num_char_total = 0
num_char_total_short = 0
num_char_total_long = 0
num_char_total_not1 = 0
num_sentence_ok = 0
num_sentence_ok_short = 0
num_sentence_ok_long = 0
num_sentence_ok_not1 = 0
num_sentence_total = 0
num_sentence_total_short = 0
num_sentence_total_long = 0
num_sentence_total_not1 = 0
while loader.has_next():
    iter_info = loader.get_iterator_info()
    print(f"Batch: {iter_info[0]} / {iter_info[1]}")
    batch = loader.get_next()
    batch = preprocessor.process_batch(batch)
    recognized, _ = model.infer_batch(batch)

    print("Ground truth -> Recognized")
    for y_true, y_pred in zip(batch.gt_texts, recognized):
        is_prediction_correct_ignoring_whitespaces = y_true.replace(
            " ", ""
        ) == y_pred.replace(" ", "")
        dist = editdistance.eval(y_true, y_pred)

        num_char_err += dist
        num_char_total += len(y_true)
        num_sentence_ok += is_prediction_correct_ignoring_whitespaces
        num_sentence_total += 1

        if len(y_true) != 1:
            num_char_err_not1 += dist
            num_char_total_not1 += len(y_true)
            num_sentence_ok_not1 += is_prediction_correct_ignoring_whitespaces
            num_sentence_total_not1 += 1

        if len(y_true) < long_threshold:
            num_char_err_short += dist
            num_char_total_short += len(y_true)
            num_sentence_ok_short += is_prediction_correct_ignoring_whitespaces
            num_sentence_total_short += 1
        else:
            num_char_err_long += dist
            num_char_total_long += len(y_true)
            num_sentence_ok_long += is_prediction_correct_ignoring_whitespaces
            num_sentence_total_long += 1

        print((f"[ERR:{dist}]" if dist else "[OK]") + f'"{y_true}" -> "{y_pred}"')

# print validation result
char_error_rate = num_char_err / num_char_total
sentence_accuracy = num_sentence_ok / num_sentence_total
char_error_rate_short = num_char_err_short / num_char_total_short
sentence_accuracy_short = num_sentence_ok_short / num_sentence_total_short
char_error_rate_long = num_char_err_long / num_char_total_long
sentence_accuracy_long = num_sentence_ok_long / num_sentence_total_long
char_error_rate_not1 = num_char_err_long / num_char_total_long
sentence_accuracy_not1 = num_sentence_ok_not1 / num_sentence_total_not1
print(
    f"Character error rate: {char_error_rate * 100.0}%. Sentence accuracy: {sentence_accuracy * 100.0}%."
)
print(
    f"Character error rate (short): {char_error_rate_short * 100.0}%. Sentence accuracy (short): {sentence_accuracy_short * 100.0}%."
)
print(
    f"Character error rate (long): {char_error_rate_long * 100.0}%. Sentence accuracy (long): {sentence_accuracy_long * 100.0}%."
)
print(
    f"Character error rate (not1): {char_error_rate_not1 * 100.0}%. Sentence accuracy (not1): {sentence_accuracy_not1 * 100.0}%."
)
\$\endgroup\$
1
\$\begingroup\$

Your question is framed as though the most fruitful avenue for code improvement lies in the area of modifying the style of iteration: maybe list comprehension, maybe for-loop, maybe double list comprehension. To my eye, however, the biggest problem with the code is its proliferation of similar variables. Whenever your code has so many variables like this, it's a sign that you should step back and get the data better organized -- either in the form of collections or data objects.

As best I can tell you have 16 variables, which come in groups of four: number of sentences, number of ok sentences, number of characters, and number of error characters. You have four of those groups: regular, long, short, and not1. Each of those groups behaves like a tally: as you loop through the data, you need to compute a few values (size of y_true, correctness, and edit distance) and then use those values to update one or more tallies. (Because you have not given us much context, some of my terminology choices might be poor, but hopefully you can understand the general point and modify the vocabulary accordingly.)

One simple idea is to create a basic data object that can perform the calculations needed for the updating process. This object is not super important for the plan I'm suggesting, but it does help to move detail out of the primary data-processing loop. That's usually a good move: shift algorithmic detail down to simple functions or classes, and leave the data-processing loop focused on reading and looping.

class Update:

    def __init__(self, y_true, y_pred):
        self.size = len(y_true)
        self.is_correct = y_true.replace(' ', '') == y_pred.replace(' ', '')
        self.dist = editdistance.eval(y_true, y_pred)

More important is some type of data object to represent your ongoing tallies. A Tally would have a type or kind, and it would know how to update its counts.

class Tally:

    REGULAR = 'regular'
    LONG = 'long'
    SHORT = 'short'
    NOT1 = 'not1'

    def __init__(self, kind):
        self.kind = kind
        self.nsent = 0
        self.nsent_ok = 0
        self.nchar = 0
        self.nchar_err = 0

    def update(self, u):
        self.nsent += 1
        self.nsent_ok += u.is_correct
        self.nchar += u.size
        self.nchar_err += u.dist

The updating portion of the data-processing loop would look like this:

reg = Tally(Tally.REGULAR)
long = Tally(Tally.LONG)
short = Tally(Tally.SHORT)
not1 = Tally(Tally.NOT1)

while loader.has_next():
    ...
    for y_true, y_pred in zip(batch.gt_texts, recognized):
        u = Update(y_true, y_pred)
        reg.update(u)
        if u.size != 1:
            not1.update(u)
        (short if u.size < long_threshold else long).update(u)

Notice that the deployment of simple data objects like Update and Tally do more than improve code readability. They also create handy vehicles for further additions to functionality. For example, if you have Tally.nsent and Tally.nsent_ok you can easily add a property to give you their difference on demand. You can add this new behavior in a single place, without having to modify multiple places in the program where you might want to create, manage, and use this additional attribute.

class Tally:

    @property
    def nsent_nok(self):
        return self.nsent - self.nsent_ok
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.