Bash script that counts the characters, frequency, and percentage of letters in Spanish sonnets

Question

This script will be used to analyze a lot of old Spanish sonnets. I have tested it thoroughly and all seems to be working perfectly. However, I want to make sure I am not missing some detail because I certainly want accurate results. For the math part it does not round up if the last decimal place is above five. I asked the person I am writing it for, and he said that did not matter for his purposes.

My operating system is Debian 9. I am glad to receive any feedback or comments.

Usage:

script sonnet.txt

Sample text:

Sancho Panza es aquéste, en cuerpo chico,
pero grande en valor, ¡milagro extraño!
Escudero el más simple y sin engaño
que tuvo el mundo, os juro y certifico.

Output:

Carácteres          Frecuencia          Porcentaje                                                                                                           
i                   6                   04.87%
e                   17                  13.82%
a                   10                  08.13%
o                   14                  11.38%
u                   7                   05.69%

b                   0                   0
c                   7                   05.69%
d                   3                   02.43%
f                   1                   00.81%
g                   3                   02.43%
h                   2                   01.62%
j                   1                   00.81%
k                   0                   0
l                   5                   04.06%
m                   4                   03.25%
n                   8                   06.50%
p                   4                   03.25%
q                   2                   01.62%
r                   9                   07.31%
s                   8                   06.50%
t                   4                   03.25%
v                   2                   01.62%
w                   0                   0
x                   1                   00.81%
y                   2                   01.62%
z                   1                   00.81%
ñ                   2                   01.62%
Carácteres: 27  Total: 123  Total: 100%

Script:

#!/bin/bash

# Check for argument. 
if [ -z "$1" ]; then
    echo "Script needs an argument."
    exit 1
fi

# Create output file name.
fname=${1%.*}.tsv

# Clear if output file already exists.
echo -n > "$fname"

# Get all alpha characters and cut new line characters.
chars=$(grep -io '[a-z]' "$1" | tr -d '\n')

# Count total alpha characters.
chars_total=$(echo -n "$chars" | wc -m)

# Get all consonants except for ñ.
consonants=$(echo {a..z} | sed 's/[ieaou] //g')

# Set long tab length.
tabs 20

# Echo header.
echo -e "Carácteres\tFrecuencia\tPorcentaje" | tee -a "$fname"

# Get letter name. Count all instances of letters including accented letters. Calculate the percent for each letter. Echo data to table.
for ltr in 'i|í' 'e|é' 'a|á' 'o|ó' 'u|ú|ü' $consonants ñ; do
    ltr_name="${ltr:0:1}"
    ltr_count=$(echo -n "$chars" | grep -Eoi "$ltr" | tr -d '\n' | wc -m)
    ltr_percent=$(echo "scale=4; $ltr_count/$chars_total" | bc | tr -d \. | sed 's/..$/.&%/')
    echo -e "$ltr_name\t$ltr_count\t$ltr_percent"
    if [ "$ltr_name" == "u" ]; then echo | tee -a "$fname"; fi
done | tee -a "$fname"

# Echo footer.
echo -e "Carácteres: 27\tTotal: $chars_total\tTotal: 100%" | tee -a "$fname"

Answer 1

The right tool for the job

Nice script! But Bash is not well-suited for this kind of computation. Bash is good as glue code, connecting different applications, bridging the gap, doing simple stuff. The task at hand would have been better written in proper programming language, for example Python.

Usability

When the first argument is missing or empty, the script exits with an error message: "Script needs an argument.". What kind of argument? It would be better if it gave a hint, for example:

if [ $# = 0 ]; then
    echo "Usage: $0 file"
    exit 1
fi

This gives a hint that the script requires a file argument.

I also changed the condition from -z "$1" to $# = 0. The former means "first argument is missing or empty", the latter means "script has no arguments". This is probably closer to your intention.

Creating empty files

A simple way to create an empty file:

> "$fname"

Don't use flags of echo

The flags of echo are not portable, for example echo -n. Try to avoid as much as possible. You can use printf instead. On the other hand, echo without any flags is fine.

Counting characters

In your input I don't see multibyte characters. If you don't need to support such characters, then you could use native Bash syntax ${#chars} to get the length of $chars, no need for wc -m.

Writing to "$fname"

The script passes some text to | tee -a "$fname" at multiple places:

• header
• loop output
  • inside the loop, blank line after the letter u -> this one looks like a bug
• footer

Writing | tee ... multiple times is tedious and error-prone. You could group commands to simplify:

{
    echo header ...
    for ...
    echo footer ...
} | tee -a "$fname"

Probably you can drop the -a from tee to truncate the file, and that will also allow you to remove the manual truncation at the beginning of the script.

Answer 2

First impressions

A well-written Bash script; nicely laid out, with appropriate level of comments. I found this easy to read and understand.

Error checking

I recommend instructing Bash to exit on failure, and to check that we don't misspell any variable names:

set -e -u

Usage test

It's good that we check for correct invocation; we can go further and test that the file is actually readable before we start work on it. I like to define a short utility function so that we can make a one-line test:

die()
{
    echo "$@" >&2; exit 1;
}

Then we can test the number of arguments (see answer by janos) and the readability of the input file like this:

test $# = 1 || die "Usage: $0 file"
test -r "$1" || die "$1: not readable"

We might want to test that our output file is not the same as the input.

Allow input from a pipe

Instead of insisting on taking input from a file and writing output to a file, consider operating as a pipe. That gives more flexibility, as we can then work directly with content retrieved from the net, or with files in directories we can't write to.

I suggest making the argument optional, and opening an output file only if standard output is a terminal:

case $# in
    0)  # Filter - keep input and output
        ;;
    2)
        exec >"$2"
        ;&
    1)
        exec <"$1"
        if test -t 1; then exec >"${1%.*}.tsv"; fi
        ;;
    *)
        die "Usage: $0 [in-file] [out-file]"
        ;;
esac

I didn't do any argument checking, as the exec lines will fail with sensible messages when the shell can't open $1 or $2. We no longer need a dummy write to clear the output.

If we really want to produce terminal output when we've written a file, we could simply cat the file when we reach the end of the script.

Counting

The for loop executes several external commands for each letter that we're counting. Instead, we can use sort and uniq -c to count letters for us. First, we should transform the input so that we have one letter per line:

sed -e 's/./&\n/g' | sort | uniq -c

We can add to this sed command to downcase all the letters, remove accents, and delete anything that's not a letter:

sed -e 's/./\l&\n/g;y/íéáóúü/ieaouu/;s/\W\n//g;s/\n$//' | sort | uniq -c

This gives us an output that looks like

 10 a
  7 c
  3 d
 17 e
  1 f

Now we can read each line, and compute a total using Bash arithmetic:

total=0
declare -A counts
while read n c
do
    let total+=n
    let counts[$c]=n
done < <(sed -e 's/./\l&\n/g;y/íéáóúü/ieaouu/;s/\W\n//g;s/\n$//' |sort |uniq -c)

Output

The tabs command is disruptive when used from a script. Assuming that the output is to a terminal, it changes the properties of that terminal and there's no way to restore the original settings. We can use spaces instead, if we specify field widths to printf. However, Bash's printf measures field widths in bytes rather than characters, so accents misalign subsequent columns if we're in a UTF-8 system. We can avoid this either by piping literal tabs through expand or by extending strings with spaces and then using string slicing in the variable expansion.

The output code isn't robust against dividing by zero if the user provides input with no letters.

We can do all the arithmetic in Bash, rather than starting bc 27 times. To do so, we need to scale up 100 times, so that we can compute in integers (i.e. if we decide that 100 means 1%; then 1 represents 0.01%). Then we can compute the whole percent and the fraction (points of a percent) from the count c like this:

    fraction=$((10000 * c / total))
    percent=$((fraction / 100))
    points=$((fraction % 100))

We can insert the gap between vowels and consonants by defining a function to output rows; we can call it separately for the vowels and the consonants, and emit our blank line in between. That makes the flow clearer, and doesn't make the reader pause and say, "What's special about u?".

---

Modified code

#!/bin/bash

set -e -u

die()
{
    echo "$@" >&2; exit 1;
}

# Check for argument.
case $# in
    0)  # Filter - keep input and output
        ;;
    2)
        exec >"$2"
        ;&
    1)
        exec <"$1"
        if test -t 1; then exec >"${1%.*}.tsv"; fi
        ;;
    *)
        die "Usage: $0 [in-file] [out-file]"
        ;;
esac

# Count occurrences of each letter
total=0
declare -A counts
while read n c
do
    let total+=n
    let counts[$c]=n
done < <(sed -e 's/./\l&\n/g;y/íéáóúü/ieaouu/;s/\W\n//g;s/\n$//' | sort | uniq -c)

test "$total" = 0 && die "No letters found"

# Output functions
format_line()
{
    # Print all arguments, in 20-char columns
    local i
    for i
    do
        i="$i                    "
        printf '%s' "${i:0:20}"
    done
    echo
}

print_lines()
{
    for i in "$@"
    do
        c="${counts[$i]:-0}"
        fraction=$((10000 * c / total))
        percent=$((fraction / 100))
        points=$((fraction % 100))
        printf -v p '%02d.%02d' $percent $points
        format_line "$i" "$c" "$p"
    done

}

# Write the output
format_line 'Carácteres' 'Frecuencia' 'Porcentaje'
print_lines i e a o u
echo
print_lines b c d f g h j k l m n \
            p q r s t v w x y z ñ
format_line 'Total: 27' "$total" '100%'