5
\$\begingroup\$

I'm trying to build an xliff for localization from 3 specific files: one contains a list of IDs, the other a list of source strings and the last, a list of translated strings.

Basically, each file contains 200,000 strings, and the process is taking so much time. How can I speed up this loop?

I use sed to replace < with &lt; and the same for >. If you have better ideas, please tell me.

FILE_ID=$1
FILE_SOURCE=$2
FILE_TARGET=$3
TOT_STRING=$(wc -l < $FILE_ID)
time=$(date +"%Y-%m-%d %H:%M")

echo "<?xml version='1.0' encoding='utf-8'?>"
echo "<xliff xmlns=\"urn:oasis:names:tc:xliff:document:1.1\" version=\"1.1\">"
echo " <file source-language=\"en-US\" datatype=\"plaintext\">"
echo "    <body>"
echo "      <trans-unit xml:space=\"preserve\" id=\"1\" approved=\"no\" restype=\"x-gettext-domain-header\">"
echo "        <source>Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "</source>"
echo "        <target state=\"translated\">Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "</target>"
echo "      </trans-unit>"

    COUNTER=1

        while [  "$COUNTER" -le "$TOT_STRING" ]; do

        ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
        ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
        ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

        if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
            echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\">"
            echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
            echo "        <target></target>"
            echo "      </trans-unit>"
        else
            echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\">"
            echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
            echo "        <target state=\"translated\">$(echo $ROW_TARGET | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</target>"
            echo "      </trans-unit>"
        fi

        COUNTER=$(( $COUNTER + 1 ))
        done

echo "       </trans-unit>"
echo "    </body>"
echo "  </file>"
echo "</xliff>"

exit
\$\endgroup\$
2
\$\begingroup\$

Improving speed

For each line, running 3 sed commands to extract the n-th line from 3 files, and then running further 2-4 sed commands is of course slow.

My first recommendation would be to implement this in another scripting language, say, Python.

If you really want to do this in Bash, you could:

  • Combine the 3 files into one, with their lines interleaved. That is, take the 1st line from each file, then the 2nd from each file, and so on. And then in each iteration of your loop, read 3 lines.
  • Instead of transforming the < and > by running sed for each line, run sed just once for the entire input

If we can assume that all 3 files have the same number of lines, then you can create the input with lines interleaved like this:

paste -d '\n' "$FILE_ID" "$FILE_SOURCE" "$FILE_TARGET" 

When replacing < and > using sed, you can do it with a single sed command using multiple -e flags, like this:

sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g'

Putting it together:

paste -d '\n' "$FILE_ID" "$FILE_SOURCE" "$FILE_TARGET" | sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g' | \
for ((COUNTER = 1; COUNTER <= TOT_STRING; ++COUNTER)); do
    read ROW_ID
    read ROW_SOURCE
    read ROW_TARGET
    # ...
done

Simplify

This is unnecessarily complicated:

ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

You can write much simpler:

ROW_ID=$(sed -n ${COUNTER}p $FILE_ID)
ROW_SOURCE=$(sed -n ${COUNTER}p $FILE_SOURCE)
ROW_TARGET=$(sed -n ${COUNTER}p $FILE_TARGET)

Counting loops in Bash

Instead of this:

COUNTER=1
while [ "$COUNTER" -le "$TOT_STRING" ]; do
    # do something
    COUNTER=$(( $COUNTER + 1 ))
done

This is equivalent, but cleaner and simpler:

for ((COUNTER = 1; COUNTER <= TOT_STRING; ++COUNTER)); do
    # do something
done

Naming

TOT_STRING is a strange name for a variable with an integer value.

Printing large text

Instead of this:

echo "<?xml version='1.0' encoding='utf-8'?>"
echo "<xliff xmlns=\"urn:oasis:names:tc:xliff:document:1.1\" version=\"1.1\">"
echo " <file source-language=\"en-US\" datatype=\"plaintext\">"
echo "    <body>"
echo "      <trans-unit xml:space=\"preserve\" id=\"1\" approved=\"no\" restype=\"x-gettext-domain-header\">"
echo "        <source>Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "</source>"
echo "        <target state=\"translated\">Project-Id-Version: 1.0"
echo " Report-Msgid-Bugs-To: email@example.com"
echo "POT-Creation-Date: $time+0200"
echo "PO-Revision-Date: $time+0200"
echo "Last-Translator: JohnnyKing"
echo "Language-Team: JohnnyKing"
echo "MIME-Version: 1.0"
echo "Content-Type: text/plain; charset=UTF-8"
echo "Content-Transfer-Encoding: 8bit"
echo "X-Generator: csv2xliff.sh"
echo "</target>"
echo "      </trans-unit>"

A simpler way to write:

cat << EOF
<?xml version='1.0' encoding='utf-8'?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:1.1" version="1.1">
 <file source-language="en-US" datatype="plaintext">
    <body>
      <trans-unit xml:space="preserve" id="1" approved="no" restype="x-gettext-domain-header">
        <source>Project-Id-Version: 1.0
 Report-Msgid-Bugs-To: email@example.com
POT-Creation-Date: +0200
PO-Revision-Date: +0200
Last-Translator: JohnnyKing
Language-Team: JohnnyKing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Generator: csv2xliff.sh
</source>
        <target state="translated">Project-Id-Version: 1.0
 Report-Msgid-Bugs-To: email@example.com
POT-Creation-Date: +0200
PO-Revision-Date: +0200
Last-Translator: JohnnyKing
Language-Team: JohnnyKing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Generator: csv2xliff.sh
</target>
      </trans-unit>
EOF

Indentation

The indentation here is odd:

echo "X-Generator: csv2xliff.sh"
echo "</target>"
echo "      </trans-unit>"

    COUNTER=1

        while [  "$COUNTER" -le "$TOT_STRING" ]; do

        ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
        ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
        ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

        if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
            echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\">"
            echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
            echo "        <target></target>"
            echo "      </trans-unit>"
        else
            echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\">"
            echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
            echo "        <target state=\"translated\">$(echo $ROW_TARGET | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</target>"
            echo "      </trans-unit>"
        fi

        COUNTER=$(( $COUNTER + 1 ))
        done

echo "       </trans-unit>"
echo "    </body>"
echo "  </file>"
echo "</xliff>"

It would be more natural this way:

echo "X-Generator: csv2xliff.sh"
echo "</target>"
echo "      </trans-unit>"

COUNTER=1

while [  "$COUNTER" -le "$TOT_STRING" ]; do
    ROW_ID=$(sed -n $(( $COUNTER ))p $FILE_ID)
    ROW_SOURCE=$(sed -n $(( COUNTER ))p $FILE_SOURCE)
    ROW_TARGET=$(sed -n $(( COUNTER ))p $FILE_TARGET)

    if [ "$ROW_SOURCE" = "$ROW_TARGET" ]; then
        echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\">"
        echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
        echo "        <target></target>"
        echo "      </trans-unit>"
    else
        echo "      <trans-unit xml:space=\"preserve\" id=\"$ROW_ID\" approved=\"yes\">"
        echo "        <source>$(echo $ROW_SOURCE | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</source>"
        echo "        <target state=\"translated\">$(echo $ROW_TARGET | sed 's/</\&lt;/g' | sed 's/>/\&gt;/g')</target>"
        echo "      </trans-unit>"
    fi

    COUNTER=$(( $COUNTER + 1 ))
done

echo "       </trans-unit>"
echo "    </body>"
echo "  </file>"
echo "</xliff>"
\$\endgroup\$
  • \$\begingroup\$ thx it works faster with the "paste" on top of the loop... much appreciated :) \$\endgroup\$ – JohnnyKing94 Dec 5 '15 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.