1
\$\begingroup\$

I have created a function that converts from UTF-16 to UTF-8.
This function converts from UTF-16 to codepoint firstly, then from codepoint to UTF-8.

void ToUTF8(char16_t *str) {
    while (*str) {
        unsigned int codepoint = 0x0;

        //-------(1) UTF-16 to codepoint -------

        if (*str <= 0xD7FF) {
            codepoint = *str;
            str++;
        } else if (*str <= 0xDBFF) {
            unsigned short highSurrogate = (*str - 0xD800) * 0x400;
            unsigned short lowSurrogate = *(str+1) - 0xDC00;
            codepoint = (lowSurrogate | highSurrogate) + 0x10000;
            str += 2;
        }

        //-------(2) Codepoint to UTF-8 -------

        if (codepoint <= 0x007F) {
            unsigned char hex[2] = { 0 };
            hex[0] = (char)codepoint;
            hex[1] = 0;
            cout << std::hex << std::uppercase << "(1Byte) " << (unsigned short)hex[0] << endl;
        } else if (codepoint <= 0x07FF) {
            unsigned char hex[3] = { 0 };
            hex[0] = ((codepoint >> 6) & 0x1F) | 0xC0;
            hex[1] = (codepoint & 0x3F) | 0x80;
            hex[2] = 0;
            cout << std::hex << std::uppercase << "(2Bytes) " << (unsigned short)hex[0] << "-" << (unsigned short)hex[1] << endl;
        } else if (codepoint <= 0xFFFF) {
            unsigned char hex[4] = { 0 };
            hex[0] = ((codepoint >> 12) & 0x0F) | 0xE0;
            hex[1] = ((codepoint >> 6) & 0x3F) | 0x80;
            hex[2] = ((codepoint) & 0x3F) | 0x80;
            hex[3] = 0;
            cout << std::hex << std::uppercase << "(3Bytes) " << (unsigned short)hex[0] << "-" << (unsigned short)hex[1] << "-" << (unsigned short)hex[2] << endl; 
        } else if (codepoint <= 0x10FFFF) {
            unsigned char hex[5] = { 0 };
            hex[0] = ((codepoint >> 18) & 0x07) | 0xF0;
            hex[1] = ((codepoint >> 12) & 0x3F) | 0x80;
            hex[2] = ((codepoint >> 6) & 0x3F) | 0x80;
            hex[3] = ((codepoint) & 0x3F) | 0x80;
            hex[4] = 0;
            cout << std::hex << std::uppercase << "(4Bytes) " << (unsigned short)hex[0] << "-" << (unsigned short)hex[1] << "-" << (unsigned short)hex[2] << "-" << (unsigned short)hex[3] << endl;
        }
    }
}

Also, you can compile and test the code from here

What do you think about that function in terms of performance, and ease?

\$\endgroup\$
1
\$\begingroup\$

First a special problem: Unicode 0 is a terminator char in strings in C/C++. Modified UTF (i.e. UTF-8) deals with this by also doing the encoding for what officially should be one byte 0. As decoding poses no problem. You might consider this. For instance simply requiring modified UTF-16 on input as you check the terminator.

    if (codepoint <= 0x007F && codepoint != 0) {

Now the actual review:

  • (Optional) To cope with modified (*str) UTF-16, generating modified UTF-8:

      if (codepoint <= 0x007F && codepoint != 0) {
    
  • You are now giving the result to cout using an extra nul byte as terminator.

  • There should be a byte output stream. An output stream parameter could be appended to per byte, without intermediate arrays; and only at the end might need a NUL byte. If str occupies N UTF bytes, then the result will at most need 2N UTF bytes. _N UTF-16 bytes ~ to N/2 code points ~ max 2N UTF-8 (N/2 * 4-byte sequences)._

    That terminator should be added outside the loop.

  • Creating arrays is superfluous, immediately return the single bytes. (This would be the case for delivering the result somewhat like cout << ((codepoint >> 6) & 0x1F) | 0xC0)

  • You nicely validate the input for illegal UTF-16 chars above the max. In java one would throw an exception, you just discard the char.

  • (A matter of taste) Maybe consider an API with a string length as input parameter instead of relying on a NUL terminator. If the area of application is file based, that would even be more natural.

\$\endgroup\$
2
  • \$\begingroup\$ Thank you for your review, but can you clarify the point 3 and 4 more. \$\endgroup\$
    – Lion King
    Jun 19 '20 at 2:05
  • \$\begingroup\$ @LionKing sorry, was abroad for a couple of days, but I hope I could be less cryptic. \$\endgroup\$
    – Joop Eggen
    Jun 22 '20 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.