2
\$\begingroup\$

While writing simple text rendering I found a lack of utf-8 decoders. Most decoders I found required allocating enough space for decoded string. In worse case that would mean that the decoded string would be four times as large as the original string.

I just needed to iterate over characters in a decoded format so I would be able to render them on the screen, so I wrote a simple function that would allow me to do that:

#include <cstdint>

// unsigned integer types
typedef uint64_t U64;
typedef uint32_t U32;
typedef uint16_t U16;
typedef uint8_t U8;

// signed integer types
typedef int64_t I64;
typedef int32_t I32;
typedef int16_t I16;
typedef int8_t I8;

U32 NextUTF8Char(const char* str, U32& idx)
{
    // https://en.wikipedia.org/wiki/UTF-8
    U8 c1 = (U8) str[idx];
    ++idx;

    U32 utf8c;

    if (((c1 >> 6) & 0b11) == 0b11)
    {
        // at least 2 bytes
        U8 c2 = (U8) str[idx];
        ++idx;
        if ((c1 >> 5) & 1)
        {
            // at least 3 bytes
            U8 c3 = (U8) str[idx];
            ++idx;

            if ((c1 >> 4) & 1)
            {
                // 4 bytes
                U8 c4 = (U8) str[idx];
                ++idx;

                utf8c = ((c4 & 0b00000111) << 18) | ((c3 & 0b00111111) << 12) |
                        ((c2 & 0b00111111) << 6) | (c1 & 0b00111111);
            } else
            {
                utf8c = ((c3 & 0b00001111) << 12) | ((c2 & 0b00111111) << 6) |
                        (c1 & 0b00111111);
            }
        } else
        {
            utf8c = ((c1 & 0b00011111) << 6) | (c2 & 0b00111111);
        }


    } else
    {
        utf8c = c1 & 0b01111111;
    }

    return utf8c;
}

Usage:

const char* text = u8"ta suhi škafec pušča";
U32 idx = 0;
U32 c;
while ((c = NextUTF8Char(text, idx)) != 0)
{
    // c is our utf-8 character in unsigned int format
}

I'm currently mostly concerned about the following :

  • Readability: The intent of every piece of code is clear to the reader.
  • Correctness: Everything is working as it should (I think it's clear what should happen).
  • Performance: Can anything be done to improve the performance of this code?
\$\endgroup\$
3
  • \$\begingroup\$ Looks like you're missing #include <cstdint> and a bunch of usings for those typedefs - would you care to include them in the code? \$\endgroup\$ Apr 6 at 11:33
  • \$\begingroup\$ @TobySpeight Included the missing header, thank you. Those types are already declared with typedef specifier on top of the function, so I do not know what do you mean by that. \$\endgroup\$
    – KlemenPl
    Apr 6 at 12:01
  • \$\begingroup\$ I meant the usings that bring std::uint64_t and friends into the global namespace. \$\endgroup\$ Apr 6 at 13:16
3
\$\begingroup\$
// unsigned integer types
typedef uint64_t U64;
typedef uint32_t U32;
typedef uint16_t U16;
typedef uint8_t U8;

// signed integer types
typedef int64_t I64;
typedef int32_t I32;
typedef int16_t I16;
typedef int8_t I8;

This has instantly made the code harder to read (as well as being incorrect, since <cstdint> declares those names in the std namespace). I'm not sure why we declare so many types, when we use just two of them anyway.

U32 NextUTF8Char(const char* str, U32& idx)

Why not return a standard std::wchar_t? Or perhaps a char32_t? Similarly, str ought to be a const char8_t* (so that the example code compiles).

I'd use a std::size_t for the index (or more likely get rid of idx altogether, and pass a reference to pointer instead).

The whole thing seems like a reinventing a lot of work that's already done for you:

#include <cwchar>

char32_t NextUTF8Char(const char8_t*& str)
{
    static const int max_utf8_len = 5;
    auto s = reinterpret_cast<const char*>(str);
    wchar_t c;
    std::mbstate_t state;
    auto len = std::mbrtowc(&c, s, max_utf8_len, &state);
    if (len > max_utf8_len) { return 0; }
    str += len;
    return c;
}
#include <iostream>
int main()
{
    std::locale::global(std::locale{"en_US.utf8"});
    const auto* text = u8"ta suhi škafec pušča";
    char32_t c;
    std::size_t i = 0;
    while ((c = NextUTF8Char(text)) != 0) {
        std::cout << '[' << i++ << "] = " << (std::uint_fast32_t)c << '\n';
        // c is our utf-8 character in unsigned int format
    }
}

I think that std::codecvt<char32_t, char8_t, std::mbstate_t> could easily do much the same:

#include <locale>

char32_t NextUTF8Char(const char8_t*& str)
{
    if (!*str) {
        return 0;
    }
    auto &cvt = std::use_facet<std::codecvt<char32_t, char8_t, std::mbstate_t>>(std::locale());
    std::mbstate_t state;

    char32_t c;
    char32_t* p = &c+1;
    auto result = cvt.in(state, str, str+6, str, &c, p, p);
    switch (result) {
    case std::codecvt_base::ok: return c;
    case std::codecvt_base::partial: return c;
    case std::codecvt_base::error: return 0;
    case std::codecvt_base::noconv: return 0;
    }
    return c;
}

Either is better than writing your own UTF-8 decoder.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.