6
\$\begingroup\$

Below is an implementation of a UTF-8 string to UTF-16 string. Kind of like MultiByteToWideChar on Win32, but it's cross-platform and constexpr. Passing null as the output simply calculates the size. It works in my testing.

#include <stddef.h>

namespace utf_converter {
namespace detail {
constexpr bool utf8_trail_byte(char8_t const in, char32_t& out) noexcept {
  if (in < 0x80 || 0xBF < in)
    return false;

  out = (out << 6) | (in & 0x3F);
  return true;
}

// Returns number of trailing bytes.
// -1 on illegal header bytes.
constexpr int utf8_header_byte(char8_t const in, char32_t& out) noexcept {
  if (in < 0x80) {  // ASCII
    out = in;
    return 0;
  }
  if (in < 0xC0) {  // not a header
    return -1;
  }
  if (in < 0xE0) {
    out = in & 0x1F;
    return 1;
  }
  if (in < 0xF0) {
    out = in & 0x0F;
    return 2;
  }
  if (in < 0xF8) {
    out = in & 0x7;
    return 3;
  }
  return -1;
}
}  // namespace detail

constexpr ptrdiff_t utf8_to_utf16(char8_t const* u8_begin,
                                  char8_t const* const u8_end,
                                  char16_t* u16out) noexcept {
  ptrdiff_t outstr_size = 0;
  while (u8_begin < u8_end) {
    char32_t code_point = 0;
    auto const byte_cnt = detail::utf8_header_byte(*u8_begin++, code_point);

    if (byte_cnt < 0 || byte_cnt > u8_end - u8_begin)
      return false;

    for (int i = 0; i < byte_cnt; ++i)
      if (!detail::utf8_trail_byte(*u8_begin++, code_point))
        return -1;

    if (code_point < 0xFFFF) {
      if (u16out)
        *u16out++ = static_cast<char16_t>(code_point);
      ++outstr_size;
    } else {
      if (u16out) {
        code_point -= 0x10000;
        *u16out++ = static_cast<char16_t>((code_point >> 10) + 0xD800);
        *u16out++ = static_cast<char16_t>((code_point & 0x3FF) + 0xDC00);
      }
      outstr_size += 2;
    }
  }
  return outstr_size;
}
}  // namespace utf_converter


// Example test on Windows where wchar_t is utf16

#include <Windows.h>

int main() {
  using utf_converter::utf8_to_utf16;

  char8_t constexpr u8[] = u8"\u7EDD\u4E0D\u4F1A\u653E\u5F03\u4F60";
  auto constexpr u16sz = utf8_to_utf16(u8, u8 + sizeof u8, nullptr);
  char16_t u16[u16sz];
  utf8_to_utf16(u8, u8 + sizeof u8, u16);

  MessageBoxW(nullptr, (wchar_t*)u16, L"UTF-16 text", MB_OK);
}
\$\endgroup\$
5
  • 1
    \$\begingroup\$ Please do not edit code after the question has been answered, read What should I not do when someone answers \$\endgroup\$
    – pacmaninbw
    Commented Jun 4, 2021 at 15:59
  • \$\begingroup\$ @pacmaninbw I intentionally didn't edit out anything answers mentioned. Not even one typo I had there. I am not sure what you're referring to. \$\endgroup\$ Commented Jun 4, 2021 at 16:00
  • 3
    \$\begingroup\$ Everyone should see what the person that answered saw, that's the reason for the don't edit rule. I didn't roll back the edit because you are right it didn't affect the answer. Every edit made after an answer is flagged in the 2nd Monitor Chat room. \$\endgroup\$
    – pacmaninbw
    Commented Jun 4, 2021 at 16:02
  • 1
    \$\begingroup\$ @pacmaninbw I see. It's nice to see this edits that invalidate answers are taken very seriously. It is, indeed, frustrating for answerers. \$\endgroup\$ Commented Jun 4, 2021 at 16:06
  • 2
    \$\begingroup\$ No one pointed it out, but code_point < 0xFFFF should have been code_point <= 0xFFFF. I am not going to edit the question to not upset mods again though. \$\endgroup\$ Commented Jun 7, 2021 at 14:01

4 Answers 4

5
\$\begingroup\$

suggested architecture

① decode UTF-8 stream

Construct with a range of UTF-8 encoded bytes, present the object as an iterator that provides complete 32-bit characters. It has another member function to give the current offset into the original input range, useful for error reporting.

A range can be a modern C++20 range object, or separate begin/end iterators, or whatever works with the libraries you are already using. I mean that loosely as a (lower-case) concept.

functional decomposition

Stand-alone utility functions used by the above are modeled on the mbchar library: is it a prefix, what is its length.

Include error detection

One issue with codecvt is the lack of error handling and the presence of UTF-8 encoding as an actual attack vector. Being an object, it is easily configured with options to consume improper byte sequences and replace with the bad-character character, or throw an exception.

② Encode 32-bit characters as UTF-16

Construct an object around an output iterator to char16_t that presents as an output iterator taking char32_t. Doing the *it++ = c32; to it will write one or two items to the wrapped iterator.

Again, configure errors to either replace an out-of-range or reserved character with the bad-character character or throw an exception.

Helper functions can cleanly be made to identify a 32-bit character as being in range for UTF-16, being in a reserved block, being in the range of surrogate pairs, etc. Those are all simple predicates that take a char and return a status, and don't modify anything or do other processing.

You can see that having simple one-line named function for returning the first half and the second half surrogates are very simple if they are called already knowing that the character is in the proper range. And it makes the final encoding logic very readable.


Note that these two jobs are separate, independant things.
You can compose them using a simple std::copy, and you can make a convenience function that wraps this conversion task. But, they can be composed along with other code as well, allowing you to do things without requiring intermediate copies to be made. It's pretty neat how iterator transforms (in Boost's Range V2 library) and now C++20 ranges let you generate an efficient all-inlined piece of code from reusable parts with the things you need to swap out being inside a loop or otherwise deep inside another piece of code.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Thank you for the review! I tried to implement your architecture in this follow-up question. \$\endgroup\$ Commented Jun 5, 2021 at 19:43
8
\$\begingroup\$

Good job on general organization: use of namespace and nested detail namespace, use of different sized char types, marking things with noexcept, etc.


Error: Why do you return false when the function expects a ptrdiff_t?


Architecture: mixing function return values and "out" parameters or, worse yet, "in/out" parameters makes it harder to understand and read through quickly.

utf8_header_byte produces the byte count through the return value and the extracted bits from this first byte via an "out" parameter. utf8_trail_byte modifies a parameter with the decoded partial result and returns a status.


The code like in utf8_header_byte always bugs me. I agree it's portable, and if it's constexpr the performance doesn't matter. But if the CPU provides a way to directly give you the number of leading 1 bits, it's just a lot of work for what could have been a single instruction. I would split the difference and implement this around a call to a function count_leading_1_bits that can have a portable implementation or call an intrinsic, through conditional compilation.

utf8_to_utf16(char8_t const* u8_begin,
              char8_t const* const u8_end,
              char16_t* u16out) noexcept {

Consider making this a template that takes any iterator, not just a raw pointer. You can use Concepts and constraints in C++20 to declare that both inputs need to be the same kind of iterator (or iterator/sentinel pair) and have char8_t as the value_type and has the necessary iterator category (does it work with forward single-pass iterators or does it require random access iterators?).

Personally, I have a more basic function that consumes bytes from UTF-8 input and returns a single char32_t. Call that from your function, also break out the surrogate pair handling into another function. Your utf8_to_utf16 function then loops calling the two helpers for input and output respectively.


I also allow more than 3 leading bits in the head byte, allowing decoding of full 31-bit characters. The input function (decode a single character) supports this; an out of range character is reported in the output step that it can't be represented as UTF-16. Note that you need checking there anyway since 4-byte codes gives 22 bits, which lets you decode things beyond the range of surrogate pairs.

You seem to be missing error checking in general. You return -1 if the number of continuation bytes is wrong, but you may have already output some characters OK. You don't report the error position, which is very handy to have in an error about being unable to load a file!

You could have a configuration where errors are automatically replaced with the invalid-character character, or throws an exception that reports the details of the error position.

\$\endgroup\$
9
  • \$\begingroup\$ Thank you for the review! The false/ptrdiff_t is a typo. I can fix it right now if you don't mind. \$\endgroup\$ Commented Jun 4, 2021 at 15:42
  • \$\begingroup\$ I used clang-format with google style for indentation. \$\endgroup\$ Commented Jun 4, 2021 at 15:43
  • \$\begingroup\$ my mistake re formatting. I saw for (int i = 0; i < byte_cnt; ++i) if (!detail::utf8_trail_byte(*u8_begin++, code_point)) return -1; and this increments the pointer as it goes, checking all the bytes ahead of time. But it doesn't operate on a copy of the pointer, so how does it decode afterwards? So I thought, on a quick reading, that it must be checking each byte before decoding and processing. \$\endgroup\$
    – JDługosz
    Commented Jun 4, 2021 at 15:48
  • \$\begingroup\$ so... the real confusion is that the decoding is done via an in/out parameter, and the return value is an error/success status. I read it as a predicate asking if it's indeed a trail byte, not expecting the "real work" to be done on the side like that. \$\endgroup\$
    – JDługosz
    Commented Jun 4, 2021 at 15:51
  • 1
    \$\begingroup\$ re optional that would be less surprising to the reader, but also complicates the code in order to use it. Just throwing an exception if it can't return something would be simplest. But I think the code can be refactored... don't test and extract and merge in a single helper function; call a function to test the high two bits, then just do the merging in the loop whose job it is to decode UTF-8. \$\endgroup\$
    – JDługosz
    Commented Jun 4, 2021 at 16:00
6
\$\begingroup\$

Adding to JDługosz's comments:

Rename utf8_to_utf16() to convert()

Why? Because the function's argument types already restrict the input to UTF-8 and the output to UTF-16, and it's in the utf_converter namespace, so there is very little room for ambiguity. It will also allow you to add overloads for the reverse transformation, and for conversions to/from UTF-32.

Make it compatible with C++20 ranges

Since you marked the question with the C++20 tag, consider that someone might want to use ranges instead of iterator pairs. Add an overload that takes the input as an input range, and writes the output to an output range. Basically, make it have the same interface as std::ranges::transform().

\$\endgroup\$
3
  • \$\begingroup\$ That reminds me... given primitives like I described (separate function to consume a single complete UTF-8 character and return it), you can simply use it with functions like transform if/when you specifically want to convert to UTF-16. I wrote mine to work as range adaptors for Boost.Range library. \$\endgroup\$
    – JDługosz
    Commented Jun 4, 2021 at 16:07
  • \$\begingroup\$ @JDługosz Are you sure? There is no 1:1 mapping between elements in a char8_t string and a char16_t string, something transform() does expect. But maybe another algorithm could be used? \$\endgroup\$
    – G. Sliepen
    Commented Jun 4, 2021 at 16:09
  • \$\begingroup\$ I don't recall exactly, @G._Slipen. It's easy to make a forward iterator (using Boost's Iterator facade library) that consumes as many items as necessary from the byte iterator it was constructed with. I had packaged some things like case conversion and UTF handling to work with Boost.Range and other things in the same ecosystem. Current Range V3 stuff is more flexible than trying to compose iterators, though. \$\endgroup\$
    – JDługosz
    Commented Jun 4, 2021 at 21:57
6
\$\begingroup\$

Re-implementing this kind of low-level conversion seems a waste of effort when C++ goes to some length to provide these facilities for us:

#include <codecvt>
#include <locale>
return std::wstring_convert<std::codecvt_utf8_utf16<char16_t>,char16_t>{}.from_bytes(s);
\$\endgroup\$
5
  • 3
    \$\begingroup\$ codecvt_utf8_utf16 is deprecated. That's kind of why I implemented this. \$\endgroup\$ Commented Jun 4, 2021 at 17:45
  • 2
    \$\begingroup\$ Yes, it's deprecated, but I don't think the intention is for everyone to reimplement basic functionality such as this. That said, I don't really understand why it's deprecated, or what the intention is... \$\endgroup\$ Commented Jun 4, 2021 at 17:53
  • 3
    \$\begingroup\$ Actually, the advice I took, when writing the above for myself a few months back, was continue using it until a new standardized, more-secure version is done. \$\endgroup\$ Commented Jun 4, 2021 at 17:57
  • 2
    \$\begingroup\$ Yes, I follow precisely the same advice. It's highly likely that I found exactly that same SO Q&A whenever I saw the spurious deprecation warning being generated by a compiler near me some time ago. I use codecvt throughout multiple production codebases, except when building on Windows, where there is already a perfectly good and non-deprecated API: MultiByteToWideChar and WideCharToMultiByte \$\endgroup\$ Commented Jun 5, 2021 at 7:21
  • \$\begingroup\$ @CodyGray Your suggestion to use MultiByteToWideChar/WideCharToMultiByte. It's documented to produce composed results from uncomposed strings. If you want simple mechanical conversion, you can't use it. \$\endgroup\$ Commented Jun 25, 2021 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.