C++ String Conversions using Iconv

Question

Since charset conversions is so darn badly implemented and most standard options are deprecated I recently wrote a wrapper around iconv that converts from system narrow / multibyte character set to UCS-2 (wstring), UTF-8, UTF-16 and UTF-32.

sconv.h

#pragma once

#include <string>
#include <string_view>

#ifdef SCONV_DLL
#define SCONV_EXPORT __declspec(dllexport)
#else
#define SCONV_EXPORT
#endif

namespace sconv
{
    //! Convert wide string to multibyte string
    //!
    //! The narrow function converts any wide or unicode string to the system's
    //! mutlibyte encoding.
    //!
    //! @{
    SCONV_EXPORT [[nodiscard]] std::string narrow(const std::string_view& input);
    SCONV_EXPORT [[nodiscard]] std::string narrow(const std::wstring_view& input);
    SCONV_EXPORT [[nodiscard]] std::string narrow(const std::u8string_view& input);
    SCONV_EXPORT [[nodiscard]] std::string narrow(const std::u16string_view& input);
    SCONV_EXPORT [[nodiscard]] std::string narrow(const std::u32string_view& input);
    //! @}

    //! Convert to wide string.
    //!
    //! @{
    SCONV_EXPORT [[nodiscard]] std::wstring widen(const std::string_view& input);
    SCONV_EXPORT [[nodiscard]] std::wstring widen(const std::wstring_view& input);
    SCONV_EXPORT [[nodiscard]] std::wstring widen(const std::u8string_view& input);
    SCONV_EXPORT [[nodiscard]] std::wstring widen(const std::u16string_view& input);
    SCONV_EXPORT [[nodiscard]] std::wstring widen(const std::u32string_view& input);
    //! @}

    //! Convert string to UTF-8 string
    //!
    //! @{
    SCONV_EXPORT [[nodiscard]] std::u8string utf8(const std::string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u8string utf8(const std::wstring_view& input);
    SCONV_EXPORT [[nodiscard]] std::u8string utf8(const std::u8string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u8string utf8(const std::u16string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u8string utf8(const std::u32string_view& input);
    //! @}

    //! Convert string to UTF-8 string
    //!
    //! @{
    SCONV_EXPORT [[nodiscard]] std::u16string utf16(const std::string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u16string utf16(const std::wstring_view& input);
    SCONV_EXPORT [[nodiscard]] std::u16string utf16(const std::u8string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u16string utf16(const std::u16string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u16string utf16(const std::u32string_view& input);
    //! @}

    //! Convert string to UTF-8 string
    //!
    //! @{
    SCONV_EXPORT [[nodiscard]] std::u32string utf32(const std::string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u32string utf32(const std::wstring_view& input);
    SCONV_EXPORT [[nodiscard]] std::u32string utf32(const std::u8string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u32string utf32(const std::u16string_view& input);
    SCONV_EXPORT [[nodiscard]] std::u32string utf32(const std::u32string_view& input);
    //! @}
}

sconv.cpp

#include "sconv.h"

#include <cassert>
#include <array>
#include <iconv.h>

namespace sconv
{
    template <typename CharT>
    struct CharTraits {};

    template <>
    struct CharTraits<char>
    {
        // Empty string is whatever the system uses as multibyte encoding
        static constexpr const char* encoding = "";
    };

    template <>
    struct CharTraits<wchar_t>
    {
        static constexpr const char* encoding = "UCS-2LE";
    };

    template <>
    struct CharTraits<char8_t>
    {
        static constexpr const char* encoding = "UTF-8";
    };

    template <>
    struct CharTraits<char16_t>
    {
        static constexpr const char* encoding = "UTF-16LE";
    };

    template <>
    struct CharTraits<char32_t>
    {
        static constexpr const char* encoding = "UTF-32LE";
    };

    template <typename ToCharT, typename FromCharT>
    std::basic_string<ToCharT> iconv_impl(const std::basic_string_view<FromCharT>& input)
    {
        auto result = std::basic_string<ToCharT>{};

        const auto from = CharTraits<FromCharT>::encoding;
        const auto to   = CharTraits<ToCharT>::encoding;

        auto cd = iconv_open(to, from);
        assert(cd);

        char*  inbuf       = reinterpret_cast<char*>(const_cast<FromCharT*>(input.data()));
        size_t inbytesleft = input.size() * sizeof(FromCharT);

        std::array<char, 512> temp;
        char*  outbuf       = temp.data();
        size_t outbytesleft = temp.size();

        while (inbytesleft > 0u)
        {
            iconv(cd, &inbuf, &inbytesleft, &outbuf, &outbytesleft);

            if (errno == EILSEQ)
            {
                throw std::runtime_error("Invalid character sequence.");
            }

            if (errno == EINVAL)
            {
                throw std::runtime_error("Incompatible character sequence.");
            }

            result.append(reinterpret_cast<ToCharT*>(temp.data()), reinterpret_cast<ToCharT*>(outbuf));
            outbuf       = temp.data();
            outbytesleft = temp.size();
        }

        int r = iconv_close(cd);
        assert(r == 0);

        return result;
    }

    std::string narrow(const std::string_view& input)
    {
        return std::string(input);
    }

    std::string narrow(const std::wstring_view& input)
    {
        return iconv_impl<char>(input);
    }

    std::string narrow(const std::u8string_view& input)
    {
        return iconv_impl<char>(input);
    }

    std::string narrow(const std::u16string_view& input)
    {
        return iconv_impl<char>(input);
    }

    std::string narrow(const std::u32string_view& input)
    {
        return iconv_impl<char>(input);
    }

    std::wstring widen(const std::string_view& input)
    {
        return iconv_impl<wchar_t>(input);
    }

    std::wstring widen(const std::wstring_view& input)
    {
        return std::wstring(input);
    }

    std::wstring widen(const std::u8string_view& input)
    {
        return iconv_impl<wchar_t>(input);
    }

    std::wstring widen(const std::u16string_view& input)
    {
        return iconv_impl<wchar_t>(input);
    }

    std::wstring widen(const std::u32string_view& input)
    {
        return iconv_impl<wchar_t>(input);
    }

    std::u8string utf8(const std::string_view& input)
    {
        return iconv_impl<char8_t>(input);
    }
    std::u8string utf8(const std::wstring_view& input)
    {
        return iconv_impl<char8_t>(input);
    }

    std::u8string utf8(const std::u8string_view& input)
    {
        return std::u8string(input);
    }

    std::u8string utf8(const std::u16string_view& input)
    {
        return iconv_impl<char8_t>(input);
    }

    std::u8string utf8(const std::u32string_view& input)
    {
        return iconv_impl<char8_t>(input);
    }

    std::u16string utf16(const std::string_view& input)
    {
        return iconv_impl<char16_t>(input);
    }

    std::u16string utf16(const std::wstring_view& input)
    {
        return iconv_impl<char16_t>(input);
    }

    std::u16string utf16(const std::u8string_view& input)
    {
        return iconv_impl<char16_t>(input);
    }

    std::u16string utf16(const std::u16string_view& input)
    {
        return std::u16string(input);
    }

    std::u16string utf16(const std::u32string_view& input)
    {
        return iconv_impl<char16_t>(input);
    }

    std::u32string utf32(const std::string_view& input)
    {
        return iconv_impl<char32_t>(input);
    }

    std::u32string utf32(const std::wstring_view& input)
    {
        return iconv_impl<char32_t>(input);
    }

    std::u32string utf32(const std::u8string_view& input)
    {
        return iconv_impl<char32_t>(input);
    }

    std::u32string utf32(const std::u16string_view& input)
    {
        return iconv_impl<char32_t>(input);
    }

    std::u32string utf32(const std::u32string_view& input)
    {
        return std::u32string(input);
    }
}

Some notes:

• I am only considering OS native narrow and wide character encodings and Unicode. For example I don't care for Shift JIS to Latin1. My inputs and outputs are some Unicode (mostly UTF-8) and I just need to interact with the OS.
• I am naively assuming all my UTF and USC encodings are little endian. This has worked for me quite well, but may be an issue.
• I have provided a pass through non converting function for symmetry. So I can do things like narrow(path.native()) in a portable way. (For display and diagnostics.)

The code is targeting C++20. I would like to hear what you think about it.

Answer 1

Pass string_views by value

Unlike std::strings, you should pass string_view by value, as it already is just a pointer and a size:

SCONV_EXPORT [[nodiscard]] std::string narrow(std::string_view input);
…

Consider avoiding const_cast<>()

The inbuf parameter of iconv() is not const in any way. This is because it can actually write to the input buffer. This could happen in case the output buffer is too small and the input has certain encodings. It probably doesn't change the input for UTF, but you could consider avoiding the cast by making a copy of the input into a mutable buffer.

Convert directly into the output string

You are converting into a temporary buffer, and then copying the result into the output string. You could instead resize the output string first, and then convert directly into that.

Related to this, consider that the output usually has a similar number of characters as the input, perhaps a few extra once in a while. You could therefore make a reasonable estimate of the size of the output string and immediately resize it to that, and if it still wasn't enough you can make it a bit bigger and do the rest of the conversion.