9
\$\begingroup\$

Due to recent need I wrote a simple main function that has the goal to convert the C-style strings and arrays into a more STL-style. Then because I also had a need for it, I created a variation that converts all the involved std::strings into std::wstrings.

So essentially this is to be seen as two parts:

  • Converting C-style main into STL-style main
  • Converting STL-style main into STL-style main with wide strings (in practice you can leave that out if you don't need wide strings)
#include <algorithm>
#include <codecvt>
#include <locale>
#include <string>
#include <vector>

int main(int argc, char* argv[]);
int stlMain(std::string&& cmd, std::vector<std::string>&& args);
int stlWMain(std::wstring&& cmd, std::vector<std::wstring>&& args);

int main(int argc, char* argv[]) {
    std::string cmd{argv[0]};

    std::vector<std::string> args;
    args.reserve(argc - 1);
    args.assign(argv + 1, argv + argc);

    return stlMain(std::move(cmd), std::move(args));
}

int stlMain(std::string&& cmd, std::vector<std::string>&& args) {
    // main with STL objects

    std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter;

    std::wstring wcmd = converter.from_bytes(cmd);

    std::vector<std::wstring> wargs;
    wargs.reserve(args.size());
    std::transform(args.cbegin(), args.cend(), std::back_inserter(wargs),
                   [&converter](const std::string& str) { return converter.from_bytes(str); });

    return stlWMain(std::move(wcmd), std::move(wargs));
}

int stlWMain(std::wstring&& cmd, std::vector<std::wstring>&& args) {
    // main with wide strings

    // Start you program here
}

I'm aware that this requires C++11 at least due to the codecvt header and that some compilers don't support it.

\$\endgroup\$
0

6 Answers 6

5
\$\begingroup\$

Frame Challenge

You probably don’t want to use wstring internally. MS Windows has supported UTF-8 since Windows 10 version 1803. On Linux, a wide character is four bytes long. On Windows, a wide character is not wide enough to hold all Unicode characters, such as emoji. If you don’t need to process the string character by character, it’s likely that all you want to do is output it as a wide string, in which case you might be able to get away with setting the std::codecvt facet on your output stream.

So, you probably want to just use UTF-8 everywhere. If, however, you don’t, you probably want to specify u16string or u32string.

Support for Obsolete Character Sets

The program as written assumes that the command-line arguments are passed as UTF-8, and does not check that they are valid UTF-8. Granted, that is probably the case on a modern system, but some systems out there are still using legacy 8-bit character encodings!

Deprecated Functionality

The wstring_convert and codecvt_utf8_utf16 class templates you use are both deprecated, and are scheduled to be removed in C++26. It is possible to do the conversion with non-deprecated parts of the standard library, but it takes a little more work.

A Portable Conversion Function

You need a bit of boilerplate to actually use the conversion facets in the standard library correctly, with a user-provided locale, without warnings on GCC, Clang or MSVC. (The Standard Library originally designed them to be inherited by the template classes that are now deprecated, so they have a protected destructor.)

struct deletable_converter : public std::codecvt_byname<wchar_t, char, std::mbstate_t> {
    explicit deletable_converter( const char* name, std::size_t refs = 0 )
      : std::codecvt_byname<wchar_t, char, std::mbstate_t>(name, refs) {};  
    explicit deletable_converter( const std::string& name, std::size_t refs = 0 )
      : std::codecvt_byname<wchar_t, char, std::mbstate_t>(name, refs) {};
    deletable_converter(const deletable_converter&) = delete;
    deletable_converter& operator=(const deletable_converter&) = delete;
    ~deletable_converter() = default;
};

This wrapper lets you write a function to convert between a byte string (in an arbitrary encoding) and a wide string. One way to do this is:

/* Converts a std::string_view to a std::wstring, according to the named
 * locale (by default, the current locale), or throws an exception.
 */
std::wstring string_view_to_wstring(const std::string_view in,
                                    const char* const locale_name = "") {
    if (in.empty()) {
        return std::wstring{};
    }

    const deletable_converter converter(locale_name);
    std::wstring out;
    out.resize(in.size());
    std::mbstate_t state;
    const char* last_in = nullptr;
    wchar_t* last_out = nullptr;
    const auto conversion_status = converter.in(
        state,
        in.data(),
        in.data() + in.size(),
        last_in,
        out.data(),
        out.data() + out.size(),
        last_out );

    if (converter.error == conversion_status) {
        throw std::runtime_error("Command-line argument not well-formed!");
    } else if (converter.ok != conversion_status) {
        throw std::logic_error("Logic error: unexpected return value from string conversion!");
    }

    assert( last_out >= out.data() &&
            last_out <= out.data() + out.size());
    out.resize(static_cast<std::size_t>(last_out - out.data()));
    return out;
}

If you know for a fact that you are converting between UTF-8 and UTF-16, or UTF-16 and UTF-32, you can use the other overrides in <locale>, such as std::codecvt<char16_t, char8_t, std::mbstate_t>. These let you ignore locales during conversion.

Other Tips

Since C++17, you can designate the arguments that some programs will ignore—not many programs care about argv[0]—as [[maybe_unused]]. You might also provide default arguments, to allow these to be called with a variable number of arguments, as main.

Your sample code doesn’t compile, because your delegated main function doesn’t return anything. The standard macro is EXIT_SUCCESS, but 0 is guaranteed to work too.

Because pointers to an array are iterators, which can be passed as the begin and end iterators to the STL, you could write main as a one-liner with no temporaries or moves. (Update: Also checking for the possibility Toby Speight points out, that argv[0] could be a null pointer.)

int main(int argc, char* argv[]) {
return stlMain(argv[0] ? std::string_view{argv[0]} : std::string_view{},
               std::vector<std::string_view>{argv+1, argv+argc});
}

You can optimize slightly by declaring stlMain and stlWMain as static functions (or in an anonymous namespace). Since this guarantees they can only be called from this source file, the compiler no longer has to make them externally-visible with the standard calling convention. It can instead inline them both, where they are called.

As @JDługosz mentioned, it is better to use string_view objects as the elements of args than string objects, which must make deep copies of the contents.

Putting it All Together

Here’s a test driver that converts the command-line arguments to all-caps and prints them out as wide characters:

#include <algorithm>
#include <cassert>
#include <cstdlib>
#include <cwctype>
#include <iostream>
#include <locale>
#include <stdexcept>
#include <string>
#include <utility>
#include <vector>

struct deletable_converter : public std::codecvt_byname<wchar_t, char, std::mbstate_t> {
    explicit deletable_converter( const char* name, std::size_t refs = 0 )
      : std::codecvt_byname<wchar_t, char, std::mbstate_t>(name, refs) {};  
    explicit deletable_converter( const std::string& name, std::size_t refs = 0 )
      : std::codecvt_byname<wchar_t, char, std::mbstate_t>(name, refs) {};
    deletable_converter(const deletable_converter&) = delete;
    deletable_converter& operator=(const deletable_converter&) = delete;
    ~deletable_converter() = default;
};

int main(int argc, char* argv[]);
static int stlMain(std::string_view cmd, std::vector<std::string_view>&& args);
static int stlWMain(std::wstring&& cmd, std::vector<std::wstring>&& args);

/* Converts a std::string_view to a std::wstring, according to the named
 * locale (by default, the current locale), or throws an exception.
 */
std::wstring string_view_to_wstring(const std::string_view in,
                                    const char* const locale_name = "") {
    if (in.empty()) {
        return std::wstring{};
    }

    const deletable_converter converter(locale_name);
    std::wstring out;
    out.resize(in.size());
    std::mbstate_t state;
    const char* last_in = nullptr;
    wchar_t* last_out = nullptr;
    const auto conversion_status = converter.in(
        state,
        in.data(),
        in.data() + in.size(),
        last_in,
        out.data(),
        out.data() + out.size(),
        last_out );

    if (converter.error == conversion_status) {
        throw std::runtime_error("Command-line argument not well-formed!");
    } else if (converter.ok != conversion_status) {
        throw std::logic_error("Logic error: unexpected return value from string conversion!");
    }

    assert( last_out >= out.data() &&
            last_out <= out.data() + out.size());
    out.resize(static_cast<std::size_t>(last_out - out.data()));
    return out;
}

int main(int argc, char* argv[]) {
return stlMain(argv[0] ? std::string_view{argv[0]} : std::string_view{},
               std::vector<std::string_view>{argv+1, argv+argc});
}

static int stlMain(std::string_view cmd, std::vector<std::string_view>&& args) {
    // main with STL objects
    std::locale::global(std::locale(""));
    const auto locale_string = std::locale().name();
    std::vector<std::wstring> wargs;
    wargs.resize(args.size());
    std::transform( args.begin(),
                    args.end(),
                    wargs.begin(),
                    [&locale_string](const auto sv)constexpr{
                        return string_view_to_wstring(sv, locale_string.data());
                    } );
    
    return stlWMain(string_view_to_wstring(cmd, locale_string.data()),
                    std::move(wargs));
}

static int stlWMain([[maybe_unused]] std::wstring&& cmd, [[maybe_unused]] std::vector<std::wstring>&& args) {
    std::wcout.imbue(std::locale());
    for (const auto& current_arg : args) {
        for (const auto wc : current_arg) {
            const auto uc = std::towupper(static_cast<std::wint_t>(wc));
            if (WEOF == uc) {
                throw std::runtime_error("Invalid case conversion!");
            }
            std::wcout << static_cast<wchar_t>(uc);
        }
        std::wcout << ' ';
    }
    std::wcout << '\n';
    return EXIT_SUCCESS;
}

Try it on the Compiler Explorer. (LANG must be set properly for the program to work; that execution environment needs en_US.utf8.) I have also tested on Windows 11. (Enable UTF-8 in a Windows command prompt with the command chcp 65001.)

\$\endgroup\$
1
  • \$\begingroup\$ The description of string_view_to_wstring wasn’t very good, since the default is the locale set by "", not the one set currently. \$\endgroup\$
    – Davislor
    Commented Apr 11 at 23:13
3
\$\begingroup\$

This is not as portable as we'd like it to be. Instead of assuming that the arguments are supplied as UTF-8 bytes, we ought to use the local character encoding, as used by std::locale{""}.

\$\endgroup\$
3
\$\begingroup\$

For just upgrading the char* arguments to object types, you should use string_view instead, and this does not require recopying the data. That's true for parameters in general.

std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> converter;

The meaning of wchar_t is implementation-dependent. I understand that on non-Windows platforms it's actually a 32-bit value. Yet you are only wanting 16-bit code points. Use char16_t instead.

I agree with the posters that said "UTF-8 everywhere" is better than wide characters. But working in UTF-16 is useful/necessary/efficient for some purposes including specific platforms (specifically Windows). I understand that the convert stuff in std is actually deprecated because it is flawed and has issues. If you're doing this because it's for a specific platform, it might be better to use the platform-specific features instead. OTOH, the supplied Win32 functions also have issues with error checking and reporting and lack of options for dealing with invalid data, so in a similar situation I used my own code.

You're not doing any error checking or trapping. It also assumes that the arguments coming into the program are encoded as UTF-8! (That's certainly not the case in Windows) It needs to check the Locale to determine how the incoming characters are encoded; or I should say how they are supposed to be encoded, as if the program was not invoked from an interactive command line it could still get it wrong.

\$\endgroup\$
1
  • \$\begingroup\$ How did you determine that the asker is "only wanting 16-bit code points"? I've re-read the question and there's no mention of UTF-16 or UCS-2 there. In fact, I'd have thought that platforms with 16-bit wchar_t would be the problem, due to their more limited range (if your users actually need to pass astral-plane characters). \$\endgroup\$ Commented Apr 11 at 6:54
2
\$\begingroup\$

I think std::vector is not a good fit for this purpose. std::span would fit a lot better. Here are the features that std::vector has that it shouldn't have:

  • Adjusting size, by pushing, popping or some other manipulation
  • Checking capacity
  • Move constructing

Here are the features that std::vector doesn't have but should for this purpose:

  • No allocation
  • Zero overhead of copying

All of the problems above are solved with std::span. Furthermore, whatever type will end up representing passed arguments as string, it would be great if the string type would have read-only nature as well, as it is not meant to be manipulatable.

\$\endgroup\$
1
\$\begingroup\$

Design:

I'd recommend going the UTF-8 Everywhere route instead.

This means avoiding std::wstrings and using std::strings in your program, encoded as utf-8. This requires a little care when processing strings, since you can't assume that one byte is one character, but it's better overall (see the link above for details).

On Windows, you then need to convert strings back to UTF-16 (what Microsoft calls "Unicode") when calling functions to open files, or using the Windows API. You can convert UTF-8 to UTF-16 with MultiByteToWideChar, and back again with WideCharToMultiByte.

You can get UTF-16 command line arguments on Windows with GetCommandLineW or CommandLineToArgvW, which should then be converted to UTF-8 for use internally.


On Linux there is no fixed encoding for command line arguments. Perhaps it's reasonable to simply require that arguments to your program are passed as UTF-8.

Otherwise I guess you're stuck with converting from the current locale... But you might run into issues with arguments in different encodings as described in that stackoverflow link.


Code:

I'd suggest avoiding the nested call to an inner main function. Extracting the first argument and passing it separately also seems unnecessary.

So perhaps:

std::vector<std::string> read_args(int argc, char* argv[]);
std::vector<std::wstring> convert_strings_utf8_to_utf16(std::vector<std::string> const& args);

int main(int argc, char* argv[]) {
    
    auto args = read_args(argc, argv);
    auto wargs = convert_strings_utf8_to_utf16(args);

    return run_program(wargs);
}
\$\endgroup\$
3
  • \$\begingroup\$ That's twice the allocations necessary... \$\endgroup\$ Commented Oct 14, 2021 at 11:37
  • \$\begingroup\$ @Deduplicator Do you mean that it should convert each argument to a std::wstring first before adding directly to the std::vector<std::wstring> ? \$\endgroup\$
    – user673679
    Commented Oct 14, 2021 at 11:44
  • \$\begingroup\$ I wanted to separate the program name from the arguments, as to me that isn't an argument. I do understand the technical reasons as to why it has been implemented like this, but personally speaking I see it as something separate enough to warrant putting it into it's own variable \$\endgroup\$
    – BrainStone
    Commented Oct 18, 2021 at 19:53
1
\$\begingroup\$

argv[0] can legally be a null pointer, depending on the platform and the way that your program is invoked. If it is null, then constructing a std::string from it is Undefined Behaviour. It's probably better not to treat this argument separately from the others; pass a single collection of zero or more strings.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.