Converting between std::wstring and std::string

Question

While researching ways to convert back and forth between std::wstring and std::string, I found this conversation on the MSDN forums.

There were two functions that, to me, looked good. Specifically, these:

std::wstring s2ws(const std::string& s)
{
    int len;
    int slength = (int)s.length() + 1;
    len = MultiByteToWideChar(CP_ACP, 0, s.c_str(), slength, 0, 0); 
    wchar_t* buf = new wchar_t[len];
    MultiByteToWideChar(CP_ACP, 0, s.c_str(), slength, buf, len);
    std::wstring r(buf);
    delete[] buf;
    return r;
}

std::string ws2s(const std::wstring& s)
{
    int len;
    int slength = (int)s.length() + 1;
    len = WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, 0, 0, 0, 0); 
    char* buf = new char[len];
    WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, buf, len, 0, 0); 
    std::string r(buf);
    delete[] buf;
    return r;
}

However, the double allocation and the need to delete the buffer concern me (performance and exception safety) so I modified them to be like this:

std::wstring s2ws(const std::string& s)
{
    int len;
    int slength = (int)s.length() + 1;
    len = MultiByteToWideChar(CP_ACP, 0, s.c_str(), slength, 0, 0); 
    std::wstring r(len, L'\0');
    MultiByteToWideChar(CP_ACP, 0, s.c_str(), slength, &r[0], len);
    return r;
}

std::string ws2s(const std::wstring& s)
{
    int len;
    int slength = (int)s.length() + 1;
    len = WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, 0, 0, 0, 0); 
    std::string r(len, '\0');
    WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, &r[0], len, 0, 0); 
    return r;
}

Unit testing indicates that this works in a nice, controlled environment but will this be OK in the vicious and unpredictable world that is my client's computer?

Answer 1

I would, and have, redesign your set of functions to resemble casts:

std::wstring x;
std::string y = string_cast<std::string>(x);

This can have a lot of benefits later when you start having to deal with some 3rd party library's idea of what strings should look like.

Answer 2

Actually my unit testing shows that your code is wrong!

The problem is that you include the zero terminator in the output string, which is not supposed to happen with std::string and friends. Here's an example why this can lead to problems, especially if you use std::string::compare:

// Allocate string with 5 characters (including the zero terminator as in your code!)
string s(5, '_');

memcpy(&s[0], "ABCD\0", 5);

// Comparing with strcmp is all fine since it only compares until the terminator
const int cmp1 = strcmp(s.c_str(), "ABCD"); // 0

// ...however the number of characters that std::string::compare compares is
// someString.size(), and since s.size() == 5, it is obviously not equal to "ABCD"!
const int cmp2 = s.compare("ABCD"); // 1

// And just to prove that string implementations automatically add a zero terminator
// if you call .c_str()
s.resize(3);
const int cmp3 = strcmp(s.c_str(), "ABC"); // 0
const char term = s.c_str()[3]; // 0

printf("cmp1=%d, cmp2=%d, cmp3=%d, terminator=%d\n", cmp1, cmp2, cmp3, (int)term);

Answer 3

It really depends what codecs are being used with std::wstring and std::string.

This answer assumes that the std::wstring is using a UTF-16 encoding, and that the conversion to std::string will use a UTF-8 encoding.

#include <codecvt>
#include <string>

std::wstring utf8ToUtf16(const std::string& utf8Str)
{
    std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> conv;
    return conv.from_bytes(utf8Str);
}

std::string utf16ToUtf8(const std::wstring& utf16Str)
{
    std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>> conv;
    return conv.to_bytes(utf16Str);
}

This answer uses the STL and does not rely on a platform specific library.

Answer 4

One thing that may be an issue is that it assumes the string is ANSI formatted using the currently active code page (CP_ACP). You might want to consider using a specific code page or CP_UTF8 if it's UTF-8.

Answer 5

I'd recommend changing this:

int len;
int slength = (int)s.length() + 1;
len = WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, 0, 0, 0, 0);

...to this:

int slength = (int)s.length() + 1;
int len = WideCharToMultiByte(CP_ACP, 0, s.c_str(), slength, 0, 0, 0, 0);

Slightly more concise, len's scope is reduced, and you don't have an uninitialised variable floating round (ok, just for one line) as a trap for the unwary.

Answer 6

I don't do any Windows development, so I can't comment on the WideCharToMultiByte part being safe.

The one thing I would say though is to ensure you are using the proper types for everything. For example, string.length() returns a std::string::size_type (most likely a size_t, the constructor also takes a std::string::size_type, but that one isn't as big of a deal). It probably won't ever bite you, but it is something to be careful of to ensure you don't have any overflows in other code you may be writing.

Answer 7

When we're using std::string and std::wstring, we need #include <string>.

There's no declaration of MultiByteToWideChar() or WideCharToMultiByte(). Their names suggest they might be some thin wrapper around std::mbstowcs() and std::wcstombs() respectively, but without seeing them, it's hard to be sure.

It would be much simpler and easier to understand if we used the standard functions to convert between null-terminated multibyte and wide-character strings.

Depending on the use case, a std::codecvt<wchar_t, char, std::mbstate_t> facet might be more appropriate. Then there's very little code to be written, and in particular, no need to guess at the possible output length.

Answer 8

I've only briefly looked over your code. I haven't worked with std::string much but I've worked a lot with the API.

Assuming you got all your lengths and arguments right (sometimes making sure the terminator and wide vs multibyte lengths are all right can be tricky), I think you're on the right track. I think the first routines you posted unnecessarily allocate an additional buffer. It isn't needed.

Answer 9

No, this is dangerous! The characters in a std::string may not be stored in a contiguous memory block and you must not use the pointer &r[0] to write to any characters other than that character! This is why the c_str() function returns a const pointer.

It might work with MSVC, but it will probably break if you switch to a different compiler or STL library.