I'm writing a Ruby extension in C. It's a string processing module working on UTF-8 encoded strings only.

One method, full_width_to_ascii!, converts full width characters to ASCII equivalents (https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms). Essentially, it subtracts an offset of 0xfee0 from any full width characters. As the bang in the name implies, it works on the string in-place.

For example:

full_width_to_ascii!('A B C')
=> 'A B C'

In UTF-8, all full-width characters are encoded in 3 bytes each, while ASCII characters are 1 byte each. So while the length of the resulting string in Unicode code points will always be the same, the new UTF-8 encoded length in bytes may be smaller.

I modify the encoded string data (retrieved using StringValueCStr()) and null-terminate it at the new end-point. Finally, I call the following function to reduce the encoded string length:

// from internal.h from MRI
#define STR_NOEMBED      FL_USER1
#define STR_SHARED       FL_USER2 /* = ELTS_SHARED */
#define STR_EMBED_P(str) (!FL_TEST_RAW((str), STR_NOEMBED))

void reduce_encoded_length(VALUE str, int length) {
  if (!STR_EMBED_P(str)) {
    RSTRING(str)->as.heap.len = length;
  } else {
    // see string.c, STR_SET_EMBED_LEN
    RBASIC(str)->flags &= ~RSTRING_EMBED_LEN_MASK;
    RBASIC(str)->flags |= (length) << RSTRING_EMBED_LEN_SHIFT;

Looking at the MRI source, the implementation of strings is surprisingly complicated, but from my understanding so far, the function above should be safe for UTF-8 encoded strings.

Does this seem reasonable? Will it break horribly on older Ruby versions? Across platforms?

See also:

https://github.com/ruby/ruby/blob/32674b167bddc0d737c38f84722986b0f228b44b/string.c http://patshaughnessy.net/2012/1/4/never-create-ruby-strings-longer-than-23-characters

  • \$\begingroup\$ I can't help much with the Ruby side of things, but I'd recommend taking a look at how iconv converts between these character sets. \$\endgroup\$
    – syb0rg
    Jun 28, 2016 at 14:12
  • \$\begingroup\$ Thanks, but my code doesn't convert character encodings. It normalizes Japanese characters in UTF-8 strings, similarly to the way that the tolower() function converts captial letters in ASCII strings to lowercase. \$\endgroup\$
    – bsa
    Jul 2, 2016 at 12:38


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.