4
\$\begingroup\$

I'm working on a program that handles UTF-8 characters. I've made the following macros to detect UTF-8. I've tested them with a few thousand words and they seem to work.

I'll add another one to do error-checking later, but for now I would like to know what mistakes I've made and how these macros can be improved.

//check if value is in range of leading byte
#define IS_UTF8_LEADING_BYTE(b) (((unsigned char)(b) >= 192) && (unsigned char)(b) < 248)

//check if value is in range of sequence byte
#define IS_UTF8_SEQUENCE_BYTE(b) ((unsigned char)(b) >= 128 && (unsigned char)(b) < 192)

//can be any utf8 byte, first, last...
#define IS_UTF8_BYTE(b) (IS_UTF8_LEADING_BYTE(b) || IS_UTF8_SEQUENCE_BYTE(b))

//no error checking, it must be used only on leading byte
#define HOW_MANY_UTF8_SEQUENCE_BYTES(b) ((((unsigned char)(b) & 64) == 64) + (((unsigned char)(b) & 32) == 32) + (((unsigned char)(b) & 16) == 16)) 
\$\endgroup\$
  • 1
    \$\begingroup\$ I'd make the brackets consistent: add some extra brackets to the first two macros to make all comparisons match the first one. The first has the form ((b) >= 192) whereas the rest are (b) < 248 \$\endgroup\$ – William Morris Oct 13 '13 at 8:45
  • \$\begingroup\$ @WilliamMorris thanks. Not sure how I missed that. \$\endgroup\$ – 2013Asker Oct 14 '13 at 20:48
1
\$\begingroup\$

If you are sure that you have only valid encoded UTF8, you can simplify

#define IS_UTF8_BYTE(b) (IS_UTF8_LEADING_BYTE(b) || IS_UTF8_SEQUENCE_BYTE(b))

to

#define IS_UTF8_BYTE(b) ((unsigned char)(b) >= 128)

because every byte that is not ASCII must be an UTF8-byte.

\$\endgroup\$
  • 1
    \$\begingroup\$ … but every byte that is not ASCII is not necessarily a valid UTF-8 byte. \$\endgroup\$ – 200_success Apr 11 '14 at 18:49
  • \$\begingroup\$ As I wrote: 'If you are sure, that you have only valid encoded UTF8' \$\endgroup\$ – MrSmith42 Apr 13 '14 at 7:33
3
\$\begingroup\$

There are a lot of magic numbers. They are numbers that would be familiar to any good programmer, but magic numbers nonetheless. Furthermore, it's more useful to think of the problem using bitwise operations, since they are more closely related to the conceptual design behind UTF-8.

You only accept two-, three-, and four-byte sequences, i.e., only code points U+0080 to U+1FFFFF, i.e., the Basic Multilingual Plane (minus the ASCII set) plus the Supplementary Multilingual Plane. That should be made clear in comments.

I suggest renaming IS_UTF8_BYTE() to IS_UTF8_MULTIBYTE(), since ASCII characters ≤ 127 are valid UTF-8 bytes too — just not members of a multibyte sequence.

//check if value is in range of leading byte (0b11?????? but not 0b11111???)
#define IS_UTF8_LEADING_BYTE(b) (((b) & 0xc0) && !((b) & 0xf8))

//check if value is in range of sequence byte (0b10??????)
#define IS_UTF8_SEQUENCE_BYTE(b) (((b) & 0x80) && !((b) & 0xc0))

//can be any byte within a UTF-8 multibyte sequence
#define IS_UTF8_MULTIBYTE(b) (IS_UTF8_LEADING_BYTE(b) || IS_UTF8_SEQUENCE_BYTE(b))

//no error checking; it must be used only on leading byte
#define HOW_MANY_UTF8_SEQUENCE_BYTES(b) (((b) & 0x40 == 0x40) + ((b) & 0x20 == 0x20) + ((b) & 0x10 == 0x10))

IS_UTF8_SEQUENCE_BYTE() could be more clever.

//check if value is in range of sequence byte (0b10??????)
#define IS_UTF8_SEQUENCE_BYTE(b) ((b) & 0xc0 == 0x80)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.