# Customised Java UTF-16

I have implemented customized encoding mechanism for javaUTF16. Does this implementation support all the characters?

public class Encoding {
public static void main(String[] args) {

byte [] arr = new byte[1000];

String str = "abcde" ; //even this encoding works supplementary characters
Encode(arr,0,str);
System.out.println(Decode(arr,str.length()));
}

public static byte[] Encode(byte[] ByteArray , int offset ,String str) {

char[] ch = str.toCharArray();
for(char c : ch) {
ByteArray[offset++] = (byte) (c >>> 8);
ByteArray[offset++] = (byte) (c & 0xff);
}
return ByteArray;
}

public static String Decode(byte[] ByteArray ,int len) {

char [] res = new char[len*2];
int i = 0;
int offset = 0;
while(i < len) {
res[i] = (char) ((ByteArray[offset++] << 8) | (ByteArray[offset++] & 0xff));
i++;
}

return new String(res);
}
}

-
I would like to be sure you already read utf8everywhere.org –  Olivier Dulac Feb 25 at 12:45

### Question of Completeness

Yes, your code covers all Unicode characters, including the supplementary characters U+10000 to U+10FFFF, because you "inherit" that functionality from the way such characters would be stored in Java's String class:

### Unicode Character Representations

The char data type (and therefore the value that a Character object encapsulates) are based on the original Unicode specification, which defined characters as fixed-width 16-bit entities. The Unicode Standard has since been changed to allow for characters whose representation requires more than 16 bits. The range of legal code points is now U+0000 to U+10FFFF, known as Unicode scalar value. (Refer to the definition of the U+n notation in the Unicode Standard.)

The set of characters from U+0000 to U+FFFF is sometimes referred to as the Basic Multilingual Plane (BMP). Characters whose code points are greater than U+FFFF are called supplementary characters. The Java platform uses the UTF-16 representation in char arrays and in the String and StringBuffer classes. In this representation, supplementary characters are represented as a pair of char values, the first from the high-surrogates range, (\uD800-\uDBFF), the second from the low-surrogates range (\uDC00-\uDFFF).

A char value, therefore, represents Basic Multilingual Plane (BMP) code points, including the surrogate code points, or code units of the UTF-16 encoding. An int value represents all Unicode code points, including supplementary code points. […]

### Reinventing the Wheel

Since you did not tag your question as , I'm obligated to mention that you could accomplish the task more simply using the built-in support for charsets.

private static final Charset UTF_16 = Charset.forName("UTF-16BE");

public static byte[] Encode(byte[] ByteArray , int offset ,String str) {
byte[] bytes = str.getBytes(UTF_16);
System.arraycopy(bytes, 0, ByteArray, offset, bytes.length);
return ByteArray;
}

public static String Decode(byte[] ByteArray ,int len) {
return new String(ByteArray, 0, 2 * len, UTF_16);
}

-
@Bradd : Thanks a lot :) –  srikanth Feb 25 at 5:59

## Style

• Encode(...) -> encode(...)
• Decode(...) -> decode(...)

• ByteArray -> byteArray

## Error Handling

Your encode method assumes there is enough space in the array....

You should have something like:

if (offset < 0 || offset + (ch.length) > byteArray.length) {
throw new IllegalArgumentException("Not enough space in array....");
}


## Functionality

The code looks functional, but there is one item that is non standard....

If you are passing in an array to a method for it to be populated, you should not also return it as the return value.... the method encode(...) should be a void method.

It is right that you are concerned about Surrogates and other esoteric encoding issues.

On the other hand, your code does a simple transformation that is 100% reversible. It simply converts a char system to a byte system, and back again. This transformation does not need to be concerned about items like surrogates.

Because the input to the encode function is always a String, the output will also decode back in to a valid string too.

-
Thanks for the reply :) But my major concern is, how this code handles Surrogate or complimentary characters? Even though i'm not checking for surrogates , it still encodes all the supplementary characters .. Does it fail in any scenario ? –  srikanth Feb 24 at 12:04
@srikanth It happens to handle surrogates correctly because Java's char is just a 16-bit code unit, not an Unicode code point. I.e. Java has already implicitly taken care of that, and you only have to translate each code unit into their corresponding bytes. Another way to interpret this is that Java's strings are actually more like UCS-2 which can be used as UTF-16, just like a byte sequence can be used as UTF-8 –  amon Feb 24 at 12:07
@amon : okay . Why java implementation of UTF16 considers Surrogate checks ? If you look this method UnicodeEncoder.encodeLoop(buffer,buffer) they are checking for surrogates. Could you please explain me why are inserting checks for surrogates? –  srikanth Feb 24 at 12:12
@srikanth That's just a few sanity checks (i.e. making sure that the whole code point can be directly encoded, instead of terminating with a half-encoded character in the output buffer). Note that in case of surrogate pairs, the whole character (two pairs) is first parsed to an int (which can hold arbitrary codepoints), and then translated back to a byte representation of the surrogates. Think: int is the new char. –  amon Feb 24 at 12:20
@srikanth ... what Amon says. Also, a fair number of checks to make sure there is space in the target area. –  rolfl Feb 24 at 12:22

Proceeding with a "regular" code review…

In Encode(), (byte)(c & 0xff) could just be (byte)c.

Your doubts about the completeness of your code could be alleviated with a more effective unit test. For example, you could try encoding 😀 (U+1F600: the GRINNING FACE emoticon).

public static void main(String[] args) {
// U+1F600: the GRINNING FACE emoticon
String orig = new String(Character.toChars(0x1f600));
// Buffer size should not be hard-coded to 1000
byte[] bytes = new byte[2 * orig.length()];

System.out.println(orig);

Encode(bytes, 0, orig);
for (byte b : bytes) {
System.out.printf("%02x ", b);
}
System.out.println();

String recovered = Decode(bytes, orig.length());
System.out.println(recovered);

if (!orig.equals(recovered)) {
System.out.println("Round trip conversion failed");
}
}


The len parameter to Decode(byte[] ByteArray, int len) is deceptive. The method signature suggests that it would decode len bytes from ByteArray, but it actually decodes ByteArray until it obtains len characters. You could provide better usage hints by reversing the parameters, giving a better parameter name, and providing JavaDoc:

/**
* Decodes the specified number of <tt>char</tt>s from
* <tt>buf</tt>, a buffer containing text encoded in UTF-16BE.
* (One <tt>char</tt> is one character in the Unicode Basic
* Multilingual Plane.  A Unicode supplementary character is
* stored as two <tt>char</tt>s using a surrogate pair.)
*/
public static String Decode(int numChars, byte[] buf) {
…
}


Alternatively, change the semantics to meet my expectations.

/**
* Decodes the first <tt>numBytes</tt> from <tt>buf</tt>,
* a buffer containing text encoded in UTF-16BE.  <tt>numBytes</tt>
* must be an even number.
*/
public static String Decode(byte[] buf, int numBytes) {
…
}


In Encode(), it would be more logical to reorder the parameters with String str first. That would be consistent with other parts of the Java API, such as System.arraycopy(), which put the source parameter before the destination.

Your Encode() lets the caller specify an offset, but not a length. Your Decode() lets the caller specify a length, but not an offset. Some design consistency would be appreciated. (The simplest approach would be to allow neither offsets nor lengths to be specified.)

As @rolfl points out, you violate naming conventions by using Capitalized names for Functions and Variables. In addition ,I'll note that your indentation is haphazard and you have odd spacing around your commas ,which is slightly annoying.

-
You are right, no idea why I did not see that it is an char[]. :( –  Thorsten S. Feb 24 at 17:57
@ThorstenS. No worries. Welcome to Code Review. –  200_success Feb 24 at 18:06
I have implemented UTF-8 too .. Please have a look on this codereview.stackexchange.com/questions/42863/… –  srikanth Feb 26 at 12:31