13
\$\begingroup\$

I am looking for some guidance and optimization pointers for my custom JavaScript function which counts the bytes in a string rather than just chars. The website uses UTF-8 and I am looking to maintain IE8 compatibility.

/**
 * Count bytes in string
 *
 * Count and return the number of bytes in a given string
 *
 * @access  public
 * @param   string
 * @return  int
 */
function getByteLen(normal_val)
{
    // Force string type
    normal_val = String(normal_val);

    // Split original string into array
    var normal_pieces = normal_val.split('');
    // Get length of original array
    var normal_length = normal_pieces.length;

    // Declare array for encoded normal array
    var encoded_pieces = new Array();

    // Declare array for individual byte pieces
    var byte_pieces = new Array();

    // Loop through normal pieces and convert to URL friendly format
    for(var i = 0; i <= normal_length; i++)
    {
        if(normal_pieces[i] && normal_pieces[i] != '')
        {
            encoded_pieces[i] = encodeURI(normal_pieces[i]);
        }
    }

    // Get length of encoded array
    var encoded_length = encoded_pieces.length;

    // Loop through encoded array
    // Scan individual items for a %
    // Split on % and add to byte array
    // If no % exists then add to byte array
    for(var i = 0; i <= encoded_length; i++)
    {
        if(encoded_pieces[i] && encoded_pieces[i] != '')
        {
            // % exists
            if(encoded_pieces[i].indexOf('%') != -1)
            {
                // Split on %
                var split_code = encoded_pieces[i].split('%');
                // Get length
                var split_length = split_code.length;

                // Loop through pieces
                for(var j = 0; j <= split_length; j++)
                {
                    if(split_code[j] && split_code[j] != '')
                    {
                        // Push to byte array
                        byte_pieces.push(split_code[j]);
                    }
                }
            }
            else
            {
                // No percent
                // Push to byte array
                byte_pieces.push(encoded_pieces[i]);
            }
        }
    }

    // Array length is the number of bytes in string
    var byte_length = byte_pieces.length;

    return byte_length;
}
\$\endgroup\$
  • \$\begingroup\$ Here is an independent and efficient method to count UTF-8 bytes of a string. Note that the method may throw error if an input string is UCS-2 malformed. \$\endgroup\$ – fuweichin Jan 21 '16 at 10:14
12
\$\begingroup\$

It would be a lot simpler to work out the length yourself rather than to interpret the results of encodeURI().

/**
 * Count bytes in a string's UTF-8 representation.
 *
 * @param   string
 * @return  int
 */
function getByteLen(normal_val) {
    // Force string type
    normal_val = String(normal_val);

    var byteLen = 0;
    for (var i = 0; i < normal_val.length; i++) {
        var c = normal_val.charCodeAt(i);
        byteLen += c < (1 <<  7) ? 1 :
                   c < (1 << 11) ? 2 :
                   c < (1 << 16) ? 3 :
                   c < (1 << 21) ? 4 :
                   c < (1 << 26) ? 5 :
                   c < (1 << 31) ? 6 : Number.NaN;
    }
    return byteLen;
}
\$\endgroup\$
  • \$\begingroup\$ Nice, I saw similar but less clean code on the site I linked to, +1 \$\endgroup\$ – konijn Dec 17 '13 at 13:23
  • \$\begingroup\$ I will give your suggestion a try as well but are the comments on this site relevant to the use of charCodeAt()? \$\endgroup\$ – MonkeyZeus Dec 17 '13 at 14:24
  • 2
    \$\begingroup\$ Good question! The code at forrst.com is bogus. Although ceil(log_256(charCode)) tells you the number of bytes it would take to represent charCode, there's nothing about UTF-8 in their byteLength() function. UTF-8 is a variable-length encoding scheme, and the few most-significant bits of every byte are necessary to indicate how many bytes form each character. Since any variable-length encoding scheme will have such padding, their byteLength() function gives a wrong answer for any encoding, including UTF-8. \$\endgroup\$ – 200_success Dec 17 '13 at 22:44
  • 1
    \$\begingroup\$ The 4-byte limit for UTF-8 derives from the decision to cap Unicode code points to U+10FFFF. However, it takes no additional effort to add two more cases, so I would code defensively. \$\endgroup\$ – 200_success Dec 18 '13 at 17:22
  • 2
    \$\begingroup\$ getByteLength( '😀' ) returns 6, but should be 4. \$\endgroup\$ – Mac May 15 '17 at 16:21
7
\$\begingroup\$

My 2 cents

  • Please do not abbreviate words, choose short words or acronyms ( Len -> Length )
  • Please lower camel case ( normal_val -> normalValue )
  • Consider using spartan conventions ( s -> generic string )
  • new Array() is considered old skool, consider var byte_pieces = []
  • You are using byte_pieces to track the bytes just to get the length, you could have just kept track of the length, this would be more efficient
  • I am not sure what abnormal pieces would be here:

if(normal_pieces[i] && normal_pieces[i] != '')

  • You check again for these here, probably not needed:

if(encoded_pieces[i] && encoded_pieces[i] != '')

  • You could just do return byte_pieces.length instead of
// Array length is the number of bytes in string
var byte_length = byte_pieces.length;

return byte_length;

All that together, I would counter propose something like this:

function getByteCount( s )
{
  var count = 0, stringLength = s.length, i;
  s = String( s || "" );
  for( i = 0 ; i < stringLength ; i++ )
  {
    var partCount = encodeURI( s[i] ).split("%").length;
    count += partCount==1?1:partCount-1;
  }
  return count;
}
getByteCount("i ♥ js");
getByteCount("abc def");

You could get the sum by using .reduce(), I leave that as an exercise to the reader.

Finally, if you are truly concerned about performance, there are some very fancy performant js libraries out there.

\$\endgroup\$
  • 1
    \$\begingroup\$ Thank you so much, looks like a lot of good stuff in your post. I will give them a go and see if I can get better performance numbers. I am not overly concerned about performance but my original code took ~6 seconds for 1200 iterations of 2400 Euro signs deduced by one char per iteration until I hit 1200 for my enforceMaxByteLength script and this code took ~3.8 so hopefully I can shave off a bit more \$\endgroup\$ – MonkeyZeus Dec 16 '13 at 18:51
  • 1
    \$\begingroup\$ Your counter proposition is genius, it shaved another .6 seconds off my benchmark, thank you. \$\endgroup\$ – MonkeyZeus Dec 16 '13 at 19:14
0
\$\begingroup\$

You can try this:

var b = str.match(/[^\x00-\xff]/g);
return (str.length + (!b ? 0: b.length));

It worked for me.

\$\endgroup\$

protected by Jamal Apr 29 at 2:12

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.