2
\$\begingroup\$

I have a situation where I have a fixed point number that I want to convert to and from a floating point number. Specifically it is the SANE_Fixed type from the SANE API. Here is the what the documentation says (I grabbed the bits I think are relavant):

SANE_Fixed is used for variables that can take fixed point values in the range -32768 to 32767.9999 with a resolution of 1/65535.

#define SANE_FIXED_SCALE_SHIFT  16 
typedef SANE_Word SANE_Fixed; 

The macro SANE_FIXED_SCALE_SHIFT gives the location of the fixed binary point. This standard defines that value to be 16, which yields a resolution of 1/65536.

SANE_Word: A word is encoded as 4 bytes (32 bits). The bytes are ordered from most-significant to least-significant byte (big-endian byte-order).

So, over the wire I get 4 bytes which I convert into a .Net int like this (you can guess what GetByte and SendByte do):

public int GetWord()
{
    int value = 0;
    value += (GetByte() << 24);
    value += (GetByte() << 16);
    value += (GetByte() << 8);
    value += (GetByte() << 0);
    return value;
}

Then I want to convert that int into a floating point number (I went for decimal) like this:

public decimal ToFloating(int source)
{
    decimal value = source / ((decimal)(1 << 16));
    return value;
}

And I will also need to go back the other way so I would convert to fixed like this:

public int ToFixed(decimal source)
{
    decimal value = source * ((decimal)(1 << 16));
    return (int)value;
}

And then convert to four bytes like this to send it like so:

public void SendWord(int word)
{
    SendByte((word >> 24) & 0xff);
    SendByte((word >> 16) & 0xff);
    SendByte((word >> 8) & 0xff);
    SendByte((word >> 0) & 0xff);
}

So, here is my question:

This seems a pretty simplistic implementation to me and I read lots of stuff on the internet about creating custom fixed point classes and whatnot but, given the constraints of my scenario, is this a safe approach to take? Could I lose precision and is the maths correct?

\$\endgroup\$
  • \$\begingroup\$ Have you tried it? Does it seem to work? \$\endgroup\$ – svick Mar 8 '13 at 9:17
  • \$\begingroup\$ Yes, I have tried it and it does seem to work \$\endgroup\$ – kmp Mar 8 '13 at 10:13
3
\$\begingroup\$

If the SANE_Fixed type has a constantly fixed precision of 1/65536, that means it always stores the fractional part of the number as a 16 bit (two-byte) unsigned integer (hence the SANE_FIXED_SCALE_SHIFT of 16 bits). Since the whole number portion of the value can range from -32768 to 32767, that means that the whole part of the number is represented with a 16 bit signed integer (making the type use a total of 4 bytes).

The decimal type in .NET, while it is a floating point type, it is far more precise that the traditional float type. It stores a 96 bit integer (plus an additional bit for the sign). It then stores the position of the decimal point as a 5-bit integer, which allows the decimal point to be located after any digit in the 96-bit integer value. As such, the larger the value gets, the lower the fractional precision becomes.

In other words, the decimal type can store a value up to 79,228,162,514,264,337,593,543,950,335 (29 digits). If it does store a 29 digit number, however, it will be unable to store any fractional value with that. It will only store the whole part of the number. If, however, it is only storing a single digit value, the fractional part can be up to 28 digits long (e.g. 0.0000000000000000000000000001).

Bearing all of that in mind, if the maximum whole number that the SANE_Fixed type can store is 32767 (a five digit number), when cast to a decimal type, that leaves 24 digits of precision (e.g. 1/10^24) in the fractional part of the number. That is still far more precise than the 1/65536 precision provided by the SANE_Fixed type.

Therefore, you can be confident that the decimal type will accurately retain the precise value. Obviously, when casting from a decimal back into a SANE_Fixed type, you will lose precision, but since the value came from that type in the first place, your values are safe.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.