Re: 0.99999998 (was: Unknown SQL)

>>[...] The value 1 does
>>not change significantly among different representations. Whether
>>represented with a character string, packed decimal, binary integer, IEEE
>>floating point number or any other representation, the value 1 remains the
>>value 1.
>
>IEEE? Interesting. Do you mean 0.99999998=1? Moving that
>value from representation to representation causes rounding
>problems. Does database recognize it as 1?

If the representation is a valid representation of 1, then one can successfully and correctly convert it from one representation to another. If one expects greater precision than the representation supports, one already commits an error. If one uses IEEE floating point with an expectation of 4 digits accuracy, the conversion from IEEE to any other representation will succeed. If one uses IEEE floating point with an expectation of unlimited digits accuracy, one errs.

In the case of the value 1, IEEE can deterministically represent this value. After all, it is (0.5 * (2 ^ 1)) which maps directly to a normalized, packed mantissa and sign of 0 with an exponent of 1. If your conversion from any other representation results in 0.99999998, your conversion routine has a bug.

>And what about TShape#1 = TShape#2? Just curious.

I am not sure what your question is. If these are variables, the equality will depend on the values and types of the variables. Keep in mind that the equality will depend on the value and not on the representation.

Regards,
Bob Received on Sun Jul 22 2001 - 01:28:31 CEST