Re: Pro*C: Loss of Decimal Places INSERTing From "double" Host Var into NUMBER(16,6) Column

From: Fleetie <fleetie_at_nospam.com>
Date: Thu, 22 May 2008 09:35:31 +0100
Message-ID: <g13b8g$ul9$1@aioe.org>


"Shakespeare" <whatsin_at_xs4all.nl> wrote
> Or it reserves one digit for the sign..
>
> Shakespeare

I don't think so, because a NUMBER(1) column can take negative values, i.e. values from -9 to +9.

Here are some more findings I've just made, in the C domain:

---

Additionally, In C domain, you start losing the 6th place right of the point
after the number to the left of the D.P. exceeds about 2^33=8589934592:

TVTime=(double)8589934590.123456; -> TVTime=8589934590.123456
TVTime=(double)8589934591.123456; -> TVTime=8589934591.123456
TVTime=(double)8589934592.123456; -> TVTime=8589934592.123455 <-- Loss 
starts now.
TVTime=(double)8589934593.123456; -> TVTime=8589934593.123455
TVTime=(double)8589934594.123456; -> TVTime=8589934594.123455

So with a C double, you can't represent a FULL 16 digits in decimal, i.e.
you can't represent 9999999999.123456 without losing precision at the right.

I guess that's why Pro*C plays safe and rounds to 15 places.

---


Martin
Received on Thu May 22 2008 - 03:35:31 CDT

Original text of this message