Oracle FAQ Your Portal to the Oracle Knowledge Grid
HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US
 

Home -> Community -> Usenet -> c.d.o.misc -> Strange precision related math errors...

Strange precision related math errors...

From: Andreas Koch <mail_at_kochandreas.com>
Date: Wed, 26 Sep 2001 19:21:26 +0200
Message-ID: <9ot325$vcj$03$1@news.t-online.com>


Hi,

i have a table test with

  a number(14,2)
  b number(14,2)
  c number(16,4)


If i (example from mind - numbers may be too small)

select a,b,c,a/b,(a/b)/c,1-((a/b)/c) from test

i get for example

19558001,1000000,1.9558,1.9558001,1.000000051129,-5.1129

well, all but the last value is correct. As soon as ((a/b)/c) has more than 4 0's before relevant digits, 1-((a/b)/c) does neither return -0.0000000051129 nor 0 (what i would have expected), but shifts the result left til it is larger(abs) than 1 !!!

This happens on both 8.0.4 and 8.1.7 ...

What is the best way to get correct results without redefining the tables.

-- 
                                       Andreas
Who doesn't live can never die
Received on Wed Sep 26 2001 - 12:21:26 CDT

Original text of this message

HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US