Re: Roundoff error question

From: Mark D Powell <Mark.Powell_at_eds.com>
Date: 27 Oct 2005 08:53:58 -0700
Message-ID: <1130428438.377423.277990@g14g2000cwa.googlegroups.com>

I have added a select after each update. The difference is where you have .09.. in your calculations the query shows a value of 1.

UT1 > set numwidth 38
UT1 > CREATE TABLE roundoff_error (N NUMBER);

Table created.

UT1 > INSERT INTO roundoff_error(N) values(1/190);

1 row created.

UT1 > SELECT * FROM roundoff_error;

                                     N


--------------------------------------

.0052631578947368421052631578947368421

UT1 > UPDATE roundoff_error SET N = N * 10;

1 row updated.

UT1 > SELECT * FROM roundoff_error;

                                     N


--------------------------------------

.0526315789473684210526315789473684211

UT1 > UPDATE roundoff_error SET N = N * 19;

1 row updated.

UT1 > SELECT * FROM roundoff_error;

                                     N


--------------------------------------

                                     1

UT1 > UPDATE roundoff_error SET N = N - 1;

1 row updated.

UT1 > SELECT * FROM roundoff_error;

                                     N


--------------------------------------

-6.0000000000000000000000000000000E-40

HTH -- Mark D Powell -- Received on Thu Oct 27 2005 - 10:53:58 CDT