Mostly resolved: Runaway pl/sql in the db from orphaned oas connections causes complete memory saturation
From: Charles Schultz <sacrophyte_at_gmail.com>
Date: Thu, 13 Dec 2012 07:57:29 -0600
Message-ID: <CAPZQniVPyOZ4M7C+j1CYeZmE0czEXBr3w9PsNrjYAUe4rh4=mg_at_mail.gmail.com>
Seems like we have run afoul Bug 14323057 (and its several clones). To recap:
Load Runner simulates 250 doing a large number of web clicks over a 10 minute span. OAS has 40 connection pools into the backend database, but between 2 and 5 of those connections seem to randomly become infected with memory leaks - the same connection properly report a timeout to the virtual end-user, but the connection between the OAS and the database not only remains open, but goes into an infinite loop consuming more memory and cpu resources, to the point that free memory is completely consumed causing the server to hang and crash databases.
Date: Thu, 13 Dec 2012 07:57:29 -0600
Message-ID: <CAPZQniVPyOZ4M7C+j1CYeZmE0czEXBr3w9PsNrjYAUe4rh4=mg_at_mail.gmail.com>
Seems like we have run afoul Bug 14323057 (and its several clones). To recap:
Load Runner simulates 250 doing a large number of web clicks over a 10 minute span. OAS has 40 connection pools into the backend database, but between 2 and 5 of those connections seem to randomly become infected with memory leaks - the same connection properly report a timeout to the virtual end-user, but the connection between the OAS and the database not only remains open, but goes into an infinite loop consuming more memory and cpu resources, to the point that free memory is completely consumed causing the server to hang and crash databases.
The bug in question, and verified by our own testing, results in HTP.htbuf_len = -1 in the call from the OAS to the database. It would appear that there is a conditional loop that waits for htbuf_len to be positive. Reading through Bug 14323057 it appears that a patch was provided to one customer that did not fix the problem, so apparently Oracle is aware of some issues and has not yet nailed it on the head.
--
Charles Schultz
--
http://www.freelists.org/webpage/oracle-l
Received on Thu Dec 13 2012 - 14:57:29 CET