A Real Stinker to Solve - EXPERTS APPLY WITHIN (DB Crashing)

From: Johne_uk <edgarj_at_tiscali.co.uk>
Date: 11 Oct 2006 09:53:49 -0700
Message-ID: <1160585629.772467.283940@h48g2000cwc.googlegroups.com>

Hi,

I have an interesting problem which has been happening for over a year on my DB instance. Oracle support has given up trying to solve it, so I have nothing to lose posting here.

I run a 9207 DB Instance on a Solaris 9 unix server (Sparc 64-bit). This is utilised by a Weblogic instance running various Java applications. Every few days the database shuts itself down and requires a manual restart.

Something is killing the LGWR process but no other logs are being generated other than what I have attached below.

I've ran truss against various unix processes (LGWR,CKPT,PMON), RDA, Statspack, init.ora changes etc. and nothing has shown up (all submitted to Metalink).

There is no pattern as to when the system crashes and I've managed to replicate the scenario on another unix server so it should not be a server failure etc.

Oracle are now asking me to apply patch 9208 but they admit that the problem is a bug and they have no idea what is causing the error. The frequency of the crash seems to be increasing also and I'm under pressure to solve.

Wondering where I can start next LOL.

Thanks in advance
John

LOGS

Alert log before and after crash / restart

Creating archive destination LOG_ARCHIVE_DEST_1: '/oracle/oradata/oranc1/arch/1_2423.arc' Wed Oct 11 16:11:52 2006
ARC1: Completed archiving log 3 thread 1 sequence 2423 Wed Oct 11 16:53:28 2006
Errors in file /oracle/admin/dump/oranc1_pmon_12316.trc: ORA-00470: LGWR process terminated with error Wed Oct 11 16:53:28 2006
PMON: terminating instance due to error 470 Instance terminated by PMON, pid = 12316 Wed Oct 11 16:56:24 2006
Starting ORACLE instance (normal)
Disable cache advisory with old cache parameters Wed Oct 11 16:56:36 2006
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0 Trace files generated

oracle_at_ncluxora1::223:/oracle/admin/dump> cat oranc1_pmon_12316.trc /oracle/admin/dump/oranc1_pmon_12316.trc Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /oracle/ora92
System name: SunOS

Node name:      ncluxora1
Release:        5.9
Version:        Generic_118558-25
Machine:        sun4u

Instance name: oranc1
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 12316, image: oracle_at_ncluxora1 (PMON)

2006-10-11 16:53:28.759
SESSION ID:(1.1) 2006-10-11 16:53:28.714 error 470 detected in background process ORA-00470: LGWR process terminated with error

/oracle/admin/dump/oranc1_arc1_20727.trc Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /oracle/ora92
System name: SunOS

Node name:      ncluxora1
Release:        5.9
Version:        Generic_118558-25
Machine:        sun4u

Instance name: oranc1
Redo thread mounted by this instance: 0 <none> Oracle process number: 11
Unix process pid: 20727, image: oracle_at_ncluxora1 (ARC1)

SESSION ID:(8.1) 2006-10-11 16:56:40.611

/oracle/admin/dump/oranc1_arc0_20725.trc Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.7.0 - Production
ORACLE_HOME = /oracle/ora92
System name: SunOS

Node name:      ncluxora1
Release:        5.9
Version:        Generic_118558-25
Machine:        sun4u

Instance name: oranc1
Redo thread mounted by this instance: 0 <none> Oracle process number: 10
Unix process pid: 20725, image: oracle_at_ncluxora1 (ARC0)

SESSION ID:(7.1) 2006-10-11 16:56:40.613

Received on Wed Oct 11 2006 - 11:53:49 CDT