Hi,
I have a rac 9.2.0.2 Database running on RedHat AS2.1,it has been up and running for about 4 monthes(One month ago, i patched it to 9202).
Last night, one instance died unexpectedly,while another instance still running.Though not much business is affected, I want to know why it died, But i am unable to find it out, so looking for your help.
Here is some information,
I tested the interconnect and the service Network card, both running fine(at 7:00 am), and the disk system is also ok.
Alert log file:
quote:
Fri Dec 20 23:38:24 2002
Thread 2 advanced to log sequence 265
Current log# 3 seq# 265 mem# 0: /dev/raw/raw8
Sat Dec 21 04:04:29 2002
Errors in file /home/oracle/admin/rac/bdump/rac2_lmon_1634.trc:
ORA-29740: evicted by member 0, group incarnation 7
Sat Dec 21 04:04:29 2002
LMON: terminating instance due to error 29740
Sat Dec 21 04:04:31 2002
Trace dumping is performing id=[cdmp_20021221040431]
Sat Dec 21 04:04:34 2002
Instance terminated by LMON, pid = 1634
Sat Dec 21 07:38:36 2002
Starting ORACLE instance (normal)
Sat Dec 21 07:38:36 2002
and this is from the trace file:
quote:
[oracle_at_rac2 bdump]$ cat /home/oracle/admin/rac/bdump/rac2_lmon_1634.trc
/home/oracle/admin/rac/bdump/rac2_lmon_1634.trc
Oracle9i Enterprise Edition Release 9.2.0.2.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.2.0 - Production
ORACLE_HOME = /home/oracle/9.2.0
System name: Linux
Node name: rac2
Release: 2.4.9-e.3smp
Version: #1 SMP Fri May 3 16:48:54 EDT 2002
Machine: i686
Instance name: rac2
Redo thread mounted by this instance: 0
Oracle process number: 4
Unix process pid: 1634, image: oracle_at_rac2 (LMON)
- SESSION ID:(3.1) 2002-11-22 03:29:38.649
Batch msg size = 2048
Batching factor: enqueue replay 48, ack 53
Batching factor: cache replay 34 size per lock 56
kjxggin: receive buffer size = 32768
kjxgmin: SKGXN ver (2 1 Oracle 9i Reference CM)
CMCLI WARNING: CMInitContext: init ctx(0xacc37e8)
- 2002-11-22 03:29:42.243
kjxgmrcfg: Reconfiguration started, reason 1
kjxgmcs: Setting state to 0 0.
- 2002-11-22 03:29:42.243
Name Service frozen
kjxgmcs: Setting state to 0 1.
kjfcpiora: publish my weight 152022
kjxgmps: proposing substate 2
kjxgmcs: Setting state to 6 2.
Performed the unique instance identification check
kjxgmps: proposing substate 3
kjxgmcs: Setting state to 6 3.
Name Service recovery started
Deleted all dead-instance name entries
kjxgmps: proposing substate 4
kjxgmcs: Setting state to 6 4.
Multicasted all local name entries for publish
Replayed all pending requests
kjxgmps: proposing substate 5
kjxgmcs: Setting state to 6 5.
Name Service normal
Name Service recovery done
- 2002-11-22 03:29:43.397
kjxgmps: proposing substate 6
kjxgmcs: Setting state to 6 6.
- 2002-11-22 03:29:43.507
- 2002-11-22 03:29:43.508
Reconfiguration started
Synchronization timeout interval: 660 sec
List of nodes: 0,1,
Global Resource Directory frozen
node 0
release 9 2 0 2
node 1
release 9 2 0 2
res_master_weight for node 0 is 152022
res_master_weight for node 1 is 152022
Total master weight = 304044
Dead inst
Join inst 0 1
Exist inst
Active Sendback Threshold = 50 %
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Resources and enqueues cleaned out
Resources remastered 0
0 GCS shadows traversed, 0 cancelled, 0 closed
0 GCS resources traversed, 0 cancelled
set master node info
Submitted all remote-enqueue requests
kjfcrfg: Number of mesgs sent to node 0 = 0
Update rdomain variables
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
- 2002-11-22 03:29:43.868
0 GCS shadows traversed, 0 replayed, 0 unopened
Submitted all GCS cache requests
0 write requests issued in 887 GCS resources
0 PIs marked suspect, 0 flush PI msgs
- 2002-11-22 03:29:44.116
Reconfiguration complete
- 2002-11-22 03:29:51.261
kjxgrtmc2: Member 1 thread 2 mounted
- 2002-12-21 04:02:05.645
kjxgrgetresults: Detect reconfig from 0, seq 6, reason 2
- 2002-12-21 04:01:57.014
kjxgrrcfgchk: Initiating reconfig, reason 2
- 2002-12-21 04:01:57.014
kjxgmrcfg: Reconfiguration started, reason 2
kjxgmcs: Setting state to 6 0.
- 2002-12-21 04:01:57.021
Name Service frozen
kjxgmcs: Setting state to 6 1.
- 2002-12-21 04:04:29.911
kjxgrdtrt: Evicted by 0, seq (7, 6)
error 29740 detected in background process
ORA-29740: evicted by member 0, group incarnation 7
ksuitm: waiting for [5] seconds before killing DIAG
--
Please see the official ORACLE-L FAQ: http://www.orafaq.net
--
Author: chao_ping
INET: chao_ping_at_vip.163.com
Fat City Network Services -- 858-538-5051 http://www.fatcity.com
San Diego, California -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from). You may
also send the HELP command for other information (like subscribing).
Received on Sat Dec 21 2002 - 06:53:41 CST
