"gc cr failure" - how to investigate further?
From: <Christopher.Taylor2_at_parallon.net>
Date: Wed, 10 Oct 2012 07:45:53 -0500
Message-ID: <F05D8DF1FB25F44085DB74CB916678E88515FD951E_at_NADCWPMSGCMS10.hca.corpad.net>
Listers,
Env:
Linux Red Hat Enterprise Linux Server release 5.6 (Tikanga) 2.6.18-128.1.16.el5 #1 SMP Fri Jun 26 10:53:31 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
SO: 0xa7ab90210, type: 2, owner: (nil), flag: INIT/-/-/0x00 (process) Oracle pidT, calls cur/top: 0xa16db03f0/0xa06577220, flag: (0) -
Process Group: DEFAULT, pseudo proc: 0xa7813f890 O/S info: user: oracle, term: UNKNOWN, ospid: 2265 OSD pid info: Unix process pid: 2265, image: oracle_at_xxxx.xxxx.xxx
...
...
transistion:0x(nil)
(FOB) flagsQ2 fib=0xa5e78d5e0 incno=0 pending i/o cnt=0
O/S info: user: XXXXXXX, term: unknown, ospid: , machine: XXXX.XXXX.XXX
..
..
...
...
Date: Wed, 10 Oct 2012 07:45:53 -0500
Message-ID: <F05D8DF1FB25F44085DB74CB916678E88515FD951E_at_NADCWPMSGCMS10.hca.corpad.net>
Listers,
Env:
Linux Red Hat Enterprise Linux Server release 5.6 (Tikanga) 2.6.18-128.1.16.el5 #1 SMP Fri Jun 26 10:53:31 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
10.2.0.4
3-node RAC
I need some assistance. I've been all over Metalink and I cannot find anything relevant to "gc cr failure" (I have found gc block lost and RAC interconnect issues but nothing specific to "gc cr failures").
Here is what happened:
From 3:00am to 7:00am+, we had a session spinning or hanging on GC CR FAILURES until I finally killed it. After it restarted everything processed fine but I need to determine what the issue was and how to make the system more robust for the future.
Anyway, here is what I have:
- "netstat -s" outputs for all 3 nodes.
- ifconfig -a outputs for all 3 nodes.
- oradebug dump systemstate level 10
- oradebug ipc (to show interconnect info)
- AWR report for the period in question
From the systemstate file:
<snip>
PROCESS 54:
SO: 0xa7ab90210, type: 2, owner: (nil), flag: INIT/-/-/0x00 (process) Oracle pidT, calls cur/top: 0xa16db03f0/0xa06577220, flag: (0) -
int error: 0, call error: 0, sess error: 0, txn error 0 (post info) last post received: 0 0 147
last post received-location: kclcomm
last process to post me: a79b96da8 1 6
last post sent: 0 0 90
last post sent-location: KJCS Post snd proxy to flush msg
last process posted by me: a79b96da8 1 6
(latch info) wait_event=0 bits=0
Process Group: DEFAULT, pseudo proc: 0xa7813f890 O/S info: user: oracle, term: UNKNOWN, ospid: 2265 OSD pid info: Unix process pid: 2265, image: oracle_at_xxxx.xxxx.xxx
...
...
SO: 0xa594ca0f0, type: 83, owner: 0xa7ab90210, flag: INIT/-/-/0x00 freelist:[a7d735a28,a7d73dc98]
KFFMOP: hash link:[a7d735a18,a7d735a18] sobj link:[a7dad1d98,a594ca110]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:7996
flags:0x0000 disk:3 au:37642 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7dad1d88,a7dad1d88] sobj link:[a7dad20e0,a7d735a28]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:7814
flags:0x0000 disk:3 au:37676 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7dad20d0,a7dad20d0] sobj link:[a7d7358c0,a7dad1d98]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:5754
flags:0x0000 disk:0 au:72024 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d7358b0,a7d7358b0] sobj link:[a7dad1618,a7dad20e0]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:7968
flags:0x0000 disk:3 au:37620 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7dad1608,a7dad1608] sobj link:[a7d9bfe98,a7d7358c0]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:7079
flags:0x0000 disk:2 au:17935 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d9bfe88,a7d9bfe88] sobj link:[a7d735aa0,a7dad1618]
map kggrp:[0x0xa7db68f88, 0, valid] map id:26
group:[4,-1918388141] file:[272,764586873] extent:30698
flags:0x0000 disk:0 au:39595 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d735a90,a7d735a90] sobj link:[a7d9c03c0,a7d9bfe98]
map kggrp:[0x0xa7db69018, 0, valid] map id:32
group:[4,-1918388141] file:[274,776515377] extent:6393
flags:0x0000 disk:2 au:33175 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d9c03b0,a7d9c03b0] sobj link:[a7d735938,a7d735aa0]
map kggrp:[0x0xa7db69018, 0, valid] map id:32
group:[4,-1918388141] file:[274,776515377] extent:6393
flags:0x0000 disk:2 au:33175 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d735928,a7d735928] sobj link:[a7d9c0000,a7d9c03c0]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:7705
flags:0x0000 disk:2 au:15504 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d9bfff0,a7d9bfff0] sobj link:[a7d7359b0,a7d735938]
map kggrp:[0x0xa7db69018, 0, valid] map id:32
group:[4,-1918388141] file:[274,776515377] extent:4033
flags:0x0000 disk:4 au:10708 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d7359a0,a7d7359a0] sobj link:[a7d9bff10,a7d9c0000]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:4511
flags:0x0000 disk:4 au:37444 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d9bff00,a7d9bff00] sobj link:[a7d9c0348,a7d7359b0]
map kggrp:[0x0xa7db69138, 0, valid] map id:34
group:[4,-1918388141] file:[276,776517837] extent:4511
flags:0x0000 disk:4 au:37444 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d9c0338,a7d9c0338] sobj link:[a7daffde8,a7d9bff10]
map kggrp:[0x0xa7db68b98, 0, valid] map id:19
group:[4,-1918388141] file:[269,741882887] extent:16385
flags:0x0000 disk:0 au:31702 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7daffdd8,a7daffdd8] sobj link:[a7d73dc98,a7d9c0348]
map kggrp:[0x0xa7db68b98, 0, valid] map id:19
group:[4,-1918388141] file:[269,741882887] extent:25479
flags:0x0000 disk:5 au:27894 lock:0 proc:0x0xa7ab90210
KFFMOP: hash link:[a7d73dc88,a7d73dc88] sobj link:[a594ca110,a7daffde8]
map kggrp:[0x0xa7db68d48, 0, valid] map id:22
group:[7,-1918388138] file:[265,747046517] extent:0
flags:0x0000 disk:5 au:28759 lock:0 proc:0x0xa7ab90210
busylist:[a594ca120,a594ca120]
transistion:0x(nil)
(FOB) flagsQ2 fib=0xa5e78d5e0 incno=0 pending i/o cnt=0
fname=+DG_XXXXXXXXX_DATA_01/ccmnasp1/datafile/general.264.719249577
fno lblksz92 fsiz932160
(FOB) flagsQ2 fib=0xa5e787710 incno=0 pending i/o cnt=0
fname=+DG_XXXXXXX_DATA_01/ccmnasp1/datafile/system.256.718811909
fno=1 lblksz92 fsiz93216
----------------------------------------
SO: 0xa77f79160, type: 4, owner: 0xa7ab90210, flag: INIT/-/-/0x00
(session) sid: 5470 trans: 0xa5c9c7988, creator: 0xa7ab90210, flag: (48100045) USR/- BSY/-/-/-/-/-
DID: 0001-0036-00008496, short-term DID: 0001-0036-00008497
txn branch: (nil)
oct: 2, prv: 0, sql: 0x6bff5df60, psql: 0x6bfe70a58, user: 92/CONCUITY
service name: XXXXXXXXX
O/S info: user: XXXXXXX, term: unknown, ospid: , machine: XXXX.XXXX.XXX
program: XXXXXXXXXXXXXX
client info: JobNo0326477.OrgId6965592.InstId=1.SIDT70.From: XXXXXXX
application name: XXXXXXXXXXX, hash value%46894660
action name: XXXXXXXXXXX, hash value016599380
waiting for 'gc cr request' blocking sess=0x(nil) seq737 wait_time=0 seconds since wait startedB
file# , block#8bdb, class#=1
Dumping Session Wait History
for 'cr request retry' count=1 wait_time=3
file# , block#8bdb, =0
for 'gc cr failure' count=1 wait_time 1
, 8bdb, =1
for 'cr request retry' count=1 wait_time=2
file# , block#8bdb, =0
for 'gc cr failure' count=1 wait_time0
, 8bdb, =1
for 'cr request retry' count=1 wait_time=3
file# , block#8bdb, =0
for 'gc cr failure' count=1 wait_time9
, 8bdb, =1
for 'cr request retry' count=1 wait_time=3
file# , block#8bdb, =0
for 'gc cr failure' count=1 wait_time9
, 8bdb, =1
for 'cr request retry' count=1 wait_time=3
file# , block#8bdb, =0
for 'gc cr failure' count=1 wait_time9
, 8bdb, =1
temporary object counter: 1
----------------------------------------
..
..
SO: 0x919e04b30, type: 39, owner: 0xa5c9c7988, flag: -/-/-/0x00
(List of Blocks) next index = 4
index itli buffer hint rdba savepoint
-----------------------------------------------------------
0 1 0x610eff610 0x7fff07ec763d 0x9c4918
1 1 0x97a0ca708 0x7fff085c627f 0x9c491d
2 1 0x1d8f080d0 0x7fff085c6281 0x9c4cf8
3 1 0x278f647a8 0x7fff085c6283 0x9c507c
----------------------------------------
SO: 0xa5eb5b1e8, type: 39, owner: 0xa5c9c7988, flag: -/-/-/0x00
(List of Blocks) next index = 20
index itli buffer hint rdba savepoint
-----------------------------------------------------------
0 1 0xa0b0da268 0x7fff07ec7636 0x9c0c06
1 1 0x4acf76668 0x7fff07ec7638 0x9c0c07
2 1 0x5aff04138 0x7fff07ec763a 0x9c0f96
3 1 0xa03008268 0x7fff07ec763c 0x9c1373
4 1 0x106f25518 0x7fff07ec763e 0x9c16f5
... MANY OF THE ABOVE...
......
SO: 0xa5c29c200, type: 36, owner: 0xa5c9c7988, flag: INIT/-/-/0x00
DML LOCK: tab$3524 flg chi=0
his[0]: mod=3 spn884819
(enqueue) TM-0003B744-00000000 DID: 0001-0036-00008496
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x7
res: 0xa75323a38, mode: SX, lock_flag: 0x0
own: 0xa77f79160, sess: 0xa77f79160, proc: 0xa7ab90210, prv: 0xa75323a48
slk: 0xa730d8c50
----------------------------------------
SO: 0xa5c29bae8, type: 36, owner: 0xa5c9c7988, flag: INIT/-/-/0x00
DML LOCK: tab$2212 flg chi=0
his[0]: mod=3 spn884819
(enqueue) TM-0003B224-00000000 DID: 0001-0036-00008496
lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 res_flag: 0x7
res: 0xa75224cf8, mode: SX, lock_flag: 0x0
own: 0xa77f79160, sess: 0xa77f79160, proc: 0xa7ab90210, prv: 0xa75224d08
slk: 0xa730d38f8
----------------------------------------
SO: 0x89679e570, type: 53, owner: 0xa77f79160, flag: INIT/-/-/0x00
LIBRARY OBJECT LOCK: lock679e570 handle1261b90 mode=N
call pin=(nil) session pin=(nil) hpc htl=0x89679e5f0[0x89679d340,0xa06594f30] htb=0xa06594f30 ssga=0xa06594688
user§7f79160 session§7f79160 count=1 flags=BRO/CBK[0120] savepoint=0x0
LIBRARY OBJECT HANDLE: handle1261b90 mtx=0x981261cc0(0) cdp=0
namespace=CRSR flags=RON/KGHP/FUL/PN0/EXP/[14010100]
kkkk-dddd-llll lwt=0x981261c38[0x981261c38,0x981261c38] ltm=0x981261c48[0x89679e5a0,0x89679e5a0]
pwt=0x981261c00[0x981261c00,0x981261c00] ptm=0x981261c10[0x981261c10,0x981261c10]
ref=0x981261c68[0x869505ab8,0x869505ab8] lnd=0x981261c80[0x981261c80,0x981261c80]
LIBRARY OBJECT: object9636c60
type=CRSR flags=EXS[0001] pflags=[0000] status=INVL load=0
DEPENDENCIES: count size
AUTHORIZATIONS: count=1 size minimum entrysize
ACCESSES: count=4 size
TRANSLATIONS: count=4 size
DATA BLOCKS:
data# heap pointer status pins change whr
----- -------- -------- --------- ---- ------ ---
0 6a9b6ffd0 909636d78 I/P/A/-/- 0 NONE 00
6 869fa91a8 8a9b05a58 I/P/A/-/E 0 NONE 00
----------------------------------------
KGX Atomic Operation Log 0x99c41f4b8
Mutex 0x869fa90b0(0, 1) idn 7ffffdad2379 oper SHRD
Cursor Pin uid 5470 efd 0 whr 5 slp 0
opr=4 pso=0x89679e570 flg=0
pcs=0x869fa90b0 nxt=0x869505b00 flg cld=0 hd=0x981261b90 par=0x96eddd0d0
ct=5 hsh=0 unp=(nil) unn=0 hvlnddd3a8 nhv=0 ses=(nil)
hep=0x869fa9130 flg ld=1 ob=0x909636c60 ptr=0x8a9b05a58 fex=0x8a9b04d68
----------------------------------------
...
-- http://www.freelists.org/webpage/oracle-lReceived on Wed Oct 10 2012 - 14:45:53 CEST