Re: Problem with starting CRS 11g on one of the node(RAC)

From: Pavel <ocp.pauler_at_gmail.com>
Date: Fri, 18 Feb 2011 20:46:47 +0300
Message-ID: <AANLkTinVUGLpjDawCuPedFkeYF+kQpNbc0AJqNkGnQ0F_at_mail.gmail.com>



Hi Martin

[root_at_hqora01n1srv crsd]# /u01/app/oracle/product/11.1.0/crs/bin/ocrcheck Status of Oracle Cluster Registry is as follows :

         Version                  :          2
         Total space (kbytes)     :     305768
         Used space (kbytes)      :       4644
         Available space (kbytes) :     301124
         ID                       :  126840750
         Device/File Name         : /dev/sddlmab
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root_at_hqora01n1srv crsd]# /u01/app/oracle/product/11.1.0/crs/bin/crsctl query css votedisk

  1. 0 /dev/sddlmaa Located 1 voting disk(s). [root_at_hqora01n1srv crsd]#

Permissions:

[root_at_srv crsd]# ls -la /dev/sddlmaa
brw-r--r-- 1 oracle dba 252, 0 Feb 17 22:35 /dev/sddlmaa [root_at_srv crsd]# ls -la /dev/sddlmab
brw-r--r-- 1 oracle oinstall 252, 16 Feb 17 22:35 /dev/sddlmab [root_at_srv crsd]#

From second node:

[root_at_srv2 ~]# ls -la /dev/sddlmaa
brw-r--r-- 1 oracle dba 252, 0 Feb 17 00:09 /dev/sddlmaa [root_at_hqora01n2srv ~]# ls -la /dev/sddlmab brw-r--r-- 1 oracle oinstall 252, 16 Feb 17 00:09 /dev/sddlmab [root_at_srv2 ~]#

crsd.log:
2010-09-05 23:40:16.158: [ CRSRES][105307040] startRunnable: setting CLI values
2010-09-05 23:40:16.173: [ CRSRES][105307040] Attempting to start `ora.srv.ASM1.asm` on member `srv`
2010-09-05 23:40:26.206: [ CRSRES][105307040] Start of `ora.srv.ASM1.asm` on member `srv` succeeded.
2010-09-05 23:40:33.841: [ CRSRES][105307040] Attempting to start `ora.srv.ASM2.asm` on member `srv`
2010-09-05 23:40:39.033: [ CRSRES][105307040] Start of `ora.srv.ASM2.asm` on member `srv` succeeded.
2010-09-05 23:41:14.427: [ CRSRES][145357728] startRunnable: setting CLI values
2010-09-05 23:41:14.435: [ CRSRES][145357728] Attempting to start `ora.boss.boss1.inst` on member `srv`
2010-09-05 23:41:14.463: [ CRSRES][132918176] Attempting to start `ora.boss.boss2.inst` on member `srv`
2010-09-05 23:41:43.512: [ CRSRES][145357728] Start of `ora.boss.boss1.inst` on member `srv` succeeded. 2010-09-05 23:41:44.383: [ CRSRES][132918176] Start of `ora.boss.boss2.inst` on member `srv` succeeded. 2010-09-05 23:41:54.448: [ CRSRES][132918176] startRunnable: setting CLI values
2010-09-05 23:41:54.449: [ CRSRES][132918176] Attempting to start `ora.boss.db` on member `srv`
2010-09-05 23:41:54.522: [ CRSRES][132918176] Start of `ora.boss.db` on member `hqora01n1srv` succeeded.
2010-09-05 23:42:36.600: [ CRSRES][132918176] startRunnable: setting CLI values
2010-09-05 23:42:36.618: [ CRSRES][132918176] Attempting to start `ora.boss.boss_taf.cs` on member `srv`
2010-09-05 23:42:36.715: [ CRSRES][141155232] Attempting to start `ora.boss.boss_taf.boss2.srv` on member `srv` 2010-09-05 23:42:36.715: [ CRSRES][143256480] startRunnable: setting CLI values
2010-09-05 23:42:36.719: [ CRSRES][143256480] Attempting to start `ora.boss.boss_taf.boss1.srv` on member `srv` 2010-09-05 23:42:36.818: [ CRSRES][143256480] Start of `ora.boss.boss_taf.boss1.srv` on member `srv` succeeded. 2010-09-05 23:42:36.861: [ CRSRES][141155232] Start of `ora.boss.boss_taf.boss2.srv` on member `srv` succeeded. 2010-09-05 23:42:37.637: [ CRSRES][132918176] Start of `ora.boss.boss_taf.cs` on member `srv` succeeded. 2010-10-17 19:44:51.879: [ COMMCRS][2778004384]clsc_receive: (0xa648e8a0) error 2

2010-11-03 08:29:01.428: [ COMMCRS][2778004384]clsc_receive: (0x9f000538) error 2

2010-11-12 05:55:32.789: [ COMMCRS][2778004384]clsc_receive: (0x9f000528) error 2

2011-01-30 16:27:49.749: [ COMMCRS][2778004384]clsc_receive: (0xa0600bc8) error 2

Last record - 30 January. That's all.

Best regards,
 Pavel.

2011/2/18 Martin Bach <development_at_the-playground.de>

> Hi Pavel,
>
> The whole clusterware stack is down, you should have a look at the
> Clusterware alert log in CRS_HOME/log/hostname/ for in indication of the
> failure.
>
> In the majority of these cases wrong permissions on the shared storage
> prevent Clusterware from starting.
>
> If this doesn't fix it, please attach the Clusterware log.
>
> Hth,
>
> Martin
>
> Martin Bach
>
> Oracle Certified Master 10g
> http://martincarstenbach.wordpress.com
> http://www.linkedin.com/in/martincarstenbach
>
> ----- Reply message -----
> From: "Pavel Ermakov" <ocp.pauler_at_gmail.com>
> Date: Fri, Feb 18, 2011 17:26
> Subject: Problem with starting CRS 11g on one of the node(RAC)
> To: "ORACLE-L" <oracle-l_at_freelists.org>
>
>
> Hi listers!
> I have 2 nodes RAC(11R1 CRS&ASM and 10.2.0.3 DB). RHEL 4. From saturday in
> alert.log:
>
> Sat Feb 12 00:17:39 2011
> Starting control autobackup
> Sat Feb 12 00:33:22 2011
> Control autobackup written to SBT_TAPE device
> comment 'API Version 2.0,MMS Version 5.4.1.0',
> media '55757'
> handle 'c-1215503348-20110212-01'
> Sat Feb 12 06:01:28 2011
> Thread 1 advanced to log sequence 4975
> Current log# 2 seq# 4975 mem# 0: +DATA/boss/redo02a.log
> Current log# 2 seq# 4975 mem# 1: +RECOVERY_AREA/boss/redo02b.log
> Sat Feb 12 13:00:03 2011
> Thread 1 advanced to log sequence 4976
> Current log# 1 seq# 4976 mem# 0: +DATA/boss/redo01a.log
> Current log# 1 seq# 4976 mem# 1: +RECOVERY_AREA/boss/redo01b.log
> Sat Feb 12 13:00:19 2011
> Starting control autobackup
> Control autobackup written to DISK device
> handle
> '+RECOVERY_AREA/boss/autobackup/2011_02_12/s_742914019.422.742914021'
> Sat Feb 12 13:03:24 2011
> Starting control autobackup
> Control autobackup written to SBT_TAPE device
> comment 'API Version 2.0,MMS Version 5.4.1.0',
> media '55757'
> handle 'c-1215503348-20110212-03'
>
> That's all. In crsd.log:
>
> 2010-09-05 23:42:36.861: [ CRSRES][141155232] Start of
> `ora.boss.boss_taf.boss2.srv` on member `hqora01n2srv` succeeded.
> 2010-09-05 23:42:37.637: [ CRSRES][132918176] Start of
> `ora.boss.boss_taf.cs` on member `hqora01n1srv` succeeded.
> 2010-10-17 19:44:51.879: [ COMMCRS][2778004384]clsc_receive: (0xa648e8a0)
> error 2
>
> 2010-11-03 08:29:01.428: [ COMMCRS][2778004384]clsc_receive: (0x9f000538)
> error 2
>
> 2010-11-12 05:55:32.789: [ COMMCRS][2778004384]clsc_receive: (0x9f000528)
> error 2
>
> 2011-01-30 16:27:49.749: [ COMMCRS][2778004384]clsc_receive: (0xa0600bc8)
> error 2
>
> [root_at_srv cssd]# /u01/app/oracle/product/11.1.0/crs/bin/crsctl start crs
> Attempting to start Oracle Clusterware stack
> The CRS stack will be started shortly
> [root_at_srv cssd]# /u01/app/oracle/product/11.1.0/crs/bin/crsctl check crs
> Failure 1 contacting Cluster Synchronization Services daemon
> Cannot communicate with Cluster Ready Services
> Cannot communicate with Event Manager
> [root_at_srv cssd]# /u01/app/oracle/product/11.1.0/crs/bin/crs_stat -t
> CRS-0184: Cannot communicate with the CRS daemon.
>
>
> Can you suggest me, what could be the reason?
>
> Best regards,
> Pavel.
>
>
>

--
http://www.freelists.org/webpage/oracle-l
Received on Fri Feb 18 2011 - 11:46:47 CST

Original text of this message