Re: Follow-up to CRSD Fails to Start
Date: Fri, 8 Aug 2014 02:38:04 -0500
Message-ID: <CAFH+ifePZsysV1LuqwzkVZNXPtdJUhZKh4sHic+8Dga0sN+7-A_at_mail.gmail.com>
I muddled my way through it. This is a single-node restore of an existing 5 Node RAC. Which started life as a 3 Node RAC. And it's in a different datacenter, Which has a different network (which didn't match any of the restored networking values) which doesn't have the server listed in ntp.conf, which doesn't have the same DNS servers (which weren't configured for reverse DNS), which didn't have ..... well you get the idea.
The problem was two-fold. First, the existing Global Plug and Play profile was not updated by config.sh. Which makes sense. Sort of. So after I deleted all those files, I had a different failure which was caused by the fact we had added two nodes. The crsconfig_params files was being 'augmented' by the crsconfig_addparams file. So once I removed that, it finally worked.
It's just that it took a long time to get to the 'simple' stuff.
Nap time.
On Fri, Aug 8, 2014 at 1:41 AM, Justin Mungal <justin_at_n0de.ws> wrote:
> Ah, sorry David I just saw this after responding to your other email. I
> don't really have anything meaningful to add, but I'm with you in that I
> would only do so much troubleshooting before re-installing. I would
> probably open a SEV-1 SR if possible to get a quick "do you know what is
> going on here" response from Oracle Support.
>
>
> On Thu, Aug 7, 2014 at 9:55 PM, David Barbour <david.barbour1_at_gmail.com>
> wrote:
>
>> Long day.
>>
>> RHEL 6.3
>> Oracle Clusterware 11.2.0.3 (with patches)
>>
>> Alrighty then:
>>
>> Trying to run $CRS_HOME/crs/config/sonfig.sh . This is a 'bare-metal
>> restore with all new disk, so no ocr, voting disks, anything. But the
>> clusterware binaries were restored from tape.
>>
>> I run through the OUI and create a disk group and set up all the
>> parameters then run root.sh. This is the result:
>>
>> Relinking oracle with rac_on option
>> Using configuration parameter file:
>> /oracle/grid/11203/crs/install/crsconfig_params
>> User ignored Prerequisites during installation
>> OLR initialization - successful
>> Adding Clusterware entries to upstart
>> CRS-2672: Attempting to start 'ora.mdnsd' on 'rchr1p01'
>> CRS-2676: Start of 'ora.mdnsd' on 'rchr1p01' succeeded
>> CRS-2672: Attempting to start 'ora.gpnpd' on 'rchr1p01'
>> CRS-2676: Start of 'ora.gpnpd' on 'rchr1p01' succeeded
>> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rchr1p01'
>> CRS-2672: Attempting to start 'ora.gipcd' on 'rchr1p01'
>> CRS-2676: Start of 'ora.cssdmonitor' on 'rchr1p01' succeeded
>> CRS-2676: Start of 'ora.gipcd' on 'rchr1p01' succeeded
>> CRS-2672: Attempting to start 'ora.cssd' on 'rchr1p01'
>> CRS-2672: Attempting to start 'ora.diskmon' on 'rchr1p01'
>> CRS-2676: Start of 'ora.diskmon' on 'rchr1p01' succeeded
>> CRS-2676: Start of 'ora.cssd' on 'rchr1p01' succeeded
>>
>> ASM created and started successfully.
>>
>> Disk Group GRID created successfully.
>>
>> clscfg: -install mode specified
>> Successfully accumulated necessary OCR keys.
>> Creating OCR keys for user 'root', privgrp 'sfcb'..
>> Operation successful.
>> Start of resource "ora.crsd" failed
>> CRS-2672: Attempting to start 'ora.crsd' on 'rchr1p01'
>> CRS-5017: The resource action "ora.crsd start" encountered the following
>> error:
>> Start action for daemon aborted. For details refer to "(:CLSN00107:)" in
>> "/oracle/grid/11203/log/rchr1p01/agent/ohasd/orarootagent_root/orarootagent_root.log".
>> CRS-2674: Start of 'ora.crsd' on 'rchr1p01' failed
>> CRS-2679: Attempting to clean 'ora.crsd' on 'rchr1p01'
>> CRS-2681: Clean of 'ora.crsd' on 'rchr1p01' succeeded
>> CRS-4000: Command Start failed, or completed with errors.
>> Grid Infrastructure exclusive mode start of Cluster Ready Services failed
>> at /oracle/grid/11203/crs/install/crsconfig_lib.pm line 6823.
>> /oracle/grid/11203/perl/bin/perl -I/oracle/grid/11203/perl/lib
>> -I/oracle/grid/11203/crs/install /oracle/grid/11203/crs/install/
>> rootcrs.pl execution failed
>>
>> The orarootagent_root.log shows:
>>
>> [ clsdmc][1386608384]Fail to connect
>> (ADDRESS=(PROTOCOL=ipc)(KEY=rchr1p01DBG_CRSD)) with status 9
>> 2014-08-06 22:13:39.844: [ora.crsd][1386608384] {0:0:2} [check] Error =
>> error 9 encountered when connecting to CRSD
>> 2014-08-06 22:13:39.846: [ora.crsd][1386608384] {0:0:2} [check]
>> DaemonAgent::check returned 1
>> 2014-08-06 22:13:39.846: [ AGFW][1388709632] {0:0:2} ora.crsd 1 1
>> state changed from: UNKNOWN to: OFFLINE
>> 2014-08-06 22:13:39.846: [ AGFW][1388709632] {0:0:2} Agent sending
>> last reply for: RESOURCE_PROBE[ora.crsd 1 1] ID 4097:96
>> 2014-08-06 22:13:39.847: [ COMMCRS][1375573760]clsc_connect:
>> (0x7f3a3c03ad50) no listener at
>> (ADDRESS=(PROTOCOL=ipc)(KEY=rchr1p01DBG_MOND))
>>
>> I have Googled and searched through MOS. I've tried cleanup and restart
>> and can't get by this. I'm ready to just re-install the software, but
>> that's not they way I'm supposed to play the game.
>>
>> Anybody seen this?
>>
>
>
-- http://www.freelists.org/webpage/oracle-lReceived on Fri Aug 08 2014 - 09:38:04 CEST