Return-Path: X-Original-To: oracle-l@orafaq.com Delivered-To: oracle-l@orafaq.com Received: from puck1183.startdedicated.com (localhost [127.0.0.1]) by puck1183.startdedicated.com (Postfix) with ESMTP id 349B0196029D for ; Fri, 8 Aug 2014 09:39:55 +0200 (CEST) Received: from turing.freelists.org (freelists-180.iquest.net [206.53.239.180]) by puck1183.startdedicated.com (Postfix) with ESMTP for ; Fri, 8 Aug 2014 09:39:55 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 10C5C25B32; Fri, 8 Aug 2014 03:39:54 -0400 (EDT) Authentication-Results: turing.freelists.org; dkim=fail (verification failed; insecure key) header.i=@gmail.com; dkim-adsp=none (insecure policy) X-Virus-Scanned: Debian amavisd-new at turing.freelists.org Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LjpVm8Mi1WUt; Fri, 8 Aug 2014 03:39:53 -0400 (EDT) Received: from turing.freelists.org (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 44AE324694; Fri, 8 Aug 2014 03:39:13 -0400 (EDT) Received: with ECARTIS (v1.0.0; list oracle-l); Fri, 08 Aug 2014 03:38:32 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 1224E2077A for ; Fri, 8 Aug 2014 03:38:32 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dX2TH92piY5a for ; Fri, 8 Aug 2014 03:38:31 -0400 (EDT) Received: from mail-ie0-f176.google.com (mail-ie0-f176.google.com [209.85.223.176]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id D28D92075F for ; Fri, 8 Aug 2014 03:38:04 -0400 (EDT) Received: by mail-ie0-f176.google.com with SMTP id tr6so5780011ieb.7 for ; Fri, 08 Aug 2014 00:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=3YiljpxqMuwqAj0IdfKL+YHljXdr2YssiKRRRCIbMEQ=; b=Tsre3lIxPitdYb9r9ncl4xa62wK5pmRHJfnLLZxo2kaYv4R4F2IvIUidBrA6trDAGY UcpiZQ2nJBhID/PMNXV0xwCI0E7FgSVYDFU6jqUMOb7Pi5uqWTwFqPpFrCRDS6TKsaJJ 3bcefizFyUfp5MGRv95Gq+XyVjc3oTUOLCcJwtNtx9tTXCLSOvVb5Ol11UgHDgIE6Yq3 sEg17Rw6vNMMVTjqXrm+pHHIuiwcSl5UGHS0Mg8ypZUZoWCmNIEk6l/AeXcLPaYBkzG0 4w2GcvVuF9SnRcMpYddkqbcW6uEmcX2eZxyFXpA7QRVObfm/UKU7Q6MfoyxZxOjhl2ot XvGQ== MIME-Version: 1.0 X-Received: by 10.43.129.74 with SMTP id hh10mr9789391icc.48.1407483484649; Fri, 08 Aug 2014 00:38:04 -0700 (PDT) Received: by 10.50.21.37 with HTTP; Fri, 8 Aug 2014 00:38:04 -0700 (PDT) In-Reply-To: References: Date: Fri, 8 Aug 2014 02:38:04 -0500 Message-ID: Subject: Re: Follow-up to CRSD Fails to Start From: David Barbour To: Justin Mungal Cc: oracle-l mailing list Content-Type: multipart/alternative; boundary=001a11c1ec9a213d0805001949b7 X-archive-position: 55856 X-ecartis-version: Ecartis v1.0.0 Sender: oracle-l-bounce@freelists.org Errors-to: oracle-l-bounce@freelists.org X-original-sender: david.barbour1@gmail.com Precedence: normal Reply-To: david.barbour1@gmail.com List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: oracle-l X-List-ID: oracle-l List-subscribe: List-owner: List-post: List-archive: X-list: oracle-l --001a11c1ec9a213d0805001949b7 Content-Type: text/plain; charset=ISO-8859-1 I muddled my way through it. This is a single-node restore of an existing 5 Node RAC. Which started life as a 3 Node RAC. And it's in a different datacenter, Which has a different network (which didn't match any of the restored networking values) which doesn't have the server listed in ntp.conf, which doesn't have the same DNS servers (which weren't configured for reverse DNS), which didn't have ..... well you get the idea. The problem was two-fold. First, the existing Global Plug and Play profile was not updated by config.sh. Which makes sense. Sort of. So after I deleted all those files, I had a different failure which was caused by the fact we had added two nodes. The crsconfig_params files was being 'augmented' by the crsconfig_addparams file. So once I removed that, it finally worked. It's just that it took a long time to get to the 'simple' stuff. Nap time. On Fri, Aug 8, 2014 at 1:41 AM, Justin Mungal wrote: > Ah, sorry David I just saw this after responding to your other email. I > don't really have anything meaningful to add, but I'm with you in that I > would only do so much troubleshooting before re-installing. I would > probably open a SEV-1 SR if possible to get a quick "do you know what is > going on here" response from Oracle Support. > > > On Thu, Aug 7, 2014 at 9:55 PM, David Barbour > wrote: > >> Long day. >> >> RHEL 6.3 >> Oracle Clusterware 11.2.0.3 (with patches) >> >> Alrighty then: >> >> Trying to run $CRS_HOME/crs/config/sonfig.sh . This is a 'bare-metal >> restore with all new disk, so no ocr, voting disks, anything. But the >> clusterware binaries were restored from tape. >> >> I run through the OUI and create a disk group and set up all the >> parameters then run root.sh. This is the result: >> >> Relinking oracle with rac_on option >> Using configuration parameter file: >> /oracle/grid/11203/crs/install/crsconfig_params >> User ignored Prerequisites during installation >> OLR initialization - successful >> Adding Clusterware entries to upstart >> CRS-2672: Attempting to start 'ora.mdnsd' on 'rchr1p01' >> CRS-2676: Start of 'ora.mdnsd' on 'rchr1p01' succeeded >> CRS-2672: Attempting to start 'ora.gpnpd' on 'rchr1p01' >> CRS-2676: Start of 'ora.gpnpd' on 'rchr1p01' succeeded >> CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rchr1p01' >> CRS-2672: Attempting to start 'ora.gipcd' on 'rchr1p01' >> CRS-2676: Start of 'ora.cssdmonitor' on 'rchr1p01' succeeded >> CRS-2676: Start of 'ora.gipcd' on 'rchr1p01' succeeded >> CRS-2672: Attempting to start 'ora.cssd' on 'rchr1p01' >> CRS-2672: Attempting to start 'ora.diskmon' on 'rchr1p01' >> CRS-2676: Start of 'ora.diskmon' on 'rchr1p01' succeeded >> CRS-2676: Start of 'ora.cssd' on 'rchr1p01' succeeded >> >> ASM created and started successfully. >> >> Disk Group GRID created successfully. >> >> clscfg: -install mode specified >> Successfully accumulated necessary OCR keys. >> Creating OCR keys for user 'root', privgrp 'sfcb'.. >> Operation successful. >> Start of resource "ora.crsd" failed >> CRS-2672: Attempting to start 'ora.crsd' on 'rchr1p01' >> CRS-5017: The resource action "ora.crsd start" encountered the following >> error: >> Start action for daemon aborted. For details refer to "(:CLSN00107:)" in >> "/oracle/grid/11203/log/rchr1p01/agent/ohasd/orarootagent_root/orarootagent_root.log". >> CRS-2674: Start of 'ora.crsd' on 'rchr1p01' failed >> CRS-2679: Attempting to clean 'ora.crsd' on 'rchr1p01' >> CRS-2681: Clean of 'ora.crsd' on 'rchr1p01' succeeded >> CRS-4000: Command Start failed, or completed with errors. >> Grid Infrastructure exclusive mode start of Cluster Ready Services failed >> at /oracle/grid/11203/crs/install/crsconfig_lib.pm line 6823. >> /oracle/grid/11203/perl/bin/perl -I/oracle/grid/11203/perl/lib >> -I/oracle/grid/11203/crs/install /oracle/grid/11203/crs/install/ >> rootcrs.pl execution failed >> >> The orarootagent_root.log shows: >> >> [ clsdmc][1386608384]Fail to connect >> (ADDRESS=(PROTOCOL=ipc)(KEY=rchr1p01DBG_CRSD)) with status 9 >> 2014-08-06 22:13:39.844: [ora.crsd][1386608384] {0:0:2} [check] Error = >> error 9 encountered when connecting to CRSD >> 2014-08-06 22:13:39.846: [ora.crsd][1386608384] {0:0:2} [check] >> DaemonAgent::check returned 1 >> 2014-08-06 22:13:39.846: [ AGFW][1388709632] {0:0:2} ora.crsd 1 1 >> state changed from: UNKNOWN to: OFFLINE >> 2014-08-06 22:13:39.846: [ AGFW][1388709632] {0:0:2} Agent sending >> last reply for: RESOURCE_PROBE[ora.crsd 1 1] ID 4097:96 >> 2014-08-06 22:13:39.847: [ COMMCRS][1375573760]clsc_connect: >> (0x7f3a3c03ad50) no listener at >> (ADDRESS=(PROTOCOL=ipc)(KEY=rchr1p01DBG_MOND)) >> >> I have Googled and searched through MOS. I've tried cleanup and restart >> and can't get by this. I'm ready to just re-install the software, but >> that's not they way I'm supposed to play the game. >> >> Anybody seen this? >> > > --001a11c1ec9a213d0805001949b7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I muddled my way through it.=A0 This is a single-node= restore of an existing 5 Node RAC.=A0 Which started life as a 3 Node RAC.= =A0 And it's in a different datacenter,=A0 Which has a different networ= k (which didn't match any of the restored networking values) which does= n't have the server listed in ntp.conf, which doesn't have the same= DNS servers (which weren't configured for reverse DNS), which didn'= ;t have ..... well you get the idea.=A0

The problem was two-fold.=A0 First, the existing Global Plug and Play p= rofile was not updated by config.sh.=A0 Which makes sense. Sort of.=A0 So a= fter I deleted all those files, I had a different failure which was caused = by the fact we had added two nodes.=A0 The crsconfig_params files was being= 'augmented' by the crsconfig_addparams file.=A0 So once I removed = that, it finally worked.

It's just that it took a long time to get to the 'si= mple' stuff.=A0

Nap time.


On Fri, Aug 8, 2014 at 1:41 AM, Jus= tin Mungal <justin@n0de.ws> wrote:
Ah, sorry David I just saw = this after responding to your other email. I don't really have anything= meaningful to add, but I'm with you in that I would only do so much tr= oubleshooting before re-installing. I would probably open a SEV-1 SR if pos= sible to get a quick "do you know what is going on here" response= from Oracle Support.


On Thu, Aug 7= , 2014 at 9:55 PM, David Barbour <david.barbour1@gmail.com><= /span> wrote:
Long day.

RHEL 6.3
Oracle Clusterware 11= .2.0.3 (with patches)

Alrighty then:

Trying to run=A0 $CRS_HOME/crs/config= /sonfig.sh .=A0 This is a 'bare-metal restore with all new disk, so no = ocr, voting disks, anything.=A0 But the clusterware binaries were restored = from tape.

I run through the OUI and create a disk group and set up all the = parameters then run root.sh. This is the result:

Relinking oracle wi= th rac_on option
Using configuration parameter file: /oracle/grid/11203/= crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - succ= essful
Adding Clusterware entries to upstart
CRS-2672: Attempting to = start 'ora.mdnsd' on 'rchr1p01'
CRS-2676: Start of '= ora.mdnsd' on 'rchr1p01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rchr1p01'
= CRS-2676: Start of 'ora.gpnpd' on 'rchr1p01' succeeded
C= RS-2672: Attempting to start 'ora.cssdmonitor' on 'rchr1p01'= ;
CRS-2672: Attempting to start 'ora.gipcd' on 'rchr1p01'
= CRS-2676: Start of 'ora.cssdmonitor' on 'rchr1p01' succeede= d
CRS-2676: Start of 'ora.gipcd' on 'rchr1p01' succeeded=
CRS-2672: Attempting to start 'ora.cssd' on 'rchr1p01'
C= RS-2672: Attempting to start 'ora.diskmon' on 'rchr1p01'CRS-2676: Start of 'ora.diskmon' on 'rchr1p01' succeeded CRS-2676: Start of 'ora.cssd' on 'rchr1p01' succeeded
ASM created and started successfully.

Disk Group GRID created succ= essfully.

clscfg: -install mode specified
Successfully accumulate= d necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'sfcb'..
Oper= ation successful.
Start of resource "ora.crsd" failed
CRS-2= 672: Attempting to start 'ora.crsd' on 'rchr1p01'
CRS-50= 17: The resource action "ora.crsd start" encountered the followin= g error:
Start action for daemon aborted. For details refer to "(:CLSN00107:)&q= uot; in "/oracle/grid/11203/log/rchr1p01/agent/ohasd/orarootagent_root= /orarootagent_root.log".
CRS-2674: Start of 'ora.crsd' on &= #39;rchr1p01' failed
CRS-2679: Attempting to clean 'ora.crsd' on 'rchr1p01'
C= RS-2681: Clean of 'ora.crsd' on 'rchr1p01' succeeded
CRS= -4000: Command Start failed, or completed with errors.
Grid Infrastructu= re exclusive mode start of Cluster Ready Services failed at /oracle/grid/11= 203/crs/install/crsco= nfig_lib.pm line 6823.
/oracle/grid/11203/perl/bin/perl -I/oracle/grid/11203/perl/lib -I/oracle/gr= id/11203/crs/install /oracle/grid/11203/crs/install/rootcrs.pl execution failed

The o= rarootagent_root.log shows:

[=A0 clsdmc][1386608384]Fail to connect (ADDRESS=3D(PROTOCOL=3Dipc)(KEY= =3Drchr1p01DBG_CRSD)) with status 9
2014-08-06 22:13:39.844: [ora.crsd][= 1386608384] {0:0:2} [check] Error =3D error 9 encountered when connecting t= o CRSD
2014-08-06 22:13:39.846: [ora.crsd][1386608384] {0:0:2} [check] DaemonAgent= ::check returned 1
2014-08-06 22:13:39.846: [=A0=A0=A0 AGFW][1388709632]= {0:0:2} ora.crsd 1 1 state changed from: UNKNOWN to: OFFLINE
2014-08-06= 22:13:39.846: [=A0=A0=A0 AGFW][1388709632] {0:0:2} Agent sending last repl= y for: RESOURCE_PROBE[ora.crsd 1 1] ID 4097:96
2014-08-06 22:13:39.847: [ COMMCRS][1375573760]clsc_connect: (0x7f3a3c03ad5= 0) no listener at (ADDRESS=3D(PROTOCOL=3Dipc)(KEY=3Drchr1p01DBG_MOND))
<= br>
I have Googled and searched through MOS.=A0 I've tried cleanup= and restart and can't get by this.=A0 I'm ready to just re-install= the software, but that's not they way I'm supposed to play the gam= e.

Anybody seen this?


--001a11c1ec9a213d0805001949b7-- -- http://www.freelists.org/webpage/oracle-l