RE: Interesting Issue with RAC - Any Advice Appreciated

From: Pete Sharman <peter.sharman_at_ORACLE.COM>
Date: Fri, 7 Oct 2005 05:02:14 +1000
Message-ID: <20051007050214984.00000002052@psharman-au>

David

Sounds like your init scripts are screwed somehow. A couple of thoughts you might want to look into:

Have a look at the various init files and see if anything looks incorrect. If it is, don’t try hand editing these files as that might really screw things up even more. J
Are there any core dumps? They seem to end up in quite a few different locations, so try checking as root for core dumps using something like:

cd $ORA_CRS_HOME ; find . -name "*core*"

cd $ORACLE_HOME ; find . -name "*core*"

Check the location and existence of the OCR file. On Linux, look at '/etc/oracle/ocr.loc'. This file should contain:

ocrconfig_loc=<shared device>

local_only=FALSE

Looking at this file on each node is useful in case one of the nodes is configured incorrectly and is referring to a different OCR device.

Perform an ocrdump as root so we can access the configuration data:

cd $ORA_CRS_HOME/bin

ocrdump ${HOST}_OCRDUMP

If you can’t find anything screwy there, try setting _USR_ORA_DEBUG=1 in racgwrap on both CRS_HOME and ORACLE_HOME. This does not require that CRSD is restarted, but should give you better diagnostic information.

Pete

"Controlling developers is like herding cats."

Kevin Loney, Oracle DBA Handbook

"Oh no, it's not. It's much harder than that!"

Bruce Pihlamae, long-term Oracle DBA

From: oracle-l-bounce@freelists.org [mailto:oracle-l-bounce@freelists.org] On Behalf Of David Barbour
Sent: Friday, 7 October 2005 12:00 AM
To: Bryan Thomas; Oracle_L
Subject: RE: Interesting Issue with RAC - Any Advice Appreciated

Thanks for the reply Bryan. I've tried a whole bunch of times,taking the node down using init 0, taking the node down using reboot, and turning off the power. Interestingly enough, this is the 'primary' node, the one from which I performed the install. The other node works perfectly. I've tried several different approaches on the 'faulty' node, including leaving the .loc_<node> and <node>.pid files in $ORA_CRS_HOME/crs/init and removing them after running a srvctl stop nodeapps -n rhlvoo4 command. Same results. Haven't tried to disable then reenable it though. Let you know how that goes.

I forgot to metion that I can bring up the node manually using either crs_ or srvctl commands, it just won't start automatically. I have altered the cluster registry to enable the ons start automatically using crs_stat -p and crs_register to make the modification to auto_start for this service (which gets set to 0 when the 10.1.0.4 patch is applied). That seems to have worked since the ons is targeted online.

Bryan Thomas <bthomas@perftuning.com> wrote:

David,

How many times did you reboot the "crashed" server? Sometimes it takes a
several tries to get RAC fully back up.

You might also want to try to disable crsd and reboot. Then enable crsd and
reboot again. That seems to fix a lot of problems.

I have not worked with RAC on RHEL V4, so I'm not exactly sure what the
problem is.

Let me know if any of this helps.

-Bryan Thomas

-----Original Message-----
From: oracle-l-bounce@freelists.org
[mailto:oracle-l-bounce@freelists.org]On Behalf Of David Barbour
Sent: Wednesday, October 05, 2005 10:37 PM
To: Oracle_L
Subject: Interesting Issue with RAC - Any Advice Appreciated

I have an Oracle RAC installed on a pair of Dell
PE6850s w/2 processors and 8GB of RAM. Running RHEL
4.0 QU1 and Oracle 10.1.0.4. Using ASM on direct
attached CX300 with qLogic HBAs.

Thought everything was just fine - until I was testing
the RAC by crashing the nodes. If I crash a node, I
see the VIP migrate to the survivor. When I bring up
the crashed box, CRSD starts and the VIP migrates back
to it's 'home' box, but the other nodeapps, gsd and
ons, don't start, and neither does the listener nor
the instance. The crsd goes bye-bye (actually
'defunct').

Here's what I see in crs_stat -t:

[oracle@rhlv005 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.prod1.db application ONLINE ONLINE
rhlv005
ora....11.inst application ONLINE OFFLINE
ora....12.inst application ONLINE ONLINE
rhlv005
ora....SM1.asm application ONLINE ONLINE
rhlv004
ora....04.lsnr application ONLINE OFFLINE
ora....004.gsd application ONLINE OFFLINE
ora....004.ons application ONLINE OFFLINE
ora....004.vip application ONLINE ONLINE
rhlv004
ora....SM2.asm application ONLINE ONLINE
rhlv005
ora....05.lsnr application ONLINE ONLINE
rhlv005
ora....005.gsd application ONLINE ONLINE
rhlv005
ora....005.ons application ONLINE ONLINE
rhlv005
ora....005.vip application ONLINE ONLINE
rhlv005

The crs log is spectacularly informative:

2005-10-05 18:18:34.127: CRS Daemon Started.
2005-10-05 18:18:34.862: Attempting to stop
`ora.rhlv004.vip` on member `rhlv005`
2005-10-05 18:18:35.499: Stop of `ora.rhlv004.vip` on
member `rhlv005` succeeded.
2005-10-05 18:18:35.720: Attempting to start
`ora.rhlv004.vip` on member `rhlv004`
2005-10-05 18:18:41.822: Start of `ora.rhlv004.vip` on
member `rhlv004` succeeded.
2005-10-05 18:18:41.922: CRS-1007: Failed after
successful dependency consideration

2005-10-05 18:18:33.090: CRSD-1: [CMDMAIN:1336832]
Restart waiting for Oracle CRSD to start
2005-10-05 18:18:42.174: CRSD-1: Complete Restart
Application Request

Any ideas?

--
http://www.freelists.org/webpage/oracle-l

-- http://www.freelists.org/webpage/oracle-l Received on Thu Oct 06 2005 - 14:06:38 CDT