Failed 10.1.0.3 CRS install on AMD Linux V20z RHEL3

From: Roberto Devo <jdevroberto_at_yahoo.com>
Date: 19 Nov 2004 17:27:45 -0800
Message-ID: <b4942975.0411191727.5b9e7f29@posting.google.com>

Hi,

I just finished a CRS install.
It seemed to go okay but I obviously boogered something up.

olsnodes -l gives me this:

PRCO-2: Unable to communicate with the clusterware

My environment:

HW:
2xV20z (AMD Opteron)
Sun 3510 FC array
FC-HBA:
2Gigabit/Sec PCI-X Single FC Network Adapter Qlogic QLA2340-CK PCI/PCI-X
http://sunsolve.sun.com/handbook_pub/Devices/Fibre_Channel/FIBRE_PCIX_2GB_Single_FC.html

SW:
RHEL 3 Upd 3
Linux kla1 2.4.21-19.ELsmp #1 SMP Thu Aug 12 23:22:47 EDT 2004 x86_64 x86_64 x86_64 GNU/Linux
CRS 10.1.0.3.0 (Currently what is up on the otn download page)

After olsnodes gave me the error,
I rebooted both nodes.

I see these processes:

node1:
root 3403 0.0 0.0 8708 1088 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.evmd run

root 3406 0.0 0.0 8424 1164 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.cssd fatal

root 3408 0.0 0.0 8832 1096 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.crsd run

root 3416 0.0 0.0 7988 1196 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.cssd startch

root 3425 0.0 0.0 8796 1168 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.cssd startch

root 3433 0.0 0.0 8836 1168 ? S 15:52 0:00 /bin/sh
/etc/init.d/init.cssd startch

node2:
root 3397 0.0 0.0 8152 1088 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.evmd run

root 3404 0.0 0.0 8420 1168 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.cssd startcheck

root 3410 0.0 0.0 8376 1168 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.cssd fatal

root 3412 0.0 0.0 9168 1096 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.crsd run

root 3419 0.0 0.0 9160 1196 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.cssd startcheck CSS

root 3425 0.0 0.0 8060 1168 ? S 15:51 0:00 /bin/sh
/etc/init.d/init.cssd startcheck

I see evidence that the OUI layed down software:

node1:
[root_at_kla1 crs]# cd $ORACLE_HOME
[root_at_kla1 crs]# ls

./            css/          inventory/  lib32/         oraInst.loc 
relnotes/       sysman/   xdk/
../           diagnostics/  jlib/       network/       oui/        
rootaddnode.sh  tg4ifmx/
bin/          evm/          jre/        nls/           plsql/      
root.sh*        tg4ingr/
cdata/        has/          JRE@        ocrconfig.log  precomp/    
slax/           tg4sybs/
cfgtoollogs/  hs/           ldap/       opmn/          racg/       
sqlplus/        tg4tera/
crs/          install/      lib/        oracore/       rdbms/      
srvm/           utl/

[root_at_kla1 crs]#

node2:
[root_at_kla1 crs]# ssh kla2
Last login: Fri Nov 19 16:46:04 2004 from kla1 [root_at_kla2 root]# cd $ORACLE_HOME
cd $ORACLE_HOME
[root_at_kla2 crs]# ls
ls
./ evm/ ldap/ oraInst.loc root.sh* tg4tera/
../ has/ lib/ oui/ slax/

utl/
bin/          hs/         lib32/         plsql/          sqlplus/ 
xdk/
cdata/        install/    network/       precomp/        srvm/
cfgtoollogs/  inventory/  nls/           racg/           sysman/
crs/          jlib/       ocrconfig.log  rdbms/          tg4ifmx/
css/          jre/        opmn/          relnotes/       tg4ingr/
diagnostics/  JRE@        oracore/       rootaddnode.sh  tg4sybs/

[root_at_kla2 crs]#

I found good clues in the installation log file:

..........................................................................

The following configuration assistants have not been successfully completed. These assistants must be completed for your product to be completely configured.

Execute file
/oracle/crs/oracle/product/10.1.0/crs/cfgtoollogs/configToolCommands
to re-run all skipped/failed configuration assistants.

..........................................................................

contents of
/oracle/crs/oracle/product/10.1.0/crs/cfgtoollogs/configToolCommands:

..........................................................................

echo Oracle Notification Server Configuration Assistant
/oracle/crs/oracle/product/10.1.0/crs/bin/racgons add_config kla1:4948
kla2:4948
echo Oracle Private Interconnect Configuration Assistant
/oracle/crs/oracle/product/10.1.0/crs/bin/oifcfg setif -global
eth0/10.6.146.0:public eth1/10.6.147.0:clust\ er_interconnect

..........................................................................

I ran the first cmd line:

[root_at_kla1 crs]# /oracle/crs/oracle/product/10.1.0/crs/bin/racgons add_config kla1:4948 kla2:4948
[root_at_kla1 crs]#

I ran the second cmd line:

[root_at_kla1 crs]# /oracle/crs/oracle/product/10.1.0/crs/bin/oifcfg setif -global eth0/10.6.146.0:public eth1\
/10.6.147.0:cluster_interconnect

PRIF-12: failed to initialize cluster support services [root_at_kla1 crs]#

I found an interesting log file:
/oracle/crs/oracle/product/10.1.0/crs/oifcfg.log

contents:
11/19/2004 17:07:34
prifcgini: LOG initialized

11/19/2004 17:07:34

        prifcgini: LEM initialized

11/19/2004 17:07:34
prifcglms - DONE

11/19/2004 17:07:34
prifcginitclss: clsssinit failed. rc = 21 11/19/2004 17:07:34
prifcgterm: Cleaning - DONE

I googled: prifcginitclss
I got no hits.

I'd appreciate any tips from CRS experts on how to troubleshoot this failed CRS installation.

Thanks,

-jd Received on Fri Nov 19 2004 - 19:27:45 CST