Fun With Scan Listener

From: David Barbour <david.barbour1_at_gmail.com>
Date: Tue, 3 Jun 2014 16:27:24 -0500
Message-ID: <CAFH+ifcQWisWDpitAp9JOYtGHr9b9SUomkchfTK1eGezCbb3yQ_at_mail.gmail.com>

Oracle 11.2.0.3 RHEL 6.3 5-Node RAC

This has me somewhat (okay - totally) baffled. I have a scan listener that is showing as follows when I run crsctl status resource -t:

---

Cluster Resources

---

ora.LISTENER_SCAN1.lsnr

      1 ONLINE UNKNOWN rchr1p01

If I check the status via srvctl I get the following:

 $ srvctl status scan_listener

SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rchr1p02
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rchr1p03

So I try to start it:

 $ srvctl start scan_listener -i 1
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr CRS-5013: Agent "/oracle/grid/11203/bin/oraagent.bin" failed to start process "/oracle/grid/11203/bin/lsnrctl" for action "clean": details at "(:CLSN00008:)" in
"/oracle/grid/11203/log/rchr1p01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-5013: Agent "/oracle/grid/11203/bin/oraagent.bin" failed to start process "/oracle/grid/11203/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in
"/oracle/grid/11203/log/rchr1p01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2680: Clean of 'ora.LISTENER_SCAN1.lsnr' on 'rchr1p01' failed

So I try to stop it:

srvctl stop scan_listener -i 1 -f
PRCR-1065 : Failed to stop resource ora.LISTENER_SCAN1.lsnr CRS-5013: Agent "/oracle/grid/11203/bin/oraagent.bin" failed to start process "/oracle/grid/11203/bin/lsnrctl" for action "clean": details at "(:CLSN00008:)" in
"/oracle/grid/11203/log/rchr1p01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-5013: Agent "/oracle/grid/11203/bin/oraagent.bin" failed to start process "/oracle/grid/11203/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in
"/oracle/grid/11203/log/rchr1p01/agent/crsd/oraagent_oracle/oraagent_oracle.log" CRS-2680: Clean of 'ora.LISTENER_SCAN1.lsnr' on 'rchr1p01' failed

So I give up and check the log.

2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][2617243392]
{1:53466:11150} [clean] clsn_agent::clean: Exception
SclsProcessSpawnException
2014-06-03 16:19:48.666: [ AGFW][3623876352] {1:53466:11150} Agent sending reply for: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:58347 2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][2617243392]
{1:53466:11150} [clean] (:CLSN00106:) clsn_agent::clean }
2014-06-03 16:19:48.666: [ AGFW][2617243392] {1:53466:11150} Command: clean for resource: ora.LISTENER_SCAN1.lsnr 1 1 completed with status: FAIL 2014-06-03 16:19:48.666: [ AGFW][3623876352] {1:53466:11150} Agent sending reply for: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:58347 2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] LsnrAgent::check {
2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] lsnrctl status LISTENER_SCAN1

2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] getOracleHomeAttrib: oracle_home =
/oracle/grid/11203
2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] getOracleHomeAttrib: oracle_home =
/oracle/grid/11203
2014-06-03 16:19:48.666: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Utils::getCrsHome crsHome /oracle/grid/11203
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Utils::execCmd 1
USR_ORA_ENV:ORACLE_BASE=/opt/oracle oracleHome:/oracle/grid/11203 CrsHome:/oracle/grid/11203
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Utils::getCrsHome crsHome /oracle/grid/11203
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Adding Environment Variables
ORACLE_HOME=/oracle/grid/11203
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Adding Environment Variables
TNS_ADMIN=/oracle/grid/11203/network/admin/ 2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Adding Environment variable from USR_ORA_ENV
ORACLE_BASE=/opt/oracle
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] Utils:execCmd action = 3 flags = 38 ohome = (null)
cmdname = lsnrctl.
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] getOracleHomeAttrib: oracle_home =
/oracle/grid/11203
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] (:CLSN00008:)Utils:execCmd scls_process_spawn()
failed 1
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] (:CLSN00008:) category: -2, operation: fork, loc:
spawnproc28, OS error: 11, other: forked failed [-1] 2014-06-03 16:19:48.667: [ AGENT][3087005440] {1:53466:11150} UserErrorException: Locale is
2014-06-03 16:19:48.667: [ora.LISTENER_SCAN1.lsnr][3087005440]
{1:53466:11150} [check] clsnUtils::error Exception type=2 string=
CRS-5013: Agent "/oracle/grid/11203/bin/oraagent.bin" failed to start process "/oracle/grid/11203/bin/lsnrctl" for action "check": details at "(:CLSN00008:)" in
"/oracle/grid/11203/log/rchr1p01/agent/crsd/oraagent_oracle/oraagent_oracle.log"

I'm willing to go with the part about 'UserErrorException', except I'm not aware of what I'm doing wrong. Looking through MOS docs but hoping someone has a suggestion?

--
http://www.freelists.org/webpage/oracle-l

Received on Tue Jun 03 2014 - 23:27:24 CEST