Home » Server Options » RAC & Failsafe » Scan VIP Failover But Node VIP Did Not Failover (11.2.0.3, SUSE Linux Enterprise Server 11 SP3)
Scan VIP Failover But Node VIP Did Not Failover [message #643095] Tue, 29 September 2015 02:19 Go to next message
juniordbanewbie
Messages: 170
Registered: April 2014
Senior Member
I've a two node RAC system. First I unplug the public cable to node 1. Next I plug back the public cable to node 1. Next I wait for the public cable to node 1 to blink.=>by now the two node vips should be in their respective nodes. Next I unplug the public cable to node 2

What happens is that scan vip failover, but not node vip. Instead both node vip is not started. How much time is required for node vip to failover? What preventive measures can be taken to assist the node vip failover.

here's my detail investigation based on MOS 975457.1 How to Troubleshoot Connectivity Issue with 11gR2 SCAN Name ( https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=264578683940305&id=975457.1&_afrWindowMode=0&_adf.ctrl-st ate=rsbaxm5r5_4)

here's my dns lookup.

grid@imdb01:~> nslookup imdb-scan.imdbworld.com
Server:         192.168.10.16
Address:        192.168.10.16#53

Name:   imdb-scan.imdbworld.com
Address: 192.168.56.65
Name:   imdb-scan.imdbworld.com
Address: 192.168.56.66
Name:   imdb-scan.imdbworld.com
Address: 192.168.56.67

grid@imdb01:~> nslookup imdb01-vip.imdbworld.com
Server:         192.168.10.16
Address:        192.168.10.16#53

Name:   imdb01-vip.imdbworld.com
Address: 192.168.56.63

grid@imdb01:~> nslookup imdb02-vip.imdbworld.com
Server:         192.168.10.16
Address:        192.168.10.16#53

Name:   imdb02-vip.imdbworld.com
Address: 192.168.56.64



here's my ping test. here the ping to cluser vip imdb01-vip and imdb02-vip fails

grid@imdb01:~> ping -c 3 imdb-scan.imdbworld.com
PING imdb-scan.imdbworld.com (192.168.56.66) 56(84) bytes of data.
64 bytes from imdb-scan.imdbworld.com (192.168.56.66): icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from imdb-scan.imdbworld.com (192.168.56.66): icmp_seq=2 ttl=64 time=0.025 ms
64 bytes from imdb-scan.imdbworld.com (192.168.56.66): icmp_seq=3 ttl=64 time=0.025 ms

--- imdb-scan.imdbworld.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 6287ms
rtt min/avg/max/mdev = 0.025/0.026/0.028/0.001 ms

grid@imdb01:~> ping -c 3 imdb01-vip.imdbworld.com
PING imdb01-vip.imdbworld.com (192.168.56.63) 56(84) bytes of data.
From imdb01.imdbworld.com (192.168.56.61): icmp_seq=1 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=1 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=2 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=3 Destination Host Unreachable

--- imdb01-vip.imdbworld.com ping statistics ---
3 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2016ms
, pipe 3
grid@imdb01:~> ping -c 3 imdb02-vip.imdbworld.com
PING imdb02-vip.imdbworld.com (192.168.56.64) 56(84) bytes of data.
From imdb01.imdbworld.com (192.168.56.61): icmp_seq=1 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=1 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=2 Destination Host Unreachable
From imdb01.imdbworld.com (192.168.56.61) icmp_seq=3 Destination Host Unreachable

--- imdb02-vip.imdbworld.com ping statistics ---
3 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2014ms
, pipe 3



here's my nodeapps status


grid@imdb01:~> srvctl status nodeapps
VIP imdb01-vip is enabled
VIP imdb01-vip is not running
VIP imdb02-vip is enabled
VIP imdb02-vip is not running
Network is enabled
Network is running on node: imdb01
Network is not running on node: imdb02
GSD is disabled
GSD is not running on node: imdb01
GSD is not running on node: imdb02
ONS is enabled
ONS daemon is running on node: imdb01
ONS daemon is not running on node: imdb02



here' my vip and scan status

grid@imdb01:~> srvctl status vip -n imdb01 -v
VIP imdb01-vip is enabled
VIP imdb01-vip is not running
grid@imdb01:~> srvctl status vip -n imdb02 -v
VIP imdb02-vip is enabled
VIP imdb02-vip is not running
grid@imdb01:~> srvctl status scan -v
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node imdb01
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node imdb01
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node imdb01
grid@imdb01:~> srvctl status scan_listener -v
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node imdb01
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node imdb01
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node imdb01

=>shows that scan ip failover but not node vip. Instead node vip suddenly stopped.

here's my scan_listener status

grid@imdb01:~> $GRID_HOME/bin/lsnrctl status LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 29-SEP-2015 10:44:33

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                29-SEP-2015 10:06:01
Uptime                    0 days 0 hr. 38 min. 31 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/11.2.0.3/grid/log/diag/tnslsnr/imdb01/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.67)(PORT=1521)))
Services Summary...
Service "ORCLSE" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "ORCLSEXDB" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "taf_basic_select" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "taf_preconnect" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...


here's my node listener status


grid@imdb01:~> $GRID_HOME/bin/lsnrctl status LISTENER

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 29-SEP-2015 10:49:02

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                29-SEP-2015 10:05:57
Uptime                    0 days 0 hr. 43 min. 4 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/imdb01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.61)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.63)(PORT=1521)))
Services Summary...
Service "ORCLSE" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "ORCLSEXDB" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "taf_basic_select" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
Service "taf_preconnect" has 1 instance(s).
  Instance "ORCLSE1", status READY, has 1 handler(s) for this service...
The command completed successfully


SYS@ORCLSE1>show parameter local_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD
                                                 DRESS=(PROTOCOL=TCP)(HOST=192.
                                                 168.56.63)(PORT=1521))))
SYS@ORCLSE1>show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      imdb-scan:1521
SYS@ORCLSE1>


grid@imdb01:~> sqlplus sys/********@imdb01-vip.imdbworld.com/ORCLSE as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 29 12:20:20 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-12543: TNS:destination host unreachable


Enter user-name: ^C
grid@imdb01:~> sqlplus sys/********@imdb02-vip.imdbworld.com/ORCLSE as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 29 12:20:34 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-12543: TNS:destination host unreachable


Enter user-name: ^C



connection through vip failed which is expected since node vip is stopped

grid@imdb01:~> sqlplus sys/********@192.168.56.65/ORCLSE as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 29 11:31:10 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-12516: TNS:listener could not find available handler with matching protocol
stack

-----------------------------------------------------------------------------------------------------------------------------------------------
Enter user-name: ^C

grid@imdb01:~> sqlplus sys/********@192.168.56.66/ORCLSE as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 29 11:33:10 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-12516: TNS:listener could not find available handler with matching protocol
stack


Enter user-name:
-----------------------------------------------------------------------------------------------------------------------------------------------
grid@imdb01:~> sqlplus sys/********@192.168.56.67/ORCLSE as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Sep 29 11:36:08 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

ERROR:
ORA-12516: TNS:listener could not find available handler with matching protocol
stack


Enter user-name:



connection through scan is expected to fail since according to MOS 975457.1 https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=264578683940305&id=975457.1&_afrWindowMode=0&_adf.ctrl-st ate=rsbaxm5r5_4

When client program connects to RAC database through SCAN name, SCAN listener will accept t he request and
redirect the connection to local listener. Unfortunately the local_instance will be listening on imdb01-vip which is stopped


2015-09-29 09:46:33.651
[crsd(13242)]CRS-2765:Resource 'ora.net1.network' has failed on server 'imdb01'.
2015-09-29 09:46:33.979
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(13377)]CRS-5016:Process "/u01/app/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/imdb01/agent/crsd/oraagent_grid/oraagent_grid.log"
2015-09-29 09:46:33.991
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(13377)]CRS-5016:Process "/u01/app/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/imdb01/agent/crsd/oraagent_grid/oraagent_grid.log"
2015-09-29 09:46:33.992
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(13377)]CRS-5016:Process "/u01/app/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/imdb01/agent/crsd/oraagent_grid/oraagent_grid.log"
2015-09-29 09:46:33.999
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(13377)]CRS-5016:Process "/u01/app/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/imdb01/agent/crsd/oraagent_grid/oraagent_grid.log"
2015-09-29 09:46:34.043
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(13377)]CRS-5016:Process "/u01/app/11.2.0.3/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/imdb01/agent/crsd/oraagent_grid/oraagent_grid.log"
2015-09-29 10:05:54.806
[crsd(13242)]CRS-2765:Resource 'ora.net1.network' has failed on server 'orac02'.



above alertimdb01.log was not very helpful as well.

Have done a check on resource "TYPE = ora.network.type"


grid@imdb01:~> crsctl stat res -w "TYPE = ora.network.type" -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.net1.network
               ONLINE  ONLINE       imdb01                                      
               ONLINE  OFFLINE      imdb02


So what are the factors affecting failover? What measure can be taken to speed up cluster node vip failover?


many many thanks in advance!
Re: Scan VIP Failover But Node VIP Did Not Failover [message #643098 is a reply to message #643095] Tue, 29 September 2015 04:46 Go to previous messageGo to next message
John Watson
Messages: 7150
Registered: January 2010
Location: Global Village
Senior Member
crsctl start resource ora.net1.network -n imdb2

However, I would not proceed at all with your release. What's wrong with 11.2.0.4? Or 12.1.0.2?
Re: Scan VIP Failover But Node VIP Did Not Failover [message #643353 is a reply to message #643098] Mon, 05 October 2015 20:32 Go to previous message
juniordbanewbie
Messages: 170
Registered: April 2014
Senior Member
What's wrong with 11.2.0.4? Or 12.1.0.2?=> nothing wrong with them, as far as company rules are concern, I need to seek permission before upgrading to higher version.

thanks and many thanks

[Updated on: Tue, 06 October 2015 20:23]

Report message to a moderator

Previous Topic: Cannot connect through single client access name scan after 1st node pubic cable is plugged off
Next Topic: CRS-2674: Start of 'ora.orcl.db' on 'node-db2' failed
Goto Forum:
  


Current Time: Sat Nov 25 03:22:48 CST 2017

Total time taken to generate the page: 0.07854 seconds