ctssd not running in observer mode in node 2

we see below error fro cluster alert.log. Cluster is running fine but, we might see alerts. We did not see this issue before, we recently got failed over from node 1 to node 2.

uname -a
SunOS rstnvahdmdb02 5.11 11.3 sun4v sparc sun4v

*Error seen*

2017-11-15 16:07:42.317 [SRVM(43764)]CRS-10051: CVU found following errors with Clusterware setup : PRVF-5408 : NTP Time Server "192.5.41.41" is common only to the following nodes "rstnvahdmdb02" PRVF-5408 : NTP Time Server "192.5.41.209" is common only to the following nodes "rstnvahdmdb01"
PRVF-5416 : Query of NTP daemon failed on all nodes PRVG-1453 : Oracle CTSS resource is not in ONLINE state on nodes "rstnvahdmdb02"

PRCW-1015 : Wallet hdmresdb does not exist.
CLSW-9: The cluster wallet to be operated on does not exist. :[1015]
PRCW-1015 : Wallet hdmresdb does not exist.
CLSW-9: The cluster wallet to be operated on does not exist. :[1015]

  Node Name                             File exists?
  ------------------------------------  ------------------------
  rstnvahdmdb02                         yes
  rstnvahdmdb01                         yes

  Node Name                             Running?
  ------------------------------------  ------------------------
  rstnvahdmdb02                         yes
  rstnvahdmdb01                         yes

  Node Name                             Port Open?
  ------------------------------------  ------------------------
  rstnvahdmdb02                         yes
  rstnvahdmdb01                         yes

NTP common Time Server Check started...
NTP Time Server "192.5.41.209" is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started... Checking on nodes "[rstnvahdmdb02, rstnvahdmdb01]"... Check: Clock time offset from NTP Time Server

Time Server: 192.5.41.209
Time Offset Limit: 1000.0 msecs
  Node Name Time Offset Status

• ------------------------ ------------------------ rstnvahdmdb02 -9.794 passed rstnvahdmdb01 14.599 passed Time Server "192.5.41.209" has time offsets that are within permissible limits for nodes "[rstnvahdmdb02, rstnvahdmdb01]". Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Oracle Cluster Time Synchronization Services check passed

Verification of Clock Synchronization across the cluster nodes was successful.

From Node 1

root_at_rstnvahdmdb01:~# ps -ef|grep -i ctssd

    root 2577 1 0 Nov 14 ? 10:37 /u01/app/12.1.0/grid/bin/octssd.bin reboot

    root 52678 39847 0 17:27:01 pts/2 0:00 grep -i ctssd root_at_rstnvahdmdb01:~# ps -ef|grep -i ntp

    root 668 1 0 Nov 14 ? 0:09 /usr/lib/inet/ntpd --pidfile /var/run/ntp.pid --panicgate

    root 52850 39847 0 17:27:07 pts/2 0:00 grep -i ntp

From Node 2

root_at_rstnvahdmdb02:~# ps -ef|grep -i ntp

    root 606 1 0 May 16 ? 18:41 /usr/lib/inet/ntpd --pidfile /var/run/ntp.pid --panicgate --logfile /var/ntp/ntp.log

    root 21578 57373 0 17:27:15 pts/3 0:00 grep -i ntp

root_at_rstnvahdmdb02:~# ps -ef|grep ctssd

    root 33426 57373 0 17:56:52 pts/3 0:00 grep ctssd

from node 1

root_at_rstnvahdmdb01:~# /usr/sbin/ntpq -pn

     remote refid st t when poll reach delay offset jitter

from node 2

root_at_rstnvahdmdb02:~# /usr/sbin/ntpq -pn

     remote refid st t when poll reach delay offset jitter

Node 1
ora.ctssd

      1 ONLINE ONLINE rstnvahdmdb01 OBSERVER,STABLE

Node 2

ora.ctssd

      1        ONLINE  OFFLINE                               STABLE

Thanks,

Balwanth

--
http://www.freelists.org/webpage/oracle-l