something kills my standby after successful switchover

From: GG <grzegorzof_at_interia.pl>
Date: Fri, 8 Apr 2016 18:29:17 +0200
Message-ID: <5707DC5D.30706_at_interia.pl>

Hi,

I've got simple dataguard configuration primary on normal fs and standby Standalone GI with role separation (grid unix user for GI+ASM and oracle for oracle) .
All generally works flawlessly until second switchover where something kills my oracle instance in GI configuration . As a workaround I can just simply srvctl start database -d orcl2 . Below some details (11.2.0.4) .

DGMGRL> show configuration verbose

Configuration - orcl_drc

Protection Mode: MaxPerformance
Databases:

     orcl  - Primary database
     orcl2 - Physical standby database

   Properties:
     FastStartFailoverThreshold      = '30'
     OperationTimeout                = '30'
     FastStartFailoverLagLimit       = '30'
     CommunicationTimeout            = '180'
     ObserverReconnect               = '0'
     FastStartFailoverAutoReinstate  = 'TRUE'
     FastStartFailoverPmyShutdown    = 'TRUE'
     BystandersFollowRoleChange      = 'ALL'
     ObserverOverride                = 'FALSE'
     ExternalDestination1            = ''
     ExternalDestination2            = ''
     PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS DGMGRL> show database verbose orcl ;

Database - orcl

   Role: PRIMARY
   Intended State: TRANSPORT-ON
   Instance(s):
     orcl

Properties:

     DGConnectIdentifier             = 'orcl'
     ObserverConnectIdentifier       = ''
     LogXptMode                      = 'ASYNC'
     DelayMins                       = '0'
     Binding                         = 'optional'
     MaxFailure                      = '0'
     MaxConnections                  = '1'
     ReopenSecs                      = '300'
     NetTimeout                      = '30'
     RedoCompression                 = 'DISABLE'
     LogShipping                     = 'ON'
     PreferredApplyInstance          = ''
     ApplyInstanceTimeout            = '0'
     ApplyParallel                   = 'AUTO'
     StandbyFileManagement           = 'auto'
     ArchiveLagTarget                = '0'
     LogArchiveMaxProcesses          = '4'
     LogArchiveMinSucceedDest        = '1'
     DbFileNameConvert               = '+DATA/ORCL2, /u02/oradata/orcl'
     LogFileNameConvert              = '+DATA/ORCL2, /u02/oradata/orcl, 
+DATA/ORCL2, /u02/fra/orcl'
     FastStartFailoverTarget         = ''
     InconsistentProperties          = '(monitor)'
     InconsistentLogXptProps         = '(monitor)'
     SendQEntries                    = '(monitor)'
     LogXptStatus                    = '(monitor)'
     RecvQEntries                    = '(monitor)'
     ApplyLagThreshold               = '0'
     TransportLagThreshold           = '0'
     TransportDisconnectedThreshold  = '30'
     SidName                         = 'orcl'
     StaticConnectIdentifier         = 


'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ol67a)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl_DGMGRL.localdomain)(INSTANCE_NAME=orcl)(SERVER=DEDICATED)))'

     StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
     AlternateLocation               = ''
     LogArchiveTrace                 = '0'
     LogArchiveFormat                = '%t_%s_%r.dbf'
     TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS DGMGRL> show database verbose orcl2 ;

Database - orcl2

   Role: PHYSICAL STANDBY
   Intended State: APPLY-ON
   Transport Lag: 0 seconds (computed 0 seconds ago)

   Apply Lag:       0 seconds (computed 0 seconds ago)
   Apply Rate:      40.00 KByte/s

Real Time Query: ON
Instance(s):

     orcl
     orcl2 (apply instance)

   Properties:
     DGConnectIdentifier             = 'orcl2'
     ObserverConnectIdentifier       = ''
     LogXptMode                      = 'ASYNC'
     DelayMins                       = '0'
     Binding                         = 'optional'
     MaxFailure                      = '0'
     MaxConnections                  = '1'
     ReopenSecs                      = '300'
     NetTimeout                      = '30'
     RedoCompression                 = 'DISABLE'
     LogShipping                     = 'ON'
     PreferredApplyInstance          = ''
     ApplyInstanceTimeout            = '0'
     ApplyParallel                   = 'AUTO'
     StandbyFileManagement           = 'AUTO'
     ArchiveLagTarget                = '0'
     LogArchiveMaxProcesses          = '4'
     LogArchiveMinSucceedDest        = '1'
     DbFileNameConvert               = '/u02/oradata/orcl, +DATA/ORCL2'
     LogFileNameConvert              = '/u02/oradata/orcl, +DATA/ORCL2, 
/u02/fra/orcl, +DATA/ORCL2'
     FastStartFailoverTarget         = ''
     InconsistentProperties          = '(monitor)'
     InconsistentLogXptProps         = '(monitor)'
     SendQEntries                    = '(monitor)'
     LogXptStatus                    = '(monitor)'
     RecvQEntries                    = '(monitor)'
     ApplyLagThreshold               = '0'
     TransportLagThreshold           = '0'
     TransportDisconnectedThreshold  = '30'
     SidName(*)
     StaticConnectIdentifier(*)
     StandbyArchiveLocation(*)
     AlternateLocation(*)
     LogArchiveTrace(*)
     LogArchiveFormat(*)
     TopWaitEvents(*)
     (*) - Please check specific instance for the property value

Database Status:
SUCCESS DGMGRL> show instance verbose orcl2

Instance 'orcl2' of database 'orcl2'

   Host Name: ol67b
   PFILE:
   Properties:

     SidName                         = 'orcl2'
     StaticConnectIdentifier         = 


'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.102)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl2_DGMGRL.localdomain)(INSTANCE_NAME=orcl2)(SERVER=DEDICATED)))'

     StandbyArchiveLocation          = '+FRA'
     AlternateLocation               = ''
     LogArchiveTrace                 = '0'
     LogArchiveFormat                = '%t_%s_%r.dbf'
     TopWaitEvents                   = '(monitor)'

Instance Status:
SUCCESS ########## starting with switchover
DGMGRL> switchover to orcl2 ;
Performing switchover NOW, please wait... Operation requires a connection to instance "orcl2" on database "orcl2" Connecting to instance "orcl2"...
Connected.
New primary database "orcl2" is opening... Operation requires startup of instance "orcl" on database "orcl" Starting instance "orcl"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "orcl2"

############# all good now, comming back DGMGRL> show configuration verbose

Configuration - orcl_drc

Protection Mode: MaxPerformance
Databases:

     orcl2 - Primary database
     orcl  - Physical standby database

   Properties:
     FastStartFailoverThreshold      = '30'
     OperationTimeout                = '30'
     FastStartFailoverLagLimit       = '30'
     CommunicationTimeout            = '180'
     ObserverReconnect               = '0'
     FastStartFailoverAutoReinstate  = 'TRUE'
     FastStartFailoverPmyShutdown    = 'TRUE'
     BystandersFollowRoleChange      = 'ALL'
     ObserverOverride                = 'FALSE'
     ExternalDestination1            = ''
     ExternalDestination2            = ''
     PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS ############# switching back
DGMGRL> switchover to orcl ;
Performing switchover NOW, please wait... Operation requires a connection to instance "orcl" on database "orcl" Connecting to instance "orcl"...
Connected.
New primary database "orcl" is opening... Operation requires startup of instance "orcl2" on database "orcl2" Starting instance "orcl2"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "orcl"

########## all good except from orcl2 (new standby) alert log shows Fri Apr 08 18:16:49 2016
RSM0 started with pid=37, OS id=4144
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='orcl2'; ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='orcl2'; ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';

ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';

ALTER SYSTEM SET db_file_name_convert='/u02/oradata/orcl','+DATA/ORCL2' SCOPE=SPFILE;
ALTER SYSTEM SET
log_file_name_convert='/u02/oradata/orcl','+DATA/ORCL2','/u02/fra/orcl','+DATA/ORCL2' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='orcl' SCOPE=BOTH; ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (orcl2) Fri Apr 08 18:16:53 2016
MRP0 started with pid=38, OS id=4146
MRP0: Background Managed Standby Recovery process started (orcl2) started logmerger process
Fri Apr 08 18:16:58 2016
Managed Standby Recovery starting Real Time Apply Parallel Media Recovery started with 2 slaves Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Clearing online redo logfile 1 +DATA/orcl2/redo01.log Clearing online log 1 of thread 1 sequence number 206 Fri Apr 08 18:16:59 2016
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATA/orcl2/redo02.log Fri Apr 08 18:17:00 2016
RFS[3]: Selected log 4 for thread 1 sequence 214 dbid 1435977192 branch 908196456
Clearing online log 2 of thread 1 sequence number 212 Fri Apr 08 18:17:00 2016
Archived Log entry 167 added for thread 1 sequence 213 ID 0x559d54e8 dest 1: Fri Apr 08 18:17:01 2016
Starting ORACLE instance (normal)
Fri Apr 08 18:17:01 2016
ALTER SYSTEM SET local_listener='
(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.102)(PORT=1521))' SCOPE=MEMORY SID='orcl2';
ALTER DATABASE MOUNT /* db agent *//* {0:0:1371} */ ORA-1154 signalled during: ALTER DATABASE MOUNT /* db agent *//* {0:0:1371} */...
Fri Apr 08 18:17:02 2016
Shutting down instance (abort)
License high water mark = 9
USER (ospid: 4191): terminating the instance Instance terminated by USER, pid = 4191
Fri Apr 08 18:17:02 2016
Instance shutdown complete

############ and the instance after Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
### goes down ?!
### as a remedy
srvctl start database -d orcl2

### and all good again
DGMGRL> show configuration verbose

Configuration - orcl_drc

Protection Mode: MaxPerformance
Databases:

     orcl  - Primary database
     orcl2 - Physical standby database

   Properties:
     FastStartFailoverThreshold      = '30'
     OperationTimeout                = '30'
     FastStartFailoverLagLimit       = '30'
     CommunicationTimeout            = '180'
     ObserverReconnect               = '0'
     FastStartFailoverAutoReinstate  = 'TRUE'
     FastStartFailoverPmyShutdown    = 'TRUE'
     BystandersFollowRoleChange      = 'ALL'
     ObserverOverride                = 'FALSE'
     ExternalDestination1            = ''
     ExternalDestination2            = ''
     PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS Suspecting GI but can't prove it right now :) . Any ideas ?
Regards
GG

--
http://www.freelists.org/webpage/oracle-l

Received on Fri Apr 08 2016 - 18:29:17 CEST

This message: [ Message body ]
Next message: Norman Dunbar: "Re: something kills my standby after successful switchover"
Previous message: Balwanth Bobilli: "RE: ACCESING ORACLE DATABASE USING SQL DEVELOPER WHEN BASHION SERVERS ARE USED"
In reply to: Upendra nerilla: "RE: ACCESING ORACLE DATABASE USING SQL DEVELOPER WHEN BASHION SERVERS ARE USED"
Next in thread: Norman Dunbar: "Re: something kills my standby after successful switchover"
Reply: Norman Dunbar: "Re: something kills my standby after successful switchover"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

Original text of this message