Re: strange error on switchover to standby

From: Mladen Gogala <gogala.mladen_at_gmail.com>
Date: Fri, 20 Jan 2017 21:07:04 -0500
Message-ID: <e3209e5d-6a32-203a-2f41-ec5c1b13cebd_at_gmail.com>



You should set the trap in sqlplus, not dgmgrl. The trace file will be written by Oracle kernel, when the error happens. Regards

On 01/20/2017 09:04 PM, Andrew Kerber wrote:
> this works in dgmgrl?
>
> Sent from my iPhone
>
> On Jan 20, 2017, at 7:55 PM, Mladen Gogala <gogala.mladen_at_gmail.com
> <mailto:gogala.mladen_at_gmail.com>> wrote:
>
>> Hi Andrew,
>> Why not trap the error:
>> ALTER SYSTEM SET EVENTS='48189 TRACE NAME ERRORSTACK FOREVER, LEVEL 16';
>> When the error occurs, it will emit a trace file which will, with any
>> luck, contain the information you need.
>> Regards
>>
>> On 01/20/2017 05:50 PM, Andrew Kerber wrote:
>>> So here is where I am. On the primary side, lets call it testp,
>>> switchover works fine using dgmgrl. (switchover to testd).
>>>
>>> I go over to the testd side, and run a switchover command in testd
>>> (switchover to testp), I get the error message:
>>>
>>> Error: ORA-48189: OS command to create directory failed
>>> Error: ORA-16625: cannot reach database "TESTD"
>>>
>>> However, if I go back over to the original primary side, and run the
>>> same switchover command there, it works fine.
>>>
>>>
>>> I am sure its some sort of privilege problem, but even with the
>>> trace level set to support, it wont tell me what command is failing.
>>> And unfortunately, I cant even tell from the error message whether
>>> its on the testp or testd side that it is trying to create a file,
>>> though it always seem to be when its communicating to testd.
>>>
>>> There is no command in the drc*.log or the alert log. Is there
>>> anywhere else I could look?
>>>
>>>
>>>
>>>
>>> On Fri, Jan 20, 2017 at 3:38 PM, Andrew Kerber
>>> <andrew.kerber_at_gmail.com <mailto:andrew.kerber_at_gmail.com>> wrote:
>>>
>>> I tried that one already. It simply wont show the command that
>>> is failing:
>>> Properties:
>>> FastStartFailoverThreshold = '30'
>>> OperationTimeout = '30'
>>> TraceLevel = 'SUPPORT'
>>> FastStartFailoverLagLimit = '30'
>>> CommunicationTimeout = '180'
>>> ObserverReconnect = '0'
>>> FastStartFailoverAutoReinstate = 'TRUE'
>>> FastStartFailoverPmyShutdown = 'TRUE'
>>> BystandersFollowRoleChange = 'ALL'
>>> ObserverOverride = 'FALSE'
>>> ExternalDestination1 = ''
>>> ExternalDestination2 = ''
>>> PrimaryLostWriteAction = 'CONTINUE'
>>>
>>>
>>> On Fri, Jan 20, 2017 at 2:47 PM, Bernhard de Cock Buning -
>>> GRID-IT <bdcbuning_at_grid-it.nl <mailto:bdcbuning_at_grid-it.nl>> wrote:
>>>
>>> Hi Andrew,
>>>
>>> Would expect to see more details in the drc logfile.
>>>
>>> If you find out it is related to one server(s) cluster, It
>>> starts to look like it is related to the log/trace directories.
>>> Have a look at the ADR on that host and the permissions,
>>> maybe it is related to the diag_dest parameter on the
>>> standby location where it defines an incorrect location. But
>>> I would in that case expect you would already have issues
>>> for some time and not only when you perform a switchover.
>>>
>>> But if you need more tracing info you can execute in dgmgrll
>>> (assume you use 11.2.0.3 or higher)
>>> Admin is the default tracelevel, change it to support.
>>>
>>> edit configuration set property tracelevel=support
>>>
>>>
>>> Hope this helps
>>>
>>> Greetings,
>>> Bernhard
>>>
>>>> Op 20 jan. 2017, om 21:32 heeft Andrew Kerber
>>>> <andrew.kerber_at_gmail.com> het volgende geschreven:
>>>>
>>>> interestingly, if I moved over to the original primary
>>>> servers to run the command, the switchover succeeds. It
>>>> only fails with the error on one cluster, which means I at
>>>> least know for sure what servers the error is coming from.
>>>> Now if I could just figure out what command.
>>>>
>>>> On Fri, Jan 20, 2017 at 2:10 PM, Andrew Kerber
>>>> <andrew.kerber_at_gmail.com> wrote:
>>>>
>>>> I was trailing that file, and it does not contain the
>>>> command that failed. Here is what is in the broker log
>>>> file:
>>>>
>>>> Failed to connect to remote database TEST. Error is
>>>> ORA-48189
>>>> Failed to send message to site TEST. Error code is
>>>> ORA-48189.
>>>> database TEST unable to contact primary database for
>>>> version check; status ORA-48189
>>>> completing bootstrap of this database
>>>> Notifying Oracle Clusterware to buildup
>>>>
>>>>
>>>> On Fri, Jan 20, 2017 at 2:05 PM, Bernhard de Cock
>>>> Buning - GRID-IT <bdcbuning_at_grid-it.nl> wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> Check out the broker logfile, when correct this
>>>> will explain where it goes wrong.
>>>> You can find this logfile drc*.log in the same
>>>> location as the alert.log of the database.
>>>> Be sure you check both location (the primary and
>>>> standby)
>>>>
>>>> You can also enable additional tracing in the
>>>> broker but as mentioned I believe the drc log will
>>>> contain the error.
>>>>
>>>> Greetings,
>>>> Bernhard
>>>>
>>>>> Op 20 jan. 2017, om 20:56 heeft Andrew Kerber
>>>>> <andrew.kerber_at_gmail.com> het volgende geschreven:
>>>>>
>>>>> I am getting a strange error when I am trying to
>>>>> switchover to my standby database. The first
>>>>> switchover worked fine, so I am switching over to
>>>>> my original primary, and I am getting this message:
>>>>>
>>>>> DGMGRL> switchover to 'TEST';
>>>>> Performing switchover NOW, please wait...
>>>>> Operation requires a connection to instance
>>>>> "TEST3" on database "TEST"
>>>>> Connecting to instance "TEST3"...
>>>>> Connected as SYSDBA.
>>>>> Error: ORA-48189: OS command to create directory
>>>>> failed
>>>>> Error: ORA-16625: cannot reach database "TEST"
>>>>>
>>>>>
>>>>> It looks like it is trying to create a directory,
>>>>> but I cannot figure out what directory it is
>>>>> trying to creat. I expect its a simple ownership
>>>>> problem, but I cannot figure out what directory I
>>>>> need to work on. I tried setting the trace level
>>>>> to support in dgmgrl, but it didnt show the
>>>>> information I needed. Any ideas?
>>>>>
>>>>>
>>>>> --
>>>>> Andrew W. Kerber
>>>>>
>>>>> 'If at first you dont succeed, dont take up
>>>>> skydiving.'
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrew W. Kerber
>>>>
>>>> 'If at first you dont succeed, dont take up skydiving.'
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrew W. Kerber
>>>>
>>>> 'If at first you dont succeed, dont take up skydiving.'
>>>
>>>
>>>
>>>
>>> --
>>> Andrew W. Kerber
>>>
>>> 'If at first you dont succeed, dont take up skydiving.'
>>>
>>>
>>>
>>>
>>> --
>>> Andrew W. Kerber
>>>
>>> 'If at first you dont succeed, dont take up skydiving.'
>>
>>
>> --
>> Mladen Gogala
>> Oracle DBA
>> Tel: (347) 321-1217

-- 
Mladen Gogala
Oracle DBA
Tel: (347) 321-1217



--
http://www.freelists.org/webpage/oracle-l
Received on Sat Jan 21 2017 - 03:07:04 CET

Original text of this message