Re: hung db

From: MJ Mody <emjay.mody_at_gmail.com>
Date: Mon, 11 Jan 2016 14:12:55 -0600
Message-Id: <91F821BF-8649-4681-BB38-DAD9C4FCB1C6_at_gmail.com>



Actual error CommVault engineer pulled from the logs is 'connection with the OraSbtThread failed, possibly due to a stuck DLL'

I think this could be the smoking gun

> On Jan 9, 2016, at 4:46 PM, Mladen Gogala <gogala.mladen_at_gmail.com> wrote:
>

>> On 01/08/2016 11:21 PM, MJ Mody wrote:
>> Suspected resource contention of sorts. Follow-up with backup vendor, CommVault, yielded a file lock with cv/oracle library (i.e. oci.dll) from an earlier backup.  Will need to review the Oracle shadow copy interface.
>> 
>> Still a mystery as to why nothing was written to alert log.

>
> Was there anything in the ClOraAgent.log? Did you open a case with CommVault support? Do you have any other backup system on the machine? Simpana client library is called ORASBT.DLL and usually resides in C:\Program Files\Commvault\Simpana\Base. If there was another backup software on the machine, it can put ORASBT.DLL either in %ORACLE_HOME%\lib or in C:\Windows\system32. Please, search the entire machine for the ORASBT.DLL libraries, since the library clash can be quite serious problem. The name of the library is mandated by Oracle Corp. and all backup suites (Commvault, TSM, NetBackup, Avamar) must use the same name.
> Regards
>
>> 
>> Cheers
>> MJ
>> 
>>> On Jan 8, 2016, at 9:26 PM, De DBA <dedba_at_tpg.com.au> wrote:
>>> 
>>> Usually I'd suspect an out-of-memory situation. The scenario reminds me of a case I had where the backup software (Tivoli, I think), would use Windows shadow copy to execute an RMAN backup. The shadow copy process never exited and after a (large) number of backups the Oracle process memory was exhausted and the database hang. From memory that was also W2K8R2/11.2.0.3. I believe there was a bug in the Oracle shadow copy interface.
>>> 
>>> You did check the Windows Event Logs (Application and System) I presume?
>>> 
>>> Hth,
>>> Tony
>>> 
>>>> On 09/01/16 11:03, MJ Mody wrote:
>>>> Nothing like ringing in a new year than with a hung database.
>>>> In this case, running 11.2.0.3 EE with ASO/TDE on W2K8-R2 with last CPU.
>>>> From what is known, external backups (tape writes) were taking place in backup software, both rman and data pump (the .dmp files were created early in the evening). Additionally, nightly EOD processing was also taking place.
>>>> 
>>>> Performed a server reboot to bring the database online. Point of concern is from 9pm onwards, until the database came back online (~1:30am), there is no entries in the alert log.
>>>> 
>>>> Feel free to share if anyone has come across this or have some words of wisdom for troubleshooting and remediation.
>>>> 
>>>> Thank you in advance.
>>>> 
>>>> Best
>>>> MJ--
>>>> http://www.freelists.org/webpage/oracle-l
>> --
>> http://www.freelists.org/webpage/oracle-l

>
>
> --
> Mladen Gogala
> Oracle DBA
> http://mgogala.freehostia.com
>
> --
> http://www.freelists.org/webpage/oracle-l
>
>
--
http://www.freelists.org/webpage/oracle-l
Received on Mon Jan 11 2016 - 21:12:55 CET

Original text of this message