Re: VKTM background process terminating the DB instance.

Hi Nenad,

I did set “_disable_hihghres_ticks” to true last week as I saw that as a workaround in one the blog but post that also instance was terminated. I will once again try in one of the non prod box and update you. Could this be an issue with VM as post moving couple of VMs to a new host I don’t see the problem again. Or do you suspect this to be an Oracle bug?

Thanks again Nenad, you have been so helpful!

Thanks,
Goti

On Thu, Aug 12, 2021 at 6:41 AM Goti <aryan.goti_at_gmail.com> wrote:

> Hi Nenad,
>
> I did set “_disable_hihghres_ticks” to true last week as I saw that as a
> workaround in one the blog but post that also instance was terminated. I
> will once again try in one of the non prod box and update you. Could this
> be an issue with VM as post moving couple of VMs to a new host I don’t see
> the problem again. Or do you suspect this to be an Oracle bug?
>
> Thanks again Nenad, you have been so helpful!
>
> Goti
>
> On Thu, 12 Aug 2021 at 6:19 AM, Noveljic Nenad <
> nenad.noveljic_at_vontobel.com> wrote:
>
>> Relink the Oracle software.
>>
>>
>>
>> *Von: *oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> im
>> namen von: Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>> *Datum *Mittwoch, 11. Aug. 2021, 8:09 PM
>> *An: *Goti <aryan.goti_at_gmail.com>, 'Oracle L' <oracle-l_at_freelists.org>
>> *Betreff: *RE: VKTM background process terminating the DB instance.
>>
>> Sltrgftime64_cgt tried to jump to an invalid program address. This
>> function is a wrapper for the OS function for getting high resolution time.
>> I‘ll check this function in detail in a couple of weeks.
>>
>> This function was called by kstmhighrestimecntk. You could try to disable
>> the high resolution ticker by setting _disable_highres_ticks to TRUE as a
>> workaround. I’ve never done this. Do it on a test system first to see if
>> there are any side-effects.
>>
>>
>>
>> *Von: *Goti <aryan.goti_at_gmail.com>
>> *Datum *Mittwoch, 11. Aug. 2021, 3:28 PM
>> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>> *Betreff: *Re: VKTM background process terminating the DB instance.
>>
>> Hi Nenad,
>>
>> Below you can find the call stack trace. Thanks again!
>>
>> https://gist.github.com/aryangoti/5734ad46a8eb513b7d239d321b7ea003
>>
>> Thanks,
>> Goti
>>
>>
>> On Wed, Aug 11, 2021 at 6:07 PM Noveljic Nenad <
>> nenad.noveljic_at_vontobel.com> wrote:
>>
>>> Could you post the call stack for ora-07445? It should be in one of the
>>> trace/incident files. This will provide the clue.
>>>
>>> With regard to the different behavior on a different host:
>>> - Are the patch levels the same? (Vktm relies on OS kernel functions,
>>> like nanosleep)
>>> - Could you compare the spfiles on both servers? (Vktm increments the
>>> time counters in SGA. There are some undocumented parameters that define
>>> the time resolution.)
>>>
>>>
>>>
>>> *Von: *Goti <aryan.goti_at_gmail.com>
>>> *Datum *Mittwoch, 11. Aug. 2021, 11:01 AM
>>> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>>> *Cc: *ORACLE-L <oracle-l_at_freelists.org>
>>> *Betreff: *Re: VKTM background process terminating the DB instance.
>>>
>>> Hi Nenad,
>>>
>>> Sorry there was typo below. Below is the correct VKTM trace file. And
>>> yes you are right there is ORA-07445 reported. But not sure why we are not
>>> getting this issue for the DB instances after we moved to a different host.
>>>
>>> kstmmainvktm: succeeded in setting elevated priority
>>> Verify: SETUID is set on ORADISM and restart the instance
>>> highres_enabled
>>>
>>> *** 2021-08-11 01:31:04.534
>>> VKTM running at (1)millisec precision with DBRM quantum (100)ms
>>> [Start] HighResTick = 35067506276
>>> kstmrmtickcnt = 0 : ksudbrmseccnt[0] = 1628645464
>>>
>>> *** 2021-08-11 04:36:31.842
>>> Exception [type: SIGSEGV, Address not mapped to object]
>>> [ADDR:0x7F03344687D0] [PC:0x7F03344687D0, {empty}] [flags: 0x0, count: 1]
>>> Incident 1800033 created, dump file:
>>> /n02/base/diag/rdbms/xxxx/xxxxx/incident/incdir_1800033/xxxx_vktm_14386_i1800033.trc
>>> ORA-07445: exception encountered: core dump [PC:0x7F03344687D0]
>>> [SIGSEGV] [ADDR:0x7F03344687D0] [PC:0x7F03344687D0] [Address not mapped to
>>> object] []
>>>
>>> ssexhd: crashing the process...
>>> Background_Core_Dump = partial
>>>
>>>
>>> Thanks,
>>> Goti
>>>
>>>
>>> On Wed, Aug 11, 2021 at 1:56 PM Goti <aryan.goti_at_gmail.com> wrote:
>>>
>>>> Hi Nenad,
>>>>
>>>> Thanks for responding.
>>>>
>>>> Below is the VKTM trace. BTW forgot to mention that, we moved a couple
>>>> of DB instances to another host post which those DB instances are not
>>>> getting terminated.
>>>>
>>>> VKTM Trace:
>>>>
>>>> kstmmainvktm: succeeded in setting elevated priority
>>>> Verify: SETUID is set on ORADISM and restart the instance
>>>> highres_enabled
>>>>
>>>> *** 2021-08-11 04:36:49.798
>>>> VKTM running at (1)millisec precision with DBRM quantum (100)ms
>>>> [Start] HighResTick = 46212770348
>>>> kstmrmtickcnt = 0 : ksudbrmseccnt[0] = 1628656609
>>>>
>>>> Thanks,
>>>> Goti
>>>>
>>>>
>>>> On Wed, Aug 11, 2021 at 1:51 PM Noveljic Nenad <
>>>> nenad.noveljic_at_vontobel.com> wrote:
>>>>
>>>>> Strace shows the segmentation fault (sigsegv) which normally
>>>>> translates to ora-7445.
>>>>>
>>>>> What are the last entries in the vktm trace?
>>>>>
>>>>>
>>>>>
>>>>> *Von: *oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org>
>>>>> im namen von: Goti <aryan.goti_at_gmail.com>
>>>>> *Datum *Mittwoch, 11. Aug. 2021, 8:54 AM
>>>>> *An: *ORACLE-L <oracle-l_at_freelists.org>
>>>>> *Betreff: *VKTM background process terminating the DB instance.
>>>>>
>>>>> Hi All,
>>>>>
>>>>> Envt Details: DB version 12.1.0.2 Apr 2019 PSU , Running on VM RHEL 7.7
>>>>>
>>>>> We have an issue wherein we have 6 DB instances running on different
>>>>> VM's sharing the same host getting terminated. The background process VKTM
>>>>> dies and subsequently PMON terminates the instance. Raised an SR with
>>>>> Oracle a couple of weeks back but so far they have not been able to find
>>>>> the root cause.Also i did follow some of the recommendations given the
>>>>> below MOS notes, but that also didn't help. Prior to VKTM getting
>>>>> terminated , the strace just shows "nanosleep({0, 1000000}, 0x7fff29abd7f0)
>>>>> = 0" in its trace file. Please find the below gist file which contains
>>>>> strace information of VKTM process just before it got terminated.
>>>>>
>>>>> https://gist.github.com/aryangoti/7fd0882352e4d6e306d212ff1ab0a432
>>>>>
>>>>> MOS Notes:
>>>>> ==============
>>>>> Customer RecommendedVKTM processes consume High CPU (Doc ID 1992652.1)
>>>>>
>>>>> Patch For Excessive VKTM And VKRM Traces In Bug 9382956 Requires
>>>>> Events (Doc ID 1381270.1)
>>>>>
>>>>> Alert log snippet:
>>>>> ========================
>>>>> Wed Aug 11 01:30:55 2021
>>>>> Instance Critical Process (pid: 4, ospid: 8566, VKTM) died unexpectedly
>>>>> PMON (ospid: 8561): terminating the instance due to error 56703
>>>>> Wed Aug 11 01:30:55 2021
>>>>> System state dump requested by (instance=1, osid=8561 (PMON)),
>>>>> summary=[abnormal instance termination].
>>>>> System State dumped to trace file
>>>>> /xxx/xxxxx/diag/rdbms/xxxx/xxxxx/trace/xxxx_diag_8576_20210811013055.trc
>>>>> Wed Aug 11 01:30:56 2021
>>>>> Dumping diagnostic data in directory=[cdmp_20210811013055], requested
>>>>> by (instance=1, osid=8561 (PMON)), summary=[abnormal instance termination].
>>>>> Wed Aug 11 01:30:57 2021
>>>>> Instance terminated by PMON, pid = 8561
>>>>> ==============================================================
>>>>>
>>>>>
>>>>> Appreciate of i can get some help here!
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Goti
>>>>>
>>>>> ____________________________________________________
>>>>>
>>>>> Please consider the environment before printing this e-mail.
>>>>>
>>>>> Bitte denken Sie an die Umwelt, bevor Sie dieses E-Mail drucken.
>>>>>
>>>>>
>>>>> Important Notice
>>>>>
>>>>> This message is intended only for the individual named. It may contain
>>>>> confidential or privileged information. If you are not the named addressee
>>>>> you should in particular not disseminate, distribute, modify or copy this
>>>>> e-mail. Please notify the sender immediately by e-mail, if you have
>>>>> received this message by mistake and delete it from your system.
>>>>> Without prejudice to any contractual agreements between you and us
>>>>> which shall prevail in any case, we take it as your authorization to
>>>>> correspond with you by e-mail if you send us messages by e-mail. However,
>>>>> we reserve the right not to execute orders and instructions transmitted by
>>>>> e-mail at any time and without further explanation.
>>>>> E-mail transmission may not be secure or error-free as information
>>>>> could be intercepted, corrupted, lost, destroyed, arrive late or
>>>>> incomplete. Also processing of incoming e-mails cannot be guaranteed. All
>>>>> liability of Vontobel Holding Ltd. and any of its affiliates (hereinafter
>>>>> collectively referred to as "Vontobel Group") for any damages resulting
>>>>> from e-mail use is excluded. You are advised that urgent and time sensitive
>>>>> messages should not be sent by e-mail and if verification is required
>>>>> please request a printed version.
>>>>> Please note that all e-mail communications to and from the Vontobel
>>>>> Group are subject to electronic storage and review by Vontobel Group.
>>>>> Unless stated to the contrary and without prejudice to any contractual
>>>>> agreements between you and Vontobel Group which shall prevail in any case,
>>>>> e-mail-communication is for informational purposes only and is not intended
>>>>> as an offer or solicitation for the purchase or sale of any financial
>>>>> instrument or as an official confirmation of any transaction.
>>>>> The legal basis for the processing of your personal data is the
>>>>> legitimate interest to develop a commercial relationship with you, as well
>>>>> as your consent to forward you commercial communications. You can exercise,
>>>>> at any time and under the terms established under current regulation, your
>>>>> rights. If you prefer not to receive any further communications, please
>>>>> contact your client relationship manager if you are a client of Vontobel
>>>>> Group or notify the sender. Please note for an exact reference to the
>>>>> affected group entity the corporate e-mail signature. For further
>>>>> information about data privacy at Vontobel Group please consult
>>>>> www.vontobel.com.
>>>>>
>>>>
>>> Important Notice
>>>
>>> This message is intended only for the individual named. It may contain
>>> confidential or privileged information. If you are not the named addressee
>>> you should in particular not disseminate, distribute, modify or copy this
>>> e-mail. Please notify the sender immediately by e-mail, if you have
>>> received this message by mistake and delete it from your system.
>>> Without prejudice to any contractual agreements between you and us which
>>> shall prevail in any case, we take it as your authorization to correspond
>>> with you by e-mail if you send us messages by e-mail. However, we reserve
>>> the right not to execute orders and instructions transmitted by e-mail at
>>> any time and without further explanation.
>>> E-mail transmission may not be secure or error-free as information could
>>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>>> processing of incoming e-mails cannot be guaranteed. All liability of
>>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>>> is excluded. You are advised that urgent and time sensitive messages should
>>> not be sent by e-mail and if verification is required please request a
>>> printed version.
>>> Please note that all e-mail communications to and from the Vontobel
>>> Group are subject to electronic storage and review by Vontobel Group.
>>> Unless stated to the contrary and without prejudice to any contractual
>>> agreements between you and Vontobel Group which shall prevail in any case,
>>> e-mail-communication is for informational purposes only and is not intended
>>> as an offer or solicitation for the purchase or sale of any financial
>>> instrument or as an official confirmation of any transaction.
>>> The legal basis for the processing of your personal data is the
>>> legitimate interest to develop a commercial relationship with you, as well
>>> as your consent to forward you commercial communications. You can exercise,
>>> at any time and under the terms established under current regulation, your
>>> rights. If you prefer not to receive any further communications, please
>>> contact your client relationship manager if you are a client of Vontobel
>>> Group or notify the sender. Please note for an exact reference to the
>>> affected group entity the corporate e-mail signature. For further
>>> information about data privacy at Vontobel Group please consult
>>> www.vontobel.com.
>>>
>>
>> Important Notice
>>
>> This message is intended only for the individual named. It may contain
>> confidential or privileged information. If you are not the named addressee
>> you should in particular not disseminate, distribute, modify or copy this
>> e-mail. Please notify the sender immediately by e-mail, if you have
>> received this message by mistake and delete it from your system.
>> Without prejudice to any contractual agreements between you and us which
>> shall prevail in any case, we take it as your authorization to correspond
>> with you by e-mail if you send us messages by e-mail. However, we reserve
>> the right not to execute orders and instructions transmitted by e-mail at
>> any time and without further explanation.
>> E-mail transmission may not be secure or error-free as information could
>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>> processing of incoming e-mails cannot be guaranteed. All liability of
>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>> is excluded. You are advised that urgent and time sensitive messages should
>> not be sent by e-mail and if verification is required please request a
>> printed version.
>> Please note that all e-mail communications to and from the Vontobel Group
>> are subject to electronic storage and review by Vontobel Group. Unless
>> stated to the contrary and without prejudice to any contractual agreements
>> between you and Vontobel Group which shall prevail in any case,
>> e-mail-communication is for informational purposes only and is not intended
>> as an offer or solicitation for the purchase or sale of any financial
>> instrument or as an official confirmation of any transaction.
>> The legal basis for the processing of your personal data is the
>> legitimate interest to develop a commercial relationship with you, as well
>> as your consent to forward you commercial communications. You can exercise,
>> at any time and under the terms established under current regulation, your
>> rights. If you prefer not to receive any further communications, please
>> contact your client relationship manager if you are a client of Vontobel
>> Group or notify the sender. Please note for an exact reference to the
>> affected group entity the corporate e-mail signature. For further
>> information about data privacy at Vontobel Group please consult
>> www.vontobel.com.
>>
>>
>> Important Notice
>>
>> This message is intended only for the individual named. It may contain
>> confidential or privileged information. If you are not the named addressee
>> you should in particular not disseminate, distribute, modify or copy this
>> e-mail. Please notify the sender immediately by e-mail, if you have
>> received this message by mistake and delete it from your system.
>> Without prejudice to any contractual agreements between you and us which
>> shall prevail in any case, we take it as your authorization to correspond
>> with you by e-mail if you send us messages by e-mail. However, we reserve
>> the right not to execute orders and instructions transmitted by e-mail at
>> any time and without further explanation.
>> E-mail transmission may not be secure or error-free as information could
>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>> processing of incoming e-mails cannot be guaranteed. All liability of
>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>> is excluded. You are advised that urgent and time sensitive messages should
>> not be sent by e-mail and if verification is required please request a
>> printed version.
>> Please note that all e-mail communications to and from the Vontobel Group
>> are subject to electronic storage and review by Vontobel Group. Unless
>> stated to the contrary and without prejudice to any contractual agreements
>> between you and Vontobel Group which shall prevail in any case,
>> e-mail-communication is for informational purposes only and is not intended
>> as an offer or solicitation for the purchase or sale of any financial
>> instrument or as an official confirmation of any transaction.
>> The legal basis for the processing of your personal data is the
>> legitimate interest to develop a commercial relationship with you, as well
>> as your consent to forward you commercial communications. You can exercise,
>> at any time and under the terms established under current regulation, your
>> rights. If you prefer not to receive any further communications, please
>> contact your client relationship manager if you are a client of Vontobel
>> Group or notify the sender. Please note for an exact reference to the
>> affected group entity the corporate e-mail signature. For further
>> information about data privacy at Vontobel Group please consult
>> www.vontobel.com.
>>
> --
> Thanks,
> Goti
>

--
http://www.freelists.org/webpage/oracle-l