Re: intermittent long "log file sync" waits

CPU util is low. we have 20 cores and only 1 active foreground process.

SQL> _at_ashtop session_id,blocking_session,event2,program2 1=1 sysdate-1/24 sysdate

   TOTALSECONDS    AAS      %This    SESSION_ID    BLOCKING_SESSION
                 EVENT2        PROGRAM2             FIRST_SEEN
 LAST_SEEN    DIST_SQLEXEC_SEEN
_______________ ______ __________ _____________ ___________________
_______________________________ _______________ ______________________
______________________ ____________________
           1964    0.5   16% |             1107                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:13
 2020-01-28 14:08:01                    1475
           1958    0.5   16% |             1351                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:13
 2020-01-28 14:08:01                    1472
            586    0.2    5% |              868                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:17
 2020-01-28 14:08:12                     368
            581    0.2    5% |              744                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:17
 2020-01-28 14:08:12                     363
            545    0.2    5% |             2085                     ON CPU
                         (sqlplus)       2020-01-28 13:08:15    2020-01-28
14:08:12                     247
            449    0.1    4% |             2205                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:34
 2020-01-28 14:08:08                       1
            440    0.1    4% |             2081                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:34
 2020-01-28 14:08:08                       1
            261    0.1    2% |              983                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:21
 2020-01-28 13:24:24                       1
            259    0.1    2% |             1103                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:08:21
 2020-01-28 13:24:17                       1
            254    0.1    2% |              131                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:45
 2020-01-28 14:08:11                       1
            251    0.1    2% |             1595                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:45
 2020-01-28 14:08:11                       1
            246    0.1    2% |             1714                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:36
 2020-01-28 14:07:59                       1
            246    0.1    2% |             1840                     Disk
file operations I/O        (TNS Vn-Vn)     2020-01-28 13:38:36
 2020-01-28 14:07:59                       1
            245    0.1    2% |             1351                     control
file sequential read    (TNS Vn-Vn)     2020-01-28 13:08:17    2020-01-28
14:08:02                     171
            242    0.1    2% |             1107                     control
file sequential read    (TNS Vn-Vn)     2020-01-28 13:08:17    2020-01-28
14:08:02                     160

15 rows selected.

On Tue, Jan 28, 2020 at 1:55 PM Noveljic Nenad <nenad.noveljic_at_vontobel.com> wrote:

> log write measures more than just io, see
> https://nenadnoveljic.com/blog/log-write-elapsed-time/
>
> Have you checked for cpu starvation as Jonathan suggested?
>
> The latest ashtop output looks differently and provides something to work
> with.
>
> Could you run « ashtop program2 » for the same time interval and all the
> sessions?
>
>
>
>
>
>
> *Von: *Chris Stephens <cstephens16_at_gmail.com>
> *Datum *Dienstag, 28. Jan. 2020, 8:44 PM
> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
> *Cc: *oracle-l <Oracle-L_at_freelists.org>
> *Betreff: *Re: intermittent long "log file sync" waits
>
> i don't have a way to reproduce on demand. is the best way to just write a
> loop hoping to catch a good/bad stack that coincides w/ log file sync? if
> so, what command to stick in loop?
>
> On Tue, Jan 28, 2020 at 1:35 PM Noveljic Nenad <
> nenad.noveljic_at_vontobel.com> wrote:
>
>> Also for any other bg processes - that’s unusual.
>>
>> Any possibility to capture pstack on lg* processes when this happens?
>>
>>
>>
>>
>> *Von: *Chris Stephens <cstephens16_at_gmail.com>
>> *Datum *Dienstag, 28. Jan. 2020, 8:20 PM
>> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>> *Cc: *oracle-l <Oracle-L_at_freelists.org>
>> *Betreff: *Re: intermittent long "log file sync" waits
>>
>> doesn't look like any ash data for lgwr:
>>
>> SQL> _at_ashtop session_id,blocking_session,event2 1=1 "TIMESTAMP'2020-01-27
>> 15:50:48'" "TIMESTAMP'2020-01-27 15:52:40'"
>> TOTALSECONDS AAS %This SESSION_ID BLOCKING_SESSION
>> EVENT2 FIRST_SEEN LAST_SEEN
>> DIST_SQLEXEC_SEEN
>> _______________ ______ __________ _____________ ___________________
>> _____________________ ______________________ ______________________
>> ____________________
>> 81 0.7 82% | 1713 1710 log
>> file sync 2020-01-27 15:50:54 2020-01-27 15:52:14
>> 1
>> 5 0 5% | 1221 ON
>> CPU 2020-01-27 15:50:59 2020-01-27 15:52:14
>> 1
>> 3 0 3% | 246 ON
>> CPU 2020-01-27 15:51:00 2020-01-27 15:52:00
>> 1
>> 3 0 3% | 1713 ON
>> CPU 2020-01-27 15:50:51 2020-01-27 15:52:27
>> 2
>> 1 0 1% | 123 ON
>> CPU 2020-01-27 15:51:35 2020-01-27 15:51:35
>> 1
>> 1 0 1% | 127 ON
>> CPU 2020-01-27 15:51:50 2020-01-27 15:51:50
>> 1
>> 1 0 1% | 252 2321 latch
>> free 2020-01-27 15:52:28 2020-01-27 15:52:28
>> 1
>> 1 0 1% | 978 ges
>> remote message 2020-01-27 15:50:48 2020-01-27 15:50:48
>> 1
>> 1 0 1% | 983 latch
>> free 2020-01-27 15:52:28 2020-01-27 15:52:28
>> 1
>> 1 0 1% | 1713
>> library cache lock 2020-01-27 15:50:48 2020-01-27 15:50:48
>> 1
>> 1 0 1% | 1831 ON
>> CPU 2020-01-27 15:50:49 2020-01-27 15:50:49
>> 1
>>
>>
>> 11 rows selected.
>>
>>
>> SQL> _at_ashtop session_id,blocking_session,event2 session_id=1710
>> "TIMESTAMP'2020-01-27 15:50:48'" "TIMESTAMP'2020-01-27 15:52:40'"
>>
>> no rows selected
>>
>> On Tue, Jan 28, 2020 at 1:09 PM Noveljic Nenad <
>> nenad.noveljic_at_vontobel.com> wrote:
>>
>>> You’re using v0.2, but v0.6 should be the latest version. 0.2 wouldn’t
>>> show idle blockers.
>>>
>>> I looked at the ashtop output once again. Strange that lgwr processes
>>> nor any other bg processes show up significantly there. The question is,
>>> what was lgwr doing while the fg processes were waiting on him.
>>>
>>>
>>>
>>>
>>> *Von: *Chris Stephens <cstephens16_at_gmail.com>
>>> *Datum *Dienstag, 28. Jan. 2020, 7:59 PM
>>> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>>> *Cc: *oracle-l <Oracle-L_at_freelists.org>
>>> *Betreff: *Re: intermittent long "log file sync" waits
>>>
>>> probably just 19c RAC
>>>
>>> SQL> _at_ash_wait_chains event2 session_id=1713 "TIMESTAMP'2020-01-27
>>> 15:50:48'" "TIMESTAMP'2020-01-27 15:52:40'"
>>>
>>> -- Display ASH Wait Chain Signatures script v0.2 BETA by Tanel Poder (
>>> http://blog.tanelpoder.com )
>>>
>>> , TO_CHAR(CASE WHEN session_state = 'WAITING' THEN p1 ELSE
>>> null END, '0XXXXXXXXXXXXXXX') p1hex
>>> *
>>> ERROR at line 14:
>>> ORA-12850: Could not allocate slaves on all specified instances: 3
>>> needed, 1 allocated
>>>
>>> SQL>
>>> Session: c2
>>>
>>> On Tue, Jan 28, 2020 at 11:30 AM Noveljic Nenad <
>>> nenad.noveljic_at_vontobel.com> wrote:
>>>
>>>> The script should work with 19. It must be something else. Could you
>>>> try with ash_wait_chains2 - 2 is the rac version? For start, I would group
>>>> only on event2 (the first parameter).
>>>>
>>>> If it doesn’t work, please post the error message.
>>>>
>>>>
>>>>
>>>> *Von: *Chris Stephens <cstephens16_at_gmail.com>
>>>> *Datum *Dienstag, 28. Jan. 2020, 5:43 PM
>>>> *An: *Noveljic Nenad <nenad.noveljic_at_vontobel.com>
>>>> *Cc: *oracle-l <Oracle-L_at_freelists.org>
>>>> *Betreff: *Re: intermittent long "log file sync" waits
>>>>
>>>> unfortunately, ash_wait_chains.sql doesn't work on 19.3 but here is
>>>> ashtop showing foreground process blocked on lgwr (1710):
>>>>
>>>> SQL> _at_ashtop inst_id,username,blocking_session,blocking_inst_id,event2
>>>> 1=1 "TIMESTAMP'2020-01-27 15:50:48'" "TIMESTAMP'2020-01-27 15:52:40'"
>>>> TOTALSECONDS AAS %This INST_ID USERNAME
>>>> BLOCKING_SESSION BLOCKING_INST_ID EVENT2
>>>> FIRST_SEEN LAST_SEEN DIST_SQLEXEC_SEEN
>>>> _______________ ______ __________ __________ ________________
>>>> ___________________ ___________________ _____________________
>>>> ______________________ ______________________ ____________________
>>>> 81 0.7 82% | 2 GEN3_MGOWER_3
>>>> 1710 2 log file sync 2020-01-27 15:50:54
>>>> 2020-01-27 15:52:14 1
>>>> 9 0.1 9% | 1 SYS
>>>> ON CPU 2020-01-27 15:50:49
>>>> 2020-01-27 15:52:14 1
>>>> 3 0 3% | 2 GEN3_MGOWER_3
>>>> ON CPU 2020-01-27 15:50:51
>>>> 2020-01-27 15:52:27 2
>>>> 2 0 2% | 2 SYS
>>>> ON CPU 2020-01-27 15:51:35
>>>> 2020-01-27 15:51:40 1
>>>> 1 0 1% | 2 GEN3_MGOWER_3
>>>> library cache lock 2020-01-27 15:50:48
>>>> 2020-01-27 15:50:48 1
>>>> 1 0 1% | 2 SYS
>>>> 2321 2 latch free 2020-01-27 15:52:28
>>>> 2020-01-27 15:52:28 1
>>>> 1 0 1% | 2 SYS
>>>> ges remote message 2020-01-27 15:50:48
>>>> 2020-01-27 15:50:48 1
>>>> 1 0 1% | 2 SYS
>>>> latch free 2020-01-27 15:52:28
>>>> 2020-01-27 15:52:28 1
>>>>
>>>>
>>>> 8 rows selected.
>>>>
>>>> SQL> _at_bg lgwr
>>>> NAME DESCRIPTION SID OPID SPID PADDR
>>>> SADDR
>>>> _______ ______________ _______ _______ ________ ___________________
>>>> ___________________
>>>> LGWR Redo etc. 1710 34 26552 00000001E8718860
>>>> 00000001D8BFF4A0
>>>>
>>>> On Tue, Jan 28, 2020 at 10:25 AM Noveljic Nenad <
>>>> nenad.noveljic_at_vontobel.com> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>>
>>>>>
>>>>> log file sync measures much more than IO.
>>>>>
>>>>>
>>>>>
>>>>> First of all, I’d run Tanel’s ash_wait_chains (
>>>>> https://github.com/tanelpoder/tpt-oracle/blob/master/ash/ash_wait_chains.sql
>>>>> ), because if often points straightly to the root cause .
>>>>>
>>>>>
>>>>>
>>>>> Here’s is a usage example, when intermittent log file sync waits where
>>>>> caused by slow control file writes:
>>>>> https://nenadnoveljic.com/blog/bad-commit-performance-control-file-writes/
>>>>> .
>>>>>
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>>
>>>>> Nenad
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> *On
>>>>> Behalf Of *Chris Stephens
>>>>> *Sent:* Dienstag, 28. Januar 2020 17:09
>>>>> *To:* oracle-l <Oracle-L_at_freelists.org>
>>>>> *Subject:* intermittent long "log file sync" waits
>>>>>
>>>>>
>>>>>
>>>>> 3-node Oracle 19.3 RAC
>>>>>
>>>>> Centos 7
>>>>>
>>>>>
>>>>>
>>>>> We have a SQLAlchemy/Python based application workload that is running
>>>>> the exact same steps with widely varying response times which appear to be
>>>>> related to varying "log file sync" wait times.
>>>>>
>>>>>
>>>>>
>>>>> Here is a profile of a "fast" run:
>>>>>
>>>>>
>>>>>
>>>>> CALL-NAME DURATION % CALLS MEAN
>>>>> MIN MAX
>>>>> ------------------------------ --------- ------ ------ --------
>>>>> -------- ---------
>>>>> SQL*Net message from client 53.197782 91.8% 10,092 0.005271
>>>>> 0.000177 28.568493
>>>>> EXEC 3.759177 6.5% 9,816 0.000383
>>>>> 0.000000 0.239592
>>>>> row cache lock 0.233153 0.4% 541 0.000431
>>>>> 0.000113 0.000941
>>>>> PARSE 0.140399 0.2% 4,867 0.000029
>>>>> 0.000000 0.006620
>>>>> DLM cross inst call completion 0.137330 0.2% 956 0.000144
>>>>> 0.000004 0.000505
>>>>> library cache lock 0.100171 0.2% 215 0.000466
>>>>> 0.000151 0.002133
>>>>> library cache pin 0.079729 0.1% 216 0.000369
>>>>> 0.000056 0.000710
>>>>> FETCH 0.058253 0.1% 1,062 0.000055
>>>>> 0.000000 0.004148
>>>>> log file sync 0.048217 0.1% 149 0.000324
>>>>> 0.000259 0.000505
>>>>> CLOSE 0.045416 0.1% 4,929 0.000009
>>>>> 0.000000 0.000073
>>>>> 20 others 0.135624 0.2% 11,854 0.000011
>>>>> 0.000000 0.001700
>>>>> ------------------------------ --------- ------ ------ --------
>>>>> -------- ---------
>>>>> TOTAL (30) 57.935251 100.0% 44,697 0.001296
>>>>> 0.000000 28.568493
>>>>>
>>>>>
>>>>>
>>>>> Here is a profile of a "slow" run:
>>>>>
>>>>>
>>>>>
>>>>> CALL-NAME DURATION % CALLS MEAN
>>>>> MIN MAX
>>>>> ------------------------------ ---------- ------ ------ --------
>>>>> -------- ----------
>>>>> SQL*Net message from client 131.186118 61.0% 10,092 0.012999
>>>>> 0.000212 106.789360
>>>>> log file sync 79.291166 36.8% 150 0.528608
>>>>> 0.000264 2.986575
>>>>> EXEC 3.728402 1.7% 9,816 0.000380
>>>>> 0.000000 0.221403
>>>>> row cache lock 0.248868 0.1% 542 0.000459
>>>>> 0.000111 0.001036
>>>>> PARSE 0.164267 0.1% 4,867 0.000034
>>>>> 0.000000 0.004652
>>>>> DLM cross inst call completion 0.146981 0.1% 957 0.000154
>>>>> 0.000005 0.001188
>>>>> library cache lock 0.104354 0.0% 218 0.000479
>>>>> 0.000160 0.000728
>>>>> library cache pin 0.082504 0.0% 202 0.000408
>>>>> 0.000157 0.000672
>>>>> FETCH 0.056687 0.0% 1,062 0.000053
>>>>> 0.000000 0.003969
>>>>> CLOSE 0.043590 0.0% 4,929 0.000009
>>>>> 0.000000 0.000180
>>>>> 20 others 0.142044 0.1% 11,866 0.000012
>>>>> 0.000000 0.001792
>>>>> ------------------------------ ---------- ------ ------ --------
>>>>> -------- ----------
>>>>> TOTAL (30) 215.194981 100.0% 44,701 0.004814
>>>>> 0.000000 106.789360
>>>>>
>>>>>
>>>>>
>>>>> looking at even histogram for that event:
>>>>>
>>>>>
>>>>>
>>>>> SQL> _at_evh "log file sync"
>>>>> EVH_EVENT EVH_WAIT_TIME_MILLI WAIT_COUNT EVH_EST_TIME
>>>>> LAST_UPDATE_TIME
>>>>> ________________ ______________________ _____________ _______________
>>>>> ______________________________________
>>>>> log file sync < 1 200051 100.026
>>>>> 27-JAN-20 11.39.57.344734 PM -06:00
>>>>> log file sync < 2 165 0.248
>>>>> 28-JAN-20 12.18.10.429089 AM -06:00
>>>>> log file sync < 4 150 0.45
>>>>> 27-JAN-20 11.18.31.158102 PM -06:00
>>>>> log file sync < 8 199 1.194
>>>>> 27-JAN-20 11.19.14.209947 PM -06:00
>>>>> log file sync < 16 253 3.036
>>>>> 28-JAN-20 08.03.17.851328 AM -06:00
>>>>> log file sync < 32 472 11.328
>>>>> 27-JAN-20 11.20.22.746033 PM -06:00
>>>>> log file sync < 64 728 34.944
>>>>> 28-JAN-20 01.13.37.364541 AM -06:00
>>>>> log file sync < 128 691 66.336
>>>>> 27-JAN-20 11.31.37.400504 PM -06:00
>>>>> log file sync < 256 414 79.488
>>>>> 28-JAN-20 12.18.10.423987 AM -06:00
>>>>> log file sync < 512 405 155.52
>>>>> 28-JAN-20 03.27.50.540383 AM -06:00
>>>>> log file sync < 1024 459 352.512
>>>>> 27-JAN-20 11.35.14.378363 PM -06:00
>>>>> log file sync < 2048 482 740.352
>>>>> 28-JAN-20 01.18.20.556248 AM -06:00
>>>>> log file sync < 4096 576 1769.472
>>>>> 27-JAN-20 11.21.05.084998 PM -06:00
>>>>> log file sync < 8192 89 546.816
>>>>> 27-JAN-20 11.57.36.436460 AM -06:00
>>>>> log file sync < 16384 60 737.28
>>>>> 25-JAN-20 07.48.31.460408 AM -06:00
>>>>> log file sync < 32768 39 958.464
>>>>> 27-JAN-20 11.59.09.869286 AM -06:00
>>>>> log file sync < 65536 27 1327.104
>>>>> 25-JAN-20 09.49.13.856563 AM -06:00
>>>>>
>>>>>
>>>>>
>>>>> The weird thing is that I don't see corresponding log I/O waits
>>>>> (awaits) in iostat output.
>>>>>
>>>>>
>>>>>
>>>>> I have a ticket open w/ oracle but does anyone have any suggestions to
>>>>> discover root cause and/or solution?
>>>>>
>>>>> ____________________________________________________
>>>>>
>>>>> Please consider the environment before printing this e-mail.
>>>>>
>>>>> Bitte denken Sie an die Umwelt, bevor Sie dieses E-Mail drucken.
>>>>>
>>>>>
>>>>> Important Notice
>>>>> This message is intended only for the individual named. It may contain
>>>>> confidential or privileged information. If you are not the named addressee
>>>>> you should in particular not disseminate, distribute, modify or copy this
>>>>> e-mail. Please notify the sender immediately by e-mail, if you have
>>>>> received this message by mistake and delete it from your system.
>>>>> Without prejudice to any contractual agreements between you and us
>>>>> which shall prevail in any case, we take it as your authorization to
>>>>> correspond with you by e-mail if you send us messages by e-mail. However,
>>>>> we reserve the right not to execute orders and instructions transmitted by
>>>>> e-mail at any time and without further explanation.
>>>>> E-mail transmission may not be secure or error-free as information
>>>>> could be intercepted, corrupted, lost, destroyed, arrive late or
>>>>> incomplete. Also processing of incoming e-mails cannot be guaranteed. All
>>>>> liability of Vontobel Holding Ltd. and any of its affiliates (hereinafter
>>>>> collectively referred to as "Vontobel Group") for any damages resulting
>>>>> from e-mail use is excluded. You are advised that urgent and time sensitive
>>>>> messages should not be sent by e-mail and if verification is required
>>>>> please request a printed version. Please note that all e-mail
>>>>> communications to and from the Vontobel Group are subject to electronic
>>>>> storage and review by Vontobel Group. Unless stated to the contrary and
>>>>> without prejudice to any contractual agreements between you and Vontobel
>>>>> Group which shall prevail in any case, e-mail-communication is for
>>>>> informational purposes only and is not intended as an offer or solicitation
>>>>> for the purchase or sale of any financial instrument or as an official
>>>>> confirmation of any transaction.
>>>>> The legal basis for the processing of your personal data is the
>>>>> legitimate interest to develop a commercial relationship with you, as well
>>>>> as your consent to forward you commercial communications. You can exercise,
>>>>> at any time and under the terms established under current regulation, your
>>>>> rights. If you prefer not to receive any further communications, please
>>>>> contact your client relationship manager if you are a client of Vontobel
>>>>> Group or notify the sender. Please note for an exact reference to the
>>>>> affected group entity the corporate e-mail signature. For further
>>>>> information about data privacy at Vontobel Group please consult
>>>>> www.vontobel.com.
>>>>>
>>>>
>>>> Important Notice
>>>> This message is intended only for the individual named. It may contain
>>>> confidential or privileged information. If you are not the named addressee
>>>> you should in particular not disseminate, distribute, modify or copy this
>>>> e-mail. Please notify the sender immediately by e-mail, if you have
>>>> received this message by mistake and delete it from your system.
>>>> Without prejudice to any contractual agreements between you and us
>>>> which shall prevail in any case, we take it as your authorization to
>>>> correspond with you by e-mail if you send us messages by e-mail. However,
>>>> we reserve the right not to execute orders and instructions transmitted by
>>>> e-mail at any time and without further explanation.
>>>> E-mail transmission may not be secure or error-free as information
>>>> could be intercepted, corrupted, lost, destroyed, arrive late or
>>>> incomplete. Also processing of incoming e-mails cannot be guaranteed. All
>>>> liability of Vontobel Holding Ltd. and any of its affiliates (hereinafter
>>>> collectively referred to as "Vontobel Group") for any damages resulting
>>>> from e-mail use is excluded. You are advised that urgent and time sensitive
>>>> messages should not be sent by e-mail and if verification is required
>>>> please request a printed version. Please note that all e-mail
>>>> communications to and from the Vontobel Group are subject to electronic
>>>> storage and review by Vontobel Group. Unless stated to the contrary and
>>>> without prejudice to any contractual agreements between you and Vontobel
>>>> Group which shall prevail in any case, e-mail-communication is for
>>>> informational purposes only and is not intended as an offer or solicitation
>>>> for the purchase or sale of any financial instrument or as an official
>>>> confirmation of any transaction.
>>>> The legal basis for the processing of your personal data is the
>>>> legitimate interest to develop a commercial relationship with you, as well
>>>> as your consent to forward you commercial communications. You can exercise,
>>>> at any time and under the terms established under current regulation, your
>>>> rights. If you prefer not to receive any further communications, please
>>>> contact your client relationship manager if you are a client of Vontobel
>>>> Group or notify the sender. Please note for an exact reference to the
>>>> affected group entity the corporate e-mail signature. For further
>>>> information about data privacy at Vontobel Group please consult
>>>> www.vontobel.com.
>>>>
>>>
>>> Important Notice
>>> This message is intended only for the individual named. It may contain
>>> confidential or privileged information. If you are not the named addressee
>>> you should in particular not disseminate, distribute, modify or copy this
>>> e-mail. Please notify the sender immediately by e-mail, if you have
>>> received this message by mistake and delete it from your system.
>>> Without prejudice to any contractual agreements between you and us which
>>> shall prevail in any case, we take it as your authorization to correspond
>>> with you by e-mail if you send us messages by e-mail. However, we reserve
>>> the right not to execute orders and instructions transmitted by e-mail at
>>> any time and without further explanation.
>>> E-mail transmission may not be secure or error-free as information could
>>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>>> processing of incoming e-mails cannot be guaranteed. All liability of
>>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>>> is excluded. You are advised that urgent and time sensitive messages should
>>> not be sent by e-mail and if verification is required please request a
>>> printed version. Please note that all e-mail communications to and from the
>>> Vontobel Group are subject to electronic storage and review by Vontobel
>>> Group. Unless stated to the contrary and without prejudice to any
>>> contractual agreements between you and Vontobel Group which shall prevail
>>> in any case, e-mail-communication is for informational purposes only and is
>>> not intended as an offer or solicitation for the purchase or sale of any
>>> financial instrument or as an official confirmation of any transaction.
>>> The legal basis for the processing of your personal data is the
>>> legitimate interest to develop a commercial relationship with you, as well
>>> as your consent to forward you commercial communications. You can exercise,
>>> at any time and under the terms established under current regulation, your
>>> rights. If you prefer not to receive any further communications, please
>>> contact your client relationship manager if you are a client of Vontobel
>>> Group or notify the sender. Please note for an exact reference to the
>>> affected group entity the corporate e-mail signature. For further
>>> information about data privacy at Vontobel Group please consult
>>> www.vontobel.com.
>>>
>>
>> Important Notice
>> This message is intended only for the individual named. It may contain
>> confidential or privileged information. If you are not the named addressee
>> you should in particular not disseminate, distribute, modify or copy this
>> e-mail. Please notify the sender immediately by e-mail, if you have
>> received this message by mistake and delete it from your system.
>> Without prejudice to any contractual agreements between you and us which
>> shall prevail in any case, we take it as your authorization to correspond
>> with you by e-mail if you send us messages by e-mail. However, we reserve
>> the right not to execute orders and instructions transmitted by e-mail at
>> any time and without further explanation.
>> E-mail transmission may not be secure or error-free as information could
>> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
>> processing of incoming e-mails cannot be guaranteed. All liability of
>> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
>> referred to as "Vontobel Group") for any damages resulting from e-mail use
>> is excluded. You are advised that urgent and time sensitive messages should
>> not be sent by e-mail and if verification is required please request a
>> printed version. Please note that all e-mail communications to and from the
>> Vontobel Group are subject to electronic storage and review by Vontobel
>> Group. Unless stated to the contrary and without prejudice to any
>> contractual agreements between you and Vontobel Group which shall prevail
>> in any case, e-mail-communication is for informational purposes only and is
>> not intended as an offer or solicitation for the purchase or sale of any
>> financial instrument or as an official confirmation of any transaction.
>> The legal basis for the processing of your personal data is the
>> legitimate interest to develop a commercial relationship with you, as well
>> as your consent to forward you commercial communications. You can exercise,
>> at any time and under the terms established under current regulation, your
>> rights. If you prefer not to receive any further communications, please
>> contact your client relationship manager if you are a client of Vontobel
>> Group or notify the sender. Please note for an exact reference to the
>> affected group entity the corporate e-mail signature. For further
>> information about data privacy at Vontobel Group please consult
>> www.vontobel.com.
>>
>
> Important Notice
> This message is intended only for the individual named. It may contain
> confidential or privileged information. If you are not the named addressee
> you should in particular not disseminate, distribute, modify or copy this
> e-mail. Please notify the sender immediately by e-mail, if you have
> received this message by mistake and delete it from your system.
> Without prejudice to any contractual agreements between you and us which
> shall prevail in any case, we take it as your authorization to correspond
> with you by e-mail if you send us messages by e-mail. However, we reserve
> the right not to execute orders and instructions transmitted by e-mail at
> any time and without further explanation.
> E-mail transmission may not be secure or error-free as information could
> be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
> processing of incoming e-mails cannot be guaranteed. All liability of
> Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively
> referred to as "Vontobel Group") for any damages resulting from e-mail use
> is excluded. You are advised that urgent and time sensitive messages should
> not be sent by e-mail and if verification is required please request a
> printed version. Please note that all e-mail communications to and from the
> Vontobel Group are subject to electronic storage and review by Vontobel
> Group. Unless stated to the contrary and without prejudice to any
> contractual agreements between you and Vontobel Group which shall prevail
> in any case, e-mail-communication is for informational purposes only and is
> not intended as an offer or solicitation for the purchase or sale of any
> financial instrument or as an official confirmation of any transaction.
> The legal basis for the processing of your personal data is the legitimate
> interest to develop a commercial relationship with you, as well as your
> consent to forward you commercial communications. You can exercise, at any
> time and under the terms established under current regulation, your rights.
> If you prefer not to receive any further communications, please contact
> your client relationship manager if you are a client of Vontobel Group or
> notify the sender. Please note for an exact reference to the affected group
> entity the corporate e-mail signature. For further information about data
> privacy at Vontobel Group please consult www.vontobel.com.
>

--
http://www.freelists.org/webpage/oracle-l