Received: (qmail 22486 invoked from network); 7 Jul 2011 10:49:19 -0500 Received: from freelists-180.iquest.net (HELO turing.freelists.org) (206.53.239.180) by static-ip-85-25-126-90.inaddr.intergenia.de with SMTP; 7 Jul 2011 10:47:54 -0500 Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 8C8DDE32DE3; Thu, 7 Jul 2011 11:46:50 -0400 (EDT) X-Virus-Scanned: Debian amavisd-new at localhost.localdomain Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SdeTKZGZv1nu; Thu, 7 Jul 2011 11:46:50 -0400 (EDT) Received: from turing.freelists.org (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id EC7E8E32CEF; Thu, 7 Jul 2011 11:46:06 -0400 (EDT) Received: with ECARTIS (v1.0.0; list oracle-l); Thu, 07 Jul 2011 11:45:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 5981DE32D57 for ; Thu, 7 Jul 2011 11:45:24 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TVMdL4dhFJAL for ; Thu, 7 Jul 2011 11:45:24 -0400 (EDT) Received: from mail-wy0-f179.google.com (mail-wy0-f179.google.com [74.125.82.179]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id F384BE32CA0 for ; Thu, 7 Jul 2011 11:45:19 -0400 (EDT) Received: by wyh21 with SMTP id 21so892092wyh.10 for ; Thu, 07 Jul 2011 08:45:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=tXRaoh4bnGgvtnwQeqOeWOf+rtg1o9/44ci9xoWhH1g=; b=cbMnf9mS0ZXFz9ZJUVrex+ljhj9C8fprReSqPkP+gv94Z+YldctdIpX0fNzBVFmWz5 LJq+9xfSuqGsI9uUXGUryKiXJ9tNoCBEv+AdaR5uUSHkdTGD423Q7CIgnka+TtF90j2A NThGCGhqqz4p3KyJIdapSHHcXAl3ypf84NQyw= MIME-Version: 1.0 Received: by 10.216.132.133 with SMTP id o5mr6649452wei.94.1310053515606; Thu, 07 Jul 2011 08:45:15 -0700 (PDT) Received: by 10.216.132.224 with HTTP; Thu, 7 Jul 2011 08:45:15 -0700 (PDT) In-Reply-To: References: <1310038670.92003.YahooMailClassic@web162012.mail.bf1.yahoo.com> Date: Thu, 7 Jul 2011 23:45:15 +0800 Message-ID: Subject: Fwd: anyone ran into ORA-00752 on dataguard node with ASM/11.1.0.7 Linux? From: "Zhu,Chao" To: ORACLE-L Content-Type: multipart/alternative; boundary=0016e6dd98fc6f3ad404a77c9ae0 X-archive-position: 37285 X-ecartis-version: Ecartis v1.0.0 Sender: oracle-l-bounce@freelists.org Errors-to: oracle-l-bounce@freelists.org X-original-sender: zhuchao@gmail.com Precedence: normal Reply-To: zhuchao@gmail.com List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: oracle-l X-List-ID: oracle-l List-subscribe: List-owner: List-post: List-archive: X-list: oracle-l --0016e6dd98fc6f3ad404a77c9ae0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable some offline-discussion with the DL member, keep others updated; and remove= d some might confidential info; in our case with oracle , SR support guy thought might be related to the AS= M rebalance bug; and my point is: They should not be related to ASm rebalance; We are on local disk(DL380G7 with 16 disks/8 raid1 diskset); So there should no such rebalance work goin= g on; And the corruption happens often: somtime once/twice a day, sometime once a week; We have been fighting with it for 2 months and i decided to upgrade it to 11.2.0.2 for a try; We are on a some-how weird combination: 11.2.0.2 ASM + 11.1.0.7.4 for RDBMS= ; (Assuming 11.2.0.2 ASM will be more stable/bug-free than the 11.1.ASM, now wondering whether this is a good idea); Also regarding the primary itself is corrupted: your analysis is very detailed and convincing, but missing one point: if it is indeed missing write in primary, then the primary database is already corrupted(some datafile); We already did bounce once or twice, nothing reported; Also, when we saw such corruption, we refrsh that corrupted datafile from primary to standby: and everything works fine: dbv/fts/analyze cascade(to b= e done, i forgot about that); If primary is indeed corrupted, standby wont recover either after refresh standby from primary(can someone confirm?), and the only solution would be, as you mentioned, to failover to a clean/consistent copy; So, my conclusion is oracle is doing some funky work(or the primary databas= e host); What we have done: 1. suspect it is ASM bug with redo, so moved redo log in primary out of ASM disks; --still fail; 2. also move standby redo out of ASM disk, fails; 3. also moved our a whole standby out of ASM to EXT3, also fails; 4. set lost_write_protect to false, still cant recovery through; Also, primary dbv/fts has been clean before/after the standby issue, and even through bounce(if it is indeed lost write, a bounce will make the problem obvious in primary as well)? Thanks for your detailed analysis! > Ask on mos if all the three standby are failing at the same time > By the way is your primary facing the same corruption issue > > On 6 Jul 2011 20:35, "Zhu,Chao" < > zhuchao@gmail.com > wrote: > > hi, guys > We ran into weird corruption in standby nodes in our first production > oracle/linux deployment, which is pretty frustrate; > Would appreciate if anyone else has ran into similar issues and share > with your solution; Working with oracle sucks, no progress in the past 2 > months and now pointing fingers to HW(hp DL380G7); Which obviously is not > the case as all 3 standby always consistently fails at the same > block#/log#, and we tried move redo to different > location/ASM/filesystem(datafile in primary still at ASM); > > > Details: > Application: Oracle RUEI on Linux(redhat 5.5) > RDBMS: 11.1.0.7.4 > ASM: 11.2.0.2 > primary is fine; > dataguard node: > > Tue Jul 05 18:10:00 2011 > > Slave exiting with ORA-752 exception > > Errors in file > /oracle/RUEICEM/home/diag/rdbms/rueicem/RUEICEM/trace/RUEICEM_pr06_18594.= trc: > > *ORA-00752: recovery detected a lost write of a data block* > > *ORA-10567: Redo is inconsistent with data block (file# 84, block# 266478= ) > * > > ORA-10564: tablespace USERS > > ORA-01110: data file 84: '+DATA/rueicem/datafile/users.404.752266499' > > ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# > 5082441 > > > or: > > Errors in file > /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM/trace/RUEICEM_pr= 0e_14272.trc > (incident=3D101514): > > *ORA-00600: internal error code, arguments: [3020], [84], [266478], > [352588014], [], [], [], [], [], [], [], []* > > *ORA-10567: Redo is inconsistent with data block (file# 84, block# 266478= ) > * > > ORA-10564: tablespace USERS > > ORA-01110: data file 84: '+DATA/rueicem/datafile/rueicem_userruei01_35.db= f' > > ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# > 5082441 > > Incident details in: > /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM/incident/incdir_= 101514/RUEICEM_pr0e_14272_i101514.trc > some of our investigation done: > > 1. All 3 standby database failed at the same log#/block#; Includin= g > the standby with datafile/redo all on filesystem; > > 2. The standby database cant be recovered even set > db_lost_write_protect =3D none; > > 3. The standby can still be open after it failed at that > block#/log#; DBV is still clean for that datafile by that time; but no w= ay > to recover through; > > 4. The standby can be recovered through through =93recover standby > database allow 1 corruption=94; But dbv did report corrupted block, and f= ts > did report corrupted block and fts fails; > Thanks for your sharing, if we dont have any progress, we will have to > upgrade RDBMS to 11.2 as well (as 3rd party application, dont really want= to > do that.) > > > -- > Regards > Zhu Chao > > > > > > -- > Regards > Zhu Chao > > > > > > -- > Regards > Zhu Chao > > > --=20 Regards Zhu Chao --=20 Regards Zhu Chao --0016e6dd98fc6f3ad404a77c9ae0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
some offline-discussion with the DL member, keep= others updated; and removed some might confidential info;

in our ca= se with oracle , SR support guy thought might be related to the ASM rebalan= ce bug; and my point is:
They should not be related to ASm rebalance; We are on local disk(DL380G7 w= ith 16 disks/8 raid1 diskset); So there should no such rebalance work going= on;
And the corruption happens often: somtime once/twice a day, sometim= e once a week;
We have been fighting with it for 2 months and i decided to upgrade it to 1= 1.2.0.2 for a try;
We are on a some-how weird combination: 11.2.0.2 ASM= + 11.1.0.7.4 for RDBMS; (Assuming 11.2.0.2 ASM will be more stable/bug-fre= e than the 11.1.ASM, now wondering whether this is a good idea);


Also regarding the primary itself is corrupted:
your analysis i= s very detailed and convincing, but missing one point:
if it is indeed m= issing write in primary,=C2=A0 then the primary database is already corrupt= ed(some datafile);
We already did bounce once or twice, nothing reported;

Also, when we saw such corruption, we refrsh that corrupted datafile fr= om primary to standby: and everything works fine: dbv/fts/analyze cascade(t= o be done, i forgot about that);
If primary is indeed corrupted, standb= y wont recover either after refresh standby from primary(can someone confir= m?), and the only solution would be, as you mentioned, to failover to a cle= an/consistent copy;

So, my conclusion is oracle is doing some funky work(or the primary dat= abase host);
What we have done:
1. suspect it is ASM bug with redo, s= o moved redo log in primary out of ASM disks; --still fail;
2. also move= standby redo out of ASM disk, fails;
3. also moved our a whole standby out of ASM to EXT3, also fails;
4. set= lost_write_protect to false, still cant recovery through;

Also, pri= mary dbv/fts has been clean before/after the standby issue, and even throug= h bounce(if it is indeed lost write, a bounce will make the problem obvious= in primary as well)?


Thanks for your detailed analysis!



Ask on mos if all the three standby are failing at the same time By the way is your primary facing the same corruption issue

On 6 Jul 2011 20:35, "Zhu,Chao" = <zhuchao@gmail.com> wrote:

hi, guys
=C2=A0=C2=A0 We ran into weird corruption in standby nodes in our firs= t production oracle/linux deployment, which is pretty frustrate;
=C2=A0=C2=A0 Would appreciate if anyone else has ran into similar issu= es and share with your solution; Working with oracle sucks, no progress in = the past 2 months and now pointing fingers to HW(hp DL380G7); Which obvious= ly is not the case as all 3 standby always consistently fails at the same b= lock#/log#,=C2=A0 and we tried move redo to different location/ASM/filesyst= em(datafile in primary still at ASM);
=C2=A0
=C2=A0
=C2=A0=C2=A0 Details:
Application: Oracle=C2=A0RUEI=C2=A0 on Linux(redhat 5.5)
RDBMS: 11.1.0.7.4
ASM: 11.2.0.2
primary is fine;
dataguard node:

Tu= e Jul 05 18:10:00 2011

Sl= ave exiting with ORA-752 exception

Er= rors in file /oracle/RUEICEM/home/diag/rdbms/rueicem/RUEICEM/trace/RUEICEM_= pr06_18594.trc:

ORA-00752: r= ecovery detected a lost write of a data block

ORA-10567: Redo is inconsistent with data block (file# 84, block# 2= 66478)

OR= A-10564: tablespace USERS

OR= A-01110: data file 84: '+DATA/rueicem/datafile/users.404.752266499'=

OR= A-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# = 5082441

=C2=A0

or:

Er= rors in file /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM/trace= /RUEICEM_pr0e_14272.trc=C2=A0 (incident=3D101514):

ORA-00600: internal error code, arguments: [3020], [84], [266478], = [352588014], [], [], [], [], [], [], [], []

ORA-10567: R= edo is inconsistent with data block (file# 84, block# 266478)=

OR= A-10564: tablespace USERS

OR= A-01110: data file 84: '+DATA/rueicem/datafile/rueicem_userruei01_35.db= f'

OR= A-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# = 5082441

In= cident details in: /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM= /incident/incdir_101514/RUEICEM_pr0e_14272_i101514.trc

some of our investigation done:

= = 1.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 All= 3 standby database failed at the same log#/block#;=C2=A0 Including the sta= ndby with datafile/redo all on filesystem;

= = 2.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The= standby database cant be recovered even set db_lost_write_protect =3D none= ;

= = 3.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The= standby can still be open after it failed at that block#/log#; =C2=A0DBV i= s still clean for that datafile by that time; but no way to recover through= ;

= = 4.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The= standby can be recovered through through =E2=80=9Crecover standby database= allow 1 corruption=E2=80=9D; But dbv did report corrupted block, and fts d= id report corrupted block and fts fails;

Thanks for your sharing, if we dont have any progress, we will have to= upgrade RDBMS to 11.2 as well (as 3rd party application, dont really want = to do that.)


--
Regards
Zhu Chao





--
Regards
Zhu Chao
=




--
Regards
Zhu Chao





--
Regards
Zhu Chao





--
Regards
Zhu Chao

--0016e6dd98fc6f3ad404a77c9ae0-- -- http://www.freelists.org/webpage/oracle-l