Fractured block ... - no ORA- in alert log

From: Herring Dave - dherri <Dave.Herring_at_acxiom.com>
Date: Wed, 19 Sep 2012 20:07:28 +0000
Message-ID: <BD475CE0B3EE894DA0CAB36CE2F7DEB401043C08D4_at_LITIGMBCRP02.Corp.Acxiom.net>



Specs: 4-node RAC, 10.2.0.2, RHEL4.6, Data Guard (physical)

We ran into a situation earlier this week where some blocks were found to be corrupted on the primary db and I'm rather disappointed at how it was "mentioned" in the alert log. We run backups on our physical standby and no corruption was found there, but on the primary we got messages similar to the following:

Corrupt block relative dba: 0x2c83c24b (file 178, block 246347) Fractured block found during buffer read Data in bad block:
 type: 6 format: 2 rdba: 0x2c83c24b
 last change scn: 0x001f.a2553430 seq: 0x1 flg: 0x06  spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x2ec10602
 check value in block header: 0x69f9
 computed block checksum: 0x1af2
Reread of rdba: 0x2c83c24b (file 178, block 246347) found same corrupted data Mon Sep 17 00:05:57 2012
Corrupt Block Found

         TSN = 180, TSNAME = <tblspace>
         RFN = 178, BLK = 246347, RDBA = 746832459
         OBJN = 56925, OBJD = 56925, OBJECT = <table>, SUBOBJECT = <partition>
         SEGMENT OWNER = <owner>, SEGMENT TYPE = Table Partition

What disturbs me is that there's no mention of an ORA- error in all that, which means unless the client starting calling and wanting to know why their job was failing, we would never have known about the corruption. To me block/datafile corruption is rather serious and we should know IMMEDIATELY if any is found. Yes, there's a message about it in the alert log but nothing automated checking would notice (such as with EM).

Does anyone else find that rather strange? Is anyone checking for corruption on a regular basis in a DG configuration against the primary, where the backup is being done on the standby? We do have a weekly job on the primary that runs an RMAN "backup validate ..." but in our case that's run on Sunday and the corruption happened afterwards that day. We can obviously run the "backup validate" job daily and also have a special script to grep on "^Corrupt block ..." in the alert log hourly, but the latter seems like a hack. I guess I just prefer to know FIRST if any issues occur with the databases I support.

DAVID HERRING
DBA
Acxiom Corporation

EML   dave.herring_at_acxiom.com
TEL    630.944.4762
MBL   630.430.5988 

1501 Opus Pl, Downers Grove, IL 60515, USA WWW.ACXIOM.COM

The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged.

If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.

If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system.

Thank You.


--
http://www.freelists.org/webpage/oracle-l
Received on Wed Sep 19 2012 - 15:07:28 CDT

Original text of this message