Oracle FAQ Your Portal to the Oracle Knowledge Grid
HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US
 

Home -> Community -> Usenet -> c.d.o.server -> Sporadic Oracle data corruptions errors

Sporadic Oracle data corruptions errors

From: <josil20_at_gmail.com>
Date: 11 Mar 2005 05:31:24 -0800
Message-ID: <1110547883.988640.224920@o13g2000cwo.googlegroups.com>


Hi,
We have a strange problem of sporadic Oracle data block corruption errors in our DSS database. Let me start with our environment (which is complicated, but certified by both Oracle and IBM).

Our environment:
· AIX 5.3 (with and without ML01) with cio enabled, JFS2 filesystem.
· Hardware: IBM P5 P590 and P570 (IBM's latest POWER 5 technology).
· Oracle 9.2.0.5, with data compression.
· SVC (SAN Volume Controller) disk (For people who are not familiar
with SVC, SVC can be connected to multiple, heterogeneous disk arrays. SVC provides any size, virtual disk to the host. Host needs to be compatible only with the SVC and not all different disk arrays. It also provides other bells and whistles like flash copy across arrays, resize disks and mostly importantly a large cache).
· A DS4100 (FastT) disk array is connected to the SVC. We are planning
to connect more disk array in future.

Actual problem:
Oracle reports sporadic data block corruption error. This is error is not permanent. For example following is a sequence of events 1. 8:05 AM (normal to low IO load on the server): Oracle reads block 5467 from datafile cposdata1998.dbf. Oracle is able to read and process the block data.
2. 8:10 AM: (heavy IO load on the server): Oracle reads block 5467 from datafile cposdata1998.dbf. Oracle is able to read the block, but reports that the block is corrupted.
3. 8:15 AM (normal to low IO load on the server): Oracle reads block 5467 from datafile cposdata1998.dbf. Oracle is able to read the block, but reports that the block is corrupted. 4. 9:45 AM (normal to low IO load on the server): Oracle reads block 5467 from datafile cposdata1998.dbf. Oracle is able to read and process the block.
5. No errors are reported, at any time, by the OS, SVC and disk array. Only Oracle reports errors.

Following are the patters that we noticed.
· Mostly, errors start when there is heavy IO.
· Sometimes it takes up to a day for the errors to disappear and
sometimes it disappears in a matter of seconds.
· If we copy the datafiles to non-SVC disk, we are not getting this
error.
· We are able recreate this error in compressed and non-compressed
Oracle tablespaces.

Is anybody using a similar environment? If so, please let me know if you getting similar errors or not. We have severity 1 tickets open with Oracle and IBM. Oracle says that this is a hardware error and IBM says that they don't see any error messages.
Thanks,
Joseph

Message from Alert log:



***
Corrupt block relative dba: 0x1000b8f9 (file 64, block 47353) Bad check value found during buffer read Data in bad block -
 type: 6 format: 2 rdba: 0x1000b8f9
 last change scn: 0x0004.669631ba seq: 0x2 flg: 0x04  consistency value in tail: 0x31ba0602
 check value in block header: 0x3242, computed block checksum: 0x2000  spare1: 0x0, spare2: 0x0, spare3: 0x0
***
Reread of rdba: 0x1000b8f9 (file 64, block 47353) found same corrupted data
Tue Mar 8 19:58:29 2005
Errors in file
/BDW/bobdwpr0/oraclebin/app/oracle/admin/udump/bobdwpr0_ora_6602760.trc: ORA-01578: ORACLE data block corrupted (file # 64, block # 47353) ORA-01110: data file 64:
'/BDW/bobdwpr0/oradataro03/data/asfividata200209_01.dbf' Received on Fri Mar 11 2005 - 07:31:24 CST

Original text of this message

HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US