Re: rman fail on backup control file

From: Jeff Chirco <backseatdba_at_gmail.com>
Date: Thu, 23 Aug 2018 07:13:28 -0700
Message-ID: <CAKsxbLobBp9f4D8w5u9w3c2-S9MkdDh54ruKW1MMdJ8SK040sQ_at_mail.gmail.com>



This is what is in the alert log

Starting control autobackup
Hex dump of (file 0, block 10014) in trace file c:\oraclexe\app\oracle\admin\xe\udump\xe_ora_4812.trc Corrupt block relative dba: 0x0000271e (file 0, block 10014) Fractured block found during backing up controlfile Data in bad block:
 type: 0 format: 2 rdba: 0x0000271e
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05  spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000000
 check value in block header: 0xe01e
 computed block checksum: 0x1
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Deleted Oracle managed file
D:\BACKUP\ORACLE\FLASH_RECOVERY_AREA\XE\AUTOBACKUP\2018_08_23\O1_MF_S_984899284_FQXHY6B8_.BKP
Autobackup failed with following error
ORA-19624: operation failed, retry possible ORA-19599: block number 10014 is corrupt in control file C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA Thu Aug 23 07:08:57 2018
Hex dump of (file 0, block 10014) in trace file c:\oraclexe\app\oracle\admin\xe\udump\xe_ora_4812.trc Corrupt block relative dba: 0x0000271e (file 0, block 10014) Fractured block found during backing up controlfile Data in bad block:
 type: 0 format: 2 rdba: 0x0000271e
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05  spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000000
 check value in block header: 0xe01e
 computed block checksum: 0x1
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data
Reread of blocknum=10014,
file=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA. found same corrupt data

On Thu, Aug 23, 2018 at 7:10 AM Jeff Chirco <backseatdba_at_gmail.com> wrote:

> Connected to:
> Oracle Database 10g Express Edition Release 10.2.0.1.0 - Production
>
> SQL> show parameter control
>
> NAME TYPE VALUE
> ------------------------------------ -----------
> ------------------------------
> control_file_record_keep_time integer 7
> control_files string
> D:\DATA\ORACLE\ORADATA\XE\CONTROL01.DBF,
> D:\DATA\ORACLE\ORADATA\XE\CONTROL02.DBF, D:\DATA\
> ORACLE\ORADATA\XE\CONTROL03.DBF
> SQL> exit
>
> Disconnected from Oracle Database 10g Express Edition Release 10.2.0.1.0 -
> Produ
> ction
>
> C:\Documents and Settings\store291\Desktop>rman target /
>
> Recovery Manager: Release 10.2.0.1.0 - Production on Thu Aug 23 07:07:14
> 2018
>
> Copyright (c) 1982, 2005, Oracle. All rights reserved.
>
> connected to target database: XE (DBID=2593989116)
>
> RMAN> backup datafile 4 format
> 'C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DAT
> ABASE\df4_%U.bk';
>
> Starting backup at 23-AUG-18
> using target database control file instead of recovery catalog
> allocated channel: ORA_DISK_1
> channel ORA_DISK_1: sid=193 devtype=DISK
> channel ORA_DISK_1: starting full datafile backupset
> channel ORA_DISK_1: specifying datafile(s) in backupset
> input datafile fno=00004 name=D:\DATA\ORACLE\ORADATA\XE\USERS.DBF
> channel ORA_DISK_1: starting piece 1 at 23-AUG-18
> channel ORA_DISK_1: finished piece 1 at 23-AUG-18
> piece
> handle=C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\DF4_PCTB8NM4_
> 1_1.BK tag=TAG20180823T070747 comment=NONE
> channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
> Finished backup at 23-AUG-18
>
> Starting Control File and SPFILE Autobackup at 23-AUG-18
> RMAN-00571: ===========================================================
> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
> RMAN-00571: ===========================================================
> RMAN-03009: failure of Control File and SPFILE Autobackup command on
> ORA_DISK_1
> channel at 08/23/2018 07:08:11
> ORA-19599: block number 10014 is corrupt in control file
> C:\ORACLEXE\APP\ORACLE\
> PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA
>
> RMAN>
>
> RMAN> backup current controlfile format
> 'C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\S
> ERVER\DATABASE\ctl_%U.bk';
>
> Starting backup at 23-AUG-18
> using channel ORA_DISK_1
> channel ORA_DISK_1: starting full datafile backupset
> channel ORA_DISK_1: specifying datafile(s) in backupset
> including current control file in backupset
> channel ORA_DISK_1: starting piece 1 at 23-AUG-18
> RMAN-00571: ===========================================================
> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
> RMAN-00571: ===========================================================
> RMAN-03009: failure of backup command on ORA_DISK_1 channel at 08/23/2018
> 07:08:
> 58
> ORA-19599: block number 10014 is corrupt in control file
> C:\ORACLEXE\APP\ORACLE\
> PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA
>
> RMAN>
>
>
>
> On Wed, Aug 22, 2018 at 5:02 PM LK <lkaing_at_gmail.com> wrote:
>
>> Hi Jeff,
>>
>> Please show me this:
>>
>> 1) In SQL*Plus as DBA:
>>
>> SQL> show parameter control
>>
>> 2) run some more RMAN backup tests to the 2 locations, choosing a small
>> sized datafile, eg. datafile 4,
>>
>> RMAN> backup datafile 4 format
>> 'C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\df4_%U.bk';
>> RMAN> backup current controlfile format
>> 'C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\ctl_%U.bk';
>>
>> Cheers,
>>
>> Leng.
>>
>>
>>
>>
>>
>> On Wed, Aug 22, 2018 at 1:57 AM, Gus Spier <gus.spier_at_gmail.com> wrote:
>>
>>> I apologize if this has been mentioned before and I missed it.
>>>
>>> You are certain that the physical disk you are trying to write to is
>>> entirely serviceable? It is not possible that the disk that houses block
>>> 10014 is failing, right?
>>>
>>> v/r,
>>>
>>> Gus
>>>
>>> On Tue, Aug 21, 2018 at 10:56 AM Jeff Chirco <backseatdba_at_gmail.com>
>>> wrote:
>>>
>>>> Thanks for your help. It would be weird if it was permissions as this
>>>> has worked for years.
>>>>
>>>> RMAN> show all;
>>>>
>>>> using target database control file instead of recovery catalog
>>>> RMAN configuration parameters are:
>>>> CONFIGURE RETENTION POLICY TO REDUNDANCY 2;
>>>> CONFIGURE BACKUP OPTIMIZATION OFF; # default
>>>> CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
>>>> CONFIGURE CONTROLFILE AUTOBACKUP ON;
>>>> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; #
>>>> default
>>>> CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; #
>>>> default
>>>> CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
>>>> CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
>>>> CONFIGURE MAXSETSIZE TO UNLIMITED; # default
>>>> CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
>>>> CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
>>>> CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
>>>> CONFIGURE SNAPSHOT CONTROLFILE NAME TO
>>>> 'C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SE
>>>> RVER\DATABASE\SNCFXE2.ORA';
>>>>
>>>> RMAN> backup datafile 1;
>>>>
>>>> Starting backup at 21-AUG-18
>>>> allocated channel: ORA_DISK_1
>>>> channel ORA_DISK_1: sid=195 devtype=DISK
>>>> channel ORA_DISK_1: starting full datafile backupset
>>>> channel ORA_DISK_1: specifying datafile(s) in backupset
>>>> input datafile fno=00001 name=D:\DATA\ORACLE\ORADATA\XE\SYSTEM.DBF
>>>> channel ORA_DISK_1: starting piece 1 at 21-AUG-18
>>>> channel ORA_DISK_1: finished piece 1 at 21-AUG-18
>>>> piece
>>>> handle=D:\BACKUP\ORACLE\FLASH_RECOVERY_AREA\XE\BACKUPSET\2018_08_21\O1_MF_
>>>> NNNDF_TAG20180821T074350_FQR996K2_.BKP tag=TAG20180821T074350
>>>> comment=NONE
>>>> channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
>>>> Finished backup at 21-AUG-18
>>>>
>>>> Starting Control File and SPFILE Autobackup at 21-AUG-18
>>>> RMAN-00571: ===========================================================
>>>> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
>>>> RMAN-00571: ===========================================================
>>>> RMAN-03009: failure of Control File and SPFILE Autobackup command on
>>>> ORA_DISK_1
>>>> channel at 08/21/2018 07:44:13
>>>> ORA-19599: block number 10014 is corrupt in control file
>>>> C:\ORACLEXE\APP\ORACLE\
>>>> PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA
>>>>
>>>> RMAN> show parameter control
>>>>
>>>> RMAN-00571: ===========================================================
>>>> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
>>>> RMAN-00571: ===========================================================
>>>> RMAN-00558: error encountered while parsing input commands
>>>> RMAN-01009: syntax error: found "parameter": expecting one of: "all,
>>>> archivelog,
>>>> auxiliary, auxname, backup, channel, controlfile, datafile, device,
>>>> default, ex
>>>> clude, encryption, maxsetsize, retention, snapshot"
>>>> RMAN-01007: at line 1 column 6 file: standard input
>>>>
>>>>
>>>> On Tue, Aug 21, 2018 at 3:59 AM LK <lkaing_at_gmail.com> wrote:
>>>>
>>>>> Hi Jeff,
>>>>>
>>>>> From the debug we can see the failure is here:
>>>>>
>>>>> DBGRPC: krmxrpc: xc=15378960 kpurpc2 rc=19624 db=target
>>>>> proc=DBMS_BACKUP_RESTORE.BACKUPPIECECREATE
>>>>> DBGMISC: ENTERED krmzejob [10:13:23.281]
>>>>> DBGMISC: Input Args(failed=1),(errnum=-19624) [10:13:23.359]
>>>>> (krmzejob)
>>>>>
>>>>> It's trying to create the backuppiece and can't do so.
>>>>>
>>>>> So I suspect that you have permission issues with the backup location.
>>>>>
>>>>> eg.
>>>>>
>>>>> How to Configure RMAN to Write to Shared Drives on Windows
>>>>> NT/2000/2003 (Doc ID 145843.1)
>>>>>
>>>>> You said that the rest of the backup works, except for the autobackup.
>>>>>
>>>>> So please show me the result of the following:
>>>>>
>>>>> RMAN> show all;
>>>>> RMAN> backup datafile 1;
>>>>>
>>>>> SQL> show parameter control
>>>>>
>>>>> I'm thinking that we need to configure the autobackup and snapshot
>>>>> controlfile locations to wherever your datafile backup goes. I
>>>>>
>>>>> But please show me the above and see what we get.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Leng Burgess.
>>>>>
>>>>>
>>>>> On Tue, Aug 21, 2018 at 3:19 AM, Jeff Chirco <backseatdba_at_gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry you are right the backup portion is running, it just last part
>>>>>> that is not.
>>>>>> Thanks for your help!
>>>>>>
>>>>>>
>>>>>> RMAN> backup current controlfile;
>>>>>>
>>>>>> Starting backup at 20-AUG-18
>>>>>> using target database control file instead of recovery catalog
>>>>>> allocated channel: ORA_DISK_1
>>>>>> channel ORA_DISK_1: sid=197 devtype=DISK
>>>>>> channel ORA_DISK_1: starting full datafile backupset
>>>>>> channel ORA_DISK_1: specifying datafile(s) in backupset
>>>>>> including current control file in backupset
>>>>>> channel ORA_DISK_1: starting piece 1 at 20-AUG-18
>>>>>> RMAN-00571:
>>>>>> ===========================================================
>>>>>> RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS
>>>>>> ===============
>>>>>> RMAN-00571: ==============================
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> =============================
>>>>>> RMAN-03009: failure of backup command on ORA_DISK_1 channel at
>>>>>> 08/20/2018 10:17:
>>>>>> 26
>>>>>> ORA-19599: block number 10014 is corrupt in control file
>>>>>> C:\ORACLEXE\APP\ORACLE\
>>>>>> PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE2.ORA
>>>>>>
>>>>>>
>>>>>> On Sun, Aug 19, 2018 at 12:39 AM Leng <lkaing_at_gmail.com> wrote:
>>>>>>
>>>>>>> What do you meany by ‘my Rman backups are failing everyday’? Is it
>>>>>>> the entire backup or just the last part where it tries to do an autobackup?
>>>>>>>
>>>>>>> You could try turning on debug as follows:
>>>>>>>
>>>>>>> Rman> debug on;
>>>>>>> Rman> spool trace to ‘c:\temp\ctl.debug;
>>>>>>> Rman> backup current control file;
>>>>>>> Rman> spool trace off
>>>>>>>
>>>>>>> Please email the trace file and output from the above and I’ll have
>>>>>>> a quick look...
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Leng Burgess.
>>>>>>>
>>>>>>> On 18 Aug 2018, at 12:58 am, Jeff Chirco <backseatdba_at_gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Thanks for the suggestions. I checked the drive space and it is not
>>>>>>> an issue. However I did change the snapshot controlfile location to a
>>>>>>> different disk and ran a backup spfile. Still getting same corruption
>>>>>>> notice. It keeps talking about corruption in (file 0, block 10014). No
>>>>>>> other errors in alert log.
>>>>>>> We use this instance in retail locations and there haven't been any
>>>>>>> issues with the database, and it gets rebooted every night. Just my rman
>>>>>>> backups are failing everyday. Good thing we also do data pump exports.
>>>>>>>
>>>>>>> On Thu, Aug 16, 2018 at 4:16 PM LK <lkaing_at_gmail.com> wrote:
>>>>>>>
>>>>>>>> Yep, agreed.
>>>>>>>>
>>>>>>>> Try using a different drive that has plenty of free space and do
>>>>>>>> the tests that I've outlined before...
>>>>>>>>
>>>>>>>> eg:
>>>>>>>>
>>>>>>>> RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'F:\DBA\SNAPCF_XE.F';
>>>>>>>>
>>>>>>>> And take an spfile backup - this should genereate a new snapshot
>>>>>>>> controlfile:
>>>>>>>>
>>>>>>>> RMAN> backup current controlfile;
>>>>>>>>
>>>>>>>> I'd also get the sysadmin to do hardware/os/firmware check of the
>>>>>>>> server in case this is just the tip of the iceberg... is there corruption
>>>>>>>> in the underlying hardware/storage/firmware etc.
>>>>>>>>
>>>>>>>> And check the alert.log - have you got other issues as well?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Leng Burgess.
>>>>>>>>
>>>>>>>> On Fri, Aug 17, 2018 at 8:38 AM, Andrew Kerber <
>>>>>>>> andrew.kerber_at_gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You may simply be out of space.
>>>>>>>>>
>>>>>>>>> Sent from my iPad
>>>>>>>>>
>>>>>>>>> On Aug 16, 2018, at 17:35, Jeff Chirco <backseatdba_at_gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Yes I have 3 control files but this is not a control file, its the
>>>>>>>>> snapshot control file. I have deleted this snapshot file and it recreates
>>>>>>>>> it as corrupt. I am wondering which control file could be corrupt.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 16, 2018 at 1:19 PM Mladen Gogala <
>>>>>>>>> gogala.mladen_at_gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On 08/15/2018 07:21 PM, Jeff Chirco wrote:
>>>>>>>>>>
>>>>>>>>>> > I have an 10g XE database that is failing during rman backup.
>>>>>>>>>> > ORA-19624: operation failed, retry possible
>>>>>>>>>> > ORA-19599: block number 10014 is corrupt in control file
>>>>>>>>>> > C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE.ORA
>>>>>>>>>> >
>>>>>>>>>> > I am not sure what SNCFXE.ORA is but I believe it is some
>>>>>>>>>> snapshot
>>>>>>>>>> > control file backup. Anybody know how to recover for this? Do
>>>>>>>>>> I need
>>>>>>>>>> > to do a full recovery from controlfile trace?
>>>>>>>>>>
>>>>>>>>>> Hopefully, you have more than one control file? If you do, please
>>>>>>>>>> copy
>>>>>>>>>> the correct one over the corrupt
>>>>>>>>>> C:\ORACLEXE\APP\ORACLE\PRODUCT\10.2.0\SERVER\DATABASE\SNCFXE.ORA
>>>>>>>>>> file.
>>>>>>>>>> Check your control_files parameter in the instance. If you don't
>>>>>>>>>> have
>>>>>>>>>> more than one, try dumping it to trace (ALTER DATABASE BACKUP
>>>>>>>>>> CONTROLFILE TO TRACE AS 'C:\temp\ctlfile.sql' NORESETLOGS) then
>>>>>>>>>> shutting
>>>>>>>>>> down the instance and re-creating the control file.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mladen Gogala
>>>>>>>>>> Database Consultant
>>>>>>>>>> Tel: (347) 321-1217
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> http://www.freelists.org/webpage/oracle-l
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>
>>

--
http://www.freelists.org/webpage/oracle-l
Received on Thu Aug 23 2018 - 16:13:28 CEST

Original text of this message