Archive Log File broken

From: FlameDance <FlameDance_at_gmx.de>
Date: Tue, 29 Jun 2004 17:52:58 +0200
Message-ID: <cbs38r$m0q$06$1@news.t-online.com>

Hello community,

today I have a tricky question.

I have built a standby database (8.1.7), using daily manual recovery. The production database is configured so that it writes 2 archived logs, one on the production machine (mandatory), one on an automounted directory on the standby machine.
There's no direct link via SQLnet, only a directory on the standby machine that the production database writes to (mounted via automount) and that the standby machine reads. For a couple of weeks now it worked fine, without any error, but yesterday we had a problem with a broken archived logfile.

The broken archived logfile has the
same size as the correct one but except for a few bytes at the beginning it's full of zeros. When I copied the productions machines copy of the defect file over to the defect one, recovery worked perfectly - but that's ultimately not the solution I desire, I want a safe guard against a crash of the production machine.

On the production machine in the alert_prod.log:

ARC0: Error 19505 creating archivelog file '/dh36adminprod/arch/fromdh11prod/arch_1_11323.arc'

ORA-19505 failed to identify file "string"

     Cause: Call to identify the file returned an error.
     Action: Check additional messages, and check whether the file exists.

On the standby machine in the alert_prod.log:

Mon Jun 28 20:52:03 2004
ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log /opt/oracle/admin/prod/arch/fromdh11prod/arch_1_11323.arc Mon Jun 28 20:52:03 2004
Errors in file /opt/oracle/admin/prod/udump/ora_26771.trc: ORA-00317: file type 0 in header is not log file ORA-00334: archived log:
'/opt/oracle/admin/prod/arch/fromdh11prod/arch_1_11323. ORA-317 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...

What can be the cause of this error?

A problem with the automounter? Unlikely, as the file was created and has the correct size.
A directory locking problem? The standby database starts recovering at 20:25 - and that's when the production databases archiver ran into the error. However, the write attempt was at 20:25, and the standby database didn't reach that file until 20:52, so it's likely not a locking problem on a file by file basis.
I inserted ALTER SYSTEM SWITCH LOGFILE right before shutting down the production database for backups to create an archived logfile where all transactions are definitely finished, even from the application viewpoint which doesn't care much about complete transactions. Could it be that shutting it down killed the archiver? Hardly imaginable - and it worked on all other days since I implemented that line.
Other ideas?

For completeness here's a longer excerpt of the production log files:

The production databases alert_prod.log: Thu Jun 24 17:05:14 2004
ARC0: Completed archiving log# 1 seq# 11322 Thu Jun 24 20:25:01 2004
Thread 1 advanced to log sequence 11324

Current log# 2 seq# 11324 mem# 0: /opt/oracle/oradata/prod/redo02a.rdo Current log# 2 seq# 11324 mem# 1: /opt/oracle/oradata/prod/redo02b.rdo Thu Jun 24 20:25:01 2004
ARC0: Beginning to archive log# 3 seq# 11323 Thu Jun 24 20:25:01 2004
Shutting down instance (immediate)
License high water mark = 52
Thu Jun 24 20:25:02 2004
ALTER DATABASE CLOSE NORMAL
Thu Jun 24 20:25:02 2004
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Jun 24 20:25:02 2004
Thread 1 closed at log sequence 11324
Thu Jun 24 20:25:02 2004
Completed: ALTER DATABASE CLOSE NORMAL
Thu Jun 24 20:25:02 2004
ALTER DATABASE DISMOUNT
Thu Jun 24 20:25:05 2004
ARC0: Error 19505 creating archivelog file '/dh36adminprod/arch/fromdh11prod/arch_1_11323.arc'

ARC0: Completed archiving log# 3 seq# 11323
ARC0: thread not mounted
ARC0: archiving disabled or thread not mounted

Thu Jun 24 20:25:05 2004
Completed: ALTER DATABASE DISMOUNT
archiving is disabled
Thu Jun 24 20:25:06 2004
ARCH shutting down
ARC0: Archival stopped

Big thanks in advance for any hints,
Stephan Received on Tue Jun 29 2004 - 10:52:58 CDT