Home » RDBMS Server » Server Administration » ORA-27044, ORA-16038, ORA-19504, ORA-00312 (Oracle 10g, Solaris 10)
ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346697] Tue, 09 September 2008 07:39 Go to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
Anacedent, Michel...
I am having issues. Oracle database went down due to archive_log destination got filled up 100%. I freed up space and started the database.

Now I am getting following errors.
ORA-27044: unable to write the header block of file,Errors in file /u01/app/oracle/admin/PROD/bdump/prod1_arc1_1459.trc
ORA-16038: log 6 sequence# 138703 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 6 thread 1


I also, tried:
alter system switch log;

but no help...

Here is the my trace file:
$ tail -100 /u01/app/oracle/admin/PROD/bdump/prod1_arc1_1459.trc

ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:02.486 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 7680
Additional information: 1048576
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 7680
Additional information: 1048576
*** 2008-09-09 05:59:03.113 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:03.113 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:03.113 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:03.123 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 16385 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 228864
Additional information: 1048576
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 16385 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 228864
Additional information: 1048576
*** 2008-09-09 05:59:04.391 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:04.391 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:04.391 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:04.399 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'
Exception 19502 encountered when closing file
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 18433 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 564736
Additional information: 1046528
ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 18433 (blocksize=512)
ORA-27063: number of bytes read/written is incorrect
Additional information: 564736
Additional information: 1046528
*** 2008-09-09 05:59:05.601 53058 kcrr.c
ARC1: Error 19502 Closing archive log file '/u02/archives/1_138704_548018336.dbf'
*** 2008-09-09 05:59:05.601 51534 kcrr.c
kcrrfail: dest:2 err:19502 force:0 blast:1
*** 2008-09-09 05:59:05.601 53058 kcrr.c
ARC1: All standby destinations failed; successful archival assumed
*** 2008-09-09 05:59:05.611 16790 kcrr.c
ORA-16038: log 1 sequence# 138704 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 1 thread 1: '+PROD_DATA1/prod/onlinelog/group_1.261.1'


I would greatly appreciate if you could take some time out and provide me the solution.

Thanks very much.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346713 is a reply to message #346697] Tue, 09 September 2008 08:41 Go to previous messageGo to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
Would following command fix the issue?

alter system archive log current;


Thanks..
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346730 is a reply to message #346697] Tue, 09 September 2008 09:35 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10672
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
>> I freed up space and started the database
Seems you nuked the log destination area including the current archive log files.
Next time, apply proper methods.
Shutdown.
Startup.
Issue
alter system switch logfile; -- do it a couple of times.

[Updated on: Tue, 09 September 2008 09:39]

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346742 is a reply to message #346730] Tue, 09 September 2008 10:18 Go to previous messageGo to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
I actually moved some old files to a different drive freeing up some 16G of space.

I first did a
crs_stat -t
and saw that the TARGET of two services was ONLINE but STATE offline. So I fired a
crs_start -all
command to bring both nodes up.

After this, I tried the
alter system switch log
command 4 times. But I am still getting the errors.

I also tried
alter system archive log current
, but no help so far.

Its live production. What should be the steps to fix this issue?

I am in a terrible situation, awaiting your reply.

Thanks

[Updated on: Tue, 09 September 2008 10:24]

Report message to a moderator

Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346744 is a reply to message #346697] Tue, 09 September 2008 10:24 Go to previous messageGo to next message
BlackSwan
Messages: 25033
Registered: January 2009
Location: SoCal
Senior Member
This comes with no guarentee, but it won't hurt if you rename /u02/archives/1_138704_548018336.dbf to a different name.
It might allow the archiver process to try again with a fresh copy & when there is free space to complete the copy.

Good Luck!
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346754 is a reply to message #346742] Tue, 09 September 2008 11:16 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10672
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
You have not mentioned that you are using RAC in any of your above posts.
>>ORA-19502: write error on file "/u02/archives/1_138704_548018336.dbf", blockno 1 (blocksize=512)
Is /u02 properly mounted across both nodes?
Are any other log destinations specified?
Repost the relevant portions of alert log in both nodes.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346779 is a reply to message #346744] Tue, 09 September 2008 12:41 Go to previous messageGo to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
It didn't help. I moved the existing file to a different folder, but I am still receiving the same error.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346782 is a reply to message #346697] Tue, 09 September 2008 12:50 Go to previous messageGo to next message
BlackSwan
Messages: 25033
Registered: January 2009
Location: SoCal
Senior Member
If the ASM volumes reside on NetApp, then this could be "known bug". Search Metalink for details.

You should consider opening a Service Request with oracle.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346787 is a reply to message #346697] Tue, 09 September 2008 13:02 Go to previous messageGo to next message
BlackSwan
Messages: 25033
Registered: January 2009
Location: SoCal
Senior Member
If you are desperate, you could create a new folder on the ASM volume & the change Archive destination in spfile to point to new directory.
Again, it can't hurt & may get around current issue involving /u02/archives.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346791 is a reply to message #346754] Tue, 09 September 2008 13:07 Go to previous messageGo to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
Yes its 2-node RAC on 10g 10.1.0.5 (node1/node2 | Active/Passive) configuration.
Quote:
Is /u02 properly mounted across both nodes?

Yes the /u02/archive folder is properly mounted on both nodes. That is the reason it was working fine until this morning when Oracle went down, and after that everything started giving errors.

Quote:
Are any other log destinations specified?
No, but /u02/archives is log_archive_dest_2 in parameter_file and log_archive_dest_1 is empty. Does this matter?

I am more interested in digging into thing rather THINGS_ARE_OK. So please guide me more.

The alert log is all about the information I posted earlier this morning. If you could give me an idea of what to look for specifically, that would be great.

Thanks for your help...

P.S. Please look at the attached file..../fa/4985/0/

Here is the Alert_log's last 100 lines...
ARC0: Evaluating archive thread 1 sequence 138761
Tue Sep  9 13:47:44 2008
ARC0: Beginning to archive thread 1 sequence 138761 (1053121623-1053125943) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:47:44 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138761_548018336.dbf' (thread 1 sequence 138761)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:47:45 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138761_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138761_548018336.dbf'
Tue Sep  9 13:47:46 2008
ARC0: Completed archiving thread 1 sequence 138761 (1053121623-1053125943) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138763
  Current log# 6 seq# 138763 mem# 0: +PROD_DATA1/prod/onlinelog/group_6.269.1
Tue Sep  9 13:53:46 2008
ARC0: Evaluating archive thread 1 sequence 138762
Tue Sep  9 13:53:46 2008
ARC0: Beginning to archive thread 1 sequence 138762 (1053125943-1053130125) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138762_548018336.dbf' (thread 1 sequence 138762)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:53:46 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138762_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138762_548018336.dbf'
Tue Sep  9 13:53:47 2008
ARC0: Completed archiving thread 1 sequence 138762 (1053125943-1053130125) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138764
  Current log# 1 seq# 138764 mem# 0: +PROD_DATA1/prod/onlinelog/group_1.261.1
Tue Sep  9 13:56:46 2008
ARC0: Evaluating archive thread 1 sequence 138763
Tue Sep  9 13:56:46 2008
ARC0: Beginning to archive thread 1 sequence 138763 (1053130125-1053131953) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138763_548018336.dbf' (thread 1 sequence 138763)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:56:46 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138763_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138763_548018336.dbf'
Tue Sep  9 13:56:47 2008
ARC0: Completed archiving thread 1 sequence 138763 (1053130125-1053131953) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138765
  Current log# 5 seq# 138765 mem# 0: +PROD_DATA1/prod/onlinelog/group_5.268.1
Tue Sep  9 13:58:10 2008
ARC0: Evaluating archive thread 1 sequence 138764
Tue Sep  9 13:58:10 2008
ARC0: Beginning to archive thread 1 sequence 138764 (1053131953-1053133576) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138764_548018336.dbf' (thread 1 sequence 138764)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 13:58:10 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138764_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138764_548018336.dbf'
Tue Sep  9 13:58:11 2008
ARC0: Completed archiving thread 1 sequence 138764 (1053131953-1053133576) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
Private_strands 0 at log switch
Thread 1 advanced to log sequence 138766
  Current log# 2 seq# 138766 mem# 0: +PROD_DATA1/prod/onlinelog/group_2.262.1
Tue Sep  9 14:00:09 2008
ARC0: Evaluating archive thread 1 sequence 138765
Tue Sep  9 14:00:09 2008
ARC0: Beginning to archive thread 1 sequence 138765 (1053133576-1053135372) (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138765_548018336.dbf' (thread 1 sequence 138765)
 (PROD1)
ARCH: Connecting to console port...
Tue Sep  9 14:00:09 2008
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_2: '/u02/archives/1_138765_548018336.dbf'
 (PROD1)
ARCH: Connecting to console port...
Committing creation of archivelog '/u02/archives/1_138765_548018336.dbf'
Tue Sep  9 14:00:10 2008
ARC0: Completed archiving thread 1 sequence 138765 (1053133576-1053135372) (PROD1)
ARCH: Connecting to console port...
  • Attachment: ARCH_Hung.JPG
    (Size: 127.92KB, Downloaded 9573 times)
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346793 is a reply to message #346697] Tue, 09 September 2008 13:11 Go to previous messageGo to next message
BlackSwan
Messages: 25033
Registered: January 2009
Location: SoCal
Senior Member
My assessment is that the logjam has broken free & all is back to normal (for now).
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346805 is a reply to message #346793] Tue, 09 September 2008 13:49 Go to previous messageGo to next message
deepmachine
Messages: 80
Registered: August 2008
Location: United States
Member
I guess, I would agree with you. The culprit is FogLight. For some reason its continuously sending me the same error/incident that took place before 6:00AM this morning.

Whereas, when I go and look at that alert_log and the *.trc file, it shows me nothing new.

Thanks for being with me. Your support is always greatly appreciated.

Thanks again..
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346806 is a reply to message #346697] Tue, 09 September 2008 13:52 Go to previous messageGo to next message
BlackSwan
Messages: 25033
Registered: January 2009
Location: SoCal
Senior Member
> The culprit is FogLight.
Those who live by the GUI, die by the GUI.

the alert_SID.log file should show you when the logjam broke free.
I still suspect things started getting better just after renaming the broken archived redo log file out of the way.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #346809 is a reply to message #346806] Tue, 09 September 2008 14:03 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10672
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
Which is why we point to read the forum rules first. Smile
Posting the relevant contents from alert log could have
saved you from some agony.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #359886 is a reply to message #346809] Tue, 18 November 2008 08:04 Go to previous messageGo to next message
khushal.k
Messages: 1
Registered: July 2008
Location: Thane
Junior Member

It clearly stated that your filesystem is full. there should be no space left on disk.
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480102 is a reply to message #346697] Thu, 21 October 2010 04:03 Go to previous messageGo to next message
ashuj20
Messages: 12
Registered: October 2005
Location: new delhi
Junior Member
Hi,

All the suggestion I see in this post are all crap!!

This problem is a know bug in Orcale (Bug 3615505). It is resolved in Release 10.2 or higher.

Thanks
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480120 is a reply to message #480102] Thu, 21 October 2010 05:15 Go to previous messageGo to next message
Michel Cadot
Messages: 64103
Registered: March 2007
Location: Nanterre, France, http://...
Senior Member
Account Moderator
Thanks for the information, why were not you there in 2008?
Insults are not necessary.

Regards
Michel
Re: ORA-27044, ORA-16038, ORA-19504, ORA-00312 [message #480135 is a reply to message #480102] Thu, 21 October 2010 06:49 Go to previous message
Mahesh Rajendran
Messages: 10672
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
>>All the suggestion I see in this post are all crap!!
Would that include your drivel too?
The said bug is windows OS.
The bug is talking about entries in Event viewer.
This thread is about entries in oracle alert log in Solaris.
Previous Topic: Clone Database can not be connected after system reboot.
Next Topic: Contention on data blocks
Goto Forum:
  


Current Time: Sat Dec 03 09:54:42 CST 2016

Total time taken to generate the page: 0.06300 seconds