Home » RDBMS Server » Server Administration » AIX random IO error (Oracle Database 9.2.0.6 AIX 5.3)
AIX random IO error [message #394850] Mon, 30 March 2009 09:57 Go to next message
blake.hall
Messages: 3
Registered: March 2009
Location: Baytown, Texas
Junior Member
I have a production AIX database server with 8 production 7X24 instances. On only one of the 8 instances, randomly a scheduled procedure fails with these errors:
ORA-01115: IO error reading block from file 10 (block # 94006)
ORA-01110: data file 10: '/u05/oracle/bupurp/aap_data.dbf'
ORA-27091: skgfqio: unable to queue I/O
ORA-27072: skgfdisp: I/O error
IBM AIX RISC System/6000 Error: 5: I/O error
The procedure populates a table with data from 2 local tables and a linked database table.
Oracle recommends upgrading the AIX to the newest patch level -we are at the most current patch level.... and applying an Oracle database patch {5496862}. Unfortunately all 8 production instances will have to be patched, since all are on the same Oracle home.
Not that I don't want to do the work, but I am looking for alternatives to applying this patch to these high activity instances, to minimize downtime.
Has anyone had a similar situation and found another approach? On one forum I saw a similar post and the recommended solution was to move the tablespace file to a different drive, on a different controller.
Thanks in advance for your help!
Re: AIX random IO error [message #394856 is a reply to message #394850] Mon, 30 March 2009 10:12 Go to previous messageGo to next message
ThomasG
Messages: 3212
Registered: April 2005
Location: Heilbronn, Germany
Senior Member
Have you checked on the OS side if there are any errors, or any limits placed on the Oracle user account?

(ulimit)

Have you checked if there are any errors in the system log or STALE PPs in the disk volume group?

(errpt / lsvg)

Re: AIX random IO error [message #394885 is a reply to message #394850] Mon, 30 March 2009 13:17 Go to previous messageGo to next message
blake.hall
Messages: 3
Registered: March 2009
Location: Baytown, Texas
Junior Member
No limitations on the Oracle user in the OS. [response was unlimited]
No obvious errors on the error report... response below:
/home/eyaba> errpt / lsvg
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
DCB47997 0330134709 T H hdisk14 DISK OPERATION ERROR
F7FA22C9 0320032809 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
F7FA22C9 0319070309 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
F7FA22C9 0308101009 I O SYSJ2 UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
C69F5C9B 0308030009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
1FB4FD21 0302054209 I S fscsi0 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi0 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi1 CONFIGURATION MISMATCH
1FB4FD21 0302054209 I S fscsi1 CONFIGURATION MISMATCH
/home/eyaba>

The recent dates and times that the procedure failed were:
03-25-2009 6:18am
03-22-2009 11:43pm
03-20-2009 5:36am
03-18-2009 2:27am

Thanks for your help!
Re: AIX random IO error [message #394888 is a reply to message #394850] Mon, 30 March 2009 13:23 Go to previous messageGo to next message
BlackSwan
Messages: 26766
Registered: January 2009
Location: SoCal
Senior Member
GOOGLE can be your friend, but only when you use it.

Results 1 - 10 of about 2,120 for IBM AIX RISC System/6000 Error: 5: I/O error oracle.

>IBM AIX RISC System/6000 Error: 5: I/O error
In my opinion the root cause is Operating System related & not really Oracle's problem.
Re: AIX random IO error [message #394891 is a reply to message #394888] Mon, 30 March 2009 13:31 Go to previous messageGo to next message
ThomasG
Messages: 3212
Registered: April 2005
Location: Heilbronn, Germany
Senior Member
No obvious errors????

What do you call this then :

DCB47997 0330134709 T H hdisk14 DISK OPERATION ERROR


Looks like "Disk kaputt" to me. Confused

Plus the "UNABLE TO ALLOCATE SPACE" and Fibrechannel errors, I say there is or was something pretty broken on the hardware side, which might have corrupted the data files.
Re: AIX random IO error [message #394909 is a reply to message #394850] Mon, 30 March 2009 17:02 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10708
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
How big is this datafile?
Is the largefile options turned on for this filesystem?
Re: AIX random IO error [message #395236 is a reply to message #394909] Tue, 31 March 2009 16:15 Go to previous messageGo to next message
blake.hall
Messages: 3
Registered: March 2009
Location: Baytown, Texas
Junior Member
The file is 28GB. I don't know if this is set up with largefile option. How to determine?
Re: AIX random IO error [message #395237 is a reply to message #395236] Tue, 31 March 2009 16:41 Go to previous message
ThomasG
Messages: 3212
Registered: April 2005
Location: Heilbronn, Germany
Senior Member
In the smitty console under "system storage management" somewhere. Where exactly would depend on how your storage is set up. But since the file is already >2GB it should be enabled.

You should have used "ulimit -a" by the way to see all user limits.

Also, errpt und lsvg are to separate commands, too, one to search the system error log, the other to list volume groups.

Both have different options (in case of lsvg also different options depending on how your storage is set up), too, to see different things.

But since the hardware looks definitely broken, you should call your hardware support anyway.

1) Fix the hardware
2) Fix the volume groups / filesystems

and only then you have any chance of doing.

3) Fix the database
Previous Topic: Multiple Connection present
Next Topic: Ramifications of ORA-01555 on Oracle Text index
Goto Forum:
  


Current Time: Tue Dec 03 21:33:13 CST 2024