RE: Excessive IOPS to shared ORACLE_HOME
Date: Fri, 22 Aug 2008 07:47:00 +0200
Oracle on netapp rings some bells for me. Oracle recommends using noac or actime=o options when mounting nfs for Datafiles, Voting Disk and OCR. Noac means "no attribute cache" means none of the file attributes are cached in the filesystem cache, which is very much needed for RAC. If you put your shared oracle home also in that mountpoint which is mounted noac, every access to a file in the oracle home requires a physical IO at the netapp. So I recommend moving all software directories ( db oracle home, asm oracle home and crs oracle home etc ) to a nfs mount which is not mounted with noac or actime=o. This will reduce your excessive IO per sec issue
Apps DBA - The Pythian Group
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Riyaj Shamsudeen
Sent: Friday, August 22, 2008 5:18 AM
Subject: Re: Excessive IOPS to shared ORACLE_HOME
I have seen similar issue in Solaris platform. Can you see any difference as to how Oracle_home file system is mounted between these servers ? It's a common fallacy that Oracle home binaries are never accessed once the database is up and running. In reality, Oracle binaries are accessed constantly and if the file system holding oracle_home is mounted with mount options bypassing UNIX buffer cache, then IOPS to underlying file system increases sharply. So, you might want to check all your DB nodes and make sure Oracle home file system is using unix buffer cache. There are other reasons too. Enormous amount of audit files, logging in listener log, CRS spitting out log entries etc. Check that out too.
Chen Shapira wrote:
> Hi Oracle Fans,
> I need some help and advice on tracking down source of excessive IO.
> Our 10.2.0.[2,3] RAC enviroments are configured with shared
> ORACLE_HOME, residing on Netapp storage. The same volume also contains
> the voting disk and OCR for the RAC.
> This volume, that we assumed will have almost no load, actually has
> about 6000 IOPS (IO operations per second), sometimes up to 13000. For
> comparison, our data volume has about 1500 IOPS.
> Interesting findings:
> 1) From Netapp reports and from some network sniffing, it appears that
> over 90% of the IO is "access" and "get attribute" operations -
> neither reads, nor writes.
> 2) This does not happen to the voting disks on RAC systems that have
> no shared home. (although there are other configuration differences as
> 3) This also does not happen in stand alone systems that have
> ORACLE_HOME on the Netapp.
> I can't use IOSTAT to debug because it does not report NFS. Network
> monitoring aggregates over all NFS traffic, so I can't seperate normal
> data traffic from strange ORACLE_HOME traffic (it is all on same
> storage interface). Linux does not report network activity per process
> (not without special patches). Wait event tables will not show traffic
> to Oracle_home or to voting disk.
> Any suggestions where to look?
> Anyone can guess what we could have configured wrong to cause such an
> issue? ( I know that we are all against random guessing, but it feels
> like all investigation channels are insufficient).
> Chen Shapira