RE: troubleshooting slow I/O performance.

From: Mark W. Farnham <mwf_at_rsiz.com>
Date: Mon, 7 May 2018 13:40:12 -0400
Message-ID: <022d01d3e62a$74243060$5c6c9120$_at_rsiz.com>



one little wrinkle to add is the redundancy level you have chosen. Stefan mentioned the underlying possible raid and lun configuration.  

IF multi-device underlying LUNs are hidden from Oracle and you have a redundancy other than external, it is possible to create an “anti-pattern” in ASM’s choices of failure group partner such that the same real physical disks are hit doubly or triply.  

This is neither a recommendation for or against external redundancy, just make sure that it is not the problem.  

If you current are NOT external redundancy, the trivial way to see if this is a problem is to create a tablespace with external redundancy and run an i/o test on it.  

If your six 30TB LUNs don’t use parts of each other’s “trays” or other components in the physical stack this is not your problem a priori. But I have observed systems where some fraction of each disk is striped with mirror pairs across many disks and that is presented as a LUN and another fraction of the same disks is likewise built into a LUN. …And then these are presented to Oracle’s ASM as two different things. Hilarity ensues.  

mwf  

From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Stefan Knecht Sent: Monday, May 07, 2018 1:12 PM
To: Chris Stephens
Cc: oracle-l
Subject: Re: troubleshooting slow I/O performance.  

By path I meant how the storage is connected to the server. Fibre channel, iSCSI, etc...  

And I assume you mean that you have 30TB LUNs - I don't think there are 30TB physical disks. The actual number of physical spinning rusty things is the number that's important. E.g. how is your LUN made up?  

And of course, it's also important to know how the disks are arranged when the LUN is built - RAID 1, RAID 5 (I hope not but that might explain the issues you're seeing) RAID 10, etc...  

I'd say a friendly chat with the storage guy is probably in order :) It could be any number of things - but what's really key to get good performance out of bare metal drives is the number of drives you got.          

On Tue, May 8, 2018 at 12:08 AM, Chris Stephens <cstephens16_at_gmail.com> wrote:

ASM 6 30TB spinning drives

type of path? we are not using asmlib or asm_fd (whatever its called).  

On Mon, May 7, 2018 at 11:54 AM Stefan Knecht <knecht.stefan_at_gmail.com> wrote:

Some details would help:  

Etc...  

On Mon, 7 May 2018, 21:00 Chris Stephens, <cstephens16_at_gmail.com> wrote:

We have a new 5 node 12.2 RAC system that is not performing the way we want.  

The glaring issue is that "db file sequential read"'s are taking ~10ms. before i lob this over to the storage administrators, are there any possible areas in the clusterware/database configuration that I should investigate first? i have root access to all of the nodes. is there any information i can collect that would expedite the process of figuring out why we have such slow I/O times?  

slow i/o was discovered by running slob. if you don't know about that tool, you should. we all owe kevin a debt of gratitude. ;)  

if nothing else, i hope to learn a little more about storage than i currently know (which isn't much).  

thanks for any help.  

chris    

-- 

//

zztat - The Next-Gen Oracle Performance Monitoring and Reaction Framework!

Visit us at  <http://zztat.net/> zztat.net | _at_zztat_oracle | fb.me/zztat | zztat.net/blog/



--
http://www.freelists.org/webpage/oracle-l
Received on Mon May 07 2018 - 19:40:12 CEST

Original text of this message