Re: troubleshooting slow I/O performance.

From: Stefan Koehler <>
Date: Wed, 9 May 2018 09:41:56 +0200 (CEST)
Message-ID: <>

Hello Mladen,
sure I agree if you already know that the disks are the bottleneck (too slow) but in this case the OP does not know and want to identify / drill-down to the bottleneck.

Let's summarize the provided information quickly:

- 5 node 12.2 RAC system
- "db file sequential read"'s are taking ~10ms
- ASM (external redundancy)
- 6 x 30TB spinning drives / 8 LUNs via FC

First of all we don't know how the 10 ms (I guess/assume average) are formed. Are most of the IOs in this latency bucket or is the majority of IO much faster and the OP just got a few very bad outliers that lead to this average.

Secondly he is using RAC and so also some kind of shared storage - even typical mid-range storage for such environments got read/write caches which improve the response time of spinning disks drastically. We just don't know how it looks like and it seems like the OP does not either.

IMHO it is better to give the OP a way to figure it out on his own (and blktrace is a good tool to do that) instead of wild guessing and already stating that the root cause of his observation is the "slow" spinning disks.  

P.S.: In the past I was able to fix several IO outlier problems with blktrace by the way ;-)

Best Regards
Stefan Koehler

Independent Oracle performance consultant and researcher Website:
Twitter: _at_OracleSK

> Mladen Gogala hat am 9. Mai 2018 um 05:14 geschrieben:
> There is rarely need to lose time by doing blktrace. If disks are too
> slow, the ONLY real answer are faster disks. I used to play with IO
> elevators, read ahead, systemtap and stuff like that, but the truth is
> that whenever I delved into that stuff, nothing useful ever came out of
> it. When IO is too slow, the faster disks are the only solution. No
> amount of tuning will turn a Ford Taurus into a Ferrari. Storage
> configuration is usually planned ahead of configuring the database. It
> may even be prudent, I apologize for using harsh language, to do some
> testing ahead of building the whole cluster. This sounds like the
> configuration done by the most interesting DBA in the world: the one who
> doesn't test his stuff often, but when he does so, he does it in production.
> --
> Mladen Gogala
> Database Consultant

Received on Wed May 09 2018 - 09:41:56 CEST

Original text of this message