Re: [OT] HOWTO's of messing up Linux cluster?

On Sat, 20 Nov 2004 18:30:12 -0500, Branimir Petrovic <branimirp_at_cpas.com> wrote:
> Situation:
> Early stages of development project; same small database runs on
> development PCs, early build is given to customer for "look and
> feel" and training purposes.
>
> Sad facts:
> "PC Vulgaris" (P4 @ 2.8 GHz with _one_ IDE drive running WinXP)
> /w Oracle 9.2 db outperforms by factor of FIVE (meaning exact same
> batch jobs run at least 5 TIMES FASTER!!!???) two node Oracle 9.2
> cluster on Linux (RH AS3). Beats it by huge margin it every single
> time.
>
> Hardware at customer's site is 6 month old (implying relatively
> new hardware) 2 P4-Xeon CPUs server forming two node Oracle 9.2
> cluster connected to dedicated (this cluster only, dedicated
> to this project only) SAN (with unknown number of drives).
>
> Question:
> If you were really incompetent SA/DBA what would be easy ways
> to duplicate the above requirement? (achieve five times
> worse performance using much "stronger" hardware)
>
> From whatever little I know of ways problematic database is set
> up and configured, I'd say it is not the Oracle that is messed up,
> but the underlying OS.
>
> Being at arms length from the problem (due to mountains of political
> bs. - "us" vs. "them"), and quite frankly due to my own shortcomings
> with Unix/Linux, I am in no position to help in any meaningful way.
> Still curiosity factor is strong - what could possibly be so wrong
> with this RAC?
>
> Wild guesses - welcomed :-)
>
> Branimir
> --
> http://www.freelists.org/webpage/oracle-l
>

I'd start with two approaches.

1. Define what is 5 times faster and compare 10046 level 8 traces - I believe this has been suggested already.
2. List important differences. From what you have said already we have

RAC vs NON-RAC.
SAN vs Local Storage.
Different OS (though I'd suspect this proves insignificant). Networked app instead of all on one box. Oracle configuration

Ignoring the OS difference for the moment (in my experience poor OS setup rarely accounts for more than 20% or so performance loss certainly not 5 times) some things you might do.

Remove RAC from the equation. (Cynics might suggest and keep it removed :)) Investigate the SAN setup (RAID levels - bad drivers for the cards etc).

Remove the network from the equation. Trace files from my most excellent 3rd party apps nearly always (like 90% of the time) show SQL*Net message from client as the biggest consumer of response time. Some of really is user 'think time' but frequently its far to many round trips between the client and the DB. I believe that others have found similar results.

Sanity check the Oracle config.

-- 
Niall Litchfield
Oracle DBA
http://www.niall.litchfield.dial.pipex.com
--
http://www.freelists.org/webpage/oracle-l