Re: RAC nodes regularly freezes for about 10 seconds

Marcin Szarek wrote:
> Hi!
>
> For a few months we suffer mysterious problem with Oracle 10g RAC (more
> details on server configuration at the bottom). At regular basis (every 5
> minutes) nodes of our cluster "freeze" - during couple of seconds
> operating system for some mysterious reason does nothing - as far as we're
> concerned - in userspace. Every single userspace process stops for this
> period. After a few seconds system comes back to life and all suspended
> processes content for CPU time and other resources, which in effect leads
> to higher load.
>
> We tried many investigations, which brougt us to following conclusions:
> - when Oracle instance is stopped, freezes disappear
> - since the moment we reduced shared_servers parameter from 60 to 20
> freezes last unsignificantly shorter (about 4 seconds shorter)
> - after instance restart freezes are unnoticeable, but as times
> goes by, they are again as long as 6-9 seconds
>
> Unfortunately investigation is very hard. /var/log/messages reports
> nothing, dmesg reports nothing, Oracle alert log also has nothing to say.
>
> Have you any idea what may be misconfigured or damaged? Could you please
> suggest us some further tests?
>
>
> Thank you in advance for any followups!
>
>
>
> Database server characteristics
> -------------------------------
> OS: RHEL 3 ES
> Kernel: 2.4.21-32.ELsmp
> Oracle: 10.2.0.1.0
> Storage: SAN accessed by QLogic HBA
> Cluster storage: OCFS v.1

A couple of thoughts the first being that HPUXRAC's advice is good and should be followed. But additionally I have seen this behaviour before when working with RAC clusters involving QLogic. I can't say that it is caused by QLogic, my trace led a different direction, but the fact that you are seeing the same behaviour with a different operating system and different version of Oracle (I saw it with 10.1.0.3) makes it worth keeping in mind.

If you have a copy of the Grid Control run it. We diagnosed our issue with the Grid far more easily than we could have with any other tool. If not run some very frequent stats packs and see if you get lucky.

-- 
Daniel A. Morgan
University of Washington
damorgan_at_x.washington.edu
(replace x with u to respond)
Puget Sound Oracle Users Group
www.psoug.org