DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth;
        b=NX82VZLe12U6JSijFq4Axy7xaZJbvlYa0Y/rhVr4LH47YHyTV5bgSuEIF/Y/28qDcTVUa0INgQSRkm3fASEo7j3qCiT+ubQ24PBMnvNggSjldiRcg7YU7YACq98sIKmBGb4E6wTTmwcciufQOjemgHqwrnH8XBS/iIpqJGCBNSk=
Message-ID: <716f7a630709211353o628b1ed9pc603ffa90c0b53e3@mail.gmail.com>
Date: Fri, 21 Sep 2007 15:53:01 -0500
From: "Don Seiler" <don@seiler.us>
To: oracle-l@freelists.org
Subject: Re: Weird database hanging
In-Reply-To: <9f0e18730709211302m3aa71e29h2ab0673b55a6d4b7@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Disposition: inline
References: <20070915071419.33C63743CB5@turing.freelists.org>
	 <009601c7f773$0d30f340$0200a8c0@Primary>
	 <716f7a630709170850w66347a8fr6f1099fbb1bda3e5@mail.gmail.com>
	 <716f7a630709170938h3403bb0dv32110c5893cf998@mail.gmail.com>
	 <716f7a630709181520p7ee8f2f4m9d4bfa7207c65d47@mail.gmail.com>
	 <c2213f680709181650y4b982fe7n2d3375a2379b0efb@mail.gmail.com>
	 <716f7a630709181902t73b910a1re2b516e9186101ca@mail.gmail.com>
	 <c2213f680709181942l79088c71x7d4140109a584509@mail.gmail.com>
	 <716f7a630709200657t5a79cbc0q6a1fb1281f087cde@mail.gmail.com>
	 <9f0e18730709211302m3aa71e29h2ab0673b55a6d4b7@mail.gmail.com>
Sender: oracle-l-bounce@freelists.org
Errors-to: oracle-l-bounce@freelists.org
Precedence: normal
Reply-to: don@seiler.us

We *think* we have found the issue, and it isn't quite Oracle-related
(of course).

The SA had been doing a Veritas online relayout on the disk partition
that is our archivelog destination.  He aborted it, but rather than
aborting, Veritas left it in a "paused" state.  This happend 20
minutes before the bulk load that caused our first instance hang.
Note that we *were* able to archive logs, it just seemed to have
caused some more waiting than normal.  This was compounded during bulk
loads, and in the end caused a crush of shared pool and library cache
latches.

This situation was discovered yesterday and the times seemed all too
coincidental.  The state was corrected and we've been happily bulk
loading anything and everything since then.

In the end, we recognize there is plenty of room for improvement in
the application code (and horrible inefficiencies in the app database
design), but were quite certain that wasn't the root cause of this
problem.  I'm still pretty upset with Oracle support over their
blinders and insistence that the problem was "properly diagnosed" and
ignored all of my input and feedback.

Don.

-- 
Don Seiler
oracle: http://ora.seiler.us
ultimate: http://www.mufc.us
--
http://www.freelists.org/webpage/oracle-l