DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth;
        b=bdOYDYRpSVJ+2HbWq8Q9/MK3dtBrxz5XJhs+uC9J89kVdGf9jyO2ta5zuSLYv7YHJbWqcInmS511SHAhTTIjyxDOjE29JG4kVIazuNQhkCJr8PvK7Zvj1TpEWXNf7UcNNoj6Yu1fZ8Dq8Q3z4P3tKF/XYxmN1SP4UUuMNvCNi74=
Message-ID: <716f7a630709190618q12674act7642b29577659d22@mail.gmail.com>
Date: Wed, 19 Sep 2007 08:18:18 -0500
From: "Don Seiler" <don@seiler.us>
To: "Alex Gorbachev" <ag@oracloid.com>
Subject: Re: Weird database hanging
Cc: oracle-l@freelists.org
In-Reply-To: <c2213f680709181942l79088c71x7d4140109a584509@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Disposition: inline
References: <20070915071419.33C63743CB5@turing.freelists.org>
	 <009601c7f773$0d30f340$0200a8c0@Primary>
	 <716f7a630709170850w66347a8fr6f1099fbb1bda3e5@mail.gmail.com>
	 <716f7a630709170938h3403bb0dv32110c5893cf998@mail.gmail.com>
	 <716f7a630709181520p7ee8f2f4m9d4bfa7207c65d47@mail.gmail.com>
	 <c2213f680709181650y4b982fe7n2d3375a2379b0efb@mail.gmail.com>
	 <716f7a630709181902t73b910a1re2b516e9186101ca@mail.gmail.com>
	 <c2213f680709181942l79088c71x7d4140109a584509@mail.gmail.com>
Sender: oracle-l-bounce@freelists.org
Errors-to: oracle-l-bounce@freelists.org
Precedence: normal
Reply-to: don@seiler.us

On 9/18/07, Alex Gorbachev <ag@oracloid.com> wrote:
> Could you paste content of /proc/meminfo?

# cat /proc/meminfo
MemTotal:     65907748 kB
MemFree:      35313952 kB
Buffers:        246220 kB
Cached:       25186036 kB
SwapCached:          0 kB
Active:       26538432 kB
Inactive:       886364 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     65907748 kB
LowFree:      35313952 kB
SwapTotal:    33551744 kB
SwapFree:     33551744 kB
Dirty:             128 kB
Writeback:           0 kB
Mapped:       14456204 kB
Slab:           673304 kB
CommitLimit:  66505616 kB
Committed_AS: 23846764 kB
PageTables:    1554896 kB
VmallocTotal: 536870911 kB
VmallocUsed:    393092 kB
VmallocChunk: 536477779 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

>
> I'm not sure not using huge pages can cause such behavior but 16 GB
> SGA without huge pages on Linux is a killer. Normally, memory is
> managed in small 4k blocks. Each process has it's own page table where
> for each page it has location, permissions, and etc. For shared
> memory, each process will have its own entry in page table but only
> when it first accesses it. What can happen with large SGA, default
> page size and many Oracle processes - page table can actually occupy
> more space than SGA in addition to overhead of on-the-fly allocation
> of entries in page table.
>
> That being said, I'm not sure if that's relevant to your case.
> If you get qualified support engineer than s/he could hopefully see a
> clue there.

My SA said he didn't think it would be an issue.  We aren't swapping
at all.  However at this point we're willing to try ANYTHING.

>
> Another option could be to OS trace the processes with high activity
> during the hang but I'd be careful on production. On the other hand -
> it's not working anyway at those times. :)

You mean like a gdb attach?  Oracle Support also asked for this due to
my system state dumps being incomplete.

Don.

>
>
> On 9/18/07, Don Seiler <don@seiler.us> wrote:
> > 5. We are not using huge pages.  My SA says he'll set it up, since it
> > can't hurt now.  I'm reading the Puschitz tuning page about it now.  I
> > suppose shrinking the SGA or shared pool is always an option.
> >
> > One other difference between old and new servers is that I have
> > enabled the default degree of parallelism on our "warehouse" tables
> > and indexes, and set NOPARALLEL on our frontend objects.  On our old
> > hardware it was kind of random which tables or indexes had it enabled
> > and what degree was set.  Also our parallel_max_servers is 32, where
> > before it was 8.  I don't think it would hurt to bring it down again.
>
>
> --
> Alex Gorbachev, Oracle DBA Brewer, The Pythian Group
> http://www.pythian.com/blogs/author/alex http://www.oracloid.com
> BAAG party - www.BattleAgainstAnyGuess.com
>


-- 
Don Seiler
oracle: http://ora.seiler.us
ultimate: http://www.mufc.us
--
http://www.freelists.org/webpage/oracle-l