Re: kernel memory keeps increasing and running out of memory (Sun T4, Solaris 10)
Date: Mon, 25 Feb 2013 06:05:55 +0200
Message-ID: <CAMHX9JL=129j9YxC2z4Q7pJHM40QQojWnxrS_XzqR747+EUaMQ_at_mail.gmail.com>
As your memory allocation seems to be increasing, then taking strack traces when kmem_alloc function is called would help to pinpoint who & when is allocating the memory.
For example, something like this (although you may want to modify it to kick in for any process):
#!/usr/sbin/dtrace -qs
#pragma D option quiet
fbt::kmem_*alloc:entry,
fbt::bkmem_alloc:entry
/execname == "oracle" && curpsinfo->pr_psargs == "oracleSOL102 (LOCAL=NO)"
/
{
_at_kmem[probefunc,stack()] = sum(arg0) }
dtrace:::END
{
printf("\n%-50s %10s time=%Y\n", "MEM_DELTA", "KB", walltimestamp); printa("%-20s %-50k %10_at_d\n", @kmem); }
(but the process/stack who allocates the most over time might not be the troublemaker (as if you always release what you use, there will be no leaking).
So you might want to measure which process keeps allocating more and more without releasing it all back. However, it's non-trivial to measure who releases memory back, as sometimes you might end up with "lazy release" in some cases (you can easily track allocations, but who/when frees the memory once it's not used can be harder to find out - if there's no 1-1 relationship between the alloc/free routine). But maybe this helps:
fbt::allocb:entry
/execname == "oracle"/
{
_at_kmem[curpsinfo->pr_psargs] = sum(arg0) }
fbt::bkmem_free:entry
/execname == "oracle"/
{
_at_kmem[curpsinfo->pr_psargs] = sum(-arg1) }
dtrace:::END, profile:::tick-5s
{
printf("\n%-50s %10s time=%Y\n", "MEM_DELTA", "KB", walltimestamp);
printa("%-50s %10_at_d\n", @kmem);
}
If some specific stack trace pops up in the top, then search for the top few functions in MOS (and google :)
-- *Tanel Poder* Enkitec (The Exadata Experts) Training <http://blog.tanelpoder.com/seminar/> | Troubleshooting<http://blog.tanelpoder.com/> | Exadata<http://www.amazon.com/Expert-Oracle-Exadata-Apress/dp/1430233923> | Voicee App <http://voic.ee/> On Mon, Feb 25, 2013 at 4:53 AM, Eagle Fan <eagle.f_at_gmail.com> wrote: > HI: > We are seeing kernel memory of the database servers keep increasing and run > out of memory. > > Server is Sun T4 (2 CPU, 128G memory), solaris version is "Oracle Solaris > 10 8/11 s10s_u10wos_17b SPARC" > > Oracle version are 10204 or 11203, running in shared server mode. > > From kmastat result, here are the top used components. > > cache buf buf buf memory alloc alloc > > name size in use total in use succeed fail > > ------------------------- ------ ------ ------ --------- --------- ----- > > streams_dblk_1040 1152 179382 186144 217841664 3232457924 0 > > kmem_alloc_8192 8192 27694 28262 231522304 57758203 0 > > kmem_alloc_131072 131072 1229 2023 265158656 175185 0 > > kmem_va_24576 24576 20875 20940 548929536 191185 0 > > kmem_va_8192 8192 71853 72608 594804736 239095 0 > > anon_cache 48 14381045 14427868 699367424 3946654430 > 0 > > streams_dblk_12112 12224 56091 57000 700416000 82708937 0 > > streams_dblk_1936 2048 887941 893324 1829527552 2303442589 0 > > streams_dblk_36688 36800 112239 112378 4142702592 923831397 0 > > Does anybody know what streams_dblk is used for? I tried to google for it > but didn't get any useful information. > > The attachment is detail information of mdb before and after server > restart. > > Thanks. > > -- > Eagle Fan (www.dbafan.com) > > > > -- > http://www.freelists.org/webpage/oracle-l > > > -- http://www.freelists.org/webpage/oracle-lReceived on Mon Feb 25 2013 - 05:05:55 CET