Re: Swapping Issues and ora_dbw
From: Frits Hoogland <frits.hoogland_at_gmail.com>
Date: Thu, 9 Oct 2014 19:50:42 +0200
Message-Id: <D6E07D90-8EB4-4FF6-868C-F1DD9C27B8F1_at_gmail.com>
Except for PGA, which doesn't use hugepages (at least on Linux)
>> On Wed, Oct 8, 2014 at 12:26 AM, Hilton Meyer <hilton_at_komit.co> wrote:
>> Thanks for the suggestions folks. I looked at the PGA and SGA targets and they were super high.
>>
>> I lowered them to SGA Target 4GB and PGA Target 2GB on each of the instances and restarted.
>>
>> Things look better already and I'll keep an eye on the issue.
>>
>>
>>
>> Thanks again for the help,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>> From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Hilton Meyer
>> Sent: Wednesday, October 08, 2014 8:29 AM
>> To: Mark W. Farnham; oracle-l_at_freelists.org
>>
>>
>> Subject: RE: Swapping Issues and ora_dbw
>>
>>
>> Here is the full commands and outputs:
>>
>>
>>
>> # top –c
>>
>> top - 08:15:23 up 26 days, 21:15, 4 users, load average: 1.80, 2.20, 2.55
>>
>> Tasks: 613 total, 5 running, 608 sleeping, 0 stopped, 0 zombie
>>
>> Cpu(s): 20.3%us, 3.9%sy, 0.0%ni, 68.2%id, 7.4%wa, 0.0%hi, 0.1%si, 0.0%st
>>
>> Mem: 49557132k total, 49264636k used, 292496k free, 49236k buffers
>>
>> Swap: 8257532k total, 7888592k used, 368940k free, 42721744k cached
>>
>>
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>> 49069 oradev 20 0 12.2g 10g 9g S 0.0 21.2 41:58.61 ora_dbw0_DEV
>>
>> 49073 oradev 20 0 12.2g 9g 9g S 0.0 21.1 41:58.67 ora_dbw1_DEV
>>
>> 4716 orapatch 20 0 12.2g 8.5g 8.5g S 0.0 17.9 34:21.58 ora_dbw1_PATCH
>>
>> 4712 orapatch 20 0 12.2g 8.5g 8.4g S 0.0 17.9 32:42.16 ora_dbw0_PATCH
>>
>> .
>>
>> .
>>
>> .
>>
>>
>>
>> # free -m
>>
>> total used free shared buffers cached
>>
>> Mem: 48395 48106 288 0 55 41525
>>
>> -/+ buffers/cache: 6525 41870
>>
>> Swap: 8063 7724 339
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>> From: Mark W. Farnham [mailto:mwf_at_rsiz.com]
>> Sent: Tuesday, October 07, 2014 5:40 PM
>> To: Hilton Meyer; oracle-l_at_freelists.org
>> Subject: RE: Swapping Issues and ora_dbw
>>
>>
>>
>> 1) tell us the command (some variety of ps?)
>>
>> 2) give us a head -2 of the command results so we can see the headings as well as the lines you care about
>>
>> 3) without knowing the tool, is it one that reports the combined sum of memory used by the process as well as any shared memory attached during its run? IF so, then easily up to your sga size could be a complete red herring in considering the amount of memory a process is holding
>>
>>
>>
>> From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Hilton Meyer
>> Sent: Tuesday, October 07, 2014 5:25 AM
>> To: oracle-l_at_freelists.org
>> Subject: Swapping Issues and ora_dbw
>>
>>
>>
>> Hi there,
>>
>>
>>
>> I have a test server with three instances running and I have been seeing that swapping issues.
>>
>> The server is running out of swap space, been flirting on the edge of 0.
>>
>> I dug a little deeper an found that the main culprits are two of the instances:
>>
>>
>>
>> 49069 oradev 20 0 12.2g 10g 10g S 0.0 21.5 40:44.19 ora_dbw0_DEV
>>
>> 49073 oradev 20 0 12.2g 10g 10g S 0.0 21.5 40:43.90 ora_dbw1_DEV
>>
>> 4716 orapatch 20 0 12.2g 8.5g 8.5g S 0.0 17.9 33:20.19 ora_dbw1_PATCH
>>
>> 4712 orapatch 20 0 12.2g 8.4g 8.4g S 0.0 17.9 31:45.49 ora_dbw0_PATCH
>>
>>
>>
>> The weird thing is the third instance does not register although they are the same clone from the Production instance.
>>
>> Few questions:
>>
>> · Is this not an unusually high percentage of memory for the process to hold?
>>
>> · Is there a way to release the memory back to the server?
>>
>> · Is there a way for me to dig deeper to find out why these two instances are holding so much memory?
>>
>>
>>
>>
>>
>> Sys Info:
>>
>> EBS R12.1.3
>>
>> Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
>>
>> Linux 2.6.39-400.210.2.el5uek
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>
Date: Thu, 9 Oct 2014 19:50:42 +0200
Message-Id: <D6E07D90-8EB4-4FF6-868C-F1DD9C27B8F1_at_gmail.com>
Except for PGA, which doesn't use hugepages (at least on Linux)
Frits Hoogland
http://fritshoogland.wordpress.com
frits.hoogland_at_gmail.com
Office : +31 20 5939953
Mobile: +31 6 14180860
(Sent from my iPhone, typo's are expected)
> Op 9 okt. 2014 om 19:39 heeft Joshua Collier <jcoll1970_at_gmail.com> het volgende geschreven: > > look into enabling huge pages for your system. it will eliminate all possibility of swapping when properly configured. with this OS Linux 2.6.39-400.210.2.el5uek you should have huge pages set up. > > How much ram on the server total? >
>> On Wed, Oct 8, 2014 at 12:26 AM, Hilton Meyer <hilton_at_komit.co> wrote:
>> Thanks for the suggestions folks. I looked at the PGA and SGA targets and they were super high.
>>
>> I lowered them to SGA Target 4GB and PGA Target 2GB on each of the instances and restarted.
>>
>> Things look better already and I'll keep an eye on the issue.
>>
>>
>>
>> Thanks again for the help,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>> From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Hilton Meyer
>> Sent: Wednesday, October 08, 2014 8:29 AM
>> To: Mark W. Farnham; oracle-l_at_freelists.org
>>
>>
>> Subject: RE: Swapping Issues and ora_dbw
>>
>>
>> Here is the full commands and outputs:
>>
>>
>>
>> # top –c
>>
>> top - 08:15:23 up 26 days, 21:15, 4 users, load average: 1.80, 2.20, 2.55
>>
>> Tasks: 613 total, 5 running, 608 sleeping, 0 stopped, 0 zombie
>>
>> Cpu(s): 20.3%us, 3.9%sy, 0.0%ni, 68.2%id, 7.4%wa, 0.0%hi, 0.1%si, 0.0%st
>>
>> Mem: 49557132k total, 49264636k used, 292496k free, 49236k buffers
>>
>> Swap: 8257532k total, 7888592k used, 368940k free, 42721744k cached
>>
>>
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>> 49069 oradev 20 0 12.2g 10g 9g S 0.0 21.2 41:58.61 ora_dbw0_DEV
>>
>> 49073 oradev 20 0 12.2g 9g 9g S 0.0 21.1 41:58.67 ora_dbw1_DEV
>>
>> 4716 orapatch 20 0 12.2g 8.5g 8.5g S 0.0 17.9 34:21.58 ora_dbw1_PATCH
>>
>> 4712 orapatch 20 0 12.2g 8.5g 8.4g S 0.0 17.9 32:42.16 ora_dbw0_PATCH
>>
>> .
>>
>> .
>>
>> .
>>
>>
>>
>> # free -m
>>
>> total used free shared buffers cached
>>
>> Mem: 48395 48106 288 0 55 41525
>>
>> -/+ buffers/cache: 6525 41870
>>
>> Swap: 8063 7724 339
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>> From: Mark W. Farnham [mailto:mwf_at_rsiz.com]
>> Sent: Tuesday, October 07, 2014 5:40 PM
>> To: Hilton Meyer; oracle-l_at_freelists.org
>> Subject: RE: Swapping Issues and ora_dbw
>>
>>
>>
>> 1) tell us the command (some variety of ps?)
>>
>> 2) give us a head -2 of the command results so we can see the headings as well as the lines you care about
>>
>> 3) without knowing the tool, is it one that reports the combined sum of memory used by the process as well as any shared memory attached during its run? IF so, then easily up to your sga size could be a complete red herring in considering the amount of memory a process is holding
>>
>>
>>
>> From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Hilton Meyer
>> Sent: Tuesday, October 07, 2014 5:25 AM
>> To: oracle-l_at_freelists.org
>> Subject: Swapping Issues and ora_dbw
>>
>>
>>
>> Hi there,
>>
>>
>>
>> I have a test server with three instances running and I have been seeing that swapping issues.
>>
>> The server is running out of swap space, been flirting on the edge of 0.
>>
>> I dug a little deeper an found that the main culprits are two of the instances:
>>
>>
>>
>> 49069 oradev 20 0 12.2g 10g 10g S 0.0 21.5 40:44.19 ora_dbw0_DEV
>>
>> 49073 oradev 20 0 12.2g 10g 10g S 0.0 21.5 40:43.90 ora_dbw1_DEV
>>
>> 4716 orapatch 20 0 12.2g 8.5g 8.5g S 0.0 17.9 33:20.19 ora_dbw1_PATCH
>>
>> 4712 orapatch 20 0 12.2g 8.4g 8.4g S 0.0 17.9 31:45.49 ora_dbw0_PATCH
>>
>>
>>
>> The weird thing is the third instance does not register although they are the same clone from the Production instance.
>>
>> Few questions:
>>
>> · Is this not an unusually high percentage of memory for the process to hold?
>>
>> · Is there a way to release the memory back to the server?
>>
>> · Is there a way for me to dig deeper to find out why these two instances are holding so much memory?
>>
>>
>>
>>
>>
>> Sys Info:
>>
>> EBS R12.1.3
>>
>> Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
>>
>> Linux 2.6.39-400.210.2.el5uek
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Hilton Meyer
>>
>> Database Administrator
>>
>>
>>
>> Phone: +972-52-7908599
>>
>> Office: +972-52-7908600 Fax: 1535-27908600
>>
>> hilton_at_komit.co | www.komit.co
>>
>>
>>
>> <image001.png>
>>
>>
>>
>
-- http://www.freelists.org/webpage/oracle-lReceived on Thu Oct 09 2014 - 19:50:42 CEST