RE: OT - Blog entry on hugepages

From: CRISLER, JON A <JC1706_at_att.com>
Date: Thu, 5 Apr 2012 22:23:23 +0000
Message-ID: <9F15274DDC89C24387BE933E68BE3FD3218BF7_at_MISOUT7MSGUSR9D.ITServices.sbc.com>



A round-about way would be to set up the /etc/security/limits.conf file so that only the "oracle" user has rights to hugepages:
  • this sets up permission to use about 80gb of memory, but does not actually allocate anything oracle hard memlock 83886080 oracle soft memlock 83886080

log out / back in, shutdown and restart oracle, then check /proc/meminfo.

Example-

HugePages_Total: 25000
HugePages_Free:   6092
HugePages_Rsvd:   5669

What I don't get is why some memory is Rsvd but not used sometimes: probably some pool has not grown much and used the memory.

Are you using Oracle 11g AMM ? If so you need to unset memory_max_size and memory_target, otherwise they try to use /dev/shm and will not use hugepages. On Linux, AMM and hugepages are mutually exclusive, but you can use both on AIX and Solaris I believe. In other words, on Oracle 11g, in order to use hugepages you have to revert to 10g style memory management. In my experience for larger systems (> 10gb SGA) this is a worthwhile tradeoff for better performance.

-----Original Message-----
From: Walker, Jed S [mailto:Jed_Walker_at_cable.comcast.com] Sent: Thursday, April 05, 2012 5:44 PM
To: CRISLER, JON A; oracle-l_at_freelists.org Subject: RE: OT - Blog entry on hugepages

Yes, I can see that hugepages is active via /proc/meminfo, I'm asking how to actually tell if it really is Oracle using it. That is why I'm asking about ipcs.

Mark sent this link which I'll read: http://yong321.freeshell.org/oranotes/HugePages.txt

-----Original Message-----
From: CRISLER, JON A [mailto:JC1706_at_att.com] Sent: Thursday, April 05, 2012 3:28 PM
To: Walker, Jed S; oracle-l_at_freelists.org Subject: RE: OT - Blog entry on hugepages

I am only familier with this on Linux- this does not seem to prove or disprove hugepages, only shared memory segments right? To look at hugepages on Linux, do a cat /proc/meminfo- other OS's require different methods, and the default hugepage size in Linux is 2mb.

-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Walker, Jed S Sent: Thursday, April 05, 2012 4:28 PM
To: oracle-l_at_freelists.org
Subject: RE: OT - Blog entry on hugepages

The Oracle documents say you can tell hugepages are being used by the following:

      To make sure that the configuration is valid, the HugePages_Free value should be smaller than HugePages_Total and there should be some HugePages_Rsvd. The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your total combined SGA as instances allocate pages dynamically and proactively as needed. This does seem to indicate they are being used, but I was wondering if there's any other way to truly see it. I'm wondering if anyone can confirm my thinking on this. I used

>ipcs

  • Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 29097988 oracle 640 4096 0 0x00000000 29130757 oracle 640 4096 0 0x0e2f3c64 29163526 oracle 640 4096 0 0x00000000 29655047 oracle 640 268435456 59 0x00000000 29687816 oracle 640 51271172096 59 0x2894b058 29720585 oracle 640 2097152 59

I could be wrong, but 51271172096/1024/1024/1000H (since pages are in kbytes, this would seem to be my 48G SGA). I'm not sure what the one above it is and it is big enough to make me wonder.

Does that look like it makes sense?

(Still wish someone had some test results to show it really helps)

Thanks,

Jed

-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Rich Jesse Sent: Wednesday, April 04, 2012 3:10 PM
To: oracle-l_at_freelists.org
Subject: Re: OT - Blog entry on hugepages

Hey Kyle,

> Does anyone have metrics to identify memory access problems on AIX
> other than paging and scanning stats that might be solved with largepages?
> Recently an AIX customer was seeing slow I/Os from an NFS filer but
> fast local I/O response times. Of course the customer blamed the NFS
> filer, but the filer reported constant speeds during good periods as
> well as bad. I

Was CPU consistent between the tests?

Does the init.ora filesystemio_options='SETALL'? I'm not sure how that affects the NFS mount, but for JFS2 mounts, that will cause Oracle to automagically use CIO (Ora10g and up) and also AIO, if enabled, IIRC. Oracle does this regardless of the mount options specified in /etc/filesystems, again for JFS2.

I'm theorizing that if CIO's not used, there could be Oracle data files in the filecache of the local storage, which could result in higher CPU while still appearing as PIO to Oracle. This would also depend on the VMO filecache settings of minperm/maxperm/maxclient. I watch the filecache constantly in AIX's excellent nmon in the memory window, as well as a vmstat monitoring script containing:

vmstat -v|grep -E 'numperm|file pages'

Hope this drivel helps! I don't claim to be an expert in this, but have been delving into this stuff now to help me determine how big I can make my buffer cache w/o causing AIX (v5.3) paging.

GL!

Rich

--
http://www.freelists.org/webpage/oracle-l




--
http://www.freelists.org/webpage/oracle-l


--
http://www.freelists.org/webpage/oracle-l
Received on Thu Apr 05 2012 - 17:23:23 CDT

Original text of this message