RE: OT - Blog entry on hugepages

From: Yong Huang <yong321_at_yahoo.com>
Date: Fri, 6 Apr 2012 11:13:25 -0700 (PDT)
Message-ID: <1333736005.87957.YahooMailClassic_at_web181210.mail.ne1.yahoo.com>



> HugePages_Total: 25000
> HugePages_Free: 6092
> HugePages_Rsvd: 5669
>
> What I don't get is why some memory is Rsvd but not used sometimes:
> probably some pool has not grown much and used the memory.

I agree. The reserved pages will be gradually used in the lifetime of the Oracle instance. In your case, you'll have 6092-5669=423 pages (846 MB if 2M page size) that will be wasted, i.e. never be used. I would decrease that down to at most 10 pages. On Linux, you can do it without shutting down anything by "echo 24587 > /proc/sys/vm/nr_hugepages" (24587 is 25000 minus 413). Of course, echo'ing a number bigger than already configured (25000 here) may not work unless the server was just rebooted.

Here's an easy way to understand the HugePages entries in /proc/meminfo.

UUUUUFFFF <-- Total split into really used (U) and free (F) UUUUURRR. <-- Total split into really used (U), reserved (R) and really free (.)

If one letter or dot is one HugePage, the above says

HugePages_Total: 9
HugePages_Free:  4
HugePages_Rsvd:  3

and you'll have 4-3=1 page completely wasted.

> I was wondering if there's any other way to truly see it [HugePages being used].

`strace -f -e trace=process sqlplus / as sysdba' and startup Oracle. Look for SHM_HUGETLB in 3rd arg to shmget(). You can also save it to a file to search for it (-o filename). Unfortunately, you can only see the HUGETLB tag during instance startup, not after.

I just set up HugePages on my Red Hat 6 running Oracle. /proc/<pid>/smaps has the page size info:

# cat /proc/<any pid of Oracle instance>/smaps ...

60000000-61000000 rwxs 00000000 00:0c 1114116                            /SYSV00000000 (deleted)
Size:              16384 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:     2048 kB
MMUPageSize:        2048 kB

[repeated for the other 2 shared memory segments]

You see the KernelPageSize and MMUPageSize with 2M page size. All other map entries show 2K page size. So now you have one more way to verify. But as of today, only Oracle Linux is certified and I don't have it to test. If you really want, I think you can apply the smaps Linux patch (see http://lkml.indiana.edu/hypermail/linux/kernel/0810.2/0226.html) to have that info in smaps. In the future, we may have a -s option for pmap on Linux to show page size just like that on Solaris (see http://www.freelists.org/post/procps/RFE-Show-the-page-size-used-for-an-allocation-in-the-pmap).

Yong Huang

--
http://www.freelists.org/webpage/oracle-l
Received on Fri Apr 06 2012 - 13:13:25 CDT

Original text of this message