Re: hugepages question

Thanks Mark, that is interesting but haven't had luck yet.

Ok one more question about memlock. In my /etc/security/limits.conf I have the following:

oracle hard memlock 237262219

oracle soft memlock 237262219

However when I run ulimits -l I get 64 returned.

ulimit -a returns:

ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals                 (-i) 1029648

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 1029648

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

> Once you allocate hugepages, it prevents you using that memory for any
> other purpose. If you're planning the use of the server, that's probably
> fine
>
> Also, to add more to hugepages without a reboot, you can (theoretically)
> do 'sudo sysctl -p' and then 'cat /proc/meminfo' to see if it fully
> allocated all the newly requested hugepages. If not all hugepages were
> allocated, you can do 'sudo sysctl -p' again, and check /proc/meminfo again.
>
> I remember, at least one time in the past, where I wanted to add a new DB,
> and I couldn't afford a bounce of the server, due to other production DBs
> on the server, where I wrote a simple script that did "grep HugePages_Total
> /proc/meminfo |awk -F: '{ print $2 }'" and kept calling 'sudo sysctl -p'
> over and over, until the output of the previous command was equal to the
> value in '/etc/sysctl.conf'. It took a few minutes, but it worked.
>
> -Mark
>
>
> On Thu, Feb 14, 2019 at 11:53 AM Jose Rodriguez <jrodriguez2_at_pythian.com>
> wrote:
>
>> I am curious to know why over allocating HP is a bad idea.
>> I mean, if you know you will end up increasing the size of an SGA or the
>> number of SGAs in the server, why not allocate HP ahead of time.
>> Save some RAM for OS, some for PGA and the rest should go to SGA, that
>> 'rest' should be in HP sooner or later, so do it sooner.
>> Does it make sense? Or am I missing something?
>>
>> [image: Pythian] <http://www.pythian.com/>
>> *Jose Rodriguez* | Oracle Project Engineer | [image: LinkedIn]
>> <https://www.linkedin.com/company/pythian>
>> *t* +1 613 565 8696 <+1+613+565+8696> *ext.* 1393
>> *m* +34 607 55 49 91 <+34+607+55+49+91>
>> jrodriguez2_at_pythian.com
>> *www.pythian.com* <https://www.pythian.com/>
>> [image: Pythian] <https://www.pythian.com/email-footer-click>
>>
>>
>> On Thu, 14 Feb 2019 at 17:19, Jeff Chirco <backseatdba_at_gmail.com> wrote:
>>
>>> Hi Andy, yes I did reboot. Oh I originally set it to 81000 and then
>>> rebooted. Yesterday I tried to increase it to 9100 without a reboot.
>>>
>>> Since it requires a reboot then that kind of is a bummer if I ever need
>>> to add another database to a server. I'd have to bring everything down to
>>> increase hugepages. I could over allocate hugepages but I read that is a
>>> bad idea.
>>>
>>> Jeff
>>>
>>> On Thu, Feb 14, 2019 at 8:05 AM Andy Wattenhofer <watt0012_at_umn.edu>
>>> wrote:
>>>
>>>> Did you reboot after the last change to hugepages?
>>>>
>>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>>> vm.nr_hugepages=*91000*
>>>>>
>>>> [...]
>>>>
>>>>> [root]# cat /proc/meminfo
>>>>> HugePages_Total: *81793*
>>>>>
>>>> HugePages_Free: 8
>>>>> HugePages_Rsvd: 7
>>>>> HugePages_Surp: 0
>>>>>
>>>>
>>>> Yes, you must reboot for 91000 to take effect. The huge page RAM is
>>>> reserved at boot time.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On Thu, Feb 14, 2019 at 9:15 AM Jeff Chirco <backseatdba_at_gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for the replies. Forgot to mention this is Oracle Linux 7
>>>>> running 12.2.0.1. These are output from my Development server but
>>>>> production is identicial except hugepage size is lower cause it has less
>>>>> database.
>>>>> Development and Production both have 256gb of RAM.
>>>>> I also noticed when I first set huegpages using the below command it
>>>>> increased hugepages but not to the amount I tried to set it to. Not until a
>>>>> server reboot did it get set the full amount. If I need to increase it
>>>>> again I would prefer to not have to reboot the server.
>>>>> # sysctl -w vm.nr_hugepages=value
>>>>>
>>>>> Regarding hugepages_settings.sh I did follow Tim's notes but I ended
>>>>> up finding an updated script under Doc ID 401749.1
>>>>>
>>>>> [root]# grep memlock /etc/security/limits.conf
>>>>> # - memlock - max locked-in-memory address space (KB)
>>>>> oracle hard memlock 237265300
>>>>> # oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit
>>>>> is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
>>>>> oracle soft memlock 237265300
>>>>>
>>>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>>> vm.nr_hugepages=91000
>>>>>
>>>>>
>>>>> [root]# ulimit -l
>>>>> 64
>>>>>
>>>>> [root]# ulimit -a
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 1029648
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 1024
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 1029648
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>>
>>>>> Alert log from one of my databases
>>>>> **********************************************************************
>>>>> 2019-02-12T21:26:58.786205-08:00
>>>>> Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
>>>>>
>>>>> 2019-02-12T21:26:58.790788-08:00
>>>>> Per process system memlock (soft) limit = 226G
>>>>> 2019-02-12T21:26:58.793885-08:00
>>>>> Expected per process system memlock (soft) limit to lock
>>>>> SHARED GLOBAL AREA (SGA) into memory: 8196M
>>>>> 2019-02-12T21:26:58.800179-08:00
>>>>> Available system pagesizes:
>>>>> 4K, 2048K
>>>>> 2019-02-12T21:26:58.806848-08:00
>>>>> Supported system pagesize(s):
>>>>> 2019-02-12T21:26:58.809846-08:00
>>>>> PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
>>>>> 2019-02-12T21:26:58.813115-08:00
>>>>> 2048K 65468 4098 4098 NONE
>>>>> 2019-02-12T21:26:58.816156-08:00
>>>>> Reason for not supporting certain system pagesizes:
>>>>> 2019-02-12T21:26:58.819060-08:00
>>>>> 4K - Large pagesizes only
>>>>> 2019-02-12T21:26:58.822058-08:00
>>>>> **********************************************************************
>>>>>
>>>>>
>>>>>
>>>>> [root]# ipcs -m
>>>>>
>>>>> ------ Shared Memory Segments --------
>>>>> key shmid owner perms bytes nattch
>>>>> status
>>>>> 0x00000000 589824 oracle 600 10485760 78
>>>>> 0x00000000 622593 oracle 600 3372220416 78
>>>>> 0x00000000 655362 oracle 600 8388608 78
>>>>> 0x76c55318 688131 oracle 600 2097152 78
>>>>> 0x00000000 11862020 oracle 600 10485760 120
>>>>> 0x00000000 11894789 oracle 600 3204448256 120
>>>>> 0x00000000 11927558 oracle 600 8388608 120
>>>>> 0x34ef509c 11960327 oracle 600 2097152 120
>>>>> 0x00000000 11993096 oracle 600 10485760 95
>>>>> 0x00000000 12025865 oracle 600 2298478592 95
>>>>> 0x00000000 12058634 oracle 600 8388608 95
>>>>> 0x507a78a4 12091403 oracle 600 2097152 95
>>>>> 0x00000000 12124172 oracle 600 10485760 207
>>>>> 0x00000000 12156941 oracle 600 6425673728 207
>>>>> 0x00000000 12189710 oracle 600 8388608 207
>>>>> 0x866b8aa0 12222479 oracle 600 2097152 207
>>>>> 0x00000000 12255248 oracle 600 12582912 81
>>>>> 0x00000000 12288017 oracle 600 8573157376 81
>>>>> 0x00000000 12320786 oracle 600 6291456 81
>>>>> 0x0c51f55c 12353555 oracle 600 2097152 81
>>>>> 0x00000000 12386324 oracle 600 12582912 79
>>>>> 0x00000000 12419093 oracle 600 8573157376 79
>>>>> 0x00000000 12451862 oracle 600 6291456 79
>>>>> 0x7afafd64 12484631 oracle 600 2097152 79
>>>>> 0x00000000 12779544 oracle 600 12582912 90
>>>>> 0x00000000 12812313 oracle 600 8573157376 90
>>>>> 0x00000000 12845082 oracle 600 6291456 90
>>>>> 0xdde48098 12877851 oracle 600 2097152 90
>>>>> 0x00000000 12910620 oracle 600 12582912 86
>>>>> 0x00000000 12943389 oracle 600 8573157376 86
>>>>> 0x00000000 12976158 oracle 600 6291456 86
>>>>> 0xbae4e74c 13008927 oracle 600 2097152 86
>>>>> 0x00000000 13303840 oracle 600 12582912 85
>>>>> 0x00000000 13336609 oracle 600 8573157376 85
>>>>> 0x00000000 13369378 oracle 600 6291456 85
>>>>> 0x4c7ac944 13402147 oracle 600 2097152 85
>>>>> 0x00000000 13434916 oracle 600 12582912 86
>>>>> 0x00000000 13467685 oracle 600 8573157376 86
>>>>> 0x00000000 13500454 oracle 600 6291456 86
>>>>> 0xf06d661c 13533223 oracle 600 2097152 86
>>>>> 0x00000000 13828136 oracle 600 12582912 87
>>>>> 0x00000000 13860905 oracle 600 8573157376 87
>>>>> 0x00000000 13893674 oracle 600 6291456 87
>>>>> 0xdb5acf68 13926443 oracle 600 2097152 87
>>>>> 0x00000000 13959212 oracle 600 12582912 87
>>>>> 0x00000000 13991981 oracle 600 8573157376 87
>>>>> 0x00000000 14024750 oracle 600 6291456 87
>>>>> 0xcdd9f634 14057519 oracle 600 2097152 87
>>>>> 0x00000000 14352432 oracle 600 10485760 102
>>>>> 0x00000000 14385201 oracle 600 6425673728 102
>>>>> 0x00000000 14417970 oracle 600 8388608 102
>>>>> 0x4bd6517c 14450739 oracle 600 2097152 102
>>>>> 0x00000000 14483508 oracle 600 10485760 116
>>>>> 0x00000000 14516277 oracle 600 6425673728 116
>>>>> 0x00000000 14549046 oracle 600 8388608 116
>>>>> 0x33ec8074 14581815 oracle 600 2097152 116
>>>>> 0x00000000 14876728 oracle 600 12582912 87
>>>>> 0x00000000 14909497 oracle 600 8573157376 87
>>>>> 0x00000000 14942266 oracle 600 6291456 87
>>>>> 0xb40541dc 14975035 oracle 600 2097152 87
>>>>> 0x00000000 1179975740 oracle 600 12582912 90
>>>>> 0x00000000 1180008509 oracle 600 8455716864 90
>>>>> 0x00000000 1180041278 oracle 600 34359738368 90
>>>>> 0x00000000 1180074047 oracle 600 123731968 90
>>>>> 0x00000000 15401024 oracle 600 12582912 91
>>>>> 0x00000000 15433793 oracle 600 8573157376 91
>>>>> 0x00000000 15466562 oracle 600 6291456 91
>>>>> 0x36329238 15499331 oracle 600 2097152 91
>>>>> 0x00000000 15532100 oracle 600 12582912 118
>>>>> 0x00000000 15564869 oracle 600 8573157376 118
>>>>> 0x00000000 15597638 oracle 600 6291456 118
>>>>> 0xa0206b88 15630407 oracle 600 2097152 118
>>>>> 0x00000000 16449608 oracle 600 12582912 86
>>>>> 0x00000000 16482377 oracle 600 8573157376 86
>>>>> 0x00000000 16515146 oracle 600 6291456 86
>>>>> 0xc402bb18 16547915 oracle 600 2097152 86
>>>>> 0x00000000 16318540 oracle 600 12582912 87
>>>>> 0x00000000 16351309 oracle 600 8573157376 87
>>>>> 0x00000000 16384078 oracle 600 6291456 87
>>>>> 0x261e431c 16416847 oracle 600 2097152 87
>>>>> 0x00000000 16187472 oracle 600 12582912 168
>>>>> 0x00000000 16220241 oracle 600 20937965568 168
>>>>> 0x00000000 16253010 oracle 600 56623104 168
>>>>> 0x35b8fa94 16285779 oracle 600 2097152 168
>>>>> 0x00000000 547127380 oracle 600 12582912 85
>>>>> 0x00000000 547160149 oracle 600 251658240 85
>>>>> 0x00000000 547192918 oracle 600 8321499136 85
>>>>> 0x00000000 547225687 oracle 600 4595712 85
>>>>> 0x9cd83398 547258456 oracle 600 40960 85
>>>>> 0x00000000 792199257 oracle 600 10485760 106
>>>>> 0x00000000 792232026 oracle 600 1644167168 106
>>>>> 0x00000000 792264795 oracle 600 4781506560 106
>>>>> 0x00000000 792297564 oracle 600 8388608 106
>>>>> 0xc78a1c8c 792330333 oracle 600 32768 106
>>>>> 0x6762e034 1180106846 oracle 600 28672 90
>>>>>
>>>>> [root]# cat /proc/meminfo
>>>>> MemTotal: 263623524 kB
>>>>> MemFree: 2169856 kB
>>>>> MemAvailable: 25248596 kB
>>>>> Buffers: 28 kB
>>>>> Cached: 68234496 kB
>>>>> SwapCached: 4392 kB
>>>>> Active: 56288772 kB
>>>>> Inactive: 32305392 kB
>>>>> Active(anon): 38139416 kB
>>>>> Inactive(anon): 28588332 kB
>>>>> Active(file): 18149356 kB
>>>>> Inactive(file): 3717060 kB
>>>>> Unevictable: 0 kB
>>>>> Mlocked: 0 kB
>>>>> SwapTotal: 16777212 kB
>>>>> SwapFree: 16732780 kB
>>>>> Dirty: 1020 kB
>>>>> Writeback: 0 kB
>>>>> AnonPages: 20491988 kB
>>>>> Mapped: 10605612 kB
>>>>> Shmem: 46232620 kB
>>>>> Slab: 2937544 kB
>>>>> SReclaimable: 2197872 kB
>>>>> SUnreclaim: 739672 kB
>>>>> KernelStack: 68544 kB
>>>>> PageTables: 1451148 kB
>>>>> NFS_Unstable: 0 kB
>>>>> Bounce: 0 kB
>>>>> WritebackTmp: 0 kB
>>>>> CommitLimit: 64832940 kB
>>>>> Committed_AS: 93046052 kB
>>>>> VmallocTotal: 34359738367 kB
>>>>> VmallocUsed: 772444 kB
>>>>> VmallocChunk: 34358900732 kB
>>>>> HardwareCorrupted: 0 kB
>>>>> AnonHugePages: 0 kB
>>>>> CmaTotal: 0 kB
>>>>> CmaFree: 0 kB
>>>>> HugePages_Total: 81793
>>>>> HugePages_Free: 8
>>>>> HugePages_Rsvd: 7
>>>>> HugePages_Surp: 0
>>>>> Hugepagesize: 2048 kB
>>>>> DirectMap4k: 514768 kB
>>>>> DirectMap2M: 41201664 kB
>>>>> DirectMap1G: 228589568 kB
>>>>>
>>>>>
>>>>> On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba_at_gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ooops sorry yes Oracle Linux 7.4
>>>>>>
>>>>>> On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh_at_gmail.com> wrote:
>>>>>>
>>>>>>> OS info might be relevant.
>>>>>>>
>>>>>>> Assuming Linux:
>>>>>>>
>>>>>>> ipcs -m
>>>>>>> cat /proc/meminfo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba_at_gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ok I just recently enabled hugepages on Production and
>>>>>>>> Development. Everything is going good but I noticed that on my development
>>>>>>>> environment which consists of a bunch of thin clones, when I remove and
>>>>>>>> recreate the same clone the database fails to create, I am getting:
>>>>>>>>
>>>>>>>> ORA-27137: unable to allocate large pages to create a shared memory segment
>>>>>>>>
>>>>>>>> I have the parameter *use_large_pages=ONLY*
>>>>>>>> So it makes sense I would get this error when there is not enough
>>>>>>>> pages. However I just removed this same database and am recreating it. I
>>>>>>>> would think that would free it up but apparently not. Does pages not get
>>>>>>>> released immediately when a database is removed? Is there something I
>>>>>>>> should run to free it? Or just not have use_large_pages=ONLY and leave it
>>>>>>>> at TRUE on Dev? I had to set it to TRUE to get this database up.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jeff
>>>>>>>>
>>>>>>>
>> --
>>
>>
>>
>>

--
http://www.freelists.org/webpage/oracle-l