Re: hugepages question

You dont necessarily need to reboot to allocate hugepages, but there is a good chance you will need to. Linux (and Windows for that matter) requires that the hugepage allocation be contiguous, so while there is a chance there is sufficient free memory for allocation outside of boot up, its pretty unlikely. I also like to set my hard and soft memlock to unlimited so I dont have to continually adjust that when I add memory to a server. Especially a VM, where adding memory is pretty easy.

On Thu, Feb 14, 2019 at 12:16 PM Andy Wattenhofer <watt0012_at_umn.edu> wrote:

> Ok, well you can see from HugePages_Free that there are not enough free
> huge pages to start another database under the 81793 limit you have active
> now. You're probably just at the cusp of having just the right amount
> reserved for all the databases on the server. So you either need to bump it
> up a smidge (and reboot) or decrease SGA in one or more databases to free
> some space. If you want you can decrease SGA_TARGET temporarily in another
> database until such time that you can get a server reboot done.
>
> Andy
>
> On Thu, Feb 14, 2019 at 10:18 AM Jeff Chirco <backseatdba_at_gmail.com>
> wrote:
>
>> Hi Andy, yes I did reboot. Oh I originally set it to 81000 and then
>> rebooted. Yesterday I tried to increase it to 9100 without a reboot.
>>
>> Since it requires a reboot then that kind of is a bummer if I ever need
>> to add another database to a server. I'd have to bring everything down to
>> increase hugepages. I could over allocate hugepages but I read that is a
>> bad idea.
>>
>> Jeff
>>
>> On Thu, Feb 14, 2019 at 8:05 AM Andy Wattenhofer <watt0012_at_umn.edu>
>> wrote:
>>
>>> Did you reboot after the last change to hugepages?
>>>
>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>> vm.nr_hugepages=*91000*
>>>>
>>> [...]
>>>
>>>> [root]# cat /proc/meminfo
>>>> HugePages_Total: *81793*
>>>>
>>> HugePages_Free: 8
>>>> HugePages_Rsvd: 7
>>>> HugePages_Surp: 0
>>>>
>>>
>>> Yes, you must reboot for 91000 to take effect. The huge page RAM is
>>> reserved at boot time.
>>>
>>> Andy
>>>
>>>
>>> On Thu, Feb 14, 2019 at 9:15 AM Jeff Chirco <backseatdba_at_gmail.com>
>>> wrote:
>>>
>>>> Thanks for the replies. Forgot to mention this is Oracle Linux 7
>>>> running 12.2.0.1. These are output from my Development server but
>>>> production is identicial except hugepage size is lower cause it has less
>>>> database.
>>>> Development and Production both have 256gb of RAM.
>>>> I also noticed when I first set huegpages using the below command it
>>>> increased hugepages but not to the amount I tried to set it to. Not until a
>>>> server reboot did it get set the full amount. If I need to increase it
>>>> again I would prefer to not have to reboot the server.
>>>> # sysctl -w vm.nr_hugepages=value
>>>>
>>>> Regarding hugepages_settings.sh I did follow Tim's notes but I ended up
>>>> finding an updated script under Doc ID 401749.1
>>>>
>>>> [root]# grep memlock /etc/security/limits.conf
>>>> # - memlock - max locked-in-memory address space (KB)
>>>> oracle hard memlock 237265300
>>>> # oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit
>>>> is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
>>>> oracle soft memlock 237265300
>>>>
>>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>> vm.nr_hugepages=91000
>>>>
>>>>
>>>> [root]# ulimit -l
>>>> 64
>>>>
>>>> [root]# ulimit -a
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 1029648
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 1024
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 1029648
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>>
>>>> Alert log from one of my databases
>>>> **********************************************************************
>>>> 2019-02-12T21:26:58.786205-08:00
>>>> Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
>>>>
>>>> 2019-02-12T21:26:58.790788-08:00
>>>> Per process system memlock (soft) limit = 226G
>>>> 2019-02-12T21:26:58.793885-08:00
>>>> Expected per process system memlock (soft) limit to lock
>>>> SHARED GLOBAL AREA (SGA) into memory: 8196M
>>>> 2019-02-12T21:26:58.800179-08:00
>>>> Available system pagesizes:
>>>> 4K, 2048K
>>>> 2019-02-12T21:26:58.806848-08:00
>>>> Supported system pagesize(s):
>>>> 2019-02-12T21:26:58.809846-08:00
>>>> PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
>>>> 2019-02-12T21:26:58.813115-08:00
>>>> 2048K 65468 4098 4098 NONE
>>>> 2019-02-12T21:26:58.816156-08:00
>>>> Reason for not supporting certain system pagesizes:
>>>> 2019-02-12T21:26:58.819060-08:00
>>>> 4K - Large pagesizes only
>>>> 2019-02-12T21:26:58.822058-08:00
>>>> **********************************************************************
>>>>
>>>>
>>>>
>>>> [root]# ipcs -m
>>>>
>>>> ------ Shared Memory Segments --------
>>>> key shmid owner perms bytes nattch status
>>>> 0x00000000 589824 oracle 600 10485760 78
>>>> 0x00000000 622593 oracle 600 3372220416 78
>>>> 0x00000000 655362 oracle 600 8388608 78
>>>> 0x76c55318 688131 oracle 600 2097152 78
>>>> 0x00000000 11862020 oracle 600 10485760 120
>>>> 0x00000000 11894789 oracle 600 3204448256 120
>>>> 0x00000000 11927558 oracle 600 8388608 120
>>>> 0x34ef509c 11960327 oracle 600 2097152 120
>>>> 0x00000000 11993096 oracle 600 10485760 95
>>>> 0x00000000 12025865 oracle 600 2298478592 95
>>>> 0x00000000 12058634 oracle 600 8388608 95
>>>> 0x507a78a4 12091403 oracle 600 2097152 95
>>>> 0x00000000 12124172 oracle 600 10485760 207
>>>> 0x00000000 12156941 oracle 600 6425673728 207
>>>> 0x00000000 12189710 oracle 600 8388608 207
>>>> 0x866b8aa0 12222479 oracle 600 2097152 207
>>>> 0x00000000 12255248 oracle 600 12582912 81
>>>> 0x00000000 12288017 oracle 600 8573157376 81
>>>> 0x00000000 12320786 oracle 600 6291456 81
>>>> 0x0c51f55c 12353555 oracle 600 2097152 81
>>>> 0x00000000 12386324 oracle 600 12582912 79
>>>> 0x00000000 12419093 oracle 600 8573157376 79
>>>> 0x00000000 12451862 oracle 600 6291456 79
>>>> 0x7afafd64 12484631 oracle 600 2097152 79
>>>> 0x00000000 12779544 oracle 600 12582912 90
>>>> 0x00000000 12812313 oracle 600 8573157376 90
>>>> 0x00000000 12845082 oracle 600 6291456 90
>>>> 0xdde48098 12877851 oracle 600 2097152 90
>>>> 0x00000000 12910620 oracle 600 12582912 86
>>>> 0x00000000 12943389 oracle 600 8573157376 86
>>>> 0x00000000 12976158 oracle 600 6291456 86
>>>> 0xbae4e74c 13008927 oracle 600 2097152 86
>>>> 0x00000000 13303840 oracle 600 12582912 85
>>>> 0x00000000 13336609 oracle 600 8573157376 85
>>>> 0x00000000 13369378 oracle 600 6291456 85
>>>> 0x4c7ac944 13402147 oracle 600 2097152 85
>>>> 0x00000000 13434916 oracle 600 12582912 86
>>>> 0x00000000 13467685 oracle 600 8573157376 86
>>>> 0x00000000 13500454 oracle 600 6291456 86
>>>> 0xf06d661c 13533223 oracle 600 2097152 86
>>>> 0x00000000 13828136 oracle 600 12582912 87
>>>> 0x00000000 13860905 oracle 600 8573157376 87
>>>> 0x00000000 13893674 oracle 600 6291456 87
>>>> 0xdb5acf68 13926443 oracle 600 2097152 87
>>>> 0x00000000 13959212 oracle 600 12582912 87
>>>> 0x00000000 13991981 oracle 600 8573157376 87
>>>> 0x00000000 14024750 oracle 600 6291456 87
>>>> 0xcdd9f634 14057519 oracle 600 2097152 87
>>>> 0x00000000 14352432 oracle 600 10485760 102
>>>> 0x00000000 14385201 oracle 600 6425673728 102
>>>> 0x00000000 14417970 oracle 600 8388608 102
>>>> 0x4bd6517c 14450739 oracle 600 2097152 102
>>>> 0x00000000 14483508 oracle 600 10485760 116
>>>> 0x00000000 14516277 oracle 600 6425673728 116
>>>> 0x00000000 14549046 oracle 600 8388608 116
>>>> 0x33ec8074 14581815 oracle 600 2097152 116
>>>> 0x00000000 14876728 oracle 600 12582912 87
>>>> 0x00000000 14909497 oracle 600 8573157376 87
>>>> 0x00000000 14942266 oracle 600 6291456 87
>>>> 0xb40541dc 14975035 oracle 600 2097152 87
>>>> 0x00000000 1179975740 oracle 600 12582912 90
>>>> 0x00000000 1180008509 oracle 600 8455716864 90
>>>> 0x00000000 1180041278 oracle 600 34359738368 90
>>>> 0x00000000 1180074047 oracle 600 123731968 90
>>>> 0x00000000 15401024 oracle 600 12582912 91
>>>> 0x00000000 15433793 oracle 600 8573157376 91
>>>> 0x00000000 15466562 oracle 600 6291456 91
>>>> 0x36329238 15499331 oracle 600 2097152 91
>>>> 0x00000000 15532100 oracle 600 12582912 118
>>>> 0x00000000 15564869 oracle 600 8573157376 118
>>>> 0x00000000 15597638 oracle 600 6291456 118
>>>> 0xa0206b88 15630407 oracle 600 2097152 118
>>>> 0x00000000 16449608 oracle 600 12582912 86
>>>> 0x00000000 16482377 oracle 600 8573157376 86
>>>> 0x00000000 16515146 oracle 600 6291456 86
>>>> 0xc402bb18 16547915 oracle 600 2097152 86
>>>> 0x00000000 16318540 oracle 600 12582912 87
>>>> 0x00000000 16351309 oracle 600 8573157376 87
>>>> 0x00000000 16384078 oracle 600 6291456 87
>>>> 0x261e431c 16416847 oracle 600 2097152 87
>>>> 0x00000000 16187472 oracle 600 12582912 168
>>>> 0x00000000 16220241 oracle 600 20937965568 168
>>>> 0x00000000 16253010 oracle 600 56623104 168
>>>> 0x35b8fa94 16285779 oracle 600 2097152 168
>>>> 0x00000000 547127380 oracle 600 12582912 85
>>>> 0x00000000 547160149 oracle 600 251658240 85
>>>> 0x00000000 547192918 oracle 600 8321499136 85
>>>> 0x00000000 547225687 oracle 600 4595712 85
>>>> 0x9cd83398 547258456 oracle 600 40960 85
>>>> 0x00000000 792199257 oracle 600 10485760 106
>>>> 0x00000000 792232026 oracle 600 1644167168 106
>>>> 0x00000000 792264795 oracle 600 4781506560 106
>>>> 0x00000000 792297564 oracle 600 8388608 106
>>>> 0xc78a1c8c 792330333 oracle 600 32768 106
>>>> 0x6762e034 1180106846 oracle 600 28672 90
>>>>
>>>> [root]# cat /proc/meminfo
>>>> MemTotal: 263623524 kB
>>>> MemFree: 2169856 kB
>>>> MemAvailable: 25248596 kB
>>>> Buffers: 28 kB
>>>> Cached: 68234496 kB
>>>> SwapCached: 4392 kB
>>>> Active: 56288772 kB
>>>> Inactive: 32305392 kB
>>>> Active(anon): 38139416 kB
>>>> Inactive(anon): 28588332 kB
>>>> Active(file): 18149356 kB
>>>> Inactive(file): 3717060 kB
>>>> Unevictable: 0 kB
>>>> Mlocked: 0 kB
>>>> SwapTotal: 16777212 kB
>>>> SwapFree: 16732780 kB
>>>> Dirty: 1020 kB
>>>> Writeback: 0 kB
>>>> AnonPages: 20491988 kB
>>>> Mapped: 10605612 kB
>>>> Shmem: 46232620 kB
>>>> Slab: 2937544 kB
>>>> SReclaimable: 2197872 kB
>>>> SUnreclaim: 739672 kB
>>>> KernelStack: 68544 kB
>>>> PageTables: 1451148 kB
>>>> NFS_Unstable: 0 kB
>>>> Bounce: 0 kB
>>>> WritebackTmp: 0 kB
>>>> CommitLimit: 64832940 kB
>>>> Committed_AS: 93046052 kB
>>>> VmallocTotal: 34359738367 kB
>>>> VmallocUsed: 772444 kB
>>>> VmallocChunk: 34358900732 kB
>>>> HardwareCorrupted: 0 kB
>>>> AnonHugePages: 0 kB
>>>> CmaTotal: 0 kB
>>>> CmaFree: 0 kB
>>>> HugePages_Total: 81793
>>>> HugePages_Free: 8
>>>> HugePages_Rsvd: 7
>>>> HugePages_Surp: 0
>>>> Hugepagesize: 2048 kB
>>>> DirectMap4k: 514768 kB
>>>> DirectMap2M: 41201664 kB
>>>> DirectMap1G: 228589568 kB
>>>>
>>>>
>>>> On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba_at_gmail.com>
>>>> wrote:
>>>>
>>>>> Ooops sorry yes Oracle Linux 7.4
>>>>>
>>>>> On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh_at_gmail.com> wrote:
>>>>>
>>>>>> OS info might be relevant.
>>>>>>
>>>>>> Assuming Linux:
>>>>>>
>>>>>> ipcs -m
>>>>>> cat /proc/meminfo
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba_at_gmail.com
>>>>>> wrote:
>>>>>>
>>>>>>> Ok I just recently enabled hugepages on Production and Development.
>>>>>>> Everything is going good but I noticed that on my development environment
>>>>>>> which consists of a bunch of thin clones, when I remove and recreate the
>>>>>>> same clone the database fails to create, I am getting:
>>>>>>>
>>>>>>> ORA-27137: unable to allocate large pages to create a shared memory segment
>>>>>>>
>>>>>>> I have the parameter *use_large_pages=ONLY*
>>>>>>> So it makes sense I would get this error when there is not enough
>>>>>>> pages. However I just removed this same database and am recreating it. I
>>>>>>> would think that would free it up but apparently not. Does pages not get
>>>>>>> released immediately when a database is removed? Is there something I
>>>>>>> should run to free it? Or just not have use_large_pages=ONLY and leave it
>>>>>>> at TRUE on Dev? I had to set it to TRUE to get this database up.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jeff
>>>>>>>
>>>>>>

-- 
Andrew W. Kerber

'If at first you dont succeed, dont take up skydiving.'


--
http://www.freelists.org/webpage/oracle-l