Re: hugepages question

Hi Andy, yes I did reboot. Oh I originally set it to 81000 and then rebooted. Yesterday I tried to increase it to 9100 without a reboot.

Since it requires a reboot then that kind of is a bummer if I ever need to add another database to a server. I'd have to bring everything down to increase hugepages. I could over allocate hugepages but I read that is a bad idea.

Jeff

On Thu, Feb 14, 2019 at 8:05 AM Andy Wattenhofer <watt0012_at_umn.edu> wrote:

> Did you reboot after the last change to hugepages?
>
> [root]# grep nr_hugepages /etc/sysctl.conf
>> vm.nr_hugepages=*91000*
>>
> [...]
>
>> [root]# cat /proc/meminfo
>> HugePages_Total: *81793*
>>
> HugePages_Free: 8
>> HugePages_Rsvd: 7
>> HugePages_Surp: 0
>>
>
> Yes, you must reboot for 91000 to take effect. The huge page RAM is
> reserved at boot time.
>
> Andy
>
>
> On Thu, Feb 14, 2019 at 9:15 AM Jeff Chirco <backseatdba_at_gmail.com> wrote:
>
>> Thanks for the replies. Forgot to mention this is Oracle Linux 7 running
>> 12.2.0.1. These are output from my Development server but production is
>> identicial except hugepage size is lower cause it has less database.
>> Development and Production both have 256gb of RAM.
>> I also noticed when I first set huegpages using the below command it
>> increased hugepages but not to the amount I tried to set it to. Not until a
>> server reboot did it get set the full amount. If I need to increase it
>> again I would prefer to not have to reboot the server.
>> # sysctl -w vm.nr_hugepages=value
>>
>> Regarding hugepages_settings.sh I did follow Tim's notes but I ended up
>> finding an updated script under Doc ID 401749.1
>>
>> [root]# grep memlock /etc/security/limits.conf
>> # - memlock - max locked-in-memory address space (KB)
>> oracle hard memlock 237265300
>> # oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit is
>> maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
>> oracle soft memlock 237265300
>>
>> [root]# grep nr_hugepages /etc/sysctl.conf
>> vm.nr_hugepages=91000
>>
>>
>> [root]# ulimit -l
>> 64
>>
>> [root]# ulimit -a
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 1029648
>> max locked memory (kbytes, -l) 64
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 1024
>> pipe size (512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 8192
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 1029648
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>>
>>
>> Alert log from one of my databases
>> **********************************************************************
>> 2019-02-12T21:26:58.786205-08:00
>> Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
>>
>> 2019-02-12T21:26:58.790788-08:00
>> Per process system memlock (soft) limit = 226G
>> 2019-02-12T21:26:58.793885-08:00
>> Expected per process system memlock (soft) limit to lock
>> SHARED GLOBAL AREA (SGA) into memory: 8196M
>> 2019-02-12T21:26:58.800179-08:00
>> Available system pagesizes:
>> 4K, 2048K
>> 2019-02-12T21:26:58.806848-08:00
>> Supported system pagesize(s):
>> 2019-02-12T21:26:58.809846-08:00
>> PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
>> 2019-02-12T21:26:58.813115-08:00
>> 2048K 65468 4098 4098 NONE
>> 2019-02-12T21:26:58.816156-08:00
>> Reason for not supporting certain system pagesizes:
>> 2019-02-12T21:26:58.819060-08:00
>> 4K - Large pagesizes only
>> 2019-02-12T21:26:58.822058-08:00
>> **********************************************************************
>>
>>
>>
>> [root]# ipcs -m
>>
>> ------ Shared Memory Segments --------
>> key shmid owner perms bytes nattch status
>> 0x00000000 589824 oracle 600 10485760 78
>> 0x00000000 622593 oracle 600 3372220416 78
>> 0x00000000 655362 oracle 600 8388608 78
>> 0x76c55318 688131 oracle 600 2097152 78
>> 0x00000000 11862020 oracle 600 10485760 120
>> 0x00000000 11894789 oracle 600 3204448256 120
>> 0x00000000 11927558 oracle 600 8388608 120
>> 0x34ef509c 11960327 oracle 600 2097152 120
>> 0x00000000 11993096 oracle 600 10485760 95
>> 0x00000000 12025865 oracle 600 2298478592 95
>> 0x00000000 12058634 oracle 600 8388608 95
>> 0x507a78a4 12091403 oracle 600 2097152 95
>> 0x00000000 12124172 oracle 600 10485760 207
>> 0x00000000 12156941 oracle 600 6425673728 207
>> 0x00000000 12189710 oracle 600 8388608 207
>> 0x866b8aa0 12222479 oracle 600 2097152 207
>> 0x00000000 12255248 oracle 600 12582912 81
>> 0x00000000 12288017 oracle 600 8573157376 81
>> 0x00000000 12320786 oracle 600 6291456 81
>> 0x0c51f55c 12353555 oracle 600 2097152 81
>> 0x00000000 12386324 oracle 600 12582912 79
>> 0x00000000 12419093 oracle 600 8573157376 79
>> 0x00000000 12451862 oracle 600 6291456 79
>> 0x7afafd64 12484631 oracle 600 2097152 79
>> 0x00000000 12779544 oracle 600 12582912 90
>> 0x00000000 12812313 oracle 600 8573157376 90
>> 0x00000000 12845082 oracle 600 6291456 90
>> 0xdde48098 12877851 oracle 600 2097152 90
>> 0x00000000 12910620 oracle 600 12582912 86
>> 0x00000000 12943389 oracle 600 8573157376 86
>> 0x00000000 12976158 oracle 600 6291456 86
>> 0xbae4e74c 13008927 oracle 600 2097152 86
>> 0x00000000 13303840 oracle 600 12582912 85
>> 0x00000000 13336609 oracle 600 8573157376 85
>> 0x00000000 13369378 oracle 600 6291456 85
>> 0x4c7ac944 13402147 oracle 600 2097152 85
>> 0x00000000 13434916 oracle 600 12582912 86
>> 0x00000000 13467685 oracle 600 8573157376 86
>> 0x00000000 13500454 oracle 600 6291456 86
>> 0xf06d661c 13533223 oracle 600 2097152 86
>> 0x00000000 13828136 oracle 600 12582912 87
>> 0x00000000 13860905 oracle 600 8573157376 87
>> 0x00000000 13893674 oracle 600 6291456 87
>> 0xdb5acf68 13926443 oracle 600 2097152 87
>> 0x00000000 13959212 oracle 600 12582912 87
>> 0x00000000 13991981 oracle 600 8573157376 87
>> 0x00000000 14024750 oracle 600 6291456 87
>> 0xcdd9f634 14057519 oracle 600 2097152 87
>> 0x00000000 14352432 oracle 600 10485760 102
>> 0x00000000 14385201 oracle 600 6425673728 102
>> 0x00000000 14417970 oracle 600 8388608 102
>> 0x4bd6517c 14450739 oracle 600 2097152 102
>> 0x00000000 14483508 oracle 600 10485760 116
>> 0x00000000 14516277 oracle 600 6425673728 116
>> 0x00000000 14549046 oracle 600 8388608 116
>> 0x33ec8074 14581815 oracle 600 2097152 116
>> 0x00000000 14876728 oracle 600 12582912 87
>> 0x00000000 14909497 oracle 600 8573157376 87
>> 0x00000000 14942266 oracle 600 6291456 87
>> 0xb40541dc 14975035 oracle 600 2097152 87
>> 0x00000000 1179975740 oracle 600 12582912 90
>> 0x00000000 1180008509 oracle 600 8455716864 90
>> 0x00000000 1180041278 oracle 600 34359738368 90
>> 0x00000000 1180074047 oracle 600 123731968 90
>> 0x00000000 15401024 oracle 600 12582912 91
>> 0x00000000 15433793 oracle 600 8573157376 91
>> 0x00000000 15466562 oracle 600 6291456 91
>> 0x36329238 15499331 oracle 600 2097152 91
>> 0x00000000 15532100 oracle 600 12582912 118
>> 0x00000000 15564869 oracle 600 8573157376 118
>> 0x00000000 15597638 oracle 600 6291456 118
>> 0xa0206b88 15630407 oracle 600 2097152 118
>> 0x00000000 16449608 oracle 600 12582912 86
>> 0x00000000 16482377 oracle 600 8573157376 86
>> 0x00000000 16515146 oracle 600 6291456 86
>> 0xc402bb18 16547915 oracle 600 2097152 86
>> 0x00000000 16318540 oracle 600 12582912 87
>> 0x00000000 16351309 oracle 600 8573157376 87
>> 0x00000000 16384078 oracle 600 6291456 87
>> 0x261e431c 16416847 oracle 600 2097152 87
>> 0x00000000 16187472 oracle 600 12582912 168
>> 0x00000000 16220241 oracle 600 20937965568 168
>> 0x00000000 16253010 oracle 600 56623104 168
>> 0x35b8fa94 16285779 oracle 600 2097152 168
>> 0x00000000 547127380 oracle 600 12582912 85
>> 0x00000000 547160149 oracle 600 251658240 85
>> 0x00000000 547192918 oracle 600 8321499136 85
>> 0x00000000 547225687 oracle 600 4595712 85
>> 0x9cd83398 547258456 oracle 600 40960 85
>> 0x00000000 792199257 oracle 600 10485760 106
>> 0x00000000 792232026 oracle 600 1644167168 106
>> 0x00000000 792264795 oracle 600 4781506560 106
>> 0x00000000 792297564 oracle 600 8388608 106
>> 0xc78a1c8c 792330333 oracle 600 32768 106
>> 0x6762e034 1180106846 oracle 600 28672 90
>>
>> [root]# cat /proc/meminfo
>> MemTotal: 263623524 kB
>> MemFree: 2169856 kB
>> MemAvailable: 25248596 kB
>> Buffers: 28 kB
>> Cached: 68234496 kB
>> SwapCached: 4392 kB
>> Active: 56288772 kB
>> Inactive: 32305392 kB
>> Active(anon): 38139416 kB
>> Inactive(anon): 28588332 kB
>> Active(file): 18149356 kB
>> Inactive(file): 3717060 kB
>> Unevictable: 0 kB
>> Mlocked: 0 kB
>> SwapTotal: 16777212 kB
>> SwapFree: 16732780 kB
>> Dirty: 1020 kB
>> Writeback: 0 kB
>> AnonPages: 20491988 kB
>> Mapped: 10605612 kB
>> Shmem: 46232620 kB
>> Slab: 2937544 kB
>> SReclaimable: 2197872 kB
>> SUnreclaim: 739672 kB
>> KernelStack: 68544 kB
>> PageTables: 1451148 kB
>> NFS_Unstable: 0 kB
>> Bounce: 0 kB
>> WritebackTmp: 0 kB
>> CommitLimit: 64832940 kB
>> Committed_AS: 93046052 kB
>> VmallocTotal: 34359738367 kB
>> VmallocUsed: 772444 kB
>> VmallocChunk: 34358900732 kB
>> HardwareCorrupted: 0 kB
>> AnonHugePages: 0 kB
>> CmaTotal: 0 kB
>> CmaFree: 0 kB
>> HugePages_Total: 81793
>> HugePages_Free: 8
>> HugePages_Rsvd: 7
>> HugePages_Surp: 0
>> Hugepagesize: 2048 kB
>> DirectMap4k: 514768 kB
>> DirectMap2M: 41201664 kB
>> DirectMap1G: 228589568 kB
>>
>>
>> On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba_at_gmail.com>
>> wrote:
>>
>>> Ooops sorry yes Oracle Linux 7.4
>>>
>>> On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh_at_gmail.com> wrote:
>>>
>>>> OS info might be relevant.
>>>>
>>>> Assuming Linux:
>>>>
>>>> ipcs -m
>>>> cat /proc/meminfo
>>>>
>>>>
>>>>
>>>> On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba_at_gmail.com wrote:
>>>>
>>>>> Ok I just recently enabled hugepages on Production and Development.
>>>>> Everything is going good but I noticed that on my development environment
>>>>> which consists of a bunch of thin clones, when I remove and recreate the
>>>>> same clone the database fails to create, I am getting:
>>>>>
>>>>> ORA-27137: unable to allocate large pages to create a shared memory segment
>>>>>
>>>>> I have the parameter *use_large_pages=ONLY*
>>>>> So it makes sense I would get this error when there is not enough
>>>>> pages. However I just removed this same database and am recreating it. I
>>>>> would think that would free it up but apparently not. Does pages not get
>>>>> released immediately when a database is removed? Is there something I
>>>>> should run to free it? Or just not have use_large_pages=ONLY and leave it
>>>>> at TRUE on Dev? I had to set it to TRUE to get this database up.
>>>>>
>>>>> Thanks,
>>>>> Jeff
>>>>>
>>>>

--
http://www.freelists.org/webpage/oracle-l