Re: hugepages question

From: Jeff Chirco <backseatdba_at_gmail.com>
Date: Thu, 14 Feb 2019 07:16:51 -0800
Message-ID: <CAKsxbLoSjFJEWRXz+99y24Yamtw=WgFf4EzMUn7j9uyR+ZsY7Q_at_mail.gmail.com>

Thanks for the replies. Forgot to mention this is Oracle Linux 7 running 12.2.0.1. These are output from my Development server but production is identicial except hugepage size is lower cause it has less database. Development and Production both have 256gb of RAM. I also noticed when I first set huegpages using the below command it increased hugepages but not to the amount I tried to set it to. Not until a server reboot did it get set the full amount. If I need to increase it again I would prefer to not have to reboot the server. # sysctl -w vm.nr_hugepages=value

Regarding hugepages_settings.sh I did follow Tim's notes but I ended up finding an updated script under Doc ID 401749.1

[root]# grep memlock /etc/security/limits.conf # - memlock - max locked-in-memory address space (KB) oracle hard memlock 237265300
# oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM oracle soft memlock 237265300

[root]# grep nr_hugepages /etc/sysctl.conf vm.nr_hugepages=91000

[root]# ulimit -l
64

[root]# ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1029648
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1029648
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Alert log from one of my databases

2019-02-12T21:26:58.786205-08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

2019-02-12T21:26:58.790788-08:00
Per process system memlock (soft) limit = 226G 2019-02-12T21:26:58.793885-08:00
Expected per process system memlock (soft) limit to lock SHARED GLOBAL AREA (SGA) into memory: 8196M 2019-02-12T21:26:58.800179-08:00
Available system pagesizes:
4K, 2048K

2019-02-12T21:26:58.806848-08:00
 Supported system pagesize(s):
2019-02-12T21:26:58.809846-08:00

PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s) 2019-02-12T21:26:58.813115-08:00

2048K 65468 4098 4098 NONE 2019-02-12T21:26:58.816156-08:00
Reason for not supporting certain system pagesizes: 2019-02-12T21:26:58.819060-08:00
4K - Large pagesizes only
2019-02-12T21:26:58.822058-08:00

[root]# ipcs -m

Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 589824 oracle 600 10485760 78 0x00000000 622593 oracle 600 3372220416 78 0x00000000 655362 oracle 600 8388608 78 0x76c55318 688131 oracle 600 2097152 78 0x00000000 11862020 oracle 600 10485760 120 0x00000000 11894789 oracle 600 3204448256 120 0x00000000 11927558 oracle 600 8388608 120 0x34ef509c 11960327 oracle 600 2097152 120 0x00000000 11993096 oracle 600 10485760 95 0x00000000 12025865 oracle 600 2298478592 95 0x00000000 12058634 oracle 600 8388608 95 0x507a78a4 12091403 oracle 600 2097152 95 0x00000000 12124172 oracle 600 10485760 207 0x00000000 12156941 oracle 600 6425673728 207 0x00000000 12189710 oracle 600 8388608 207 0x866b8aa0 12222479 oracle 600 2097152 207 0x00000000 12255248 oracle 600 12582912 81 0x00000000 12288017 oracle 600 8573157376 81 0x00000000 12320786 oracle 600 6291456 81 0x0c51f55c 12353555 oracle 600 2097152 81 0x00000000 12386324 oracle 600 12582912 79 0x00000000 12419093 oracle 600 8573157376 79 0x00000000 12451862 oracle 600 6291456 79 0x7afafd64 12484631 oracle 600 2097152 79 0x00000000 12779544 oracle 600 12582912 90 0x00000000 12812313 oracle 600 8573157376 90 0x00000000 12845082 oracle 600 6291456 90 0xdde48098 12877851 oracle 600 2097152 90 0x00000000 12910620 oracle 600 12582912 86 0x00000000 12943389 oracle 600 8573157376 86 0x00000000 12976158 oracle 600 6291456 86 0xbae4e74c 13008927 oracle 600 2097152 86 0x00000000 13303840 oracle 600 12582912 85 0x00000000 13336609 oracle 600 8573157376 85 0x00000000 13369378 oracle 600 6291456 85 0x4c7ac944 13402147 oracle 600 2097152 85 0x00000000 13434916 oracle 600 12582912 86 0x00000000 13467685 oracle 600 8573157376 86 0x00000000 13500454 oracle 600 6291456 86 0xf06d661c 13533223 oracle 600 2097152 86 0x00000000 13828136 oracle 600 12582912 87 0x00000000 13860905 oracle 600 8573157376 87 0x00000000 13893674 oracle 600 6291456 87 0xdb5acf68 13926443 oracle 600 2097152 87 0x00000000 13959212 oracle 600 12582912 87 0x00000000 13991981 oracle 600 8573157376 87 0x00000000 14024750 oracle 600 6291456 87 0xcdd9f634 14057519 oracle 600 2097152 87 0x00000000 14352432 oracle 600 10485760 102 0x00000000 14385201 oracle 600 6425673728 102 0x00000000 14417970 oracle 600 8388608 102 0x4bd6517c 14450739 oracle 600 2097152 102 0x00000000 14483508 oracle 600 10485760 116 0x00000000 14516277 oracle 600 6425673728 116 0x00000000 14549046 oracle 600 8388608 116 0x33ec8074 14581815 oracle 600 2097152 116 0x00000000 14876728 oracle 600 12582912 87 0x00000000 14909497 oracle 600 8573157376 87 0x00000000 14942266 oracle 600 6291456 87 0xb40541dc 14975035 oracle 600 2097152 87 0x00000000 1179975740 oracle 600 12582912 90 0x00000000 1180008509 oracle 600 8455716864 90 0x00000000 1180041278 oracle 600 34359738368 90 0x00000000 1180074047 oracle 600 123731968 90 0x00000000 15401024 oracle 600 12582912 91 0x00000000 15433793 oracle 600 8573157376 91 0x00000000 15466562 oracle 600 6291456 91 0x36329238 15499331 oracle 600 2097152 91 0x00000000 15532100 oracle 600 12582912 118 0x00000000 15564869 oracle 600 8573157376 118 0x00000000 15597638 oracle 600 6291456 118 0xa0206b88 15630407 oracle 600 2097152 118 0x00000000 16449608 oracle 600 12582912 86 0x00000000 16482377 oracle 600 8573157376 86 0x00000000 16515146 oracle 600 6291456 86 0xc402bb18 16547915 oracle 600 2097152 86 0x00000000 16318540 oracle 600 12582912 87 0x00000000 16351309 oracle 600 8573157376 87 0x00000000 16384078 oracle 600 6291456 87 0x261e431c 16416847 oracle 600 2097152 87 0x00000000 16187472 oracle 600 12582912 168 0x00000000 16220241 oracle 600 20937965568 168 0x00000000 16253010 oracle 600 56623104 168 0x35b8fa94 16285779 oracle 600 2097152 168 0x00000000 547127380 oracle 600 12582912 85 0x00000000 547160149 oracle 600 251658240 85 0x00000000 547192918 oracle 600 8321499136 85 0x00000000 547225687 oracle 600 4595712 85 0x9cd83398 547258456 oracle 600 40960 85 0x00000000 792199257 oracle 600 10485760 106 0x00000000 792232026 oracle 600 1644167168 106 0x00000000 792264795 oracle 600 4781506560 106 0x00000000 792297564 oracle 600 8388608 106 0xc78a1c8c 792330333 oracle 600 32768 106 0x6762e034 1180106846 oracle 600 28672 90

[root]# cat /proc/meminfo

MemTotal:       263623524 kB
MemFree:         2169856 kB
MemAvailable:   25248596 kB
Buffers:              28 kB
Cached:         68234496 kB
SwapCached:         4392 kB
Active:         56288772 kB
Inactive:       32305392 kB

Active(anon): 38139416 kB
Inactive(anon): 28588332 kB
Active(file): 18149356 kB
Inactive(file): 3717060 kB

Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16777212 kB
SwapFree:       16732780 kB
Dirty:              1020 kB
Writeback:             0 kB
AnonPages:      20491988 kB
Mapped:         10605612 kB
Shmem:          46232620 kB
Slab:            2937544 kB
SReclaimable:    2197872 kB
SUnreclaim:       739672 kB
KernelStack:       68544 kB
PageTables:      1451148 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB

CommitLimit: 64832940 kB
Committed_AS: 93046052 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 772444 kB
VmallocChunk: 34358900732 kB

HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:   81793
HugePages_Free:        8
HugePages_Rsvd:        7
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      514768 kB

DirectMap2M: 41201664 kB
DirectMap1G: 228589568 kB

On Thu, Feb 14, 2019 at 7:15 AM Jeff Chirco <backseatdba_at_gmail.com> wrote:

> Thanks for the replies. Forgot to mention this is Oracle Linux 7 running
> 12.2.0.1. These are output from my Development server but production is
> identicial except hugepage size is lower cause it has less database.
> Development and Production both have 256gb of RAM.
> I also noticed when I first set huegpages using the below command it
> increased hugepages but not to the amount I tried to set it to. Not until a
> server reboot did it get set the full amount. If I need to increase it
> again I would prefer to not have to reboot the server.
> # sysctl -w vm.nr_hugepages=value
>
> Regarding hugepages_settings.sh I did follow Tim's notes but I ended up
> finding an updated script under Doc ID 401749.1
>
> [root]# grep memlock /etc/security/limits.conf
> # - memlock - max locked-in-memory address space (KB)
> oracle hard memlock 237265300
> # oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit is
> maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
> oracle soft memlock 237265300
>
> [root]# grep nr_hugepages /etc/sysctl.conf
> vm.nr_hugepages=91000
>
>
> [root]# ulimit -l
> 64
>
> [root]# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 1029648
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1029648
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Alert log from one of my databases
> **********************************************************************
> 2019-02-12T21:26:58.786205-08:00
> Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
>
> 2019-02-12T21:26:58.790788-08:00
> Per process system memlock (soft) limit = 226G
> 2019-02-12T21:26:58.793885-08:00
> Expected per process system memlock (soft) limit to lock
> SHARED GLOBAL AREA (SGA) into memory: 8196M
> 2019-02-12T21:26:58.800179-08:00
> Available system pagesizes:
> 4K, 2048K
> 2019-02-12T21:26:58.806848-08:00
> Supported system pagesize(s):
> 2019-02-12T21:26:58.809846-08:00
> PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
> 2019-02-12T21:26:58.813115-08:00
> 2048K 65468 4098 4098 NONE
> 2019-02-12T21:26:58.816156-08:00
> Reason for not supporting certain system pagesizes:
> 2019-02-12T21:26:58.819060-08:00
> 4K - Large pagesizes only
> 2019-02-12T21:26:58.822058-08:00
> **********************************************************************
>
>
>
> [root]# ipcs -m
>
> ------ Shared Memory Segments --------
> key shmid owner perms bytes nattch status
> 0x00000000 589824 oracle 600 10485760 78
> 0x00000000 622593 oracle 600 3372220416 78
> 0x00000000 655362 oracle 600 8388608 78
> 0x76c55318 688131 oracle 600 2097152 78
> 0x00000000 11862020 oracle 600 10485760 120
> 0x00000000 11894789 oracle 600 3204448256 120
> 0x00000000 11927558 oracle 600 8388608 120
> 0x34ef509c 11960327 oracle 600 2097152 120
> 0x00000000 11993096 oracle 600 10485760 95
> 0x00000000 12025865 oracle 600 2298478592 95
> 0x00000000 12058634 oracle 600 8388608 95
> 0x507a78a4 12091403 oracle 600 2097152 95
> 0x00000000 12124172 oracle 600 10485760 207
> 0x00000000 12156941 oracle 600 6425673728 207
> 0x00000000 12189710 oracle 600 8388608 207
> 0x866b8aa0 12222479 oracle 600 2097152 207
> 0x00000000 12255248 oracle 600 12582912 81
> 0x00000000 12288017 oracle 600 8573157376 81
> 0x00000000 12320786 oracle 600 6291456 81
> 0x0c51f55c 12353555 oracle 600 2097152 81
> 0x00000000 12386324 oracle 600 12582912 79
> 0x00000000 12419093 oracle 600 8573157376 79
> 0x00000000 12451862 oracle 600 6291456 79
> 0x7afafd64 12484631 oracle 600 2097152 79
> 0x00000000 12779544 oracle 600 12582912 90
> 0x00000000 12812313 oracle 600 8573157376 90
> 0x00000000 12845082 oracle 600 6291456 90
> 0xdde48098 12877851 oracle 600 2097152 90
> 0x00000000 12910620 oracle 600 12582912 86
> 0x00000000 12943389 oracle 600 8573157376 86
> 0x00000000 12976158 oracle 600 6291456 86
> 0xbae4e74c 13008927 oracle 600 2097152 86
> 0x00000000 13303840 oracle 600 12582912 85
> 0x00000000 13336609 oracle 600 8573157376 85
> 0x00000000 13369378 oracle 600 6291456 85
> 0x4c7ac944 13402147 oracle 600 2097152 85
> 0x00000000 13434916 oracle 600 12582912 86
> 0x00000000 13467685 oracle 600 8573157376 86
> 0x00000000 13500454 oracle 600 6291456 86
> 0xf06d661c 13533223 oracle 600 2097152 86
> 0x00000000 13828136 oracle 600 12582912 87
> 0x00000000 13860905 oracle 600 8573157376 87
> 0x00000000 13893674 oracle 600 6291456 87
> 0xdb5acf68 13926443 oracle 600 2097152 87
> 0x00000000 13959212 oracle 600 12582912 87
> 0x00000000 13991981 oracle 600 8573157376 87
> 0x00000000 14024750 oracle 600 6291456 87
> 0xcdd9f634 14057519 oracle 600 2097152 87
> 0x00000000 14352432 oracle 600 10485760 102
> 0x00000000 14385201 oracle 600 6425673728 102
> 0x00000000 14417970 oracle 600 8388608 102
> 0x4bd6517c 14450739 oracle 600 2097152 102
> 0x00000000 14483508 oracle 600 10485760 116
> 0x00000000 14516277 oracle 600 6425673728 116
> 0x00000000 14549046 oracle 600 8388608 116
> 0x33ec8074 14581815 oracle 600 2097152 116
> 0x00000000 14876728 oracle 600 12582912 87
> 0x00000000 14909497 oracle 600 8573157376 87
> 0x00000000 14942266 oracle 600 6291456 87
> 0xb40541dc 14975035 oracle 600 2097152 87
> 0x00000000 1179975740 oracle 600 12582912 90
> 0x00000000 1180008509 oracle 600 8455716864 90
> 0x00000000 1180041278 oracle 600 34359738368 90
> 0x00000000 1180074047 oracle 600 123731968 90
> 0x00000000 15401024 oracle 600 12582912 91
> 0x00000000 15433793 oracle 600 8573157376 91
> 0x00000000 15466562 oracle 600 6291456 91
> 0x36329238 15499331 oracle 600 2097152 91
> 0x00000000 15532100 oracle 600 12582912 118
> 0x00000000 15564869 oracle 600 8573157376 118
> 0x00000000 15597638 oracle 600 6291456 118
> 0xa0206b88 15630407 oracle 600 2097152 118
> 0x00000000 16449608 oracle 600 12582912 86
> 0x00000000 16482377 oracle 600 8573157376 86
> 0x00000000 16515146 oracle 600 6291456 86
> 0xc402bb18 16547915 oracle 600 2097152 86
> 0x00000000 16318540 oracle 600 12582912 87
> 0x00000000 16351309 oracle 600 8573157376 87
> 0x00000000 16384078 oracle 600 6291456 87
> 0x261e431c 16416847 oracle 600 2097152 87
> 0x00000000 16187472 oracle 600 12582912 168
> 0x00000000 16220241 oracle 600 20937965568 168
> 0x00000000 16253010 oracle 600 56623104 168
> 0x35b8fa94 16285779 oracle 600 2097152 168
> 0x00000000 547127380 oracle 600 12582912 85
> 0x00000000 547160149 oracle 600 251658240 85
> 0x00000000 547192918 oracle 600 8321499136 85
> 0x00000000 547225687 oracle 600 4595712 85
> 0x9cd83398 547258456 oracle 600 40960 85
> 0x00000000 792199257 oracle 600 10485760 106
> 0x00000000 792232026 oracle 600 1644167168 106
> 0x00000000 792264795 oracle 600 4781506560 106
> 0x00000000 792297564 oracle 600 8388608 106
> 0xc78a1c8c 792330333 oracle 600 32768 106
> 0x6762e034 1180106846 oracle 600 28672 90
>
> [root]# cat /proc/meminfo
> MemTotal: 263623524 kB
> MemFree: 2169856 kB
> MemAvailable: 25248596 kB
> Buffers: 28 kB
> Cached: 68234496 kB
> SwapCached: 4392 kB
> Active: 56288772 kB
> Inactive: 32305392 kB
> Active(anon): 38139416 kB
> Inactive(anon): 28588332 kB
> Active(file): 18149356 kB
> Inactive(file): 3717060 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 16777212 kB
> SwapFree: 16732780 kB
> Dirty: 1020 kB
> Writeback: 0 kB
> AnonPages: 20491988 kB
> Mapped: 10605612 kB
> Shmem: 46232620 kB
> Slab: 2937544 kB
> SReclaimable: 2197872 kB
> SUnreclaim: 739672 kB
> KernelStack: 68544 kB
> PageTables: 1451148 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 64832940 kB
> Committed_AS: 93046052 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 772444 kB
> VmallocChunk: 34358900732 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> CmaTotal: 0 kB
> CmaFree: 0 kB
> HugePages_Total: 81793
> HugePages_Free: 8
> HugePages_Rsvd: 7
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 514768 kB
> DirectMap2M: 41201664 kB
> DirectMap1G: 228589568 kB
>
>
> On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba_at_gmail.com> wrote:
>
>> Ooops sorry yes Oracle Linux 7.4
>>
>> On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh_at_gmail.com> wrote:
>>
>>> OS info might be relevant.
>>>
>>> Assuming Linux:
>>>
>>> ipcs -m
>>> cat /proc/meminfo
>>>
>>>
>>>
>>> On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba_at_gmail.com wrote:
>>>
>>>> Ok I just recently enabled hugepages on Production and Development.
>>>> Everything is going good but I noticed that on my development environment
>>>> which consists of a bunch of thin clones, when I remove and recreate the
>>>> same clone the database fails to create, I am getting:
>>>>
>>>> ORA-27137: unable to allocate large pages to create a shared memory segment
>>>>
>>>> I have the parameter *use_large_pages=ONLY*
>>>> So it makes sense I would get this error when there is not enough
>>>> pages. However I just removed this same database and am recreating it. I
>>>> would think that would free it up but apparently not. Does pages not get
>>>> released immediately when a database is removed? Is there something I
>>>> should run to free it? Or just not have use_large_pages=ONLY and leave it
>>>> at TRUE on Dev? I had to set it to TRUE to get this database up.
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>

--
http://www.freelists.org/webpage/oracle-l

Received on Thu Feb 14 2019 - 16:16:51 CET

This message: [ Message body ]
Next message: Mark J. Bobak: "Re: hugepages question"
Previous message: Jeff Chirco: "Re: hugepages question"
Maybe in reply to: Jeff Chirco: "hugepages question"
In reply to Andy Wattenhofer: "Re: hugepages question"
Next in thread: Jeff Chirco: "Re: hugepages question"
Reply: Jeff Chirco: "Re: hugepages question"
Reply: Jeff Chirco: "Re: hugepages question"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

Original text of this message