Re: Strange Memory Information on AWR
Date: Tue, 22 Oct 2013 17:43:31 -0400
Message-ID: <CACNsJncvDTD6vqNaWjkZZaYHA2C6=BqzTrVH+je-knQOOcRung_at_mail.gmail.com>
I could possibly be "kswapd"..
If you have oswatcher installed you can do something like this on that hour where you had the high load average...
less
/opt/oracle.oswatcher/osw/archive/oswtop/pd01db03_top_12.03.08.1600.dat.bz2 |
grep -A20 "load average:" > loadspike.txt
Some of my notes below:
PGA usage maxing out the memory and the kswapd kicks in and the server starts to swap like crazy causing high on CPU WAIT IO
top - 12:58:20 up 132 days, 42 min, 2 users, load average: 13.68, 10.22, 9.07 Tasks: 995 total, 42 running, 919 sleeping, 0 stopped, 34 zombie Cpu(s): 48.5%us, 28.4%sy, 0.0%ni, 10.5%id, 11.2%wa, 0.0%hi, 1.3%si, 0.0%st Mem: 98848968k total, 98407164k used, 441804k free, 852k buffers Swap: 25165816k total, 2455968k used, 22709848k free, 383132k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13483 oracle 25 0 12.9g 509m 43m R 80.1 0.5 214:25.43
oraclemtaprd111 (LOCAL=NO)
24308 oracle 25 0 13.4g 1.0g 97m R 77.1 1.1 15:58.80
oraclemtaprd111 (LOCAL=NO)
16227 oracle 25 0 13.4g 1.0g 95m R 74.1 1.1 1312:47
oraclemtaprd111 (LOCAL=NO)
1401 root 11 -5 0 0 0 R 67.8 0.0 113:21.15 [kswapd0]
-- top - 12:59:48 up 132 days, 44 min, 2 users, load average: 116.16, 43.81, 20.96 Tasks: 985 total, 73 running, 879 sleeping, 0 stopped, 33 zombie Cpu(s): 8.6%us, 90.1%sy, 0.0%ni, 0.6%id, 0.7%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 98848968k total, 98407396k used, 441572k free, 2248k buffers Swap: 25165816k total, 2645544k used, 22520272k free, 370780k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32349 oracle 18 0 9797m 1.1g 33m S 493.3 1.2 0:36.76 oraclebiprd2 (LOCAL=NO) 29495 oracle 15 0 216m 26m 11m S 466.6 0.0 3:01.86Received on Tue Oct 22 2013 - 23:43:31 CEST
/u01/app/11.2.0/grid/bin/diskmon.bin -d -f
32726 oracle 16 0 8788m 169m 37m R 447.6 0.2 0:24.86 oraclebiprd2 (LOCAL=NO) 32338 oracle 18 0 9525m 905m 42m R 407.0 0.9 0:33.20 oraclebiprd2 (LOCAL=NO) -- top - 12:59:54 up 132 days, 44 min, 2 users, load average: 107.27, 44.31, 21.37 Tasks: 991 total, 16 running, 942 sleeping, 0 stopped, 33 zombie Cpu(s): 30.3%us, 3.6%sy, 0.0%ni, 14.3%id, 51.6%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 98848968k total, 98167188k used, 681780k free, 5264k buffers Swap: 25165816k total, 2745440k used, 22420376k free, 369676k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1401 root 10 -5 0 0 0 S 77.8 0.0 114:36.03 [kswapd0] <-- KSWAPD kicked in 19163 oracle 15 0 2152m 72m 16m S 74.9 0.1 9:42.45
/u01/app/11.2.0/grid/bin/oraagent.bin
3394 oracle 15 0 436m 23m 14m S 33.8 0.0 12:23.44
/u01/app/11.2.0/grid/bin/oraagent.bin
2171 root 16 0 349m 28m 12m S 28.6 0.0 1:50.29
/u01/app/11.2.0/grid/bin/orarootagent.bin
> vmstat 1 5000 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 2 69 9893340 442000 9032 346328 1160 2760 1264 2904 2083 13656 10 0 34 56 0 3 67 9894700 443524 9036 345768 1204 3236 1268 3300 1930 13332 7 0 47 46 0 1 72 9895936 446228 9052 346484 1052 3156 1220 3648 1819 13674 5 0 47 48 0 2 74 9897156 448732 9064 346616 1724 3432 2128 3436 1936 14598 7 0 44 49 0 3 73 9897724 446904 9068 347580 1524 2468 1636 2480 1730 13363 6 0 32 61 0 7 65 9898208 448312 9080 347472 1328 1944 1660 1952 2496 14019 16 0 32 52 0 8 61 9898500 444836 9092 347904 2128 2004 2464 2208 3381 16093 29 1 23 47 0 1 79 9899372 441588 9104 348048 1236 2684 1424 3300 2774 14103 23 0 24 53 0 13 54 9909828 551780 9224 349588 36124 63296 37608 64800 126067 443473 18 0 23 59 0 16 40 9910136 536048 9260 350004 4208 988 5076 2044 5434 17055 51 3 10 36 0 http://karlarao.tiddlyspot.com/#kswapd -- Karl Arao Blog: karlarao.wordpress.com Wiki: karlarao.tiddlyspot.com Twitter: _at_karlarao <http://twitter.com/karlarao> -- http://www.freelists.org/webpage/oracle-l