Oracle8.1.7 : Very high CPU load but NO alert.log record
Date: Mon, 5 May 2008 10:13:47 -0700 (PDT)
Message-ID: <62a1552a-5702-42de-a5a9-c49f14f5b40b@q1g2000prf.googlegroups.com>
Hi,
two of our DB server suffered very high CPU load today. we could
not identify the exact reason
for it.
We run a main DB whose table content update frequently. Both of the two questionable servers maintein Snapshot of main DB, they refresh snapshot every 5 min. The OS of two stallite server is HP UX 11i. Normally, the two stalitte servers' CPU is just around 20%.
Today, both of them run suffered very high CPU load (around 96% and above), application on stalite DB respond very slow with a lot of service timeout. we have to stop the application which make the two stallite DB CPU come back within 20min. Restarting application does not generate the same problem at all.
Looking into sar ball stored, we just find there is high CPU and a lot of semops. But there is NO alert.log record and Archive was executed successfully.
What's the possible reason for this ?
sar record information:
# sar -f sa05 -u | more
HP-UX idb-dcb B.11.00 U 9000/800 05/05/08
00:02:00 %usr %sys %wio %idle 19:41:59 46 8 6 40 19:46:59 79 15 1 5 19:51:59 80 15 1 4 19:56:59 80 15 1 4 20:02:00 80 15 1 4 20:07:00 80 16 1 4 20:12:00 80 15 1 4 20:17:00 78 15 1 6 20:22:00 76 15 1 8 20:27:00 44 8 3 44 20:32:00 8 1 3 88 20:37:00 7 1 4 88 20:42:00 8 1 11 79 20:47:00 7 1 6 86 20:52:00 7 1 8 84
# sar -b -f sa05 | more
HP-UX idb-dcb B.11.00 U 9000/800 05/05/08
00:02:00 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/
s
19:31:59 0 3141 100 2 2 32 65
45
19:36:59 0 3152 100 1 2 26 65
37
19:41:59 0 5233 100 2 3 32 165
67
19:46:59 0 8475 100 2 3 38 141
15
19:51:59 0 8814 100 2 3 35 131
53
19:56:59 0 8448 100 2 4 36 125
57
20:02:00 0 8413 100 2 4 37 119
59
20:07:00 1 8448 100 2 3 37 117
53
20:12:00 1 8463 100 3 4 32 109
51
20:17:00 1 8404 100 2 3 37 114
52
20:22:00 0 8279 100 2 3 37 115
49
20:27:00 0 5226 100 1 2 26 72
37
20:32:00 0 1157 100 2 2 28 31
30
20:37:00 0 1078 100 1 2 32 43
26
20:42:00 0 1026 100 2 4 30 276
32
# sar -c -f sa05 | more
HP-UX idb-dcb B.11.00 U 9000/800 05/05/08
00:02:00 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
19:26:59 16598 7558 4519 0.38 0.36 11036224
7397
19:31:59 16984 7616 4553 0.50 0.48 11296768
11547
19:36:59 16616 7569 4522 0.38 0.36 11412110
8113
19:41:59 31015 12931 7664 0.54 0.51 14109634
13459
19:46:59 53605 20837 12369 0.59 0.55 12035902
13353
19:51:59 54295 21122 12326 0.53 0.50 498452 12947 19:56:59 53569 20748 12314 0.67 0.64 11791916 15763 20:02:00 53674 20657 12267 0.63 0.61 11718527 14961 20:07:00 53529 20728 12309 0.61 0.58 11805223 13598 20:12:00 53757 20775 12346 0.60 0.57 11921990 17148 20:17:00 52144 20625 12246 0.60 0.56 11452784 14296 20:22:00 50627 20276 12043 0.56 0.53 10370528 13101 20:27:00 29580 12678 7521 0.42 0.40 14077691 9100 20:32:00 6054 2516 1462 0.48 0.47 9361588 11318 20:37:00 5173 2211 1271 0.40 0.38 8727180 8103 20:42:00 5876 2413 1253 0.59 0.57 8267531 17158 20:47:00 5626 2203 1210 0.65 0.63 8142735 15047
# sar -w -f sa05 | more
HP-UX idb-dcb B.11.00 U 9000/800 05/05/08
00:02:00 swpin/s bswin/s swpot/s bswot/s pswch/s 19:36:59 0.00 0.0 0.00 0.0 7057 19:41:59 0.00 0.0 0.00 0.0 16138 19:46:59 0.00 0.0 0.00 0.0 31574 19:51:59 0.00 0.0 0.00 0.0 31989 19:56:59 0.00 0.0 0.00 0.0 31973 20:02:00 0.00 0.0 0.00 0.0 31700 20:07:00 0.00 0.0 0.00 0.0 31929 20:12:00 0.00 0.0 0.00 0.0 31971 20:17:00 0.00 0.0 0.00 0.0 31420 20:22:00 0.00 0.0 0.00 0.0 30354 20:27:00 0.00 0.0 0.00 0.0 16026 20:32:00 0.00 0.0 0.00 0.0 2100 20:37:00 0.00 0.0 0.00 0.0 1831 20:42:00 0.00 0.0 0.00 0.0 2106 20:47:00 0.00 0.0 0.00 0.0 1824 20:52:00 0.00 0.0 0.00 0.0 1881
# sar -f sa05 -d | more
HP-UX idb-dcb B.11.00 U 9000/800 05/05/08
00:02:00 device %busy avque r+w/s blks/s avwait avserv
20:02:00 c1t6d0 2.78 0.55 4 37 4.26 12.33
c2t6d0 1.88 0.54 3 32 4.55 10.07
c40t0d1 13.61 0.80 32 278 5.10 4.74
c40t0d2 13.56 2.23 41 348 5.32 4.06
c40t0d3 13.02 1.76 38 328 5.23 4.02
c40t0d4 13.25 1.04 35 302 5.16 4.44
c40t0d5 13.78 0.87 33 289 5.13 4.90
20:07:00 c1t6d0 2.49 0.81 4 43 5.75 12.55
c2t6d0 1.74 0.83 3 39 6.41 10.87
c40t0d1 13.10 1.07 30 262 5.07 4.82
c40t0d2 12.79 2.44 37 321 5.38 4.23
c40t0d3 13.04 2.05 37 316 5.33 4.25
c40t0d4 13.12 1.50 33 290 5.23 4.63
c40t0d5 13.57 1.03 33 287 5.08 4.87
20:12:00 c1t6d0 3.50 0.62 5 72 4.76 11.16
c2t6d0 2.08 0.75 3 39 5.74 10.99
c40t0d1 12.56 1.52 29 257 5.19 4.82
c40t0d2 12.33 2.89 36 310 5.48 4.15
c40t0d3 11.86 2.76 34 294 5.44 4.21
c40t0d4 12.30 2.08 31 269 5.38 4.58
c40t0d5 12.84 1.64 31 270 5.24 5.08
regards
Joe Received on Mon May 05 2008 - 12:13:47 CDT