Re: 11.2.0.3 Grid causing high load on servers

From: Shastry(DBA) <"Shastry>
Date: Sat, 11 Aug 2012 12:32:45 +0530
Message-ID: <CAFQkUXR0n2eRAg4wVk17rtP2UPK_R9ZwzFHzBxAx01cj-RqVoA_at_mail.gmail.com>

Apologies on earlier blank email which was sent accidentally due to key board short-cuts:
Hi Gurus,

              Recently, 2 months back we upgraded the 11g CRS from 11.2.0.1 to 11.2.0.3 version. It was running absolutely fine and couple of days back the load on linux box went to 600+ from 25 avg load causing the server to hung. We are working with Oracle SR support too but not recieved any inputs so far. Below are some of the observations by SA. please suggest

Oracle DB - 11.2.0.3
Oracle CRS/Grid - 11.2.0.3
Platform - Linux

oracle 32659 1 0 Aug08 ? 00:00:01 /oracle/product/grid_home/jdk/bin/java -Djava.net.preferIPv4Stack=true –classpath
/oracle/product/grid_home/jdk/lib/rt.jar:/oracle/product/grid_home/jlib/cvu.jar:/oracle/product/grid_home/jlib/srvm.jar:/oracle/product/grid_home/oui/jlib/OraInstaller.jar:/oracle/product/grid_home/oui/jlib/OraPrereq.jar:/oracle/product/grid_home/oui/jlib/prov_fixup.jar:/oracle/product/grid_home/oui/jlib/xmlparserv2.jar:/oracle/product/grid_home/oui/jlib/share.jar:/oracle/product/grid_home/oui/jlib/orai18nmapping.jar:/oracle/product/grid_home/jlib/srvmhas.jar:/oracle/product/grid_home/jdbc/lib/ojdbc5.jar:/oracle/product/grid_home/jlib/netcfg.jar -DCV_DESTLOC=/tmp -DCV_HOME=/oracle/product/grid_home oracle.ops.verification.client.CluvfyDriver comp crs -display_status

[8833_at_testdb] [main] [ 2012-08-09 05:53:15.233 PDT ]
[RuntimeExec.runCommand:75] Calling Runtime.exec() with the command
[8833_at_testdb] [main] [ 2012-08-09 05:53:15.233 PDT ]
[RuntimeExec.runCommand:77] /bin/rm
[8833_at_testdb] [main] [ 2012-08-09 05:53:15.233 PDT ]
[RuntimeExec.runCommand:77] -f
[8833_at_testdb] [main] [ 2012-08-09 05:53:15.234 PDT ]
[RuntimeExec.runCommand:77] /tmp/localnode.olr.loc
[8833_at_testdb] [Thread-21] [ 2012-08-09 05:53:15.237 PDT ]
[StreamReader.run:61] In StreamReader.run
[8833_at_testdb] [main] [ 2012-08-09 05:53:15.237 PDT ]
[RuntimeExec.runCommand:142] runCommand: Waiting for the process
[8833_at_testdb] [Thread-20] [ 2012-08-09 05:53:15.237 PDT ]
[StreamReader.run:61] In StreamReader.run
[7329_at_testdb] [main] [ 2012-08-09 05:53:21.340 PDT ]
[RuntimeExec.runCommand:142] runCommand: Waiting for the process
[7329_at_testdb] [Thread-30] [ 2012-08-09 05:53:21.340 PDT ]
[StreamReader.run:61] In StreamReader.run
[7329_at_testdb] [Thread-31] [ 2012-08-09 05:53:21.340 PDT ]
[StreamReader.run:61] In StreamReader.run

SA suspected that the "file got deleted: /var/tmp/.oracle/sprocr_local_conn_0_PROC"

But there are no crons or manual removal of the files in /var/tmp/.oracle area by anyone, what else can cause this issue since we are not able to stop and start the CRS services and reboot of all cluster nodes is fixing it, rather we are looking for the actual cause and fix. Please share your views and suggestions.

[root_at_testdb ~]# uptime

13:12:33 up 1 day, 5:23, 1 user, load average: *627.15, 624.81, 619.83*

oracle 32742 1 0 Aug09 ? 00:00:01 /oracle/product/grid_home/jdk/bin/java -Djava.net.preferIPv4Stack=tru -classpath
/oracle/product/grid_home/jdk/lib/rt.jar:/oracle/product/grid_home/jlib/cvu.jar:/oracle/product/grid_homejlib/srvm.jar:/oracle/product/grid_home/oui/jlib/OraInstaller.jar:/oracle/product/grid_home/oui/jlib/OraPrereq.jar:/oacle/product/grid_home/oui/jlib/prov_fixup.jar:/oracle/product/grid_home/oui/jlib/xmlparserv2.jar:/oracle/product/gri_home/oui/jlib/share.jar:/oracle/product/grid_home/oui/jlib/orai18n-mapping.jar:/oracle/product/grid_home/jlib/srvmha.jar:/oracle/product/grid_home/jdbc/lib/ojdbc5.jar:/oracle/product/grid_home/jlib/netcfg.jar -DCV_DESTLOC=/tmp -DCV_HME=/oracle/product/grid_home oracle.ops.verification.client.CluvfyDriver comp crs -display_status

[root_at_testdb ~]# strace -p 32742

Process 32742 attached - interrupt to quit futex(0x2aaaac06ad14, FUTEX_WAIT_PRIVATE, 59, NULL <unfinished ...> Process 32742 detached

Thanks,
Shastry

On Sat, Aug 11, 2012 at 12:23 PM, Shastry(DBA) <shastry17_at_gmail.com> wrote:

> Hi gurus,
>
>

--
http://www.freelists.org/webpage/oracle-l

Received on Sat Aug 11 2012 - 02:02:45 CDT