RE: Slow GC response for x64 bit 2008
Date: Wed, 7 Dec 2011 09:30:50 -0600
Message-ID: <C5533BD628A9524496D63801704AE56D6A34134F89_at_SPOBMEXC14.adprod.directory>
If you have the Microsoft SysInternals Suite downloaded to your GC box, I think you can tell what options Weblogic is running with:
1.) Download Sysinternals Suite (if you don't have it)
2.) Execute PROCEXP.EXE
3.) Right click on one of the COLUMN HEADERS and choose "SELECT COLUMNS"
[cid:image001.png_at_01CCB4C1.A266B6B0]
4.) Choose PROCESS IMAGE tab
5.) Then CHECK the COMMAND LINE box
[cid:image002.png_at_01CCB4C1.A266B6B0]
6. CLICK OK 7. Now, SORT by COMMAND LINE and scroll down till you see your Weblogic Drive & Directory
8. You should see something like DRIVE:\MIDDLE~1\...\...\...
9. Look for Process Type Java.Exe and Command Line like EMGC_OMS1 and EMGC_ADMINSERVER - These are the 2 Weblogic Domains (You may have to expand the COMMAND LINE column to see it)
[cid:image003.png_at_01CCB4C2.970739B0]
[cid:image004.png_at_01CCB4C2.970739B0]
- RIGHT CLICK on each of the Processes and choose properties
[cid:image005.png_at_01CCB4C2.970739B0]
- SELECT the COMMAND LINE text (shift-click with mouse at the beginning of the line and then press END key to quickly select entire line)
- Copy the text
- Paste it to an editor such as notepad etc
- examine the command line for the debug switch I believe
- Here's my FULL command line
F:\Oracle\MIDDLE~1\JDK160~1.5-3\bin\java -client -Xms256m -Xmx1024m -XX:MaxPermSizeQ2m -XX:CompileThreshold€00 -XX:PermSize8m -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=F:\Oracle\MIDDLE~1\WLSERV~1.3\server\lib\weblogic.policy -Dweblogic.system.BootIdentityFile=F:\Oracle\gc_inst\user_projects\domains\GCDomain\servers\EMGC_OMS1\data\nodemanager\boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowedúlse -DINSTANCE_HOME=F:\Oracle\gc_inst\em\EMGC_OMS1 -DORACLE_HOME=F:\Oracle\Middleware\oms11g -Ddomain.home=F:\Oracle\gc_inst\user_projects\domains\GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Docm.repeater.home=F:\Oracle\Middleware\oms11g -Xrs -Xverify:none -da -Dplatform.home=F:\Oracle\MIDDLE~1\WLSERV~1.3 -Dwls.home=F:\Oracle\MIDDLE~1\WLSERV~1.3\server -Dweblogic.home=F:\Oracle\MIDDLE~1\WLSERV~1.3\server -Ddomain.home=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain -Dcommon .components.home=F:\Oracle\MIDDLE~1\ORACLE~1 -Djrf.version.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=F:\Oracle\MIDDLE~1\ORACLE~1\modules\oracle.jrf_11.1.1\jrocket_optfile.txt -Doracle.domain.config.dir=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain\config\FMWCON~1 -Doracle.server.config.dir=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain\config\FMWCON~1\servers\EMGC_OMS1 -Doracle.security.jps.config=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain\config\fmwconfig\jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain\config\FMWCON~1\carml -Digf.arisidstack.home=F:\Oracle\gc_inst\USER_P~1\domains\GCDomain\config\FMWCON~1\arisidprovider -Dweblogic.alternateTypesDirectory=\modules\oracle.ossoiap_11.1.1,\modules\oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabledúlse -Dweblogic.management.discoverúlse -Dweblogic.management.server=https://spob mgrd01:7101 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=F:\Oracle\MIDDLE~1\patch_wls1032\profiles\default\sysext_manifest_classpath weblogic.Server
- Notice near the end are some switches:
-Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole
These look to be not set on mine. I think there is an ADDITIONAL switch for DEBUGGING that shows up when DEBUG is enabled but I might be mistaken
Chris Taylor
Sr. Oracle DBA
Ingram Barge Company
Nashville, TN 37205
"Quality is never an accident; it is always the result of intelligent effort."
- John Ruskin (English Writer 1819-1900)
CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender immediately and delete the contents of this message without disclosing the contents to anyone, using them for any purpose, or storing or copying the information on any medium.
-----Original Message-----
From: Storey, Robert (DCSO) [mailto:RStorey_at_DCSO.nashville.org]
Sent: Wednesday, December 07, 2011 9:11 AM
To: Taylor, Chris David; 'Mark W. Farnham'
Subject: RE: Slow GC response for x64 bit 2008
I would agree but I can't find a good way to test that.
Let me check the process.
-----Original Message-----
From: Taylor, Chris David [mailto:ChrisDavid.Taylor_at_ingrambarge.com]
Sent: Wednesday, December 07, 2011 9:09 AM
To: Storey, Robert (DCSO); 'Mark W. Farnham'; 'oracle-l_at_freelists.org'
Subject: RE: Slow GC response for x64 bit 2008
I still think the performance issue is in the Weblogic rendering BUT:
Examine the task manager of the Windows 2008 target box (particularly the EMAGENT.EXE process) and run your tests while observing the task on the monitored host. If the emagent.exe process begins chewing up CPU then you're more likely on the right track. However, if the emagent process on the monitored host remains mostly idle, then the problem is on the GC host itself (and most likely in the Weblogic 10 layer).
I would bet the Production Mode (from Development Mode) switch didn't work as it should have in Weblogic and it is still running with DEBUG set.
Chris Taylor
Sr. Oracle DBA
Ingram Barge Company
Nashville, TN 37205
"Quality is never an accident; it is always the result of intelligent effort."
- John Ruskin (English Writer 1819-1900)
CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender immediately and delete the contents of this message without disclosing the contents to anyone, using them for any purpose, or storing or copying the information on any medium.
-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Storey, Robert (DCSO)
Sent: Wednesday, December 07, 2011 7:35 AM
To: 'Mark W. Farnham'; oracle-l_at_freelists.org
Subject: RE: Slow GC response for x64 bit 2008
Since I'm new to GC, I'm still not fully understanding what is happening when you select a Host target homepage. Is the agent being asked to provide a huge amount of data all over? Is it reading from the repository.
I definitely agree it's something wacky on the 2008 machines.
Ping was the first thing I did. No network latency. Less than 1ms response. The two machines are on the same switch, so no weird routing.
If I understood the sequence of steps better for building that Target page, I could do some tracing.
Which tracing level would be the one to set? One for the OMS? OMA? Agent on the box? Since I'm not (as best I can tell) seeing an error in the process, it leads me to believe that it is taking the agent that long to gather and send the info.
-----Original Message-----
From: Mark W. Farnham [mailto:mwf_at_rsiz.com]
Sent: Wednesday, December 07, 2011 7:22 AM
To: Storey, Robert (DCSO); oracle-l_at_freelists.org
Subject: RE: Slow GC response for x64 bit 2008
I'd start with divide and conquer regarding where in the stack the long elapsed time is.
If you open a command window and use ping you should be able to do a traceroute to see if the network is going some stupid way to your targets.
If you examine the objects on each displayed page, can you verify you're not reloading wallpaper, logos, or pictures from a remote source?
Is there anything funky about the DNS and/or security on W2008 (haven't had the displeasure yet) that might be taking a long time to resolve?
22 seconds is approximately forever, but not failing, so I think we're looking for some strange latency here.
Is it possible W2008R2 is trying to refresh the whole network topology for the workgroup, or something like that?
Good luck.
-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org]
On Behalf Of Storey, Robert (DCSO)
Sent: Tuesday, December 06, 2011 1:19 PM
To: oracle-l_at_freelists.org
Subject: Slow GC response for x64 bit 2008
Afternoon all,
I've been working with Oracle on an issue rendering of the target homepage in grid control for 3, x64bit boxes running 64bit W2008R2. But, since its Pri4, I should hear from them in the next 30 days or so.
I'm running GC 11 on W2008 x32. I have 22 targets. Two of my targets are x32bit. A W2003 x32 running 32bit os and my GC box that is running W2008
x32 bit OS on a x64bit hardware. Rendering these Host target homepages takes about 1.2 seconds. My other 3 boxes are all W2008R2 x64 on 64bit Dell Powedges. When I click on the link for their Host homepage, its takes 22 seconds to render.
I have looked through every log that I can find but can find no errors. I installed the agent on a 64 bit workstation running Windows7 64bit and it works just fine. Renders in about a second.
Once suggestion I got from another DBA was that I was in Demo mode vice Production on Weblogic. I reset and rebooted that so it is now in production mode.
But, I can't find any reason as to why the render takes so long on these boxes.
Anyone run up against this or have a thought? (lol moving to linux is not an option).
-- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-lReceived on Wed Dec 07 2011 - 09:30:50 CST