RE: OPatchAuto failed

From: <dimensional.dba_at_comcast.net>
Date: Fri, 4 Oct 2019 01:50:57 -0700
Message-ID: <0e8601d57a90$d79e6920$86db3b60$_at_comcast.net>



You set the PATH twice

[root_at_uat-db-002 .ssh]# export PATH=$PATH:$ORACLE_HOME/OPatch

[root_at_uat-db-002 .ssh]# export PATH=$PATH:/u01/g01/grid/12.1/OPatch

Was the original $ORACLE_HOME = /u01/g01/grid/12.1/OPatch or something else  

Run

Env | grep PATH

Does the path have different paths to OPatch?

Whatever is the first one in the PATH variable will be used first.  

Is the version of opatchauto/opatch the same on each node and the required version?        

From: oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> On Behalf Of Sameer Choudhari Sent: Thursday, October 3, 2019 8:41 PM
To: Oracle-L_at_freelists.org
Subject: OPatchAuto failed  

Team  

I am applying JULY 2019 Proactive Bunble Patch on our UAT env.  

Env :  

DB : 12.1.0.2.0 (2 node rac)

OS : Oracle Linux 7  

I am able to patch node-1 i.e. uat-db-001 without any issue but got below error while patching node -2 i.e. uat-db-002 .    

[root_at_uat-db-002 .ssh]# export PATH=$PATH:$ORACLE_HOME/OPatch

[root_at_uat-db-002 .ssh]# export PATH=$PATH:/u01/g01/grid/12.1/OPatch
[root_at_uat-db-002 .ssh]# cd  /u01/stage/COMBO_JVM_DBBP_JUL2019/29699255
[root_at_uat-db-002 29699255]# $ORACLE_HOME/OPatch/opatchauto apply /u01/stage/COMBO_JVM_DBBP_JUL2019/29699255/29698629

OPatchauto session is initiated at Fri Oct 4 13:13:54 2019

System initialization log file is /u01/g01/grid/12.1/cfgtoollogs/opatchautodb/systemconfig2019-10-04_01-14-00PM.log.

Session log file is /u01/g01/grid/12.1/cfgtoollogs/opatchauto/opatchauto2019-10-04_01-14-44PM.log The id for this session is LBTP
Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Execution failed for host uat-db-001 due to : java.lang.Exception: Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:
OPatchAuto failed.

OPatchauto session completed at Fri Oct 4 13:15:08 2019 Time taken to complete the session 1 minute, 14 seconds

 opatchauto failed with error code 42
[root_at_uat-db-002 29699255]#  

Checked MOS and Google but no help. I have shared partial session log below i.e. /u01/g01/grid/12.1/cfgtoollogs/opatchauto/opatchauto2019-10-04_01-14-44PM.log . Could you please let me know what could be the issue.      

Oct 04, 2019 1:15:06 PM com.oracle.cie.remote.linux.LinuxCommandExecutionHelper getSSHCmd
INFO: SSH path::/bin/ssh
Oct 04, 2019 1:15:06 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand
INFO: Command execution command: /bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 uat-db-001 ls /u01/g01/grid/12.1 > /dev/null Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: Command execution return code: 0
Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: command execution message:
Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: command executon failure message:
*** Authorized uses only. All activity may be monitored and reported. ***

Oct 04, 2019 1:15:07 PM com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify INFO: Executing command on host uat-db-001 : /u01/g01/grid/12.1//perl/bin/perl /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl -GRID_HOME=/u01/g01/grid/12.1 -OBJECTLOC=/u01/g01/grid/ 12.1//cfgtoollogs/opatchautodb/hostdata.obj -CRS_ACTION=check -CLUSTERNODES=uat-db-001,uat-db-002 -JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper

Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxCommandExecutionHelper getSSHCmd
INFO: SSH path::/bin/ssh
Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand
INFO: Command execution command: /bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o NumberOfPasswordPrompts=0 uat-db-001 /u01/g01/grid/12.1//perl/bin/perl /u01/g0 1/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl -GRID_HOME=/u01/g01/grid/12.1 -OBJECTLOC=/u01/g01/grid/12.1//cfgtoollogs/opatchautodb/hostdata.obj -CRS_ACTION=check -CLUSTERNODES=uat-db-001 ,uat-db-002 -JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: Command execution return code: 2
Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: command execution message:
Oct 04, 2019 1:15:07 PM com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl executeCommand INFO: command executon failure message:
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.

Oct 04, 2019 1:15:07 PM com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify WARNING: Error while executing command "/u01/g01/grid/12.1//perl/bin/perl /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl -GRID_HOME=/u01/g01/grid/12.1 -OBJECTLOC=/u01/g01/grid/12.1/ /cfgtoollogs/opatchautodb/hostdata.obj -CRS_ACTION=check -CLUSTERNODES=uat-db-001,uat-db-002 -JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper " on host uat-db-001 :  Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:
Oct 04, 2019 1:15:07 PM com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify WARNING: Exception while executing command "" on host uat-db-001 : Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:
Oct 04, 2019 1:15:07 PM com.oracle.helper.remote.ssh.SSHEquivalenceThreadHandler$SSHEquiCommandListener notifyResult INFO: Received command execution result for host uat-db-001 Oct 04, 2019 1:15:07 PM oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector notifyResult INFO: Consolidating result from host : uat-db-001 Oct 04, 2019 1:15:07 PM oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector notifyResult INFO: Execution failed for host uat-db-001 due to : Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:
Oct 04, 2019 1:15:07 PM oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector performRemoteOperForActions INFO: Remote node list for home /u01/g01/grid/12.1 : []

2019-10-04 13:15:07,859 INFO [1] oracle.dbsysmodel.driver.sdk.util.OsysUtility - Output message ::: 2019-10-04 13:15:07,870 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter - Reporting console output : Message{id='null', message='Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output: '}
2019-10-04 13:15:07,873 INFO [1] com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport - Space available after session: 33580 MB 2019-10-04 13:15:07,996 SEVERE [1] com.oracle.glcm.patch.auto.OPatchAuto - OPatchAuto failed. com.oracle.glcm.patch.auto.OPatchAutoException: oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Execution failed for host uat-db-001 due to : java.lang.Exception: Remote command exe cution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:

        at com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:775)
        at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:397)
        at com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:344)
        at com.oracle.glcm.patch.auto.OPatchAuto.main(OPatchAuto.java:212)
Caused by: oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException: Execution failed for host uat-db-001 due to : java.lang.Exception: Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output:

        at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.checkForException(OUIDriver.java:846)
        at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.handleExceptions(OUIDriver.java:832)
        at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteHostData(OUIDriver.java:791)
        at oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteNodeData(OUIDriver.java:690)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.DBPatchingHelper.fetchPreInitiatedNodes(DBPatchingHelper.java:899)
        at com.oracle.glcm.patch.auto.db.product.validation.validators.RemoteGIStatusValidation.isFirstNodeOfPatching(RemoteGIStatusValidation.java:274)
        at com.oracle.glcm.patch.auto.db.product.validation.validators.RemoteGIStatusValidation.validate(RemoteGIStatusValidation.java:185)
        at com.oracle.glcm.patch.auto.db.product.validation.DBValidationController.invoke(DBValidationController.java:91)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.CRSTopologyBuilder.validate(CRSTopologyBuilder.java:367)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.AbstractTopologyBuilder.buildTopology(AbstractTopologyBuilder.java:117)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.CRSTopologyBuilder.buildTopology(CRSTopologyBuilder.java:112)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.loadTopology(DBBaseProductSupport.java:192)
        at com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBProductSupport.loadTopology(DBProductSupport.java:69)
        at com.oracle.glcm.patch.auto.OPatchAuto.loadTopology(OPatchAuto.java:1732)
        at com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:730)
        ... 3 more

2019-10-04 13:15:07,999 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter - Reporting console output : Message{id='null', message='oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverE xception: Execution failed for host uat-db-001 due to : java.lang.Exception: Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in _at_INC (_at_INC contains: /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi /u01/g01/grid/12.1/perl/lib/site_perl/5.14.1 /u01/g01/grid/12.1/perl/lib/5.14.1/x86_6 4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2. BEGIN failed--compilation aborted at /u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.  Command output: '}
2019-10-04 13:15:08,003 INFO [1] com.oracle.cie.common.util.reporting.CommonReporter - Reporting console output : Message{id='null', message='OPatchAuto failed.'} [root_at_uat-db-002 29699255]#      

So far , I tried  

For "Can't locate Cwd.pm in _at_INC" , I checked it's there. Also, checked the permission for /u01/g01/grid/12.1/cfgtoollogs/opatchautodb and its owned by grid home user.  

[root_at_uat-db-002 29699255]# find / -name Cwd.pm

/usr/lib64/perl5/vendor_perl/Cwd.pm
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi/Cwd.pm
/u01/g01/grid/12.1/perl/lib/5.14.1/Cwd.pm
/u01/d01/db/12.1.0/perl/lib/5.14.1/x86_64-linux-thread-multi/Cwd.pm
/u01/em_agent/agent_13.2.0.0.0/perl/lib/5.14.4/x86_64-linux-thread-multi/Cwd.pm
[root_at_uat-db-002 29699255]#  

Cheers

Sameer  

This is an email from Ezidebit Pty Ltd. The email and any attachments may be confidential, legally privileged and/or subject to copyright. If you are not the intended recipient, you must not disclose or use the information contained in this e-mail. If you have received this email in error, please notify us immediately and delete the email and all copies. We do not guarantee that this email and any attachments are free from virus or other errors. We will not be responsible for loss or damage resulting (either directly or indirectly) from any such virus or error. We will only be responsible for a change to a document if we made the change. The content of and opinions expressed in non-business emails are not that of Ezidebit Pty Ltd and are that of the sender.

--
http://www.freelists.org/webpage/oracle-l
Received on Fri Oct 04 2019 - 10:50:57 CEST

Original text of this message