RE: Oracle 12.2 is junk!
Date: Tue, 22 Oct 2019 12:04:34 -0400
Message-ID: <13cc01d588f2$6d967de0$48c379a0$_at_rsiz.com>
Just curious: Is the root cause that the first bit of the patch fubar’d your inventory? I dimly recall a case where that happened and then it checked some default location for a file to check which had (no surprise) corrupt blocks as viewed by the new version, being a leftover from a long ago move by copy of a file that had not been deleted (probably because it was part of a cold backup of a before upgrade online source.) Something like that.
This would have been somewhere around the time I posted a suggestion to back up your Oracle environment and configuration files before doing any patches because there was a bug that could fubar your oraInventory that that could not be undone.
Another dim memory is a patch did something like make a new oraInventory, do the patch, and then run around “cleaning up” all the oraInventory files it could find, presuming it was the only oracle configuration on the machine.
sigh. (and good luck).
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Mladen Gogala
Sent: Monday, October 21, 2019 8:55 PM
To: niall.litchfield_at_gmail.com; dimensional.dba_at_comcast.net
Cc: Noveljic Nenad; oracle-l
Subject: Re: Oracle 12.2 is junk!
Well, I don't have corruption:
RMAN> backup validate tablespace sysaux;
Starting backup at 21-OCT-19
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=28 device type=DISK channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00003 name=/home/oradata/o122/data_D-O122_I-1534876540_TS-SYSAUX_FNO-3_7su6tgso channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03 List of Datafiles
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN ---- ------ -------------- ------------ --------------- ---------- 3 OK 0 16198 102419 21663082 File Name: /home/oradata/o122/data_D-O122_I-1534876540_TS-SYSAUX_FNO-3_7su6tgso Block Type Blocks Failing Blocks Processed
- -------------- ---------------- Data 0 17937 Index 0 24343 Other 0 43922
Finished backup at 21-OCT-19
RMAN> As you can see, in "marked corrupt" column, the number is 0. I also did another check:
SQL> select file#,block#,corruption_type from v$database_block_corruption;
no rows selected
SQL> No corruption there. There should be a utility to fix logical problems. I will unplug my tenants and create a new database and patch the database. However, I will NOT risk crap like this with a production DB.
On 10/21/19 11:55 AM, niall.litchfield_at_gmail.com wrote:
It's also worth mentioning that Note 1609718.1 mentioned in your output has at least a couple of possible causes in addition to the one you settled on. If you do have corruption in SYSAUX it's quite hard to blame datapatch for that.
On Mon, Oct 21, 2019 at 4:25 PM <dimensional.dba_at_comcast.net> wrote:
There are many of us who patch all the time for different clients and have no problems at all with datapatch, quarterly or one-offs.
You definitely need to look at your log files as to what the specific error is.
From: oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> On Behalf Of Noveljic Nenad
Sent: Monday, October 21, 2019 7:06 AM
Hi Mladen,
The patching indeed became awkward since the good old sql upgrade scripts were replaced by datapatch. But I’m afraid, there is no way back, and leaving the databases unpatched isn’t a sustainable option.
datapatch invokes “opatch lsinventory” which basically generates some shell scripts for loading the inventory.xml into the database to check what needs to be done. In the past, there have been concurrency problems, because the generated files weren’t unique ( https://nenadnoveljic.com/blog/simultaneous-patching-not-possible-datapatch/ ) . The files in /u00/oracle/orabase/product/12.2.0.x.x/cfgtoollogs/opatch/lsinv/ should provide you more information about the error.
What worked for me is to start the datapatch with -apply and –force options to bypass the opatch invocations. You can specify the patch numbers after –apply. I tested this on many releases for non-cdb and 18.7.0.0.190716 multi-tenant.
Alternatively, if your database is multi-tenant, you could try patching the containers one by one – to avoid concurrency. (I haven’t tried this yet.)
Best regards,
Nenad
From: oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> On Behalf Of Mladen Gogala
Sent: Montag, 21. Oktober 2019 01:36
I tried patching plain vanilla Oracle 12.2 on OL 7.7 and patch installation went fine. However "datapatch" part did not work:
[oracle_at_ora122 30133625]$ $ORACLE_HOME/OPatch/datapatch -verbose
Log file for this invocation: /oracle/cfgtoollogs/sqlpatch/sqlpatch_4981_2019_10_20_19_19_07/sqlpatch_invocation.log
Connecting to database...OK
Queryable inventory could not determine the current opatch status.
Execute 'select dbms_sqlpatch.verify_queryable_inventory from dual'
and/or check the invocation log
Please refer to MOS Note 1609718.1 and/or the invocation log
/oracle/cfgtoollogs/sqlpatch/sqlpatch_4981_2019_10_20_19_19_07/sqlpatch_invocation.log
for information on how to resolve the above errors.
SQL Patching tool complete on Sun Oct 20 19:19:20 2019
So, I did what the message suggests:
[oracle_at_ora122 30133625]$ sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Sun Oct 20 19:19:40 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
SQL> select dbms_sqlpatch.verify_queryable_inventory from dual;
VERIFY_QUERYABLE_INVENTORY
To: gogala.mladen_at_gmail.com; oracle-l <oracle-l_at_freelists.org>
Subject: RE: Oracle 12.2 is junk!
To: oracle-l <oracle-l_at_freelists.org>
Subject: Oracle 12.2 is junk!
SQL Patching tool version 12.2.0.1.0 Production on Sun Oct 20 19:19:07 2019
Copyright (c) 2012, 2019, Oracle. All rights reserved.
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
/oracle/cfgtoollogs/sqlpatch/sqlpatch_4981_2019_10_20_19_19_07/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
ORA-20001: Latest xml inventory is not loaded into table
So, I went to Oracle Support and looked for the message above. No problems, I am not the first one to encounter this:
datapatch Fails with "ORA-20001: Latest xml inventory is not loaded into table" due to Block Corruption "ORA-01578" (Doc ID 2364930.1)
Here is the solution:
- Fix the corrupted blocks and validate SYSAUX datafile. Note: You need to restore from valid rman backups or datapump exp/imp.
- Re-run datapatch
cd $ORACLE_HOME/OPatch
./datapatch -verbose
So, I've had a fully functional Oracle DB, I install October patches and now I have to do recovery? What idiot got rid of catbundle PSU apply? I have been patching Oracle 11g for years and now I have all sorts of problems with Oracle 12.2 and Oracle 18.8? Do we really need a patching utility that causes unnecessary problems? I rolled back the patches and will not apply any more patches to 12.2 until Oracle comes up with the sure fire way to fix the "datapatch" problems and repopulate the tables from "lsinventory".
-- Mladen Gogala Database Consultant Tel: (347) 321-1217 ____________________________________________________ Please consider the environment before printing this e-mail. Bitte denken Sie an die Umwelt, bevor Sie dieses E-Mail drucken. Important Notice This message is intended only for the individual named. It may contain confidential or privileged information. If you are not the named addressee you should in particular not disseminate, distribute, modify or copy this e-mail. Please notify the sender immediately by e-mail, if you have received this message by mistake and delete it from your system. Without prejudice to any contractual agreements between you and us which shall prevail in any case, we take it as your authorization to correspond with you by e-mail if you send us messages by e-mail. However, we reserve the right not to execute orders and instructions transmitted by e-mail at any time and without further explanation. E-mail transmission may not be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also processing of incoming e-mails cannot be guaranteed. All liability of Vontobel Holding Ltd. and any of its affiliates (hereinafter collectively referred to as "Vontobel Group") for any damages resulting from e-mail use is excluded. You are advised that urgent and time sensitive messages should not be sent by e-mail and if verification is required please request a printed version. Please note that all e-mail communications to and from the Vontobel Group are subject to electronic storage and review by Vontobel Group. Unless stated to the contrary and without prejudice to any contractual agreements between you and Vontobel Group which shall prevail in any case, e-mail-communication is for informational purposes only and is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. The legal basis for the processing of your personal data is the legitimate interest to develop a commercial relationship with you, as well as your consent to forward you commercial communications. You can exercise, at any time and under the terms established under current regulation, your rights. If you prefer not to receive any further communications, please contact your client relationship manager if you are a client of Vontobel Group or notify the sender. Please note for an exact reference to the affected group entity the corporate e-mail signature. For further information about data privacy at Vontobel Group please consult www.vontobel.com. -- Niall Litchfield Oracle DBA http://www.orawin.info -- Mladen Gogala Database Consultant Tel: (347) 321-1217 -- http://www.freelists.org/webpage/oracle-lReceived on Tue Oct 22 2019 - 18:04:34 CEST