Re: [Non-DoD Source] 12.1-12.2 PDB Upgrade with TDE

Hi Thomas,

With a few different SR's and some serious volume of testing differing methods of sequencing. i.e. luck. I have the entire end-to-end process, pre-upgrade, post upgrade etc. scripted now quite well in bash, to take batches of PDBs at a time, and it works very successfully and is robust.

We export the TDE keys as usual from each PDB on the source CDB side. Once you clone/create the PDB into the 12.2 CDB from the 12.1 CDB - and I mean immediately following the 'create pluggable database etc' command, you open in upgrade mode. (i.e. MIGRATE mode). Its worth noting at this point that you may still *not* have a PDB_PLUG_IN_VIOLATIONS entry for a required key. However, you should essentially proceed with the upgrade process prior to looking to do any key import steps, so we initially focus on the PDB version upgrade rather than any complaints around the key.

**One of the points at which we were constantly falling down was attempting to open the PDB read-write immediately post the clone operation, in order to try and resolve the key situation prior to the upgrade execution. This problem point in the sequence also caused us issues with automatic local UNDO tablespace creation as well in the 12.2 CDB. *

Then, you execute the catupgrd scripted upgrade process. Once complete successfully, we perform a normal pluggable database OPEN on the *local* RAC instance only. This should result in and you should see a pending entry for a PDB_PLUG_IN_VIOLATIONS 'wallet key required'. At which point you import the key as per usual AKM command. A final close/open of the PDB then to get it out of restricted mode.

One of our other nuances we later observed was that we had a mixed bag of non-TDE PDB's in a container alongside TDE PDB's. The special thing I also discovered with that setup, is that with 12.2 DB_LINK cloning you can "carry" the key with the 'create pluggable database...' command. i.e. ship the TDE key with the clone operation. However, if you do this (12.2 online clone via DB_LINK from CDB1 to CDB2) for a non-TDE PDB, you will figure out quite fast that even if the specific PDB does *not* have a key associated with it, the online nature of the clone operation (redo between start-finish) results in a backwards dependency to the CDB$ROOT key of the original source CDB. This took us a while to figure.

Kind regards,
Ruan

On Tue, Apr 28, 2020 at 2:26 PM Terrian, Thomas J CTR DLA INFO OPERATIONS (USA) <Tom.Terrian.ctr_at_dla.mil> wrote:

> Ruan, I am facing the same problem. How did you fix this?
>
>
>
> *From:* oracle-l-bounce_at_freelists.org <oracle-l-bounce_at_freelists.org> *On
> Behalf Of *Ruan Linehan
> *Sent:* Wednesday, January 1, 2020 6:40 AM
> *To:* Oracle-L oracle-l <oracle-l_at_freelists.org>
> *Subject:* [Non-DoD Source] 12.1-12.2 PDB Upgrade with TDE
>
>
>
> Hi list,
>
> I am very stuck and really beginning to tread water on an SR with the
> below scenario. Would anyone have experience of a 12.1-12.2 Multitenant PDB
> upgrade path via remote clone including TDE??
>
> Source 12.1 Multitenant PDB on Exadata RAC with TDE.
> Destination 12.2 Multitenant PDB on Exadata RAC (Separate CDB on the
> same cluster with separate software wallet).
>
> I am attempting to perform a 12.1 --> 12.2 individual PDB upgrade via
> remote clone between CDB containers. I have verified the process as
> successful when using a non-TDE [Tablespace] specific PDB. i.e. perform the
> necessary preupgrade.jar steps in the 12.1 source side, clone the PDB over
> a DB link into the 12.2 destination side and then proceed with the
> catctl/catupgrd scripting to perform the upgrade. No issues and all works
> fine... without TDE.
>
> However, when TDE is in the mix on the source PDB, I cannot manage to
> succeed in transitioning the TDE key properly from source side over into
> the destination side (Either before, during or post the upgrade scripting).
> Below is a summary of steps for context but the issues comes down to; how
> do I get a TDE key imported successfully when cloning from a 12.1 CDB to a
> 12.2 CDB...
>
>
> *High Level summary:*
>
> 1. Export the TDE key from the source 12.1 side (*From within the PDB
> itself)...
> - administer key management set keystore open identified by
> "password1";
>
> - administer key management export encryption keys with secret
> "password1" to '/tmp/source_side.p12' identified by "password1";
>
> *Note1* - I believe the export operation is 100% clean as the same steps
> used for this export work seamlessly when performing clones of TDE specific
> PDBs between *12.1* CDBs.
>
> 2. Put the source PDB in ReadOnly...
> - alter pluggable database ORCL close immediate instances=all;
> - alter pluggable database ORCL open read only instances=all;
>
> 3. Perform DB Link clone between containers (*From the CDB$ROOT of the
> destination 12.2 CDB)
> - create pluggable database ORCL122 from ORCL_at_CLONE_LINK standbys =
> none;
>
> *Note2* - The DB link is fine as the PDB creates in restricted mode. I
> have also attempted incorporating the "keystore identified by" syntax as
> part of the create statement.
> i.e. create pluggable database ORCL122 from ORCL_at_CLONE_LINK keystore
> identified by "password1" standbys = none;
>
> 4. Open the cloned pluggable...
> - alter pluggable database ORCL122 open instances=all;
>
> *Note3* - At this point, the absence of the TDE required key is evident
> in the alert log messaging, "PDB needs to import keys from source". But
> there is no PDB_PLUG_IN_VIOLATION entry present yet, for the TDE key. Only
> an error as expected for the upgrade path, "PDB's version does not match
> CDB's version", which is quite expected. The PDB exists in restricted mode,
> ready for upgrade.
>
> 5. It's from this point that I run into difficulty. I can try to proceed
> with the catupgrd script operations firstly. Once that completes (seemingly
> cleanly), there is a PDB_PLUG_IN_VIOLATION entry for the TDE key
> requirement. I can then try and import the TDE (*From within the 122 PDB
> itself) key per:
> - administer key management import encryption keys with secret
> "password1" from '/tmp/source_side.p12' identified by "password1" with
> backup;
>
> Alternatively, I could try and perform the import 'ahead of time' at the
> CDB$ROOT container, prior to the clone operation at all. This command
> succeeds as I can see the additional entry populated into the
> v$encryption_keys view and within the software wallet, detailing the source
> of the key.
>
> Alternatively, I can try and transition the key, from the CDB$ROOT
> container levels between CDBs instead using the "WITH IDENTIFIER" clause
> syntax as I note that certain operations are directed to be performed at
> the root level from 12.2 forward, but the very same result occurs in the
> end...
>
>
> I can observe the messaging "Keys imported: PDB ORCL122 can now open in
> normal mode" recorded in the alert log and the PDB_PLUG_IN_VIOLATION entry
> disappears. Great!
> However, no matter how I "get the Key in", post a close/open of the PDB
> (To get it out of restricted mode) - I immediately start receiving errors
> in the alert log around master key not found in current keystore and typed
> master key not found in wallet. It seems no matter how I try to transition
> the correct key INTO the 12.2 destination side, the AKM commands appear to
> succeed but upon proper opening of the PDB, I am flooded with TDE errors in
> the alert logs.
>
>
> Mr. Pachot made it look frustratingly easy here - my experience differs
> significantly, and not sure where I am going wrong:
> https://blog.dbi-services.com/12cr2-upgrade-by-remote-clone-with-tde-in-dbaas/
>
>
> So the TL;DR version of the above is simply...
> a) Is this manner of migration path supported?
>
> b) At what stage/point in the sequence should I look to perform the TDE
> key IMPORT on the destination side when cloning between 12.1-12.2 CDBs for
> an upgrade in this manner?
> c) And should the tried and trusted 12.1 key import syntax work
> effectively for importing the key on the 12.2 destination side or am
> missing something?
>
> "administer key management import encryption keys with secret
> "password1" from '/tmp/source_side.p12' identified by "password1" with
> backup;""
>
> Regards,
>
> Ruan
>

--
http://www.freelists.org/webpage/oracle-l