CPU April 2005 patching madness

Boy, do I love patching.
Especially Oracle.

Especially Oracle on Windows.

You could refer to it as the Full Employment for DBA's Under Pressure act.

Having made the determination to patch the 9.2.0.6 <http://9.2.0.6>databases on a
new server that is supposed to go live at the end of this week ( OK, it was determined for me. I didn't really want to) I began the patching task Friday Evening, having first practiced it on a test server.

There are three Oracle Homes on this Win23k Server, and three Oracle databases.

Let's call them dev, qa and prd for simplicity.

Applying the patch on dev and qa went fairly smoothly.

Opatch apply worked on both, run catcpu.sql - no problems. The only errors were ORA-942's for already existing objects.

Both databases and databases services were stopped and restarted without incident. The importance of this point will become apparent later.

At this time I started the patch process for the PRD database and ran into difficulty.

The patch would not succeed due to some files being in use. I could not find any Oracle processes running, so I put them all on manual start and rebooted the server.

Normally, no big deal.

But, the server did not come back up properly.

I could ping it.

I could connect to it with the Windows Management Console (t).

But there were several services down that could not be started.

I could not connect to the box with RDP, LanDesk or PC Anywhere.

Nice.

So, I called my friendly neighborhood SA and he kindly rebooted the box for me.

This is where the fun starts.

Recall that the QA and DEV databases were successfully stopped and started prior to the reboot.

Now they will not come up.

Assume the DEV database.

Errors are:

ORA-01078: failure in processing system parameters LRM-00109: could not open parameter file
'D:\ORACLE\ORA92\PRD\DATABASE\INITDEV.ORA'
What is interesting about this is that the init.ora file location listed with the error message is incorrect.

I checked the registry. All correct.
I recreated the instance via ORADIM - no joy.

These databases are all using spfile's. So I tried starting up the database using a pfile.

SQL> startup pfile="d:\oracleora92\admin\dev\pfile\initdev.ora"

Now there is an error opening the password file:

ORA-1990 - error opening password file.

The password file is D:\oracle\ora92\dev\database\pwddev.ora

The error message for the ORA-1990 would be something like this: " Could not open password file D:\oracle\ora92\prd\database\pwddev.ora"

The file name is correct, but again, the location is incorrect.

Only by setting remote_login_passwordfile=none was I able to start the database.

Both the QA and DEV database must be now started with a script that explicitly
names the pfile, and there can be no password file.

Trying to create a new spfile is no good:

SQL> create spfile='spfile_from_pfile.ora' from pfile; create spfile='spfile_from_pfile.ora' from pfile *
ERROR at line 1:
ORA-01078: failure in processing system parameters LRM-00109: could not open parameter file
'D:\ORACLE\ORA92\PRD\DATABASE\INITDEV.ORA'
SQL> Again, the location is wrong.

All registry entries are correct.

All environment variables are correct.
ORACLE_HOME=D:\oracle\ora92\dev
ORACLE_SID=DEV
PATH contains Oracle paths from only one ORACLE_HOME, DEV.

I have opened a TAR on this, but nothing useful has yet come of it.

This server now has one database that operates correctly, the unpatched one.

At this point, I don't know if it is Oracle, Windows, Dell HW or the phase of the moon.

If you have any ideas on this, don't hesitate to share them.

If I'm lucky, I'm just overlooking something glaringly obvious and easy to fix. :)

Thanks for reading this far.

-- 
Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist

--
http://www.freelists.org/webpage/oracle-l