| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
 |
 | |||
| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
|
| |||
Home -> Community -> Usenet -> c.d.o.server -> Re: DBA Experiences from the edge
Andreas Sheriff wrote:
>
Similar experience about 10 years ago. Only this was not development. It was a major production warehouse. Only this was not just one machine. It was a cluster.
The vendor was suppose to upgrade the kernel on all cluster nodes on the weekend. Contacted the Unix sysadmin and me for more space. There were none. So we told them they can break the TEMP space mirror. Which they did and continued to do their thing.
On Sunday I was contacted and told that everything was done and I can startup the database. Similar errors as yours. Tried the same thing too. TEMP space datafiles "corrupted". Weird I thought as the broken mirror should not be causing it. Anyway, I did a offline drop and one more.. and one more.. and then suddenly it was no longer TEMP space, but user space too..
To cut a long story short, this o/s upgrade somehow screwed up every single raw device on that cluster - because a shortcut was attempted (they had to do two o/s upgrades in essence and tried to hack it doing the o/s upgrade to the new version without going through the intermediate o/s version)
The vendor blamed the breaking of the mirror for the problem - and the Unix sysadmin and me for allowing it. To the extend that their CEO wrote an official letter pointing fingers at the two of us. Needless to say I threw my toys and grabbed my old lead pipe. Made so much noise and rock the boat so hard that both Unix sysadmin and I got letters from the chairman of the holding company of the company we worked for, saying that they do not blame us. Nor think it our fault.
Worse was the backups. Backups were done by running dd. Both Unix sysadmin and I did not like. Were forced to assign the acceptance certificate for this (multi million project and were told by our bosses to sign it as we were holding it up when the system was installed). So we did and added on the certificate that we do not believe this will work.
And we were right. Backups were restored and these did not work either. Oh boy.. did the shit hit the fan (and did the vendor get splattered).
The vendor? Does not exist anymore - has folded long ago and pieces of it is operating for many years now with a new name and Japanese partnership.
Moral of the story: happiness is a warm well-used and bloodied lead pipe.
-- BillyReceived on Wed Jun 22 2005 - 01:11:27 CDT
 |
 |