Re: Quick Tale: Lost production database because of keyboard and Veritas Cluster Server

In comp.sys.sun.hardware RSH <RSH_Oracle_at_worldnet.att.net> wrote:
> As Yoda said, "There is no 'try', there is just DO.", when it comes to
> production.

        I'm not so sure about that. Let me tell you about one of my production horror stories. I was going to hot-swap a drive with some software on it, and I went through all the proper procedure, shutting down the necessary software and unmounting the partitions and quiescing the bus, and then I pulled the drive. And then I realized I had the root drive in my hand.

        The difference? I was doing this on an idle box in the server room to make sure I could do it /before/ I did it on the live system. After the initial "oh shit" moment, I said, "well, I'm not going to do /that/ again! Next time I will make sure I have the right drive in hand." And I did. So yes, you only get one chance to do it for real... but you can make damn sure by doing a dress rehearsal.

        (A dress rehearsal, if such a thing is possible. In too many places I've worked, the DBAs have development environments, the coders have development environments, the content developers have dev environments, and the sysadmins get nothing. You want to try out some strange new hardware or something? Tough. The dev environments aren't "dev" for you - people will scream bloody murder if they can't get to their stuff for one damn hour because you were trying something out. Other people are allowed to break their toys all they want, but you aren't given anything you're allowed to break. (To be completely honest, I suspect the DBAs often feel the same - it's called a dev database, but the coders expect it to be available all the time just the same, and complain that they can't do their work if it isn't there.) If I ever run an IT department, I'm going to ensure that everyone has something they can mess around with, something they don't have to answer to anybody else for.)

JDW

-- 
If mail to me bounces, try removing the "+STRING" part of the address.