Re: Quick Tale: Lost production database because of keyboard and Veritas Cluster Server

• RSH <RSH_Oracle_at_worldnet.att.net> wrote:
  > Well, as being one of your defenders...
  >
  > You really didn't help yourself out a lot with a
  > "try and try again" sort of
  > attitude as regards PRODUCTION.
  

If I was trying to win, I wouldn't have posted, I knew people were going to lay into me. I was hoping someone might give me a hint as to what I should have done. If I didn't get advice, it'd be many years until I saw the same thing with more experience, or even heard about the same thing.

Production really has no meaning to anyone (especially management) until the thing breaks. Like why did two hosting center techs install a KVM on a production server? I didn't even think about it until it caused a problem.

Certainly RAID, Tape backup drivers, medicine, air bags, seatbelts, floatation devices didn't come from perfect pre-planning. Certainly these safety devices didn't come from a need to save more test beds / testing environments. They came about because of demand from production systems. Which means someone is having a problem.

> That would rather sort of be like an open-heart
> surgery patient on the table
> overhearing the last words before anaesthesia sets
> in like : "Oh well, if it
> doesn't work on this guy, next time let's not split
> open the next patient's
> chest, we'll try and implant the electrodes
> internally; maybe the survival
> rate will improve somewhat."

Medicine is an applied science. The above is exactly what I'm talking about. This is exactly how doctors learn.

You only laugh at this because you are informed from others (hundreds of years of work) that the chest is full of organs, biology and the basics of the human body.

Instead, imagine if you didn't know these things and you belived that the Sun (your god) was causing you to have heartburn. Would you try to invent TUMS or would you pray to the Sun?

I think you would pray.

In fact, element belief (earth, wind, fire and air) can be found in "educated" medical practices in Europe in the 1800s.  

> As Yoda said, "There is no 'try', there is just
> DO.", when it comes to
> production.

Yes, I DID. I DID unplug the keyboard. Had I taken a class, I would have TRIED not to do such a thing. Had I been more involved in Sun Hardware instead of software configs, I'd probably know better. However, in some respects I'm starting from scratch so my experience was from PS/2 keyboards on x86 Linux. The applying of the logic didn't work this time. And I'm not the only one. People make this mistake all the time. Our Oracle rep has seen _the exact same thing_ with other clients.

It's like me pulling the trigger of a spray water bottle vs a handgun.  The action is the same but the consequences are very different.

What if I'm a bushman. I'd probably accidently blow my own head off:

**BLAM!**
"But I thought it was a spray bottle!"

If I did, someone would learn something and document it, put it in a book that would somehow make it in to a school system. And what do you know? Science!

Science works because engineers are figuring things out in the field. You can't completely reason and think your way through problems without trying. Yoda is a great character, but I'd like to see him compile 1000 lines of Java code without getting exceptions. Or make drastic DNS zone changes on a production system. I think he would DO it on a test bed, and then DO it on the production system. Sounds a lot like trying to me ...

But it comes back to my fault. I should have tested the keyboard plug on an extra 420r. Unfortunately, we didn't have one ... though I could have pulled the keyboard on the other 420r that wasn't live ...

But:

1. Testing it on a non-live system
2. Reading docs
3. Asking someone who knows
4. Telling people I'm about to do it
5. Stopping the database
6. Configuring Veritas failover
7. Doing an export just in case

all assumes that I knew that pulling a keyboard plug sends a Stop-A. And I didn't know that.

Doctors know how to save your life because they've killed people just like you and me. It's a good thing we all have the same specification / design ... else, I don't think I'd be paying for BlueCross / BlueShield.

Just for the record, I'm sick of talking about this crap already. I'm going to go home to my non-production Linux box, design JSP/Servlets for the hell of it, have a coffee, get it working, feel accomplished and have something go right for once. And when I do get it working, I don't have to tell management all about it in layman's terms, I don't have to get permission to delete partitions and user accounts.

And I certainly can unplug my USB keyboard in and out all day without crashing jack shiznit.

THE END --Milk Received on Tue Mar 05 2002 - 16:22:59 CST