Re: Quick Tale: Lost production database because of keyboard and Veritas Cluster Server

In comp.sys.sun.hardware RSH <RSH_Oracle_at_worldnet.att.net> wrote:

> But the poor guy SAID he was NOT a DBA and wasn't really familiar with how
> Sun's work, or Oracle, or Veritas, or failover. And all the things you
> mention are things that you have to know, to be able to do them, which he
> didn't and therefore couldn't.

        This is an issue with his management (where the real fault appears to lie). It sounds like they were being cheap asses and rushing to get something out the door without providing the right people (or to be fair, maybe he is the right person for the job without the time/training/experience yet).

> That's the trouble with these USENET etc forums, I hate seeing people in the
> middle of a disaster being yelled at; that's kind of like using a megaphone
> to yell at a family being taken away by the National Guard, in a boat from
> their home, that's now under water, and saying, "YOU SHOULDN'T HAVE BUILT IN
> A KNOWN FLOOD PLAIN AREA! I HOPE YOU AT LEAST TOOK OUT NATIONAL FLOOD
> INSURANCE!".

> As I said, all you said was useful information, but don't you think it's a
> little mean to dump a whole ton of woulda / coulda / shouldas onto the
> shoulders of a guy who probably already feels pretty lousy, and is just
> begging for help to get out of a mess, and not lectured about all the things
> done wrong or not done that led to this crisis? Over many of which he
> probably had little or no control?

        I wasn't trying to make him feel bad over his mistake. Maybe it came across as such. The point I was trying to drive home was: He was blaming the software, which was not at fault, it was a mis-understanding (or non-understanding) of how VCS works, it's capabilities, and it's limitations.

> Well, I guess its a matter of different management philosophies. If someone
> on my team makes a big boo-boo, we get it fixed working together, then the
> team gets together to do a post mortem, and we turn tragedy into an
> educational experience; making someone feel worse about something they did
> that they already know was wrong, is not the way I handle my DBA's,
> programmers, SA's, and network folks. The man feels bad enough already.

        Well, although you seem to think I was just being rude to him I was actually giving some constructive comments of how to handle this. Is that not educational? So, next time this guy probably knows better than to diddle with a production machine that's not backed up and if the thing does fail the cluster not to leave the failed side up even if you do "revive" it. Maybe he can also convince his management with some of the comments here that they shouldn't be so tight and get someone in for training if this is critical to them.

-- 
B.M. Wright
bmwright_at_xmission.com