Re: Bribing the SA (Mythos vs Reality)

Jack,

Fair enough... I should have been more specific... OPS is not an HA solution to loss of a site (that city grid going down -- we had a Con Ed substation blow a transformer -- OPS wouldn't have helped in that case, you can't bring up an instance without power and I can't restore and roll forward either). I do tend to use HA solution interchangeably with BC solution. I just want to make sure I can continue business under any conditions short of nuclear war. :)

Not all failures are hardware related... some are code, some are "fat fingers", hey, I freely admit to dropping production tablespaces accidentally... I ALWAYS admit when I screw up, gives me credibility when I say "this time it's not me"

Car washes -- depends on if your SA/DBA/admin likes his/her car shiny or not :)

Raw -- I don't know how to work it or administer it (from a hardware/OS level) so I would need an extremely competent SA to work with. There are many of those. Unfortunately, not every company has one. Or the one they have is so overworked that he/she is unable to help when and as needed. There is some more flexibility in using file system files, in that as long as there is disk space available somewhere, and I have access to it, I do not have to disturb my SA to add space to my database. And there are ways to make file systems work almost as fast as raw.

Having said all that.... it is ALWAYS a pleasure to work with someone who knows what they are doing. I have worked with SAs I've adored, and with DBAs I've hated. The only thing I really hate is someone who is incompetent who tries to pass him/herself off as an expert.

My SAs don't NEED bribes to get them to help me, but I DO appreciate the work they do and bring them presents -- like those Hershey's kisses -- to let them know that. Come holiday time, I'll bake and bring in cookies too.

I am glad you stopped lurking, even if only for the moment... you keep us, or at least me, honest here :)

Rachel
>From: Jack L Bell <jack_at_loyola.comconn.com>
>Reply-To: ORACLE-L_at_fatcity.com
>To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
>Subject: Re: Bribing the SA (Mythos vs Reality)
>Date: Mon, 16 Oct 2000 20:45:29 -0800
>
>
>time to come out of the woodwork.. where I've lurked, lo, these many
>years..
>
>A confession- I'm what you folks call "an SA". Been supporting DBAs in
>their efforts for "a while". Have a passing familiarity with several
>databases, and how to make them happy on the hardware- and have no
>interest in formatting in TeX, or programming in SQL. Enjoy my work- find
>it extremely challenging... *my turn..* :-)
>
>And my response...
>
>We (well, at least, *I*) prefer a professional bit of give and take-
>
>ok- we're running a small OPS cluster (5 TB) on a pair of nodes-
>50+ CPUs each. We do it for 1) HA 2) resource sharing and 3)
>cause we figured out how to make it work *for* us, rather than against us.
>
>myth..er.. points to address...
>
>1) - all failures are hardware related
> *WRONG*- most are *admin related (hey- he admitted to.. no, wait a
> second- he didn't say it was a *system* admin- just "admin"..)
> Yep- restores in the * years due to SAs- 1 (and it was at the behest
>of the
> DBA..- "..we don't need those volumes anymore.."). the rest were-
>well-
> lots'a data? lots'a database manipulation. Who does that? Hardware
> reasons? uhh.. *who* set up the system? There aren't that many *true*
> "double disasters"- and if you've dealt with all the single disasters,
> ( and have two of everything, besides..) just what could *possibly* go
> wrong? (ever notice the "nice" way the naturalists describe ****? -
> animal "drop"pings... but, still, it happens..:-) )
>
>2) OPS is not an HA solution
> uhh... you can bring *up* an instance, and roll back faster than I
> can roll back one that's already up? Ok- if you say so.
>
>3) A single point of failure does not an HA solution make- the drives are
> all a single point of failure..
>
> .. as is the power grid, city, block, building...
> we don't have many "HA" solutions that have 15 miles of cable strung
> between them (minimum recommended distance)- those are called
> "Business Continuance" (BC) solutions- HA solutions are *extremely*
> similar to an OP$ $olution- they ju$t don't involve a$ much in the
> way of licen$e fee$- but both must *$ee* the same di$k drive$...
> please see #1, above. You can make a *lot* more use out of an
> OPS cluster than something running "active-standby" (please note
> that the BC $ite - if running an OP$ cluster- can also (8i) be
> open for reporting in read-only mode while in automated recovery..)
>
>4) car washes are a decent bribe..
> Latex paint only lasts so long- why cut the life by washing??
>
>I'm also a firm believer in RAW I/O, .. RAID 10, and the value of someone
>who *really* knows what they're doing... in whatever. Whew.
>(now, where was that dark corner that I was... there it is...
>(bye for now) )
>
>-Jack
>
>
> >
> > Steve,
> >
> > The point is to find out what bribes YOUR SAs. In my case, the SAs want
> > chocolate (and we stock M&M's, pretzels, any General Mills cereal that
>has
> > the free CD from our company offer on the box, popcorn, etc). One of
>the
> > programmers wants "healthy snacks" -- I'd bring him celery and carrot
> > sticks.
> >
> > You might have to bribe with things like, oh say, car washes :)
> >
> > Rachel
> >
> >
> > >From: "Steve Orr" <sorr_at_arzoo.com>
> > >Reply-To: ORACLE-L_at_fatcity.com
> > >To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
> > >Subject: RE:Bribing the SA and company bribes in general
> > >Date: Mon, 16 Oct 2000 16:50:39 -0800
> > >
> > >Hi Rachel,
> > >
> > >That doesn't work here... we have a nearly unlimited supply of
>chocolate,
> > >candy bars, power bars, snacks, popcorn, pretzels and the like, the
> > >refrigerator is constantly stocked with soft drinks, bottled water,
> > >Snapple,
> > >etc. Sigh... the waste-line hazards of working for a startup where they
>try
> > >to bribe EVERYONE. I understand that Microsoft's increased employee
> > >productivity secret is free espresso.
> > >
> > >Whining all the way to the fridge again...
> > >Steve Orr
> > >
> > >
> > >
> > >-----Original Message-----
> > >Carmichael
> > >Sent: Monday, October 16, 2000 5:21 PM
> > >To: Multiple recipients of list ORACLE-L
> > >
> > >
> > >or at least finding out what bribes the SA will accept (which is why I
> > >brought in a large bag of Hershey's kisses today <G>)
> > >
> > >
> > >
> > >
> > > >From: "Steve Orr" <sorr_at_arzoo.com>
> > > >Reply-To: ORACLE-L_at_fatcity.com
> > > >To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
> > > >Subject: RE: SAC NORAD .... how to break it?
> > > >Date: Mon, 16 Oct 2000 14:00:48 -0800
> > > >
> > > >Apart from the obvious it's going to vary so you'll need to create
>your
> > >own
> > > >checklist of things to test for your particular environment. You'll
>also
> > > >want to test that you have good monitoring in place. I once thought I
>was
> > > >protected with mirrored redo logs only to find out one drive had
>failed a
> > > >month before and the sysadmin wasn't monitoring the mirror. You
>should
> > > >probably start by taking your sysadmin out to lunch.
> > > >
> > > >Steve
> > > >
> > > >
> > > >-----Original Message-----
> > > >Linda
> > > >Sent: Monday, October 16, 2000 1:50 PM
> > > >To: Multiple recipients of list ORACLE-L
> > > >
> > > >
> > > >Steve/Anyone -
> > > >
> > > >We're going to be doing HA in the near future. I'm sure I can come
>up
> > >with
> > > >obvious tests, like pull a disk, turn off a machine.... But do you
>have
> > >a
> > > >'break it' checklist? (sorry if this has been asked before)
>Especially
> > > >how
> > > >it affects Oracle performance. I want this thing to run smoothly.
> > > >
> > > >Unless of course we have a 'real' test of Norad in which case I'm
>close
> > > >enough that I don't care!
> > > >
> > > >Linda
> > > >
> > > >-----Original Message-----
> > > >Sent: Monday, October 16, 2000 11:36 AM
> > > >To: Multiple recipients of list ORACLE-L
> > > >systems
> > > >
> > > >
> > > >Actually, NORAD was designed to survive a direct hit as was capable
> > >during
> > > >the time it was build. However, with more accurate delivery systems
>now
> > >it
> > > >is conceivable that a missle could navigate part way through the
>entrance
> > > >tunnel so as to make the facility inoperable. Then there are multiple
> > > >direct
> > > >hits...
> > > >
> > > >But of course, none of this has been tested and sadly, this is often
>the
> > > >case with HA 24X7 systems. You need sufficient pre-production quiet
>time
> > >to
> > > >test your HA solution. I call it the "pseudo sledge hammer" testing
> > >period.
> > > >Have you ever taken a drive out of your RAID and replaced it to see
>how
> > > >long
> > > >it takes for resilvering and what happens to I/O performance? How
>much
> > >time
> > > >does it take to test the entire HA implementation and how much time
>will
> > > >you
> > > >be given? The trouble is that you get all this expensive equipment in
>the
> > > >data center and install Oracle then damagement is anxious to get the
> > >entire
> > > >application up and running ASAP and asks you to take short cuts or
>just
> > > >trust that everything will work. But really you haven't finished the
>job
> > > >until you've reasonably tested everything end to end.
> > > >
> > > >IMHO,
> > > >Steve Orr
> > > >
> > > >
> > > >-----Original Message-----
> > > >Sent: Monday, October 16, 2000 7:06 AM
> > > >To: Multiple recipients of list ORACLE-L
> > > >systems
> > > >
> > > >
> > > >That's why they say that SAC/NORAD ( Strategic
> > > >Air Command HQ, North American Defense ) buried
> > > >deep into a mountain in Colorado is a "single point
> > > >of failure" for the US NationalDefense:
> > > >
> > > >All it takes is a direct hit by one nuclear
> > > >bomb to bring down the whole facility! :-)
> > > >
> > > >In the words of the Marathon Man's tormentor:
> > > >
> > > >"Is it safe?"
> > > >
> > > ><evil laughter>
> > > >
> > > >
> > > >-----Original Message-----
> > > >Sent: Friday, October 13, 2000 7:45 PM
> > > >To: Multiple recipients of list ORACLE-L
> > > >
> > > >
> > > >Sorry Ross. Yes I am familiar with enterprise
> > > >class storage systems.
> > > >
> > > >It still isn't HA.
> > > >
> > > >It only takes one bumbling SA ( or DBA ) to bring
> > > >the system down, one neanderthalic techie in the
> > > >computer room to push the 'OFF' switch.
> > > >
> > > >Simultaneous failure of both of the controllers for
> > > >an array, or of enough disks to bring the array down
> > > >are not unheard of.
> > > >
> > > >Jared
> > > >
> > > >On Fri, 13 Oct 2000, Mohan, Ross wrote:
> > > >
> > > > > I have to say this "disk is a single point of failure"
> > > > > is jangling to the cognitive logic subsystem.
> > > > >
> > > > > Why?
> > > > >
> > > > > Well, the disk farms i have seen have redundant controllers,
> > > > > with redundant channels, TRIPLE power supplies, at least a
> > > > > single mirror with dual porting. There's your "single" disk
> > > > > point of failure for you.
> > > > >
> > > > > Now, try this: Take your two "redundant" nodes....put them
> > > > > in a really really big rack and then inside ONE big box. <G>
> > > > >
> > > > > Are the two nodes ( which now have at least redundant CPUs,
> > > > > power supplies, etc. ) a "single point of failure"?
> > > > >
> > > > > Come on, guys, if you've worked with this stuff a bunch you know:
> > > > >
> > > > > (a) properly configured diskfarms have a great MTBF, better
> > > > > than the other hardware, and
> > > > > (b) to REALLY answer Mary's class of questions, you need to
> > > > > calculate MTBFs and MTTRs.
> > > > >
> > > > > The rest is armchair clustering!
> > > > >
> > > > > hope this pertains,
> > > > >
> > > > > Ross Mohan
> > > > >
> > > > > p.s. HA is the latest marketspeak for "failover" or "redundant"
>or
> > > > > whatever...
> > > > > please try to browse a copy of "In Search of Clusters" by Gregory
> > > >Pfister
> > > > > from
> > > > > IBM. It's a cult classic, a helluva fun read, and one of the best
> > > > > thought-out
> > > > > technical books i have ever seen, period.
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > Sent: Thursday, October 12, 2000 2:00 PM
> > > > > To: Multiple recipients of list ORACLE-L
> > > > >
> > > > >
> > > > >
> > > > > Mary,
> > > > >
> > > > > OPS is not an HA solution. While you may still have
> > > > > an instance running if a node goes down, the storage
> > > > > medium is still a single point of failure.
> > > > >
> > > > > Jared
> > > > >
> > > > > On Thu, 12 Oct 2000, Ruiz, Mary A (CAP, CDI) wrote:
> > > > >
> > > > > > I need a little advice. We have a fairly new (< 1 year) 8.1.5
> > > >instance
> > > > > to
> > > > > > support my company's internet business. We recently changed our
> > > >network
> > > > > > solutions provider and now my management wants to achieve a
>higher
> > > >level
> > > > > of
> > > > > > redundancy than it currently does with mirrored disks. The
>solution
> > > >being
> > > > > > proposed by my Sysadmin is an Oracle Parallel Server solution.
>Some
> > > > > > background is in order here - we have always shut our databases
>down
> > > >at
> > > > > > night for backups. I am not highly skilled in backup and
>recovery
> > > > > although
> > > > > > I tried some of the hot backup techniques from this list and was
> > >able
> > > >to
> > > > > > recover successfully to another server. I noticed that the
>course
> > > >offered
> > > > > > by Oracle in OPS has backup and recovery as well as performance
> > >tuning
> > > >as
> > > > > > pre-requisites, which indicates to me that OPS could be
>extremely
> > > > > > challenging. Also, I have read mainly unfavorable comments
>about
> > >OPS
> > > >from
> > > > > > this list, but most of those comments were based on the Oracle 7
> > > > > > implementations (High administrative costs, difficult to
>implement,
> > > >etc.).
> > > > >
> > > > > >
> > > > > > Have things improved with Oracle 8i ? Is OPS worth pursuing? Or
> > > >should
> > > >I
> > > > > > convince my management that extra $$ spent in, say, a hot
>standby
> > > >database
> > > > > > is well worth it? Is there any other solution that would not
> > >involve
> > > >a
> > > > > > second set of disks, rather a second database on the same set of
> > >disks
> > > >??
> > > > > >
> > > > > > Thanks in advance,
> > > > > > Mary Ruiz / Atlanta
> > > > > >
> > > > > > --
> > > > > > Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > > >also send the HELP command for other information (like subscribing).
> > > >
> > > >--
> > > >Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > > >--
> > > >Author: Steve Orr
> > > > INET: sorr_at_arzoo.com
> > > >
> > > >Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > > >San Diego, California -- Public Internet access / Mailing
>Lists
> > > >--------------------------------------------------------------------
> > > >To REMOVE yourself from this mailing list, send an E-Mail message
> > > >to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > >the message BODY, include a line containing: UNSUB ORACLE-L
> > > >(or the name of mailing list you want to be removed from). You may
> > > >also send the HELP command for other information (like subscribing).
> > > >--
> > > >Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > > >--
> > > >Author: Seley, Linda
> > > > INET: LSeley_at_IQNavigator.com
> > > >
> > > >Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > > >San Diego, California -- Public Internet access / Mailing
>Lists
> > > >--------------------------------------------------------------------
> > > >To REMOVE yourself from this mailing list, send an E-Mail message
> > > >to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > >the message BODY, include a line containing: UNSUB ORACLE-L
> > > >(or the name of mailing list you want to be removed from). You may
> > > >also send the HELP command for other information (like subscribing).
> > > >
> > > >--
> > > >Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > > >--
> > > >Author: Steve Orr
> > > > INET: sorr_at_arzoo.com
> > > >
> > > >Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > > >San Diego, California -- Public Internet access / Mailing
>Lists
> > > >--------------------------------------------------------------------
> > > >To REMOVE yourself from this mailing list, send an E-Mail message
> > > >to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > >the message BODY, include a line containing: UNSUB ORACLE-L
> > > >(or the name of mailing list you want to be removed from). You may
> > > >also send the HELP command for other information (like subscribing).
> > >
> >
> >_________________________________________________________________________
> > >Get Your Private, Free E-mail from MSN Hotmail at
>http://www.hotmail.com.
> > >
> > >Share information about yourself, create your own public profile at
> > >http://profiles.msn.com.
> > >
> > >--
> > >Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > >--
> > >Author: Rachel Carmichael
> > > INET: carmichr_at_hotmail.com
> > >
> > >Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > >San Diego, California -- Public Internet access / Mailing Lists
> > >--------------------------------------------------------------------
> > >To REMOVE yourself from this mailing list, send an E-Mail message
> > >to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > >the message BODY, include a line containing: UNSUB ORACLE-L
> > >(or the name of mailing list you want to be removed from). You may
> > >also send the HELP command for other information (like subscribing).
> > >
> > >
> > >--
> > >Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > >--
> > >Author: Steve Orr
> > > INET: sorr_at_arzoo.com
> > >
> > >Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > >San Diego, California -- Public Internet access / Mailing Lists
> > >--------------------------------------------------------------------
> > >To REMOVE yourself from this mailing list, send an E-Mail message
> > >to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > >the message BODY, include a line containing: UNSUB ORACLE-L
> > >(or the name of mailing list you want to be removed from). You may
> > >also send the HELP command for other information (like subscribing).
> >
> >
>_________________________________________________________________________
> > Get Your Private, Free E-mail from MSN Hotmail at
>http://www.hotmail.com.
> >
> > Share information about yourself, create your own public profile at
> > http://profiles.msn.com.
> >
> > --
> > Please see the official ORACLE-L FAQ: http://www.orafaq.com
> > --
> > Author: Rachel Carmichael
> > INET: carmichr_at_hotmail.com
> >
> > Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
> > San Diego, California -- Public Internet access / Mailing Lists
> > --------------------------------------------------------------------
> > To REMOVE yourself from this mailing list, send an E-Mail message
> > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > the message BODY, include a line containing: UNSUB ORACLE-L
> > (or the name of mailing list you want to be removed from). You may
> > also send the HELP command for other information (like subscribing).
> >
>
>--
>Please see the official ORACLE-L FAQ: http://www.orafaq.com
>--
>Author: Jack L Bell
> INET: jack_at_loyola.comconn.com
>
>Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
>San Diego, California -- Public Internet access / Mailing Lists
>--------------------------------------------------------------------
>To REMOVE yourself from this mailing list, send an E-Mail message
Received on Tue Oct 17 2000 - 10:26:09 CDT