| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
 |
 | |||
| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
|
| |||
Home -> Community -> Mailing Lists -> Oracle-L -> RE: Taking your time when a crisis occurs
You have to be like an ankylosaurus, with armour plating everywhere, and spikes on your tail. The armour plating to deal automatically with frets while you continue concentrating on your work, and spikes to keep people away to give you room to get the job done.
If you can't bring the database back up you KNOW you will be out of a job.
If you can bring it back, they can fire you but at least you will know that the database came back up. Be prepared to explain everything you do. If you can demonstrate that you took the time that it would have taken, then the monkey is off your back and if people are not happy they will look for other causes (OPS? Other vendor? More staff? More hardware? Better, more thorough testing procedures before implementing change?).
Your only option is to be methodical, and bring the system back up. It will take the time that it takes - think things through, do it right the first time. One step at a time, until it's complete. Focus. There's no time to panic anyway, do that after the crisis is over. Fear and anger just waste time. Life is too short for that nonsense on "regular" days, it's even more obvious while there is a crisis. Chickens flying around in a wild frenzy never manage to achieve anything constructive. Listen to their concerns, if there is a good comment use it, otherwise just let them lose feathers.
If some people need help leaving you alone, arrange for that to happen - tell your boss that interference is slowing down recovery time, please keep this person away from me. Maybe appoint someone to filter messages and have one contact person.
It also helps others to see that the person responsible for bringing the system back up is methodical and calm. You can tell people in many cases: "We haven't lost any data, it's all there. I am recovering the database now as fast as I can, but it WILL take some time. Can't say how long right now, but I am going as fast as possible, believe me I want this system back up as much as you do."
Regards,
Patrice Boivin
Systems Analyst (Oracle Certified DBA)
Systems Admin & Operations | Admin. et Exploit. des systèmes Technology Services | Services technologiques Informatics Branch | Direction de l'informatique Maritimes Region, DFO | Région des Maritimes, MPO
E-Mail: boivinp_at_mar.dfo-mpo.gc.ca <mailto:boivinp_at_mar.dfo-mpo.gc.ca>
Ph: (902) 426-4774
-----Original Message-----
From: Rachel Carmichael [SMTP:carmichr_at_hotmail.com]
Sent: Monday, May 07, 2001 11:36 AM
To: Multiple recipients of list ORACLE-L
Subject: Re: Taking your time when a crisis occurs
want to tell me how you hold off the CEO who is breathing down your
neck on
the 24x7 database that's down?
>From: "William Beilstein" <BeilstWH_at_obg.com>
>Reply-To: ORACLE-L_at_fatcity.com
>To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
>Subject: Re: Taking your time when a crisis occurs
>Date: Mon, 07 May 2001 05:30:57 -0800
>
>I agree, many serious problems have been created by DBA's that act
before
>they think things through. When I have a problem with the database,
I get a
>cup of coffee, put my feet up, and think about what happened and
what to do
>to solve the problem. Between corrupted data files, hardware
crashed, bad
>data in tables and other nasties, I have never failed to take care
of the
>problem, because I figured out what to do before starting the
repair.
>
> >>> stephane_paquette_at_yahoo.com 05/07/01 04:30AM >>>
>The golden rule when there is a big crash is :
>1. Panic
>2. Stop panic
>3. Fix the problem
>
>
>--- "Hallas, John" <HallasJ_at_logicae.com> a écrit : >
>FOR YOUR INFORMATION
> >
> > ESIS and EPFAL are now part of Logica. The Internet
> > email addresses of the staff has changed to the
> > following - lastnameinitial_at_logica.com eg
> > SMITHK_at_logica.com. Emails using the old format will
> > continue to be delivered until 30th June 2001.
> >
> > David,
> > I support what you say about taking your time
> > entirely. In fact at any
> > interviews I attend backup/recovery question(s)n are
> > always asked. My
> > standard answer is the at then first thing I will do
> > is go for a cup of
> > coffee. After their jaws have finished dropping I
> > explain how thinking time
> > is required etc.
> >
> > On a similar theme a few years ago I was
> > interviewing for a contract DBA
> > and he made the statement along the lines of 'you
> > are paying me more because
> > I have made mistakes before and I have learnt from
> > them so you will be safe
> > with me'. ( I am sure he phrased it more eloquently
> > than that).
> > After the interview the senior manager at the
> > interview said that he would
> > not have anyone as self-obsessed and over-confident
> > as that on board. I
> > disagreed and said that what the contractor was
> > offering was exactly what we
> > wanted. We took him on and he fitted in very well.
> > This story fits in with
> > the concept of getting a coffee and thinking about
> > things first, which is
> > all about using your experience well.
> >
> > John
> >
> > Logica/ESIS Tel 0115 945 6643
> >
> > -----Original Message-----
> > From: David A. Barbour
> > [mailto:dbarbour_at_nucentrix.net]
> > Sent: 03 May 2001 18:46
> > To: Multiple recipients of list ORACLE-L
> > Subject: Re: Fwd: please help
> >
> > Jared,
> >
> > I think you hit the nail on the head when you said
> > "Best
> > practice of
> > course is to make a backup of your database in
> > it's current
> > condition
> > prior to restoring it."
> >
> > Too many recoveries are failures because DBAs tend
> > to forget
> > basics when
> > confronted with the pressures from management,
> > users, and
> > the
> > constraints of time (primary key). I made this
> > mistake once
> > early on.
> > Now if I have a possible recovery scenario, the
> > first thing
> > I do is take
> > a deep breath, get a cup of coffee, and THINK
> > about what I'm
> > going to do
> > before I ever touch the keyboard.
> >
> > Absent all that, I still make a copy of the redo
> > logs
> > whenever I do a
> > backup. Yeah, you could mess up and apply them
> > inadvertently, but
> > hopefully you will have practiced recovery
> > scenarios (see
> > "Training a
> > DBA" by Kimberly Smith) and be comfortable with
> > your tapes,
> > disks,
> > commands, systems administrator, etc. At least if
> > you've
> > got them, and
> > everything goes to h*%$ in a handbasket, you can
> > always give
> > 'them' back
> > something.
> >
> > David A. Barbour
> >
> >
> > Jared Still wrote:
> > >
> > > Dick,
> > >
> > > Backing up the redo logs can have some serious
> > consequences.
> > >
> > > Let's say you are restoring the database files,
> > and a
> > number of
> > > archived logs to roll forward through.
> > >
> > > Following that, you are going to roll forward
> > through all
> > archived logs
> > > that are still online, and then through your
> > current redo
> > logs for a
> > > complete recovery.
> > >
> > > Restoring old redo logs would render this
> > strategy
> > ineffective.
> > >
> > > Backing them up can be a good thing, but it
> > would be very
> > easy
> > > to inadvertently wipe out the current ones when
> > restoring
> > from tape.
> > >
> > > Best practice of course is to make a backup of
> > your
> > database in
> > > it's current condition prior to restoring it.
> > >
> > > It would also be prudent to make copies of the
> > redo logs
> > locally
> > > so you don't have to restore them from tape.
> > >
> > > Jared
> > >
> > > On Wednesday 02 May 2001 07:24, dgoulet_at_vicr.com
> > wrote:
> > > > Jonathan,
> > > >
> > > > It would appear that your friend has hit
> > upon one of
> > the problems of
> > > > hot backups that everyone misses and actually
> > Oracle
> > recommends against.
> > > > That is backing up your online redo log files
> > and doing
> > that LAST. The
> > > > reason is that there are more than likely
> > active
> > transactions that were
> > > > recorded therein and those logs are not
> > available. Can
> > he complete the
> > > > recovery, maybe if he has the remaining logs
> > from the
> > active system, I'm
> > > > assuming he is recovering to somewhere other
> > than his
> > production system.
> > > > Otherwise his only recourse is OTS.
> > > >
> > > > Dick Goulet
> > > > Oracle Certified 8i DBA
> > > >
> > > > ____________________Reply
> > Separator____________________
> > > > Author: Jonathan Gennick
> > <jonathan_at_gennick.com>
> > > > Date: 5/1/2001 8:55 PM
> > > >
> > > > Fellow list members, I received the following
> > email from
> > a
> > > > reader a few minutes ago. If you skip down to
> > where he
> > talks
> > > > about backup, you'll see that he's in trouble
> > with a
> > > > database that won't recover. I've already
> > suggested that
> > he
> > > > open a TAR, and that he supply more specifics
> > as to
> > error
> > > > messages and the like, but maybe someone on
> > this list
> > can
> > > > draw some conclusions from what he's told me
> > so far. If
> > > > you're good at recovery, have a look at what
> > he says.
> > I'll
> > > > post his email address later if he says its
> > ok,
>=== message truncated ===
>
>
>=====
>Stéphane Paquette
>DBA Oracle, consultant entrepôt de données
>Oracle DBA, datawarehouse consultant
>stephane_paquette_at_yahoo.com
>
>___________________________________________________________
>Do You Yahoo!? -- Pour faire vos courses sur le Net,
>Yahoo! Shopping : http://fr.shopping.yahoo.com
>--
>Please see the official ORACLE-L FAQ: http://www.orafaq.com
>--
>Author: =?iso-8859-1?q?paquette=20stephane?=
> INET: stephane_paquette_at_yahoo.com
>
>Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
>San Diego, California -- Public Internet access / Mailing
Lists
>--------------------------------------------------------------------
>To REMOVE yourself from this mailing list, send an E-Mail message
>to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
>the message BODY, include a line containing: UNSUB ORACLE-L
>(or the name of mailing list you want to be removed from). You may
>also send the HELP command for other information (like
subscribing).
>--
>Please see the official ORACLE-L FAQ: http://www.orafaq.com
>--
>Author: William Beilstein
> INET: BeilstWH_at_obg.com
>
>Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
>San Diego, California -- Public Internet access / Mailing
Lists
>--------------------------------------------------------------------
>To REMOVE yourself from this mailing list, send an E-Mail message
>to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
>the message BODY, include a line containing: UNSUB ORACLE-L
>(or the name of mailing list you want to be removed from). You may
>also send the HELP command for other information (like
subscribing).
--
Please see the official ORACLE-L FAQ: http://www.orafaq.com
--
Author: Rachel Carmichael
INET: carmichr_at_hotmail.com
Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051
San Diego, California -- Public Internet access / Mailing
Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from). You may
also send the HELP command for other information (like subscribing).
-- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Boivin, Patrice J INET: BoivinP_at_mar.dfo-mpo.gc.ca Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051 San Diego, California -- Public Internet access / Mailing Lists -------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).Received on Mon May 07 2001 - 10:15:36 CDT
 |
 |