Re: 1 minutes: best downtime story

From: Sandra Becker <sbecker6925_at_gmail.com>
Date: Thu, 28 Mar 2013 14:18:33 -0600
Message-ID: <CAJzM94B=Txc1E5aibyNGueN63zTPwynoO56fXcGm3_uesY=2Gg_at_mail.gmail.com>

Component: Entire customer network went down. (We're an Saas company.) Cause: IT guy told consultant, "Sure go ahead and shut that down. We don't need it." Wrong console.
Length: Down for over 2 hours because neither the consultant nor the IT guy knew how to restart everything.
Worst part: IT guy realized right away what happened and never notified anyone. I, the lone DBA, noticed issues and tried to raise a flag. Shot down at every turn. 30 minutes after I tried to raise a flag, customer services was flooded with emails and phone calls. That's when they decided to investigate.
Sandy

On Thu, Mar 14, 2013 at 3:07 PM, Jeremy Schneider < jeremy.schneider_at_ardentperf.com> wrote:

> Hey all -
>
> I'm writing a paper about top causes of downtime. As one component
> of research, I'd like to get some input from you!
>
> One minute, two sentences. First sentence: describe what went down.
> Second sentence: describe why. (I have to categorize all of these.)
> Everyone should have at least one downtime story so I'm hoping for a
> lot of feedback!
>
> Answer about any technology - database, operating system, etc.
>
> Thanks!
>
> -Jeremy
>
>
> --
> Jeremy Schneider
> Pythian Consulting Group
> Chicago
>
> +1 312-725-9249
> http://www.pythian.com
>
> --
> http://www.freelists.org/webpage/oracle-l
>
>
>

-- 
Sandy
Transzap, Inc.


--
http://www.freelists.org/webpage/oracle-l

Received on Thu Mar 28 2013 - 21:18:33 CET