Re: 1 minutes: best downtime story
From: Dba DBA <oracledbaquestions_at_gmail.com>
Date: Mon, 8 Apr 2013 15:14:19 -0400
Message-ID: <CAE-dsO+XQU4S87tEUxN-XtUX7ZhYSvjZ_Sui_qvKAj_xsDMpow_at_mail.gmail.com>
Last spring, late on a friday night, some guys from EMC came in to add some more storage to our SAN. They closed a drawer full of disks, some wire was lose or something (never did get the details of what exactly happened) and the entire SAN went down. This SAN had several hundred servers on it. These servers included the ones that ran the VPN software. So I had to drive into the office at 2 AM to work on this. Many of our people are remote so they don't have access to an office and the first guy that was called said he was at a bar and was drunk. I was asleep, so I was not alert enough to make an excuse.
This SAN included at least 50 production servers and 10-15 production DBs (not to mention non-prod stuff).
Date: Mon, 8 Apr 2013 15:14:19 -0400
Message-ID: <CAE-dsO+XQU4S87tEUxN-XtUX7ZhYSvjZ_Sui_qvKAj_xsDMpow_at_mail.gmail.com>
Last spring, late on a friday night, some guys from EMC came in to add some more storage to our SAN. They closed a drawer full of disks, some wire was lose or something (never did get the details of what exactly happened) and the entire SAN went down. This SAN had several hundred servers on it. These servers included the ones that ran the VPN software. So I had to drive into the office at 2 AM to work on this. Many of our people are remote so they don't have access to an office and the first guy that was called said he was at a bar and was drunk. I was asleep, so I was not alert enough to make an excuse.
This SAN included at least 50 production servers and 10-15 production DBs (not to mention non-prod stuff).
To make matters worse someone fell asleep and was snoring on the conference call all night Never did figure out who it was. I can still here the snoring when I think about it. So my manager was sending me IMs telling me to stop snoring and then told everyone it was me.
No data was lost. We didn't have to do any DB recovers. However, the SAs had to remount the drives, etc... Then we had to deal with constant 'status' checks and 'are you done yets' from executives.
if you are wondering this is not anything that made the news.
-- http://www.freelists.org/webpage/oracle-lReceived on Mon Apr 08 2013 - 21:14:19 CEST