Re: Controlfiles just got overwritten

From: Maureen English <maureen.english_at_alaska.edu>
Date: Fri, 18 Jul 2014 10:26:58 -0800
Message-ID: <53C966F2.7060208_at_alaska.edu>



Tim, rest of list,

THANK YOU! I was ready to start cleaning out my desk after realizing what I did. My manager, though, was concerned about what to tell the users and letting our director know what had happened. Our director, someone I've worked with for almost 20 years, always remains calm and offered some suggestions for quick fixes since he was a DBA years ago. Our CITO, someone I expected to be very upset, responded with something like 'Mistakes happen, you're only human.'

  • Maureen

On 7/17/2014 3:28 PM, Tim Gorman wrote:
> Maureen,
>
> About 4 years ago, or 26 years into my IT career, I dropped an index on a 60 Tb table with 24,000 hourly partitions;
> the index was over 15 Tb in size. It was the main table in that production application, of course.
>
> Over a quarter century of industry experience as a developer, production support, systems administrator, and database
> administrator; if that's not enough time to have important lessons pounded into one's head, then how much time is needed?
>
> My supervisor at the time was amazing. After the shock of watching it all happen and still not quite believing it had
> happened, I called him at about 9pm local time, and told him what occurred. I finished speaking, and waited for the axe
> to drop, for the entirely-justified anger to crash down on my head. He was silent for about 3 seconds, then just said
> calmly, "Well, I guess we need to fix it." And that was it. No anger, no recriminations, no humiliating
> micro-management. We launched straight into planning what needed to happen to fix it.
>
> He got to work notifying the organization what had happened, and I got started on the rebuild, which eventually took
> almost 2 weeks to complete.
>
> I hope you had the same experience. Because it truly happens to all of us. And anyone who pretends otherwise simply
> hasn't been doing important work.
>
> Hope this helps...
>
> -Tim
>
>
>
>
> On 7/17/14, 17:01, Maureen English wrote:
>> I ended up recovering the database from a backup done the day before and rolling forward.
>>
>> I also modified my instructions for recreating a standby database. Instead of 'cp -p' to copy
>> my standby control file to the appropriate directory on the standby server, I will now use 'cp -pi'.
>>
>> I'm still feeling badly for making this mistake, but I'm amazed at how understanding my coworkers
>> have been.
>>
>> - Maureen
>>

--
http://www.freelists.org/webpage/oracle-l
Received on Fri Jul 18 2014 - 20:26:58 CEST

Original text of this message