Re: Oracle: how to demonstrate successful restore?

On Wed, 25 Jan 2006 16:02:03 -0800, Tiff wrote:

> You can see I have no experience in disaster recovery.

Companies normally have DR tests, just like fire drills. Typically, those tests go from primary and standby databases switching places all the way to going to remote location and restoring 1.1TB database at a spare location which can be provided by companies like IBM or SunGuard. It is important to understand that DR document should include the possibility of failure of any component, like router, name server, firewall, application server, web server, VPN server or LDAP server. Without the name server or router your clients will not be able to find the database server, even if the latter is perfectly operational.

The first thing to decide when writing a DR document is how far do you want to go and what do you want to protect the company from? Are we talking about major malfunction (like the 2003 big power outage) or total loss of a location as after Katrina or 9/11? What is the cost of the downtime and what are the time constraints? If there is an outage, how long can the company afford to stay down? If the allowed downtime is long enough, you can get away with restoring from backup in the moment of a problem or simply activating a standby which can be located at the other end of the same building.
The second thing to decide is how much data can you afford to lose? If it is a dating database that keeps "compatibility points" and 10 million addresses of ineligible and undesired bachelors, you can afford to lose more data then if you are operating an on-line banking database. Last but not least is how much does the company want to spend on data protection? DR plans are frequently the first casualty of cost cutting. Suddenly, non-technical people (CFO, for instance) bring in this wonderful sales guy from EMC telling you that RAID-5 is just fine and is equally as fast as RAID 1+0 or that those nasty old PA RISC boxes can easily be replaced by nice new thingys with quad-Opteron motherboard and running RH. When you find an unknown coffee bags in the kitchen in the place where Maxwell House coffee bags used to be, it's time to get your resume on the Dice. Maxwell House coffee is critical for DR plans as it enables your DBA to perform database recovery at 03:30 AM. It's good to the last drop. CIO is usually extremely touchy when it comes to signing checks, especially for disaster recovery. Cheap solutions are abundant and normally do not work without an extreme effort and many hours of pointless toiling for which nobody will thank you. DR is primarily a business decision for which you will need support of the senior management of the company.

Why am I telling you all this?
1) You are a junior DBA person and obviously are in charge of the

   company's DR plan. That is usually a grave mistake and a sign of the    company trying to save money on the wrong thing. Nothing personal,    but as a long time DBA, I don't think that a DR plan should be    tasked to a junior person. That is the job for your team lead. 2) You don't have a DR drill. Fire drills are required by law, but

   not the DR drills. To be effective, DR drill must be performed at    least once a year. A&E, the company that I had a consulting gig with    for a few months, performed DR drills monthly. They have two locations    (NYC and Stamford, CT) and they switch roles of the primary and standby    databases, name servers, routers and they go through the whole 9 yards.    Now that's the company conscious of the data security. You don't even    know whether your backup can be restored. That leads me to the    conclusion that RESTORE DATABASE VALIDATE is not a part of your weekly    backup routine. (This RMAN command scans the last backup and checks    whether it can be restored)

If your company is trying to save some money and if you, as a junior DBA person, are in charge of DR plan, then you are what is commonly known as a "scapegoat". Anything happens, and it's your job on the line. For a DBA, an item like "I was fired when I lost company's production database" doesn't look too nice on the resume and somewhat diminishes chances of getting hired again as a DBA. Try clarifying the points mentioned above with the management and if you don't get clear and satisfactory answers, run like hell.

I humbly apologize if this sounds harsh and cynical, but I assure you, it's a cold world out there. You are free to hate me if you so desire.

-- 
http://www.mgogala.com