Re: Bank Databases

From: Dba DBA <oracledbaquestions_at_gmail.com>
Date: Tue, 3 Jul 2012 11:41:51 -0400
Message-ID: <CAE-dsOLYvEgzu37QB2zR1ijtwabFLYsMMovFy2oxPRe_RPhzDg_at_mail.gmail.com>



Fluke things can happen. I worked on a project where we lost 200 servers. EMC came in to add additional storage to the SAN on a Friday night. Fairly routine maintenance. When they closed one of the bays the whole SAN went down. Apparently some wires were loose. We lost everything including the VPN. The whole team had to drive into the office in the middle of the night to work. The VPN was back up in a few hours, but there was alot to do to bring all those servers up, the software that runs on them, and make sure the file systems were mounted and who knows what else. We had the core production stuff back up over night, but I am not sure if the systems administrators got more than a few hours sleep in the next week.

7-8 years ago Orbitz(I think it was orbitz) went down for 3 days. This is big, since their whole business is their website. They had some issues with a RAC upgrade, then their disk based backups didn't work, so they had to recover from tape. They publicly blamed Oracle. A couple of years later I ran into a guy who worked on that issue and he said it was because they didn't test some patches/upgrades to both oracle and the servers before going to production. This is not necessarily the staffs fault. Small business startups often go at ridiculous paces and you are not allowed to test. I worked for a financial company that did not want to pay for development servers or licenses. We had been using transportable tablespaces for several years. We upgraded from 8i to 9.2 RAC in product. There was a bug with transportable tablespaces and RAC. So we could not publish data. Took 2 months to get a patch from Oracle. It only came that fast because I worked for a fortune 500 company and the client was Wachovia. So the executives put alot of pressure on Oracle to get this punched out. Oracle did not want to release a patch and wanted us to wait until the next version of 9.2 RAC was out.

--
http://www.freelists.org/webpage/oracle-l
Received on Tue Jul 03 2012 - 10:41:51 CDT

Original text of this message