Re: RAC for dev env

From: Guillermo Alan Bort <>
Date: Tue, 2 Feb 2010 21:32:50 -0200
Message-ID: <>

I have horror stories of Siebel on RAC (windows)... they complained that in test/dev it worked fine, but in prod they got disconnected very often. We soon found out that there were about 3 instance evictions per week... and sometimes the servers would just freeze. This turned out to be a RAC bug on windows (nobody sensible ever used RAC on windows, so it was discovered in this environment. No patch for it... it's a 9i RAC.

Oh, a load testing as well... they were performing separate tests... one for RW operations and one for RO. the RO was on prod the RW on dev... and they called THAT load testing...


On Tue, Feb 2, 2010 at 8:30 PM, Thomas Roach <> wrote:

> I will give you a story where this bit me BIGTIME.
> We had a Solaris 10 RAC cluster ( for production but the business
> skimped because they didn't want to pay "all that money" for RAC just for a
> Dev/QA environment. This was a system that was generating 600+ million
> dollars in revenue, but they didn't want to spend a few hundred thousand to
> get RAC. Sounds like the tail wagging the dog, and it was!
> So time came around to do security patches and what not. They also wanted
> CRS Bundle patches. So we patched Dev/QA, no problem. When we went to apply
> the CRS_BUNDLE patch there was an error with the patchset and it wiped out
> the inittab. (I didn't know what happened, but eventually uncovered
> inittab). The upgrade script bombed multiple times. I got Oracle support on
> the line and we troubleshooted it. 8 hours later we got the systems back
> online.
> The owner of this business segment was pissed. We had a post mortem that
> was not pretty. She looked straight at me and said, why didn't we see these
> problems in QA or even Dev? It cost us (a lot of money, more than Oracle RAC
> licensing for Dev/QA would have cost). I told her that her environments did
> not match. She said why? I said because no one wanted to spend the money to
> make the environments match. That shut her up real quick.
> Needless to say it was a high priority to make them match and they spared
> no money to get environments up to speed.
> I knew the lesson learned, but it was a lesson for the business. You need
> to explicitly tell them that unless environments match, there is a risk that
> not everything can be pretested prior to going into production. Let them
> decide if it is worth it. If they say no, get it in writing.
> Tom
> On Tue, Feb 2, 2010 at 4:51 PM, Guillermo Alan Bort <>wrote:
>> Hi Ram
>> I have seen this in several places and it always leads to severe downtime
>> in production. non RAC Dev environments could work for functional
>> development. For performance and concurrency testing and for staiblity
>> testing, you need a RAC environment. Furthermore, I'd suggest you get
>> someone with a lot of experience in RAC before actually going to production
>> with it. Having a TEST/DEV/QA RAC could help you gain that experience.
>> I cannot speak as to licensing, but if you have the DB up, then it doesn't
>> matter what you use it for... you still have to have a license, at least
>> that is how I understand it.
>> One suggestion, if you can't get someone with experience... get a couple
>> of virtual machines and toy around with them. You can delete them
>> afterwards, or burn them to DVD and hide them under your desk in case of an
>> audit by Oracle, and still have a RAC sandbox... you wouldn't be able to
>> perform stress tests here, but you can test functionality and the quirks of
>> RAC.
>> That being said, perhaps reviewing WHY they want RAC would reveal that
>> they actually don't need RAC but a simple DataGuard. I've recently
>> recommended moving from a two-node RAC to a single instance on a better
>> server (better hardware, 64 bit instead of 32 bit and AIX instead of
>> windows) and setting up a DataGuard for fast recovery and minimal data loss
>> And just a word of advice... never, EVER, use RAC on Windows. Actually...
>> never user windows for a database server...
>> RAC is like exadata... it's useful only in very specific situations, other
>> than that is just a waste of money and resources.
>> hth
>> Alan.-
>> On Tue, Feb 2, 2010 at 6:30 PM, Ram Raman <> wrote:
>>> Hi
>>> A decision has been made by upper management to use RAC for production
>>> environment and nonRAC for dev/test environments. I would prefer to have a
>>> parallel environment which is identical to prod. Is it possible to run a
>>> dev environment without having to worry about license. If license might
>>> be an issue for dev, can we have one environment where we just install RAC
>>> for practice before deploying it in production.
>>> Any thoughts or guidances on the issue would be highly appreciated.
>>> Thanks,
>>> Ram.
> --
> Thomas Roach
> 813-404-6066

Received on Tue Feb 02 2010 - 17:32:50 CST

Original text of this message