Re: Is anyone using non GigE for the RAC Interconnect?

From: Andy Colvin <>
Date: Fri, 11 Nov 2011 07:47:12 -0600
Message-Id: <>

I've worked with a couple of clients that used IB for their interconnect (outside of Exadata). They were both running on Linux x86_64. Neither of them chose to use RDS, simply because they didn't find it to be entirely reliable on 10.2. One of the clients had an overly "chatty" application that flooded the interconnect. This was back in 2008, and they've been happily using IB for their interconnects ever since. The other client had a database with a very large block size and wanted to be able to send entire blocks in a single packet over the interconnect. We were unsuccessful in getting the RDS implementation to work with the database, but tests with 11.2 were successful with RDS. Unfortunately, I didn't have enough time to test the performance gains from RDS vs UDP.

Overall, Infiniband has become MUCH easier to implement with OEL5.5 and up. The OFED package is included in the OS distribution, so there aren't many gotcha moments any more. Back on RHEL/OEL4, it was much more vendor specific.

Andy Colvin

Principal Consultant
Office - 972-607-3744
Mobile - 214-763-8140

On Nov 10, 2011, at 1:18 AM, Martin Bach wrote:

> Good morning all,
> I'm putting material together for a presentation and in this context was
> wondering if anyone was using anything other than 1GigE and UDP for the
> RAC Interconnect. Exadata - users need not reply ;)
> Most new hardware you order such as the BL68x blades (which I love!)
> come with 10G Ethernet adapters but I assume the majority of RAC
> deployments hasn't made the transition to 10GE yet. Or am I wrong?
> Now while 1GE is probably the rule with 10GE quickly catching up, what's
> the situation for Infiniband-anyone outside the Exadata world using it?
> One of the companies I worked for a while back used IP over Infiniband,
> but that seems quite wasteful compared to the use of the RDS protocol
> (which also uses Infiniband).
> All opinions and observations greatly appreciated!
> Martin Bach
> --
> --

Received on Fri Nov 11 2011 - 07:47:12 CST

Original text of this message