RE: RAC system Calls

From: Matthew Zito <mzito_at_gridapp.com>
Date: Thu, 10 Jul 2003 12:50:56 -0700
Message-ID: <F001.005C44B6.20030710124426@fatcity.com>

Jumbo frames are the use of larger than normal MTU (Maximum Transmission Unit) settings on gigabit Ethernet links. The traditional limit for Ethernet frames is 1500 bytes, which was fine for 10 and 100 megabit Ethernet links. With gigabit, however, since you lose a certain minimum amount of bandwidth to signaling overhead (preamble, postamble, header info, etc.) and that the Ethernet card has to do a certain minimum processing for each Ethernet frame it receives, a huge amount of CPU overhead can be spent on trying to fill a gigabit pipe. The other problem is that if the host(s) are sending/receiving data larger than 1500 bytes, the data packet has to be fragmented into multiple, smaller packets, which then have to be reassembled on the far side. Since this all has to be done on the host CPU rather than the Ethernet card, it increases both bus overhead and CPU time.

With jumbo frames, you use a >1500 byte MTU - the exact amount varies by implementation, but they're generally in the 9000-9200 byte range. That's a 6x improvement in the amount of data per ethernet frame, plus there's less reassembly. Unfortunately, Sun never really embraced it as a technology, so unless you're running one of a couple of third-party gigabit cards, I think you're probably out of luck.

The specific relevance to RAC, which I somehow managed to mention, is that data blocks being shuttled 'tween nodes (depending on the blocksize) can be placed into a smaller number of ethernet frames, reducing both latency and overhead. Ideally, each block will fit into one ethernet frame, but as always, YMMV.

Thanks,
Matt

--
Matthew Zito
GridApp Systems
Email: [EMAIL PROTECTED]
Cell: 646-220-3551
Phone: 212-358-8211 x 359
http://www.gridapp.com



> -----Original Message-----

> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On 

> Behalf Of Ravi Kulkarni

> Sent: Thursday, July 10, 2003 2:49 PM

> To: Multiple recipients of list ORACLE-L

> Subject: RE: RAC system Calls

> 

> 

> Matt,

> What are jumbo frames? Are these assigning private

> network IPs to cluster_interconnects parameter?

> -Ravi.

> 

> 

> --- Matthew Zito <[EMAIL PROTECTED]> wrote:

> > 

> > And are you using jumbo frames on your interconnect?

> >  That can make a

> > significant contribution to reducing overhead from a

> > system standpoint.

> > 

> > Thanks,

> > Matt

> > 

> > --

> > Matthew Zito

> > GridApp Systems

> > Email: [EMAIL PROTECTED]

> > Cell: 646-220-3551

> > Phone: 212-358-8211 x 359

> > http://www.gridapp.com

> > 

> > > -----Original Message-----

> > > From: [EMAIL PROTECTED]

> > [mailto:[EMAIL PROTECTED] On

> > > Behalf Of K Gopalakrishnan

> > > Sent: Thursday, July 10, 2003 11:44 AM

> > > To: Multiple recipients of list ORACLE-L

> > > Subject: RE: RAC system Calls

> > > 

> > > 

> > > Ravi:

> > > 

> > > Do you have a statspack report? I would like to

> > see that. But

> > > in any case, 45% kernel is just too much?

> > > 

> > > BTW have you verified the private interconnect is

> > used

> > > for cache fusion transfer.. Make sure the cache

> > fusion

> > > is not going thru the public network.

> > > 

> > > 

> > > 

> > > Best Regards,

> > > K Gopalakrishnan

> > > 

> > >  

> > > 

> > > 

> > > -----Original Message-----

> > > Ravi Kulkarni

> > > Sent: Thursday, July 10, 2003 9:30 AM

> > > To: Multiple recipients of list ORACLE-L

> > > 

> > > 

> > > Hello List,

> > > 

> > > We are running Benchmark tests on Solaris 2-Node

> > RAC.

> > > Consistently noticed the following :

> > > - Very high Kernel usage (averaging 45%) on TOP

> > > - Statspack has "IPC Send Completion sync" waits

> > (70%

> > > Total ela time)

> > > - On trussing top process, found Oracle to be

> > issuing

> > > huge number of "times" system calls in addition to

> > 

> > > read/writes(which I think are select/inserts). Has

> > anyone

> > > noticed this in your environment. I am guessing

> > these to be

> > > inter-instance pings, but could not get any hits

> > in

> > > Doc/Metalink to confirm this. "times" call is

> > clocking lot of

> > > CPU. Is this normal ?

> > > Any pointers would be helpful ? If this is out of

> > > context, is there a separate list for RAC?

> > > 

> > > Thanks,

> > > Ravi.

> > > 

> > > 

> > > 

> > > __________________________________

> > > Do you Yahoo!?

> > > SBC Yahoo! DSL - Now only $29.95 per month!

> > http://sbc.yahoo.com

> > > --

> > > Please see the official ORACLE-L FAQ:

> > http://www.orafaq.net

> > > --

> > > Author: Ravi Kulkarni

> > >   INET: [EMAIL PROTECTED]

> > > 

> > > Fat City Network Services    -- 858-538-5051

> > http://www.fatcity.com

> > > San Diego, California        -- Mailing list and

> > web hosting services

> > >

> >

> ---------------------------------------------------------------------

> > > To REMOVE yourself from this mailing list, send an

> > E-Mail message

> > > to: [EMAIL PROTECTED] (note EXACT spelling of

> > 'ListGuru')

> > > and in the message BODY, include a line

> > containing: UNSUB

> > > ORACLE-L (or the name of mailing list you want to

> > be removed

> > > from).  You may also send the HELP command for

> > other

> > > information (like subscribing).

> > > 

> > > 

> > > --

> > > Please see the official ORACLE-L FAQ:

> > http://www.orafaq.net

> > > --

> > > Author: K Gopalakrishnan

> > >   INET: [EMAIL PROTECTED]

> > > 

> > > Fat City Network Services    -- 858-538-5051

> > http://www.fatcity.com

> > > San Diego, California        -- Mailing list and

> > web hosting services

> > >

> >

> ---------------------------------------------------------------------

> > > To REMOVE yourself from this mailing list, send an

> > E-Mail message

> > > to: [EMAIL PROTECTED] (note EXACT spelling of

> > 'ListGuru')

> > > and in the message BODY, include a line

> > containing: UNSUB

> > > ORACLE-L (or the name of mailing list you want to

> > be removed

> > > from).  You may also send the HELP command for

> > other

> > > information (like subscribing).

> > > 

> > 

> > --

> > Please see the official ORACLE-L FAQ:

> > http://www.orafaq.net

> > -- 

> > Author: Matthew Zito

> >   INET: [EMAIL PROTECTED]

> > 

> > Fat City Network Services    -- 858-538-5051

> > http://www.fatcity.com

> > San Diego, California        -- Mailing list and web

> > hosting services

> >

> ---------------------------------------------------------------------

> > To REMOVE yourself from this mailing list, send an

> > E-Mail message

> > to: [EMAIL PROTECTED] (note EXACT spelling of

> > 'ListGuru') and in

> > the message BODY, include a line containing: UNSUB

> > ORACLE-L

> > (or the name of mailing list you want to be removed

> > from).  You may

> > also send the HELP command for other information

> > (like subscribing).

> 

> 

> __________________________________

> Do you Yahoo!?

> SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com

> -- 

> Please see the official ORACLE-L FAQ: http://www.orafaq.net

> -- 

> Author: Ravi Kulkarni

>   INET: [EMAIL PROTECTED]

> 

> Fat City Network Services    -- 858-538-5051 http://www.fatcity.com

> San Diego, California        -- Mailing list and web hosting services

> ---------------------------------------------------------------------

> To REMOVE yourself from this mailing list, send an E-Mail message

> to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') 

> and in the message BODY, include a line containing: UNSUB 

> ORACLE-L (or the name of mailing list you want to be removed 

> from).  You may also send the HELP command for other 

> information (like subscribing).

> 


-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Matthew Zito
  INET: [EMAIL PROTECTED]

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Received on Thu Jul 10 2003 - 14:50:56 CDT