Hey Matt,
Thanks for the useful info.
>into a smaller number of ethernet frames,
> reducing both latency and overhead. Ideally, each
> block will fit into one ethernet frame,
We are using 16K blocksize, so we cannot avoid
datagram splits with 9k frames - but certainly, sounds
better than with 1500 frames.
>so unless you're running one of a couple of
third-party gigabit cards, I think you're probably out
of luck.
Will check with SysAdmins if the GigE we have
supports. Are there any risks/disadvantages of using
9k frames for interconnect?
Thanks,
-Ravi.
- Matthew Zito <mzito_at_gridapp.com> wrote:
>
> Jumbo frames are the use of larger than normal MTU
> (Maximum Transmission
> Unit) settings on gigabit Ethernet links. The
> traditional limit for
> Ethernet frames is 1500 bytes, which was fine for 10
> and 100 megabit
> Ethernet links. With gigabit, however, since you
> lose a certain minimum
> amount of bandwidth to signaling overhead (preamble,
> postamble, header
> info, etc.) and that the Ethernet card has to do a
> certain minimum
> processing for each Ethernet frame it receives, a
> huge amount of CPU
> overhead can be spent on trying to fill a gigabit
> pipe. The other
> problem is that if the host(s) are sending/receiving
> data larger than
> 1500 bytes, the data packet has to be fragmented
> into multiple, smaller
> packets, which then have to be reassembled on the
> far side. Since this
> all has to be done on the host CPU rather than the
> Ethernet card, it
> increases both bus overhead and CPU time.
>
> With jumbo frames, you use a >1500 byte MTU - the
> exact amount varies by
> implementation, but they're generally in the
> 9000-9200 byte range.
> That's a 6x improvement in the amount of data per
> ethernet frame, plus
> there's less reassembly. Unfortunately, Sun never
> really embraced it as
> a technology, so unless you're running one of a
> couple of third-party
> gigabit cards, I think you're probably out of luck.
>
>
> The specific relevance to RAC, which I somehow
> managed to mention, is
> that data blocks being shuttled 'tween nodes
> (depending on the
> blocksize) can be placed into a smaller number of
> ethernet frames,
> reducing both latency and overhead. Ideally, each
> block will fit into
> one ethernet frame, but as always, YMMV.
>
> Thanks,
> Matt
>
> --
> Matthew Zito
> GridApp Systems
> Email: mzito_at_gridapp.com
> Cell: 646-220-3551
> Phone: 212-358-8211 x 359
> http://www.gridapp.com
>
> > -----Original Message-----
> > From: ml-errors_at_fatcity.com
> [mailto:ml-errors_at_fatcity.com] On
> > Behalf Of Ravi Kulkarni
> > Sent: Thursday, July 10, 2003 2:49 PM
> > To: Multiple recipients of list ORACLE-L
> > Subject: RE: RAC system Calls
> >
> >
> > Matt,
> > What are jumbo frames? Are these assigning private
> > network IPs to cluster_interconnects parameter?
> > -Ravi.
> >
> >
> > --- Matthew Zito <mzito_at_gridapp.com> wrote:
> > >
> > > And are you using jumbo frames on your
> interconnect?
> > > That can make a
> > > significant contribution to reducing overhead
> from a
> > > system standpoint.
> > >
> > > Thanks,
> > > Matt
> > >
> > > --
> > > Matthew Zito
> > > GridApp Systems
> > > Email: mzito_at_gridapp.com
> > > Cell: 646-220-3551
> > > Phone: 212-358-8211 x 359
> > > http://www.gridapp.com
> > >
> > > > -----Original Message-----
> > > > From: ml-errors_at_fatcity.com
> > > [mailto:ml-errors_at_fatcity.com] On
> > > > Behalf Of K Gopalakrishnan
> > > > Sent: Thursday, July 10, 2003 11:44 AM
> > > > To: Multiple recipients of list ORACLE-L
> > > > Subject: RE: RAC system Calls
> > > >
> > > >
> > > > Ravi:
> > > >
> > > > Do you have a statspack report? I would like
> to
> > > see that. But
> > > > in any case, 45% kernel is just too much?
> > > >
> > > > BTW have you verified the private interconnect
> is
> > > used
> > > > for cache fusion transfer.. Make sure the
> cache
> > > fusion
> > > > is not going thru the public network.
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > > K Gopalakrishnan
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > Ravi Kulkarni
> > > > Sent: Thursday, July 10, 2003 9:30 AM
> > > > To: Multiple recipients of list ORACLE-L
> > > >
> > > >
> > > > Hello List,
> > > >
> > > > We are running Benchmark tests on Solaris
> 2-Node
> > > RAC.
> > > > Consistently noticed the following :
> > > > - Very high Kernel usage (averaging 45%) on
> TOP
> > > > - Statspack has "IPC Send Completion sync"
> waits
> > > (70%
> > > > Total ela time)
> > > > - On trussing top process, found Oracle to be
> > > issuing
> > > > huge number of "times" system calls in
> addition to
> > >
> > > > read/writes(which I think are select/inserts).
> Has
> > > anyone
> > > > noticed this in your environment. I am
> guessing
> > > these to be
> > > > inter-instance pings, but could not get any
> hits
> > > in
> > > > Doc/Metalink to confirm this. "times" call is
> > > clocking lot of
> > > > CPU. Is this normal ?
> > > > Any pointers would be helpful ? If this is out
> of
> > > > context, is there a separate list for RAC?
> > > >
> > > > Thanks,
> > > > Ravi.
>
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com
--
Please see the official ORACLE-L FAQ: http://www.orafaq.net
--
Author: Ravi Kulkarni
INET: nandagokul_at_yahoo.com
Fat City Network Services -- 858-538-5051 http://www.fatcity.com
San Diego, California -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from). You may
also send the HELP command for other information (like subscribing).
Received on Thu Jul 10 2003 - 19:04:25 CDT
