Re: Slower performance after enabling async io on Oracle Linux
From: Matthias Hoys <matthias.hoys_at_gmail.com>
Date: Tue, 15 May 2012 07:30:17 -0700 (PDT)
Message-ID: <13115770.1692.1337092217169.JavaMail.geo-discussion-forums_at_vbyb13>
On Tuesday, May 15, 2012 4:07:47 PM UTC+2, Matthias Hoys wrote:
> On Tuesday, May 15, 2012 3:16:46 PM UTC+2, Matthias Hoys wrote:
> > On Tuesday, May 15, 2012 2:52:33 AM UTC+2, Mladen Gogala wrote:
> > > Matthias, I the problem may as well lie on the network side. Check netstat
> > > -s for timeouts, retransmits or packets being dropped. Depending on the
> > > hardware, you may want to use jumbo packets, increase rmem and wmem
> > > parameters. Below are two articles about tuning the network:
> > >
> > > http://fasterdata.es.net/host-tuning/linux/
> > > http://www.ibm.com/developerworks/linux/library/l-adfly/index.html
> > >
> > > Problem with Linux is that it is not well instrumented, there is no wait
> > > event interface so you cannot use the same technique as with the
> > > database. You can use systemtap to explore the performance of the network
> > > layer and figure out what is taking so long.
> > > Also, hopefully your iSCSI traffic doesn't have to go through any
> > > firewall? Packet inspection can take time, just as with the customs
> > > office.
> > >
> > > So, if you want a recipe, measure the duration of the system calls by
> > > using strace and systemtap. Check netstat -s, make sure that your network
> > > is optimally configured.
> > >
> >
> > Using netstat -s, I think I found something:
> >
> > Tcp:
> > 1168305 active connections openings
> > 264105 passive connection openings
> > 910210 failed connection attempts
> >
> > I'm no network specialist, but it seems like there are a huge amount of "failed connection attempts"?? Or is this normal for iSCSI, I never worked with iSCSI storage before...
> >
> > Btw: we don't have a local disk, everything is on VMWare and the disks are on the iSCSI box.
> >
> > Thanks for the help,
> > Matthias Hoys
>
> Update: the Oracle Listener is causing all the "failed connection attempts". When I stop it, no more failed connection attempts. I'm now further debugging this...
>
> Matthias
Date: Tue, 15 May 2012 07:30:17 -0700 (PDT)
Message-ID: <13115770.1692.1337092217169.JavaMail.geo-discussion-forums_at_vbyb13>
On Tuesday, May 15, 2012 4:07:47 PM UTC+2, Matthias Hoys wrote:
> On Tuesday, May 15, 2012 3:16:46 PM UTC+2, Matthias Hoys wrote:
> > On Tuesday, May 15, 2012 2:52:33 AM UTC+2, Mladen Gogala wrote:
> > > Matthias, I the problem may as well lie on the network side. Check netstat
> > > -s for timeouts, retransmits or packets being dropped. Depending on the
> > > hardware, you may want to use jumbo packets, increase rmem and wmem
> > > parameters. Below are two articles about tuning the network:
> > >
> > > http://fasterdata.es.net/host-tuning/linux/
> > > http://www.ibm.com/developerworks/linux/library/l-adfly/index.html
> > >
> > > Problem with Linux is that it is not well instrumented, there is no wait
> > > event interface so you cannot use the same technique as with the
> > > database. You can use systemtap to explore the performance of the network
> > > layer and figure out what is taking so long.
> > > Also, hopefully your iSCSI traffic doesn't have to go through any
> > > firewall? Packet inspection can take time, just as with the customs
> > > office.
> > >
> > > So, if you want a recipe, measure the duration of the system calls by
> > > using strace and systemtap. Check netstat -s, make sure that your network
> > > is optimally configured.
> > >
> >
> > Using netstat -s, I think I found something:
> >
> > Tcp:
> > 1168305 active connections openings
> > 264105 passive connection openings
> > 910210 failed connection attempts
> >
> > I'm no network specialist, but it seems like there are a huge amount of "failed connection attempts"?? Or is this normal for iSCSI, I never worked with iSCSI storage before...
> >
> > Btw: we don't have a local disk, everything is on VMWare and the disks are on the iSCSI box.
> >
> > Thanks for the help,
> > Matthias Hoys
>
> Update: the Oracle Listener is causing all the "failed connection attempts". When I stop it, no more failed connection attempts. I'm now further debugging this...
>
> Matthias
Problem solved: I saw the following in the listener.log file: "WARNING: Subscription for node down event still pending". The solution is to add "SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF" to the listener.ora file, and restart the listener. Apparently, the listener service constantly tries to contact a RAC service, but this is a non-RAC installation. It was already a known issue with Oracle 10g. Way to go, Oracle ;-) Let's see if this improves the async io performance.
Matthias Hoys Received on Tue May 15 2012 - 09:30:17 CDT