Re: Slower performance after enabling async io on Oracle Linux
From: joel garry <joel-garry_at_home.com>
Date: Tue, 15 May 2012 09:15:51 -0700 (PDT)
Message-ID: <9e9693f1-a217-40cd-bb95-24374878fa15_at_e9g2000pbh.googlegroups.com>
On May 15, 7:30 am, Matthias Hoys <matthias.h..._at_gmail.com> wrote:
> On Tuesday, May 15, 2012 4:07:47 PM UTC+2, Matthias Hoys wrote:
> > On Tuesday, May 15, 2012 3:16:46 PM UTC+2, Matthias Hoys wrote:
> > > On Tuesday, May 15, 2012 2:52:33 AM UTC+2, Mladen Gogala wrote:
> > > > Matthias, I the problem may as well lie on the network side. Check netstat
> > > > -s for timeouts, retransmits or packets being dropped. Depending on the
> > > > hardware, you may want to use jumbo packets, increase rmem and wmem
> > > > parameters. Below are two articles about tuning the network:
>
> > > >http://fasterdata.es.net/host-tuning/linux/
> > > >http://www.ibm.com/developerworks/linux/library/l-adfly/index.html
>
> > > > Problem with Linux is that it is not well instrumented, there is no wait
> > > > event interface so you cannot use the same technique as with the
> > > > database. You can use systemtap to explore the performance of the network
> > > > layer and figure out what is taking so long.
> > > > Also, hopefully your iSCSI traffic doesn't have to go through any
> > > > firewall? Packet inspection can take time, just as with the customs
> > > > office.
>
> > > > So, if you want a recipe, measure the duration of the system calls by
> > > > using strace and systemtap. Check netstat -s, make sure that your network
> > > > is optimally configured.
>
> > > Using netstat -s, I think I found something:
>
> > > Tcp:
> > > 1168305 active connections openings
> > > 264105 passive connection openings
> > > 910210 failed connection attempts
>
> > > I'm no network specialist, but it seems like there are a huge amount of "failed connection attempts"?? Or is this normal for iSCSI, I never worked with iSCSI storage before...
>
> > > Btw: we don't have a local disk, everything is on VMWare and the disks are on the iSCSI box.
>
> > > Thanks for the help,
> > > Matthias Hoys
>
> > Update: the Oracle Listener is causing all the "failed connection attempts". When I stop it, no more failed connection attempts. I'm now further debugging this...
>
> > Matthias
>
> Problem solved: I saw the following in the listener.log file: "WARNING: Subscription for node down event still pending". The solution is to add "SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF" to the listener.ora file, and restart the listener. Apparently, the listener service constantly tries to contact a RAC service, but this is a non-RAC installation. It was already a known issue with Oracle 10g. Way to go, Oracle ;-) Let's see if this improves the async io performance.
>
> Matthias Hoys
Date: Tue, 15 May 2012 09:15:51 -0700 (PDT)
Message-ID: <9e9693f1-a217-40cd-bb95-24374878fa15_at_e9g2000pbh.googlegroups.com>
On May 15, 7:30 am, Matthias Hoys <matthias.h..._at_gmail.com> wrote:
> On Tuesday, May 15, 2012 4:07:47 PM UTC+2, Matthias Hoys wrote:
> > On Tuesday, May 15, 2012 3:16:46 PM UTC+2, Matthias Hoys wrote:
> > > On Tuesday, May 15, 2012 2:52:33 AM UTC+2, Mladen Gogala wrote:
> > > > Matthias, I the problem may as well lie on the network side. Check netstat
> > > > -s for timeouts, retransmits or packets being dropped. Depending on the
> > > > hardware, you may want to use jumbo packets, increase rmem and wmem
> > > > parameters. Below are two articles about tuning the network:
>
> > > >http://fasterdata.es.net/host-tuning/linux/
> > > >http://www.ibm.com/developerworks/linux/library/l-adfly/index.html
>
> > > > Problem with Linux is that it is not well instrumented, there is no wait
> > > > event interface so you cannot use the same technique as with the
> > > > database. You can use systemtap to explore the performance of the network
> > > > layer and figure out what is taking so long.
> > > > Also, hopefully your iSCSI traffic doesn't have to go through any
> > > > firewall? Packet inspection can take time, just as with the customs
> > > > office.
>
> > > > So, if you want a recipe, measure the duration of the system calls by
> > > > using strace and systemtap. Check netstat -s, make sure that your network
> > > > is optimally configured.
>
> > > Using netstat -s, I think I found something:
>
> > > Tcp:
> > > 1168305 active connections openings
> > > 264105 passive connection openings
> > > 910210 failed connection attempts
>
> > > I'm no network specialist, but it seems like there are a huge amount of "failed connection attempts"?? Or is this normal for iSCSI, I never worked with iSCSI storage before...
>
> > > Btw: we don't have a local disk, everything is on VMWare and the disks are on the iSCSI box.
>
> > > Thanks for the help,
> > > Matthias Hoys
>
> > Update: the Oracle Listener is causing all the "failed connection attempts". When I stop it, no more failed connection attempts. I'm now further debugging this...
>
> > Matthias
>
> Problem solved: I saw the following in the listener.log file: "WARNING: Subscription for node down event still pending". The solution is to add "SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF" to the listener.ora file, and restart the listener. Apparently, the listener service constantly tries to contact a RAC service, but this is a non-RAC installation. It was already a known issue with Oracle 10g. Way to go, Oracle ;-) Let's see if this improves the async io performance.
>
> Matthias Hoys
lol, I ran into that on hp-ux, but since you are linux, I had no clue to say anything. The symptoms and async i/o are different, but the message is the same.
jg
-- _at_home.com is bogus. http://www.pcworld.com/businesscenter/article/253980/oracle_considered_buying_rim_and_palm_ellison_says.html http://tkyte.blogspot.com/2005/09/lefties-unite.html#112653167997033408Received on Tue May 15 2012 - 11:15:51 CDT