RAC network interface and CPU INTERRUPTS

From: <Christopher.Taylor2_at_parallon.net>
Date: Thu, 3 Jan 2013 09:29:56 -0600
Message-ID: <F05D8DF1FB25F44085DB74CB916678E8856A2DCF60_at_NADCWPMSGCMS10.hca.corpad.net>

OS: RH EL 5.6 64-bit 2.6.18-128.1.16.el5 Any of you guys/gals troubleshooted network interface issues on RAC regarding CPU interrupts?

We have 3 dell servers with Broadcom 5708 cards with bnx2 drivers (blech!) and the interrupts are ALL landing on 1 CPU core.

It "seems" to me that you would want the interrupts spread across cores and the way to do this is to set disable_msi=1 in modprobe.conf.

We have 1 particular server that is giving us network issues with frame errors (1 out of 3) and we've disabled the msi on that server and the interrupts did spread across all cores.

What I'm trying to verify is that we would want that behavior versus all interrupts landing on 1 core?

Example:
node1: cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
107: 1192156702          0          0          0          0          0          0          0         PCI-MSI  eth1

node2: cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
210:         27          0   12022876          0          0  563692916   69534064 2446826584   IO-APIC-level  eth0

Based on what I've read it seems you do want to balance interrupts across cores but wanted to see what your thoughts were.

Chris Taylor
Oracle DBA
Parallon IT&S

--
http://www.freelists.org/webpage/oracle-l

Received on Thu Jan 03 2013 - 16:29:56 CET