ASM, block devices and SCSI emulation

Quite recently, I learned that Oracle ASM is now supporting block devices. The whole process is described here:

http://docs.oracle.com/cd/E11882_01/install.112/e17212/ storage.htm#CDEBFDEH

This deserves few additional observations. The main engine that enables one to add block devices to SCSI is udev. Essentially, one creates SCSI devices by using a configuration file like this one, pasted directly from the manual above:

# vi /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL==”sd?1″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”, RESULT==”14f70656e66696c00000000″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd?2″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”, RESULT==”14f70656e66696c00000001″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″
KERNEL==”sd?3″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id”, RESULT==”14f70656e66696c00000002″, OWNER=”grid”, GROUP=”asmadmin”, MODE=”0660″

What does that do? This enables udev to create “SCSI devices” for each and every SATA or FC disk on the system. You sse, when Linux was much, much younger, there was a problem with now largely forgotten devices called “CD burners”. Those vere EIDE devices and each and every one had to have its own driver. Commercial packages like Roxio were used to support as many of those devices as possible and, needless to say, Linux was lagging behind. Then the Linux guys got an excellent idea: why not emulate SCSI on all EIDE devices and write one program to burn them all, using just SCSI command set. The generic SCSI device emulator, embodied in the “sg” module was born and it still lives on today. That is what the configuration file above is utilizing to create SCSI disks from the standard SATA devices. The data is sent to the device using IOCTL defined in /usr/include/scsi/scsi_ioctl. That is the same interface once used by the program X-CD-Roast, the first popular Linux CD burning program.

There is, however, a little thing that one must be aware about SCSI emulation. It is an additional driver and driver requests are usually spawned by the processor interrupts. Device driver, itself activated by an interrupt, spawns another interrupt, frequently called “soft interrupt” because it’s not spawned by a hardware, with the arguments which direct kernel to select the apropriate drives, in this case “sg”. That is happening for every request to device. Using “sg” doubles the number of disk interrupts caused by the particular device it is attached to. And interrupts are expensive. Not only do they operate in kernel mode, they also prevent other interrupts from being delivered. Each interrupt has its IPL (Interrupt Priority Level) and during the interrupt handler execution (aka “driver”), all interrupts with the IPL lower than the currently executing one are blocked. The clock interrupt is at the lowest priority level, which means that any disk interrupt will prevent the time sharing system during its execution. Rasing the number of interrupts is not a trivial matter. It doesn’t really matter for my desktop computer and burning CD’s, which nobody does any more, but for my database server it greatly increases the price of my IO. On LinkedIn RAC group, a guy reported performance drop as much as 25% as compared to ASMLib, which uses raw devices. He was talking about zLinux on an IBM mainframe which is particularly sensitive because of the mainframe IO architecture which is all about the smart channels and avoiding interrupts. Raw devices, which is the interface presented to ASM by ASMLib, do not increase the number of interrupts as they are not emulating anything.

-- 
Mladen Gogala
The Oracle Whisperer
http://mgogala.byethost5.com