Re: RAC on OCFS2 acceptance testing

Comments in-line:

On 12/30/2006 12:17:57 PM, Steve Perry wrote:
> A customer ran into a simlar problem(s) with OCFS2 and RHEL4 upd 4
> (smp kernel).
> heavy db updates or mixed io (cp from ocfs to ext3, oracle export to
> ext3) would cause the cluster to become unresponsive and crash a node.
> cp and exp caused a high load avg and heavy swapping. We couldn't
> even ssh to the host.
> I didn't understand the heavy swapping because there was 3GB of cache
> mem available (shown by free -m).
> something to do with ocfs and low mem usage. I never got a clear
> answer on it.
>
> the ended up setting "vm.lower_zone_protection=100" which helped the
> swapping issue.

The vm.lower_zone_protection parameter makes certain portion of physical memory non-pageable. On MVS, it used to be known as "VIRTUAL=REAL boundary". Conveniently, the units are megabytes, which means that you precluded 100M of memory from being pageable. In particular, that means that OCFS kernel module will not be able to allocate user buffers from the memory below 100M boundary. The reason for that are "features" in Linux kernel, more or less openly admitted in the documentation for this parameter. Here is an excerpt from the documentation:

/usr/share/doc/kernel-doc-2.6.17/Documentation/filesystems/proc.txt

lower_zone_protection

For some specialised workloads on highmem machines it is dangerous for the kernel to allow process memory to be allocated from the "lowmem" zone. This is because that memory could then be pinned via the mlock() system call, or by unavailability of swapspace.

And on large highmem machines this lack of reclaimable lowmem memory can be fatal.

So the Linux page allocator has a mechanism which prevents allocations which _could_ use highmem from using too much lowmem. This means that a certain amount of lowmem is defended from the possibility of being captured into pinned user memory.

(The same argument applies to the old 16 megabyte ISA DMA region. This mechanism will also defend that region from allocations which could use highmem or lowmem).

The `lower_zone_protection' tunable determines how aggressive the kernel is in defending these lower zones. The default value is zero - no protection at all.

If you have a machine which uses highmem or ISA DMA and your applications are using mlock(), or if you are running with no swap then you probably should increase the lower_zone_protection setting.

The units of this tunable are fairly vague. It is approximately equal to "megabytes". So setting lower_zone_protection=100 will protect around 100 megabytes of the lowmem zone from user allocations. It will also make those 100 megabytes unavaliable for use by applications and by pagecache, so there is a cost.

>
> The fencing problem was attributed to the following init.ora parms.
> filesystemio_options = asynch
> disk_asynch_io = TRUE
>
> they were changed to:
> disk_asynch_io=FALSE
> filesystemio_options='DIRECTIO'

Neither OCFS nor OCFS2 support asynchronous I/O. They both allow only direct I/O. By attempting to use asynchronous I/O, you may crash your system or your database. That is well documented on the OCFS site.

>
> Things have improved since.
>
> I asked Oracle for a good document for OCFS2 and RAC and still
> haven't got a response.
> I also asked for optimal kernel parameter settings for OCFS2.
>
> The closest I got was the following list, but no values.
> - vm.swappiness
> - vm.lower_zone_protection
> - vm.vfs_cache_pressure
> - vm.dirty_ratio
> - vm.dirty_background_ratio

Here we have to deal with the fact that Linux kernel is less then perfect, to say the least. From those parameters, swappiness and vfs_cache_pressure are so called "composite parameters" which regulate "tendency", which means that you don't get to see an accurate parameter description without plunging into the kernel code. Both of these parameters regulate "aggressiveness" of the OS with swapping/page stealing or replacing inodes and directory entries. I find them best set to 0. I was playing with the "swappiness" and I found that it will turn on aggressive page swapping which will slow down your system. Dirty ratio and background ratio are parameters for modified page write-back. Due to Linux kernel problems, you don't have any tools which would help you diagnose problems with the page write-back. You don't have anything even remotely like VMS "monitor page" and "monitor pool" commands. Linux itself is inferior to SYSVR4 Unix derivatives like AIX or HP-UX and is certainly inferior to Solaris which took a significant step ahead and removed itself from the pervasive SYSVR4 standard. Without being able to monitor effects, those parameters should be left alone.

Parameter that you should set to at least 15% of your memory, to ensure ease of memory allocation is vm.min_free_kbytes. This parameters sets the target value for the page replacement daemon to keep free and available for "malloc" calls.

>
> I'm not sure about "unbreakable" Oracle/Linux combo. I'd be happy if
> they focused on "stable" Oracle/Linux.

They first have to make Linux as capable as the other operating systems and add instrumentation for monitoring and diagnostics. When that is done, the first Mogens theorem will apply:
"Anything that is sufficiently instrumented is obsolete". That is the theorem from the Oak Table book.

>
> It comes back to "You get what you pay for". Customers think that
> Oracle spends as much money on the "freebies" (i.e. OCFS) as they do
> the database.

That has always been the case. Anybody with the right mind should expect to pay for decent things. There is a reason why Cadillac costs more then a Yugo.

-- 
Mladen Gogala
http://www.mladen-gogala.com

--
http://www.freelists.org/webpage/oracle-l