RE: Solaris 10 shmmax

At the time, we considered it best practice to max out the Solaris shmmax setting for multiple reasons:

• to avoid reboots when up-sizing SGAs
• to reduce frivolous SGA fragmentation
• to dispel users of the notion that it was *ever* a really useful knob to begin with

...I know we are discussing an old document, but you never said it was
wrong and the document clearly states on page 10:

"For optimal Oracle performance, the SGA must be allocated in a single shared memory segment as ISM. If the SGA cannot be built this way, Oracle will use multiple segments, and will settle for non-ISM allocations"

...so the list of 3 reasons you provide here is either missing mention
of the ISM angle, or the document pushed the ISM concern erroneously. Regardless of how old the document is it has to be one or the other.

The notion that having a monolithic SGA made a big performance impact was never actually characterized, AFAIK. I suspect it was only ever measurably important when something else was wrong - like a pathologically high rate of client process creation and exiting

...unless the document is wrong, it must have been characterized as
making a performance impact at some point in time.

In short, Oracle 10G uses Solaris MPO APIs to split its SGA purposefully across lgroups on NUMA systems. It also dedicates one DBWR per lgroup, with each exploiting local memory access for efficiency.

...Please explain. DBWR does not actually peek into SGA buffers (unless
block checking is turned on) so how can pairing DBWR with buffers on the same memory hierarchy exploit anything? And as far as I/O adaptors go there should be fair access to all from any CPU in these architectures, no? Making DBWR NUMA aware is not a very large step. Making foreground processes NUMA aware is where the big gains come in. After all, foregrounds actually use buffer contents so having them prefer local memory for their buffers speeds up post-I/O accesses, block cloning copies, sort spill and so on...

...On the other hand, the LRU structures need to be co-located in the
same memory hierarchy as the buffers for there to be significant efficiency gained. The LRU structures are in the variable SGA component so that would have to be chopped up and allocated one per memory hierarchy as well for NUMA placement. That's been done before. The things that would make this all nice and clean would be a shmget/shmat variant that places regions of an IPC shared memory segment onto memory hierarchy as per a description mask. That way it *looks* like a single segment to the admin but *feels* like a NUMA optimized segment to Oracle...I described this back in 1998 in brief here:

http://www.scaleabilities.co.uk/book/CH08.html

but for others to use the concept:

http://www.freepatentsonline.com/6505286.html

...Don't get me wrong, I'm glad to see 10g on Solaris at least taking
steps to exploit NUMA, but there is nothing new about the concept since it was done in Oracle8 on Sequent and (IIR) 8i on SGI, DEC and DG to varying degrees.

--
http://www.freelists.org/webpage/oracle-l