deep-geek discussion: config for Sun E10K OPS w/mirrored A3500 Arrays?

I am open to suggestions from anyone experienced with the Sun A3500 and D5000 arrays about laying out a very hot and critical OLTP database on mirrored A3500 arrays on a Sun Starfire E10000. I will be using hardware striping and will mirror one entire array to the other. I'm not looking for basic advice on how to allocate segments to tablespaces, how to "separate data from indexes" (it may be moot anyway) or other such advice. The books - Oracle's and others - are all full of generic advice for lesser systems. Few address

I'm
looking for deep-geek info/validation on stripe sizing, cache trashing, and some well-reasoned discussion/religion on some basic intense I/O issues.

Facts:

1. There is also a D1000 array for system disks, $ORACLE_HOME, apps, archive logs, etc.
2. The database consists of several catagories of objects:
   ~40 GB of large tables that are extremely read intensive during critical
   performance periods, but inserted into during an hour-long "batch'ish" process.
   ~1 GB of infrequently updated/inserted/deleted data that is read critical
   ~4-8 GB of intensely updated, inserted, and read data - most critical!
   ~1-2 GB of non-persistent(!) data that is inserted -> read -> deleted
   
3. This database sustaines 200 TPS and peaks at 800-2000 TPS. It generates 10-20 GB of archive log every day (depending on activity)
4. Each A3500 has 128 MB cache, 50 9GB 10k RPM drives, 2 controllers (Disk space obviously isn't the issue - the spindle count is for speed!)
5. Memory and CPU are tunable - starting at 12 GB & 12 337MHz CPUs (Could go significantly larger (~3x) in this domain.)
6. Will probably use 8k Oracle block size (perhaps 16k after testing)

---

Assume that everything wil be on raw devices (except archived logs of course) since OPS is involved. The questions are:

1. Cache-trashing seems to be a potential issue here. Since the entire array has only 128 MB of cache, what are thoughts about caching only the redo logs? What about turning off all read cache and reserving it for writes only? Eh?
2. Online redo logs will be ping-ponged between at least two drives, Peak is a 100MB log switch every 45-90 seconds, during less critical hours. Most critical hours see 100MB log switch every 2-5 minutes.
3. Rule of thumb is a log switch every 15/20/<pick your religion> minutes, but are near-gigabyte redo logs even feasible? What are the largest you've used for ultra hot OLTP? What is your religion here?
4. Given the nature of redo logs, has anyone really seen significant benefit from striping them? (As in ping-ponging between two striped n-disk LUNs?)
5. Assume that 4 of the 50 disks in each A3500 are dedicated to redo logs, That leaves 46 disks and 6 available LUNs. It could be set up as anything between "stripe everything across everything" ( a newer "start-up" religion) and 6 striped LUNs. For example, the latter might be 5 LUNS of 8-disk wide stipe sets plus one LUN of a 6-disk wide stripe set. Traditionally, I would have gone with something like this, but the appeal of "everything everywhere" is intriguing - especially since the I/O is so random. Experiences? Religion? Horror stories?
6. Now, to complicate matters, consider if you had three of the A3500 arrays with 50 disks each and one D5000 (?) array (100 MB/sec, but no cache). How would you split the redo and the various type of segments across the various arrays? Dedicate one A3500 array to redo logs?!!??!?!? (For caching efficiency primarily...)
7. Does the introduction of OPS change anything significant in your model? (Please! No obvious and generic "partitioning" warnings.)
8. What would be your optimal stripe size for each segment type? (Please specify whether per disk column or per stripe width.)

I've been laying out Oracle servers for many, many years and I've been laying out fairly high-end Solaris Oracle servers for years, and I've been working with parallel server for a few years. But nothing quite like this! And I've never worked with the A3500 arrarys before.

I'd hate to miss anything with this one!

• OraSaurus --
• Remove "not_" to reply... --