Oracle FAQ Your Portal to the Oracle Knowledge Grid

Home -> Community -> Mailing Lists -> Oracle-L -> Re: RMAN impact

Re: RMAN impact

From: Steve Perry <>
Date: Fri, 10 Mar 2006 06:32:24 -0600
Message-ID: <02aa01c6443e$afa5d620$0301a8c0@desk1>

thanks Mladen,

Feel free to berate, it's not my doing :)

I wasn't saying that it was a good configuration. I didn't set it up and I'm only watching from the sidelines. I said a lot of what you did, but some people know better. I didn't want to get into why it's configured this way or why it hasn't been changed. In fact, "they" won't allow any changes. It would take too long to explain it all, but you'd have a good laugh :) the box is a 4-way Dell 6850. side note: It has 2-2 ghz HBAs coming in, but can only do 1GB to the storage node.

What is interesting to me is that this RMAN backup has this kind of impact. at my last job, I had BCVs attached a single windows backup host (HP DL580) and ran 3 concurrent backups (2 TB, 1TB and 250GB) and it ran/responded fine. They completed in a little over 4 hours. They were file backups (we did hot backups and split the BCVs).
The backup host had HBAs attached to 2 SANs (1 was tape and the other was the data). The RMAN backup has data coming off the SAN and being written over the network.

To sum up what you said "You get what you pay for" and I totally agree.


On 03/09/2006 09:14:21 PM, Steve Perry wrote:
> we have a rman backup for 500 GB 9.2 RAC on RHEL3 (3 LTO II drives) that
> takes 3 hours to complete, but kills the node it runs on.
> io wait % goes to 60%, cpu is low. the server is pretty much unresponsive
> until it completes.
> We allocate 4 channels and are not using the large_pool or tape slaves.
> We tried the same thing but used Disk (seperate LUN from the database) and
> the server crashed.

Steve, it is not my intention to berate you or anything of the sort, but this
is some weird stuff. PC equipment, even the one with the SMP motherboards is not made for high volume I/O. Now you are discovering the difference between a
PC and mini-computers like HP 9000/rp4400 or IBM p520: those can do massive amounts of I/O while not even the best Dell PC can do that. The problem is in
the fact that PC buss, even with the best SMP motherboard, doesn't have enough
capacity to allow simultaneous traffic between multiple CPU boards, peripheral
devices and memory. Disks normally use DMA and deposit the result of I/O directly into the memory. Disks also notify CPU that I/O is done by sending interrupts which must be handled. With 4 channels, you have 4 active RMAN processes, each performing reads from your disks, depositing the result into memory and notifying any available CPU that it has completed I/O. It communicates with Oracle processes,sends data to and from network, which results in some more interrupts and DMA traffic between NIC and memory and your system bus is saturated. System is unresponsive because simple interrupts,
like pressing enter, must wait to be handled. Steve, there is a reason why PC
equipment is so much cheaper: it cannot do massive amounts of I/O. What you pay
for when you buy p595 is a massive backplane which can sustain almost a TB/second and will allow your system to operate normally, even if you are writing 300MB/sec. The secret is in the fact that fiber channel adapter for IBM
p595 is attached to memory and not the central system bus. When you issue an I/O request, you deposit IORB on one location in the memory where smart I/O controller reads it, executes and deposits results into memory. It then issues
a single interrupt saying that it's done. Central system bus, the one used to
carry data between CPU and RAM isn't used at all. Drivers for that kind of equipment are standard on AIX or HP-UX and require some work on Linux, where
they're known as I2O. Also, architecturally speaking, those machines are much
more balanced. When you have 4 screaming 3GHZ Intels inside, it is tough to feed them with memory. The fastest available memories are 30-50ns. That means
that CPU can ask memory for more data less then 40 million times per second. Translated into megahertz, it corresponds to bus frequency of 40MHZ, needed of
course, to feed each of the processors. Unless we can feed them faster, our screaming 3GHZ chip is useless as it, like the angel in the movie "Barbarella", has no memory to work on.
The situation can be improved by large L2 and L1 caches as well as TLB buffers. The efficiency of those is severely impacted by things like long jumps
across the address space and context switches like, for instance, ones caused
by interrupts or normal multiprocess work. Add, on that same bus, additional
lines for cache synchronization, which are on a separate bus on the proper minis, needed to keep caches coherent. Those lines must exist between each L1
and L2 cache, so that it doesn't happen that one address has one value in one
cache and another one in another. Now, add peripheral devices: video adapter,
disk controllers and NIC. Your central bus has frequency of 233MHZ and is 64 wires wide ("64 bits"). When you calculate the maximum speed, it gives you 1864
megabytes/second for EVERYTHING. That is theoretical data transfer speed. The
real one, due to retransmits and synchronizations between devices (Which device
should transmit first? This is resolved by so called "bus arbitration", as well
as the question which CPU should handle an interrupt) is significantly lower,
only around 1.2 GB/sec. That is called "sustained data rate". The massive amount of I/O that you are trying to make your poor PC perform will
simply consume the central bus and nothing will work. In addition to that, if
interrupt handlers and software components start to detect timeouts, your machine will think that there is something wrong with the motherboard and will
crash. Having only two channels would probably finish sooner and with less problems then having 4 channels. Machine doesn't crash with the tape as tapes
are slower and cannot do I/O fast enough to endanger the central bus. Multiple
disk drives are just about fast enough to crash the system. Detecting problems
like this is precisely the purpose of benchmarking and testing before you buy a
machine. For things like that you should use a proper mini. Do you know why they call them Infinitely Boring Machines? Nothing ever happens. They don't crash, they don't go down, they just quietly work and don't provide any excitement or adventure in your life.

Mladen Gogala

Received on Fri Mar 10 2006 - 06:32:24 CST

Original text of this message