RE: Looking for Suggestions - 5 TB DB WHSE Backup options

Also Chris, you have to determine if your disk subsystem on your database can support the 12 streams from your calculation and does the increase in the number fo streams affect the congestion on your network coming off of that server. There is a point that more streams simply makes it worse from the server or network side.  

Matthew Parker

Chief Technologist

Dimensional DBA

425-891-7934 (cell)

D&B 047931344

CAGE 7J5S7 Dimensional.dba_at_comcast.net

<http://www.linkedin.com/pub/matthew-parker/6/51b/944/> View Matthew Parker's profile on LinkedIn

www.dimensionaldba.com <http://www.dimensionaldba.com/>  

From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Dimensional DBA Sent: Monday, August 1, 2016 1:05 PM
To: christopherdtaylor1994_at_gmail.com; gogala.mladen_at_gmail.com Cc: 'ORACLE-L'
Subject: RE: Looking for Suggestions - 5 TB DB WHSE Backup options  

The Oracle advice was only accurate if you used direct attached tape drives to your servers (circa 1990’s).

As to Seth’s comment on “Shoe Shining”, that only applies to slow writes to the tape device where the head is having to backtrack to the last write point over and over again because the stream of data can’t keep up with the streaming speed of the tape drives. If the streams are maintaining the pipe to the tape drive itself, then you will not get “shoe shining”.  

The math shows 1Gb link, but are your links larger than that from DB server to Netbackup media server? If it is then yes multiplexing at the Netbackup side for tape drives can help boost performance. This is where I originally stated there is architecture and Netbackup tuning that has to be done.

What version of tape drive do you have?

How many tape drives are dedicated to the Netbackup pool for your backups?

How many tape drivers are attached to each media server?

Intel or AMD architecture on the Netbackup media servers?

How many Netbackup media servers do you have to participate in the backup?

How much free memory is available on the Netbackup media server?

What is the chunk size setting of the Netbackup controlled tape drives.

How are you sending the data via RMAN? Are you simply letting Oracle chunk the data or are you breaking the database backup apart into manageable chunks?  

No Mladen, there is still a large portion of the world that uses tape directly for backups except for slow streaming or glitchy data flow backups (like from many windows servers). Yes there are customers who use disk too.  

As to your question Chris

“Let's say we have 6 tapes and he's configured it to do 2 threads per tape. Does that mean I can open 12 channels in RMAN? I mean, I would assume so but I have no idea LOL“  

Your math is correct but that assumes no one else is performing backups to those tape drives. This is where you have to work with your Netbackup admin as to what overall resources are available to you versus the rest of the backup system to get the optimal flow.

At Amazon we had Netbackup tape multiplexing set to 8 and I have many clients that use multiplexing at 4 to 6. IT really depends on what performance you are getting for each incoming stream and how much shared memory is dedicated to Netbackup on the media servers and number of tape drives per media server to help maintaining streaming performance to the drive.  

Matthew Parker

Chief Technologist

Dimensional DBA

425-891-7934 (cell)

D&B 047931344

CAGE 7J5S7 Dimensional.dba_at_comcast.net

<http://www.linkedin.com/pub/matthew-parker/6/51b/944/> View Matthew Parker's profile on LinkedIn

www.dimensionaldba.com <http://www.dimensionaldba.com/>  

From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Chris Taylor Sent: Monday, August 1, 2016 11:31 AM
To: gogala.mladen_at_gmail.com
Cc: ORACLE-L
Subject: Re: Looking for Suggestions - 5 TB DB WHSE Backup options  

Yeah Seth and I have been discussing this offline.  

If a "tape device" has multiple tape writers, then you can allocate multiple channels equal to the tape writers available in the "tape device". So this advice in Oracle Doc does seem to be dated in regard to Backup systems like Commvault, Netbackup etc.  

Chris    

On Mon, Aug 1, 2016 at 1:28 PM, Mladen Gogala <gogala.mladen_at_gmail.com> wrote:

Hi
One little clarification: that only applies if the tape device is really tape, which is increasingly rare. All backup suites that I have ever worked with (Commvault, NetBackup, EMC NetWorker) allocate SBT_TAPE device for writing to disk storage, too. If the device type SBT channel is allocated for writing to disk, it is reasonable to use multiple channels. Regards
On 08/01/2016 12:36 PM, Seth Miller wrote:

Duplexing wouldn't make much sense since that would mean copying the same file to multiple destinations. He is probably referring to multiplexing or parallelism. There is a difference between RMAN multiplexing and parallelism and the term "stream" usually relates to the latter. RMAN multiplexing allows multiple files to be read at the same time within a single channel and defaults to 8. RMAN parallelism refers to multiple channels writing to the same destination and is often referred to as multiplexing on the media management side.  

Oracle suggests that no more than 1 channel be used per tape device to prevent inefficient writes and shoe-shining.  

"Do not utilize media management multiplexing (multiple channels per tape drive). RMAN backup pieces will not be efficiently restored due to the interleaving of pieces on the same tape volume, which may necessitate the forward and backward movement of the tape."    

Seth Miller  

On Mon, Aug 1, 2016 at 11:20 AM, Chris Taylor <christopherdtaylor1994_at_gmail.com> wrote:

*Basically* - He keeps talking about Streams/Duplexing on the tape side - where 2 threads can write to the same tape apparently? I *think* that part is done on the Backup device outside of RMAN but I'm not sure how many channels I should allocate within RMAN to take advantage.  

For example I'm wondering (and I'll discuss with him) about the following scenario:  

Let's say we have 6 tapes and he's configured it to do 2 threads per tape. Does that mean I can open 12 channels in RMAN? I mean, I would assume so but I have no idea LOL  

Chris    

On Mon, Aug 1, 2016 at 10:44 AM, Seth Miller <sethmiller.sm_at_gmail.com> wrote:

Chris,  

It sounds like you are currently parallelizing the backup to disk with 4 channels and your NBU guy is suggesting parallelizing your backup directly to tape using multiple channels and you're asking how that is going to be different than what you are doing now. Is this correct?    

Seth Miller  

On Mon, Aug 1, 2016 at 9:31 AM, Chris Taylor <christopherdtaylor1994_at_gmail.com> wrote:

So we ran the first FULL on Friday and it took 9 hours - so your math is excellent.  

We did run into an ancillary problem however and that was that Netbackup was configured to use a storage pool which filled up on the Netbackup server and caused subsequent unrelated RMAN backups to fail for other databases. So, we're working through that. Our Netbackup guy wants us to write directly to tape and use multiple streams and is going to help us configure the RMAN script to take advantage of this. I'm unclear on the relationship between RMAN threads and Netbackup streams so I'm a little in the dark on what might need to be changed at this point. Currently, I'm using 4 RMAN threads/channels to write the backup.  

Thanks,

Chris    

On Tue, Jul 12, 2016 at 5:32 PM, Dimensional DBA <dimensional.dba_at_comcast.net> wrote:

It really depends on your netbackup infrastructure and your network infrastructure in between your database server and your backup infrastructure or your remote copy on disk and your backup infrastructure.  

If you have a 1GB backup network from your db server to your Netbackup Infrastrucutre with proper tuning you can normally achieve 104MB/sec assuming proper tuning and architecture or about 432GB/hr or in your case a 9hr full backup.  

If you have a 10Gb backup network from your db server to your Netbackup Infrastructure with proper tuning you can normally achieve 780MB/sec assuming proper tuning and architecture or about 2.88TB/hr or in your case a 2hr full backup.  

This is without any special equipment or SW licenses.    

You also could

1. If you have the Oracle ASO for advanced compression then you can also turn on compression in your RMAN backups and decrease the amount of data that has to be transferred across the link.
2. look at performing mixed incrementals and partial fulls to spread a full backup over multiple days.
3. Use the Incremental forever, although I wouldn’t recommend this.

Matthew Parker

Chief Technologist

Dimensional DBA

425-891-7934 (cell)

D&B 047931344

CAGE 7J5S7 Dimensional.dba_at_comcast.net

<http://www.linkedin.com/pub/matthew-parker/6/51b/944/> View Matthew Parker's profile on LinkedIn

www.dimensionaldba.com <http://www.dimensionaldba.com/>  

From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Ilmar Kerm Sent: Tuesday, July 12, 2016 3:20 PM
To: christopherdtaylor1994_at_gmail.com
Cc: ORACLE-L
Subject: Re: Looking for Suggestions - 5 TB DB WHSE Backup options  

Hi  

We do "incremental forever" backups using incrementally updated image copies that are located on a separate disk storage than the primary database. Before updating the image copy, we snapshot the backup filesystem to provide backup history. Also store a second copy of the archivelogs in that same filesystem where the image copy is located - LOG_ARCHIVE_DEST_x parameter, so no separate archivelog backup is needed.  

Tape is ruled out in this case and you also need either a storage system that can do NFS, snapshots, compression (optional) and thin clones (Oracle ZFSSA, Netapp, ...) or a filesystem that can do these things (ACFS, ZFS, ..?). But all the steps needed (except storage snapshots) can be done using RMAN and no 3rd party libraries are needed - just need to write a script to orchestrate the steps.      

Ilmar    

On Tue, Jul 12, 2016 at 7:42 PM, Chris Taylor <christopherdtaylor1994_at_gmail.com> wrote:

Ok, guys & gals, I'm looking for suggestions for the following challenge. I'm very familiar with RMAN fulls w/ incrementals writing either to disk or directly to tape. What I'm NOT familiar with is other options that I may not know of and that's where I need your help.  

Objective:
Nightly backups of 5 TB Data Warehouse that is currently being snapshotted weekly at the SAN Layer instead of tape or disk based backups.  

Hardware/OS:

IBM XIV Storage (not sure of model #)

RedHat Linux OS (5.x)

Oracle 11.2.0.4

Netbackup is tape solution  

Method Options:

1. RMAN Fulls on Weekend, (either to disk or direct to tape) with nightly incrementals. I'm leaning toward disk based backups which are then written to tape and using parallel threads for the disk based backup to prevent overwhelming our tape library
2. Other options?

Chris  

-- 

Ilmar Kerm

 

 

 

 

 

 

-- 
Mladen Gogala
Oracle DBA
Tel: (347) 321-1217 <tel:%28347%29%20321-1217> 

 



--
http://www.freelists.org/webpage/oracle-l