RE: OCR / VD external vs. normal redundancy using NFS.
Date: Sat, 10 Jul 2010 01:11:10 +0200
Message-ID: <4814386347E41145AAE79139EAA3989810265B1CE0_at_ws03-exch07.iconos.be>
David,
See following oracle documentation part: http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/storage.htm#sthref536
When using normal redundancy, Oracle will create 3 voting disks and 1 ocr device, which will be mirrored. When using high redundancy, Oracle will create 5 voting disks and 1 ocr device, which will be mirrored 2 times.
The output below comes from a setup where I have a separate normal redundancy diskgroup to hold my cluster files. I needed 3 different luns in this diskgroup before I could create it.
Also note that you can't see the voting disks as "files" within the diskgroup. My guess is that Oracle uses a header within each asm disk to store the voting disk.
[grid_at_arsvorats1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 7855e73b3d2a4fb3bf901beecad62093 (/dev/oracle/asm_clfilests1p1) [DG_CLUSTER]2. ONLINE 213259f135934f27bf86b91559a44df5 (/dev/oracle/asm_clfilests2p1) [DG_CLUSTER] 3. ONLINE 1b12d9dc6d2b4fb8bfe63b365e2db555 (/dev/oracle/asm_clfilests3p1) [DG_CLUSTER] Located 3 voting disk(s).
[grid_at_arsvorats1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2800
Available space (kbytes) : 259320
ID : 1074808153
Device/File Name : +DG_CLUSTER
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
[grid_at_arsvorats1 ~]$ export ORACLE_SID=+ASM1
[grid_at_arsvorats1 ~]$ asmcmd
ASMCMD> ls
DG_CLUSTER/ DG_DWHTS_DATA/ DG_FRA/
ASMCMD> cd DG_CLUSTER
ASMCMD> ls
arclorats/
ASMCMD> cd arclorats
ASMCMD> ls
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> cd OCRFILE
ASMCMD> ls -l
Type Redund Striped Time Sys Name OCRFILE MIRROR COARSE JUL 10 00:00:00 Y REGISTRY.255.721844181
SQL> set linesize 120
SQL> column full_alias_path format a70
SQL> column file_type format a15
SQL> select concat('+'||gname, sys_connect_by_path(aname, '/')) full_alias_path,
2 system_created, alias_directory, file_type
3 from ( select b.name gname, a.parent_index pindex, a.name aname,
4 a.reference_index rindex , a.system_created, a.alias_directory,
5 c.type file_type
6 from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
7 where a.group_number = b.group_number
8 and a.group_number = c.group_number(+)
9 and a.file_number = c.file_number(+)
10 and a.file_incarnation = c.incarnation(+)
11 )
12 where alias_directory = 'N'
13 and system_created = 'Y'
14 and file_type like 'OCR%'
15 start with (mod(pindex, power(2, 24))) = 0
16 and rindex in
17 ( select a.reference_index
18 from v$asm_alias a, v$asm_diskgroup b
19 where a.group_number = b.group_number
20 and (mod(a.parent_index, power(2, 24))) = 0
21 )
22 connect by prior rindex = pindex;
FULL_ALIAS_PATH S A FILE_TYPE ---------------------------------------------------------------------- - - --------------- +DG_CLUSTER/arclorats/OCRFILE/REGISTRY.255.721844181 Y N OCRFILE
Freek D'Hooge
Uptime
Oracle Database Administrator
email: freek.dhooge_at_uptime.be
tel +32(0)3 451 23 82
http://www.uptime.be
disclaimer: www.uptime.be/disclaimer
-----Original Message-----
From: David Robillard [mailto:david.robillard_at_gmail.com]
Sent: vrijdag 9 juli 2010 18:36
To: D'Hooge Freek
Cc: oracle-l mailing list; LS Cheng
Subject: Re: OCR / VD external vs. normal redundancy using NFS.
Hi Freek,
On Fri, Jul 9, 2010 at 12:20 PM, D'Hooge Freek <Freek.DHooge_at_uptime.be> wrote:
> DirectNFS can't be used for cluster files (ocr or voting disks).
> When using AMS with external redundancy you only have 1 vd and 1 ocr device.
> I only have setup such a configuration once. There I used 3 luns to create a separate diskgroup that only holds my voting disks and ocr devices.
> The other diskgroups I created where set to external redundancy.
Well, I found the answer to my question...
> "How then do you create the OCR mirror and multiple Voting Disks with ASM using External redundancy?"
...by RTFM. Chapter 2 of "Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2)" states that:
<quote>
If OCR is stored in an Oracle ASM disk group with external redundancy,
then Oracle recommends that you add another OCR location to another
disk group to avoid the loss of OCR, if a disk fails in the disk
group.
</quote>
There you have it! I think I'll create a very small ASM disk group called +GRID in which I'll store one OCR disk and one Voting Disk. I'll use the +DATA disk group to store another Voting Disk and the +FRA disk group will have the second OCR disk and third Voting Disk.
I guess that should have me covered.
> Note that when you create the diskgroup for the ocr / voting disks during the rac installation, normal redundancy means 3 way mirror and high redundancy means 5 way mirroring.
Ah, not exactly :) If you read chapter 1 of "Oracle Database Storage Administrator's Guide 11g Release 2 (11.2)" [2], you see:
<quote>
- Normal for 2-way mirroring
- High for 3-way mirroring
- External to not use Oracle ASM mirroring, such as when you configure hardware RAID for redundancy </quote>
Many thanks for your help, I appreciated it.
David
[1] http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/votocr.htm#CHDJIIJH
[2] http://download.oracle.com/docs/cd/E11882_01/server.112/e10500/asmcon.htm#BABGDGAA
> regards,
>
> Freek D'Hooge
> Uptime
> Oracle Database Administrator
> email: freek.dhooge_at_uptime.be
> tel +32(0)3 451 23 82
> http://www.uptime.be
> disclaimer: www.uptime.be/disclaimer
> ________________________________________
> From: David Robillard [david.robillard_at_gmail.com]
> Sent: 09 July 2010 17:26
> To: D'Hooge Freek
> Cc: oracle-l mailing list; LS Cheng
> Subject: Re: OCR / VD external vs. normal redundancy using NFS.
>
> Hello Freek,
>
> On Thu, Jul 8, 2010 at 2:26 AM, D'Hooge Freek <Freek.DHooge_at_uptime.be> wrote:
>> Yes and no, when using a single ocr device then the cluster would hang waiting on an io response instead of failing. >> But I don't think many people would care for the difference when encountering such a situation. >
> Indeed, I totally agree with you.
>
> Do you know if this bug is also present with Oracle Direct NFS ?
> >> The chance that you run into this bug depends on your storage configuration. >> So, you would best test what happens in you configuration. >
> Well, you sort of convinced me to swith from NFSv3 to ASM with iSCSI
> over GbE. I've watched the webinar "ASM: advantages and disadvantages"
> presented by Frits Hoogland [1]. Besides the very important
> problematic behavior you thankfully pointed out to me, I like the fact
> that ASM can help you migrate from one stoage array to another without
> taking the cluster down.
>
> The way I see it, ASM takes the file management "problem" out, but
> introduces a new set of complexities in managing LUNs and iSCSI
> configurations. NFS is still a lot less complicated to manage.
>
> Anyway, I'll switch to ASM now and see how it goes.
> >> I like having multiple ocr and voting devices, even when I only have a single storage array as it >> protects you against file corruptions and against some "administrator oepses" >> (which are these days more command than hardware failures). >
> I also agree with you here. However, I plan to use External Redundancy
> with ASM. But in "Metalink note 810394.1 - "RAC Assurance Support
> Team: RAC and Oracle Clusterware Starter Kit and Best Practices
> (Generic)" [2], you found this:
>
> <quote>
> For Oracle 11gR2 it is a best practice to store the OCR and Voting
> Disk within ASM and to maintain the ASM best practice of having no
> more than 2 diskgroups (Flash Recovery Area and Database Area). This
> means that the OCR and Voting disk will be stored along with the
> database related files. If you are utilizing external redundancy for
> your disk groups this means you will have 1 Voting Disk and 1 OCR.
> </quote>
>
> How then do you create the OCR mirror and multiple Voting Disks with
> ASM using External redundancy?
>
> And do you create a disk group to store only the OCR + voting disks or
> you really just drop them into the DATA and FRA diskgroups as stated
> above?
>
> Many thanks,
>
> David
>
> [1] http://www.oracleracsig.org/pls/apex/f?p=105:200:3736449467233452::NO:::
>
> [2] https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx?id=810394.1
> > >> Regards, >> >> >> Freek D'Hooge >> Uptime >> Oracle Database Administrator >> email: freek.dhooge_at_uptime.be >> tel +32(0)3 451 23 82 >> http://www.uptime.be >> disclaimer: www.uptime.be/disclaimer >> -----Original Message----- >> From: David Robillard [mailto:david.robillard_at_gmail.com] >> Sent: maandag 5 juli 2010 22:29 >> To: D'Hooge Freek >> Cc: oracle-l mailing list; LS Cheng >> Subject: Re: OCR / VD external vs. normal redundancy using NFS. >> >> On Thu, Jul 1, 2010 at 1:31 AM, D'Hooge Freek <Freek.DHooge_at_uptime.be> wrote: >>> I'm usually using the normal redundancy, putting each voting disk / cluster registry in its own volume. >>> You might want to read http://freekdhooge.wordpress.com/2009/11/03/two-oracle-rac-bugs-on-the-wall-two-oracle-bugs-take-one-down/. Because of this "bug" I needed to switch to iscsi for the cluster registry volumes. It is probably still valid with 11gR2 >> >> Interesting. Thanks for the heads-up. >> >> I've read your post and the ones from Geert de Paep. What I understand >> is that problem seems to exist when you have several OCR disks. But if >> you choose "External redundancy" instead of "Normal redundancy" when >> you install the Grid Infrastructure software, you end up with a single >> OCR disk. Hence this bug would not happen. Right? >> >> But is using "External redundancy" for OCR a bad idea when you have a >> single storage array? >> >> Because if that single storage array suffers a catastrophic failure, >> you end up loosing not only the OCR disk, but the voting disk and all >> database files as well. The entire cluster would be down even with >> several OCR disks. >> >> So even if I switch to iSCSI, I would still have the same "single >> storage array" problem. Since all LUNs would be stored on that storage >> array. >> >> Unless there are other ways for an OCR disk to become corrupted? >> >> Thanks, >> >> David >> >>> Regards, >>> >>> Freek D'Hooge >>> Uptime >>> Oracle Database Administrator >>> email: freek.dhooge_at_uptime.be >>> tel +32(0)3 451 23 82 >>> http://www.uptime.be >>> disclaimer: www.uptime.be/disclaimer >>> >>> -----Original Message----- >>> From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of David Robillard >>> Sent: woensdag 16 juni 2010 16:44 >>> To: oracle-l mailing list >>> Subject: OCR / VD external vs. normal redundancy using NFS. >>> >>> Hello everyone, >>> >>> I'd like to know how does each of us configure redundancy for both the >>> Grid Infrastructure's Oracle Cluster Registry (OCR) and Voting disks >>> (VD) when they're stored over NFSv3 on an enterprise grade storage >>> array with RAID disks. Do you use external or normal redundancy for >>> OCR and VD? >>> >>> I'm looking to install Grid Infrastructure 11gR2 on RedHat Enterprise >>> Linux 5 x86_64 over NFSv3 on a clustered Sun/Oracle Unified Storage >>> 7410. The storage array has built-in redundancy on every components. >>> Each cluster node has two quad-ethernet network interface cards (NIC). >>> All network interfaces are built with two ports, one on each NIC, to >>> create a private (bond0), public (bond1) and storage (bond2) >>> interface. There are two redundant GbE switches dedicated for a >>> storage-only non-routable subnet. >>> >>> Quick note, I'm not using ASM over iSCSI for various reasons. Mostly >>> because every paper I read from NetApp/Sun/Oracle/EMC says the >>> performance of ASM with software initiator iSCSI is not as good as >>> NFS. Also because it's easier to manage the storage array's total disk >>> space when using NFS instead of iSCSI. I'm also not using Oracle's >>> dNFS feature simply because I haven't had the time to look at it and >>> I've been working with NFS for over 10 years. Plus I'm also lucky (?) >>> enough to be the UNIX sysadmin, the storage array administrator and >>> the DBA, so I don't need to configure everything from the Oracle >>> stand-point (i.e. I can't do finger pointing in case things go wrong, >>> I'm the only one to blame). >>> >>> With that in mind, which option would you choose and why? >>> >>> Option A) >>> >>> Create an NFS share for OCR and another one for VD then use external >>> redundancy. That would generate the following mount points: >>> >>> OCR = /u01/ocr/cluster.registry >>> VD = /u01/vd/voting.disk >>> >>> -or- >>> >>> Option B) >>> >>> Create three different NFS shares for OCR and three other shares for >>> VD then use normal redundancy. That would create the following mount >>> points: >>> >>> OCR 1 = /u01/ocr/ocr1/cluster.registry >>> OCR 2 = /u02/ocr/ocr2/cluster.registry >>> OCR 3 = /u03/ocr/ocr3/cluster.registry >>> VD 1 = /u01/vd/vd1/voting.disk >>> VD 2 = /u02/vd/vd2/voting.disk >>> VD 3 = /u03/vd/vd3/voting.disk >>> >>> Of course there are other options and variations. I welcome all >>> comments and critics on that setup :) >>> >>> Many thanks, >>> >>> David >>> -- >>> David Robillard, UNIX team leader and Oracle DBA >>> CISSP, RHCE, SCSA & SCSECA >>> -- >>> http://www.freelists.org/webpage/oracle-l >>> >>> >>> >> >
-- http://www.freelists.org/webpage/oracle-lReceived on Fri Jul 09 2010 - 18:11:10 CDT