RE: ORA-15042: ASM disk "xx" is missing

From: Amaral, Rui <Rui.Amaral_at_tdsecurities.com>
Date: Wed, 2 Jan 2013 13:12:19 -0500
Message-ID: <CC45762506E4C84A8612307613CDCB2560D33ACF_at_EX7T2-SV08.TDBFG.COM>



No specific view that I am aware of myself my Oracle has a note on how to do this using os tools:

Note id: 452770.1

Use truss on the rbal process when doing the select * from v$asm_disk.

hth

Rui Amaral
Senior Oracle DBA
Oracle Database Engineering, TDBFG
E-Mail: rui.amaral_at_tdsecurities.com| Blackberry: 416-625-3280 |

-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Fergal Taheny Sent: Wednesday, January 02, 2013 12:45 PM To: oracle-l-freelists
Subject: ORA-15042: ASM disk "xx" is missing

Hi,
ASM 11.1.0.7.0
Solaris 10

We have a diskgoup that won't mount. I'm believe it a SAN/disk issue as opposed to an ASM issue but I'm trying to eliminate some guesswork in troubleshooting this.

In the +ASM alert log we have:

NOTE: cache registered group ORADATA number=1 incarn=0x2bc38c91
NOTE: cache began mount (first) of group ORADATA number=1 incarn=0x2bc38c91 Wed Jan 02 14:16:33 2013
NOTE: Assigning number (1,15) to disk (/dev/oradsk/ASMDA7)
NOTE: Assigning number (1,10) to disk (/dev/oradsk/ASMDA2)
NOTE: Assigning number (1,18) to disk (/dev/oradsk/ASMDA10)
NOTE: Assigning number (1,9) to disk (/dev/oradsk/ASMDA1)
NOTE: Assigning number (1,19) to disk (/dev/oradsk/ASMDA11)
NOTE: Assigning number (1,20) to disk (/dev/oradsk/ASMDA12)
NOTE: Assigning number (1,21) to disk (/dev/oradsk/ASMDA13)
NOTE: Assigning number (1,22) to disk (/dev/oradsk/ASMDA14)
NOTE: Assigning number (1,23) to disk (/dev/oradsk/ASMDA15)
NOTE: Assigning number (1,24) to disk (/dev/oradsk/ASMDA16)
NOTE: Assigning number (1,25) to disk (/dev/oradsk/ASMDA17)
NOTE: Assigning number (1,26) to disk (/dev/oradsk/ASMDA18)
NOTE: Assigning number (1,12) to disk (/dev/oradsk/ASMDA4) Wed Jan 02 14:16:37 2013
NOTE: start heartbeating (grp 1)

kfdp_query(): 12
kfdp_queryBg(): 12
NOTE: Assigning number (1,11) to disk ()
NOTE: Assigning number (1,13) to disk ()
NOTE: Assigning number (1,14) to disk ()
NOTE: Assigning number (1,16) to disk ()
NOTE: Assigning number (1,17) to disk ()
kfdp_query(): 13
kfdp_queryBg(): 13
NOTE: cache dismounting group 1/0x2BC38C91 (ORADATA)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x2BC38C91 (ORADATA)
NOTE: cache ending mount (fail) of group ORADATA number=1 incarn=0x2bc38c91
kfdp_dismount(): 14
kfdp_dismountBg(): 14
NOTE: De-assigning number (1,9) from disk (/dev/oradsk/ASMDA1)
NOTE: De-assigning number (1,10) from disk (/dev/oradsk/ASMDA2)
NOTE: De-assigning number (1,12) from disk (/dev/oradsk/ASMDA4)
NOTE: De-assigning number (1,15) from disk (/dev/oradsk/ASMDA7)
NOTE: De-assigning number (1,18) from disk (/dev/oradsk/ASMDA10)
NOTE: De-assigning number (1,19) from disk (/dev/oradsk/ASMDA11)
NOTE: De-assigning number (1,20) from disk (/dev/oradsk/ASMDA12)
NOTE: De-assigning number (1,21) from disk (/dev/oradsk/ASMDA13)
NOTE: De-assigning number (1,22) from disk (/dev/oradsk/ASMDA14)
NOTE: De-assigning number (1,23) from disk (/dev/oradsk/ASMDA15)
NOTE: De-assigning number (1,24) from disk (/dev/oradsk/ASMDA16)
NOTE: De-assigning number (1,25) from disk (/dev/oradsk/ASMDA17)
NOTE: De-assigning number (1,26) from disk (/dev/oradsk/ASMDA18)
ERROR: diskgroup ORADATA was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "17" is missing
ORA-15042: ASM disk "16" is missing
ORA-15042: ASM disk "14" is missing
ORA-15042: ASM disk "13" is missing
ORA-15042: ASM disk "11" is missing

ERROR: alter diskgroup oradata mount

So ASM identified 13 disks and couldn't find 5 disks.

The first problem is how to map these missing ASM disks to unix device files.

Looking in /dev/oradsk I find device files for the 13 good disks listed above and also 5 other device files not listed above. I assume these are the device files for the 5 problem disks but I want to prove that.

When I use the od command I can read the file headers for the 13 good disks:

> od -c -N 128 /dev/oradsk/ASMDA1

0000000  \0 202 001 001  \0  \0  \0  \0 200  \0  \0  \t 326 331 223 236
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040   O   R   C   L   D   I   S   K  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000100  \n 020  \0  \0  \0  \t 001 003   O   R   A   D   A   T   A   _
0000120   0   0   0   9  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 O R A D A T A \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200

And I get an IO error for the other 5 device files

> od -c -N 128 /dev/oradsk/ASMDA3
od: cannot open ASMDA3: I/O error

Ok still promising but still guessing a bit.

So I look in v$asm_disk and I only see the 13 good disks (same for v$asm_disk_stat and X$KFDSK)

sys_at_+ASM> select path from v$asm_disk;

PATH



/dev/oradsk/ASMDA1
/dev/oradsk/ASMDA2
/dev/oradsk/ASMDA4
/dev/oradsk/ASMDA7
/dev/oradsk/ASMDA10
/dev/oradsk/ASMDA11
/dev/oradsk/ASMDA18
/dev/oradsk/ASMDA13
/dev/oradsk/ASMDA14
/dev/oradsk/ASMDA15
/dev/oradsk/ASMDA16
/dev/oradsk/ASMDA17
/dev/oradsk/ASMDA12

13 rows selected.

Eventually I go back through old alert logs to find the last time the diskgroup was successfully mounted and I find that indeed the five disks I suspected were mounted in the disk group at that time. So that's fairly conclusive but my questions are:

Is there a view in ASM that lists all the disks that belong in a disk group? Including ones asm can't see at present.

Or is this information stored in the file headers? If so how could I read it?

Or does ASM just probe all the disks (as defined in asm_diskstring) to find all the disks in a disk group? And if it does how does it know how many disks to expect? I don't see the "numbers of disks" displayed anywhere in V$ASK_DISKGROUP or in the output of kfed (shown below). Where would I find the number of disks in each disk group?

> kfed read /dev/oradsk/ASMDA1

kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483657 ; 0x008: TYPE=0x8 NUMB=0x9
kfbh.check:                  3604583326 ; 0x00c: 0xd6d9939e
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000
kfdhdb.dsknum:                        9 ; 0x024: 0x0009
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:            ORADATA_0009 ; 0x028: length=12
kfdhdb.grpname:                 ORADATA ; 0x048: length=7
kfdhdb.fgname:             ORADATA_0009 ; 0x068: length=12
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32975350 ; 0x0a8: HOUR=0x16 DAYS=0xf
MNTH=0xa YEAR=0x7dc
kfdhdb.crestmp.lo:           1831326720 ; 0x0ac: USEC=0x0 MSEC=0x1f5
SECS=0x12 MINS=0x1b
kfdhdb.mntstmp.hi:             32975859 ; 0x0b0: HOUR=0x13 DAYS=0x1f
MNTH=0xa YEAR=0x7dc
kfdhdb.mntstmp.lo:           3870520320 ; 0x0b4: USEC=0x0 MSEC=0xdd
SECS=0x2b MINS=0x39
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                  204760 ; 0x0c4: 0x00031fd8
kfdhdb.pmcnt:                         3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      0 ; 0x0d4: 0x00000000
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000
kfdhdb.dbcompat:              168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi:             32942731 ; 0x0e4: HOUR=0xb DAYS=0x14
MNTH=0xa YEAR=0x7da
kfdhdb.grpstmp.lo:           3678353408 ; 0x0e8: USEC=0x0 MSEC=0x3ce
SECS=0x33 MINS=0x36
kfdhdb.ub4spare[0]:                   0 ; 0x0ec: 0x00000000
kfdhdb.ub4spare[1]:                   0 ; 0x0f0: 0x00000000
kfdhdb.ub4spare[2]:                   0 ; 0x0f4: 0x00000000
kfdhdb.ub4spare[3]:                   0 ; 0x0f8: 0x00000000
kfdhdb.ub4spare[4]:                   0 ; 0x0fc: 0x00000000
kfdhdb.ub4spare[5]:                   0 ; 0x100: 0x00000000
kfdhdb.ub4spare[6]:                   0 ; 0x104: 0x00000000
kfdhdb.ub4spare[7]:                   0 ; 0x108: 0x00000000
kfdhdb.ub4spare[8]:                   0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[9]:                   0 ; 0x110: 0x00000000
kfdhdb.ub4spare[10]:                  0 ; 0x114: 0x00000000
kfdhdb.ub4spare[11]:                  0 ; 0x118: 0x00000000
kfdhdb.ub4spare[12]:                  0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[13]:                  0 ; 0x120: 0x00000000
kfdhdb.ub4spare[14]:                  0 ; 0x124: 0x00000000
kfdhdb.ub4spare[15]:                  0 ; 0x128: 0x00000000
kfdhdb.ub4spare[16]:                  0 ; 0x12c: 0x00000000
kfdhdb.ub4spare[17]:                  0 ; 0x130: 0x00000000
kfdhdb.ub4spare[18]:                  0 ; 0x134: 0x00000000
kfdhdb.ub4spare[19]:                  0 ; 0x138: 0x00000000
kfdhdb.ub4spare[20]:                  0 ; 0x13c: 0x00000000
kfdhdb.ub4spare[21]:                  0 ; 0x140: 0x00000000
kfdhdb.ub4spare[22]:                  0 ; 0x144: 0x00000000
kfdhdb.ub4spare[23]:                  0 ; 0x148: 0x00000000
kfdhdb.ub4spare[24]:                  0 ; 0x14c: 0x00000000
kfdhdb.ub4spare[25]:                  0 ; 0x150: 0x00000000
kfdhdb.ub4spare[26]:                  0 ; 0x154: 0x00000000
kfdhdb.ub4spare[27]:                  0 ; 0x158: 0x00000000
kfdhdb.ub4spare[28]:                  0 ; 0x15c: 0x00000000
kfdhdb.ub4spare[29]:                  0 ; 0x160: 0x00000000
kfdhdb.ub4spare[30]:                  0 ; 0x164: 0x00000000
kfdhdb.ub4spare[31]:                  0 ; 0x168: 0x00000000
kfdhdb.ub4spare[32]:                  0 ; 0x16c: 0x00000000
kfdhdb.ub4spare[33]:                  0 ; 0x170: 0x00000000
kfdhdb.ub4spare[34]:                  0 ; 0x174: 0x00000000
kfdhdb.ub4spare[35]:                  0 ; 0x178: 0x00000000
kfdhdb.ub4spare[36]:                  0 ; 0x17c: 0x00000000
kfdhdb.ub4spare[37]:                  0 ; 0x180: 0x00000000
kfdhdb.ub4spare[38]:                  0 ; 0x184: 0x00000000
kfdhdb.ub4spare[39]:                  0 ; 0x188: 0x00000000
kfdhdb.ub4spare[40]:                  0 ; 0x18c: 0x00000000
kfdhdb.ub4spare[41]:                  0 ; 0x190: 0x00000000
kfdhdb.ub4spare[42]:                  0 ; 0x194: 0x00000000
kfdhdb.ub4spare[43]:                  0 ; 0x198: 0x00000000
kfdhdb.ub4spare[44]:                  0 ; 0x19c: 0x00000000
kfdhdb.ub4spare[45]:                  0 ; 0x1a0: 0x00000000
kfdhdb.ub4spare[46]:                  0 ; 0x1a4: 0x00000000
kfdhdb.ub4spare[47]:                  0 ; 0x1a8: 0x00000000
kfdhdb.ub4spare[48]:                  0 ; 0x1ac: 0x00000000
kfdhdb.ub4spare[49]:                  0 ; 0x1b0: 0x00000000
kfdhdb.ub4spare[50]:                  0 ; 0x1b4: 0x00000000
kfdhdb.ub4spare[51]:                  0 ; 0x1b8: 0x00000000
kfdhdb.ub4spare[52]:                  0 ; 0x1bc: 0x00000000
kfdhdb.ub4spare[53]:                  0 ; 0x1c0: 0x00000000
kfdhdb.ub4spare[54]:                  0 ; 0x1c4: 0x00000000
kfdhdb.ub4spare[55]:                  0 ; 0x1c8: 0x00000000
kfdhdb.ub4spare[56]:                  0 ; 0x1cc: 0x00000000
kfdhdb.ub4spare[57]:                  0 ; 0x1d0: 0x00000000
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000

Thanks,
Fergal

--
http://www.freelists.org/webpage/oracle-l



NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. If received in error, please go to www.td.com/legal for instructions.
AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. Si reçu par erreur, prière d'aller au www.td.com/francais/avis_juridique pour des instructions.
--
http://www.freelists.org/webpage/oracle-l
Received on Wed Jan 02 2013 - 19:12:19 CET

Original text of this message