ORA-15042: ASM disk "xx" is missing

From: Fergal Taheny <ftaheny_at_gmail.com>
Date: Wed, 2 Jan 2013 17:45:18 +0000
Message-ID: <CAOuMUT4Y1tBQjfSEphH5USebwymCGdfHwLw9VkMJKWhCfgbB4Q_at_mail.gmail.com>

Hi,
ASM 11.1.0.7.0
Solaris 10

We have a diskgoup that won't mount. I'm believe it a SAN/disk issue as opposed to an ASM issue but I'm trying to eliminate some guesswork in troubleshooting this.

In the +ASM alert log we have:

NOTE: cache registered group ORADATA number=1 incarn=0x2bc38c91 NOTE: cache began mount (first) of group ORADATA number=1 incarn=0x2bc38c91 Wed Jan 02 14:16:33 2013

NOTE: Assigning number (1,15) to disk (/dev/oradsk/ASMDA7)
NOTE: Assigning number (1,10) to disk (/dev/oradsk/ASMDA2)
NOTE: Assigning number (1,18) to disk (/dev/oradsk/ASMDA10)
NOTE: Assigning number (1,9) to disk (/dev/oradsk/ASMDA1)
NOTE: Assigning number (1,19) to disk (/dev/oradsk/ASMDA11)
NOTE: Assigning number (1,20) to disk (/dev/oradsk/ASMDA12)
NOTE: Assigning number (1,21) to disk (/dev/oradsk/ASMDA13)
NOTE: Assigning number (1,22) to disk (/dev/oradsk/ASMDA14)
NOTE: Assigning number (1,23) to disk (/dev/oradsk/ASMDA15)
NOTE: Assigning number (1,24) to disk (/dev/oradsk/ASMDA16)
NOTE: Assigning number (1,25) to disk (/dev/oradsk/ASMDA17)
NOTE: Assigning number (1,26) to disk (/dev/oradsk/ASMDA18)
NOTE: Assigning number (1,12) to disk (/dev/oradsk/ASMDA4)

Wed Jan 02 14:16:37 2013
NOTE: start heartbeating (grp 1)
kfdp_query(): 12
kfdp_queryBg(): 12

NOTE: Assigning number (1,11) to disk ()
NOTE: Assigning number (1,13) to disk ()
NOTE: Assigning number (1,14) to disk ()
NOTE: Assigning number (1,16) to disk ()
NOTE: Assigning number (1,17) to disk ()

kfdp_query(): 13
kfdp_queryBg(): 13

NOTE: cache dismounting group 1/0x2BC38C91 (ORADATA)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x2BC38C91 (ORADATA)
NOTE: cache ending mount (fail) of group ORADATA number=1 incarn=0x2bc38c91

kfdp_dismount(): 14
kfdp_dismountBg(): 14

NOTE: De-assigning number (1,9) from disk (/dev/oradsk/ASMDA1)
NOTE: De-assigning number (1,10) from disk (/dev/oradsk/ASMDA2)
NOTE: De-assigning number (1,12) from disk (/dev/oradsk/ASMDA4)
NOTE: De-assigning number (1,15) from disk (/dev/oradsk/ASMDA7)
NOTE: De-assigning number (1,18) from disk (/dev/oradsk/ASMDA10)
NOTE: De-assigning number (1,19) from disk (/dev/oradsk/ASMDA11)
NOTE: De-assigning number (1,20) from disk (/dev/oradsk/ASMDA12)
NOTE: De-assigning number (1,21) from disk (/dev/oradsk/ASMDA13)
NOTE: De-assigning number (1,22) from disk (/dev/oradsk/ASMDA14)
NOTE: De-assigning number (1,23) from disk (/dev/oradsk/ASMDA15)
NOTE: De-assigning number (1,24) from disk (/dev/oradsk/ASMDA16)
NOTE: De-assigning number (1,25) from disk (/dev/oradsk/ASMDA17)
NOTE: De-assigning number (1,26) from disk (/dev/oradsk/ASMDA18)

ERROR: diskgroup ORADATA was not mounted

ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "17" is missing
ORA-15042: ASM disk "16" is missing
ORA-15042: ASM disk "14" is missing
ORA-15042: ASM disk "13" is missing
ORA-15042: ASM disk "11" is missing

ERROR: alter diskgroup oradata mount

So ASM identified 13 disks and couldn't find 5 disks.

The first problem is how to map these missing ASM disks to unix device files.

Looking in /dev/oradsk I find device files for the 13 good disks listed above and also 5 other device files not listed above. I assume these are the device files for the 5 problem disks but I want to prove that.

When I use the od command I can read the file headers for the 13 good disks:

> od -c -N 128 /dev/oradsk/ASMDA1

0000000  \0 202 001 001  \0  \0  \0  \0 200  \0  \0  \t 326 331 223 236
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040   O   R   C   L   D   I   S   K  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000100  \n 020  \0  \0  \0  \t 001 003   O   R   A   D   A   T   A   _
0000120   0   0   0   9  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

0000140 \0 \0 \0 \0 \0 \0 \0 \0 O R A D A T A \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200

And I get an IO error for the other 5 device files

> od -c -N 128 /dev/oradsk/ASMDA3
od: cannot open ASMDA3: I/O error

Ok still promising but still guessing a bit.

So I look in v$asm_disk and I only see the 13 good disks (same for v$asm_disk_stat and X$KFDSK)

sys_at_+ASM> select path from v$asm_disk;

PATH

---

/dev/oradsk/ASMDA1
/dev/oradsk/ASMDA2
/dev/oradsk/ASMDA4
/dev/oradsk/ASMDA7
/dev/oradsk/ASMDA10
/dev/oradsk/ASMDA11
/dev/oradsk/ASMDA18
/dev/oradsk/ASMDA13
/dev/oradsk/ASMDA14
/dev/oradsk/ASMDA15
/dev/oradsk/ASMDA16
/dev/oradsk/ASMDA17
/dev/oradsk/ASMDA12

13 rows selected.

Eventually I go back through old alert logs to find the last time the diskgroup was successfully mounted and I find that indeed the five disks I suspected were mounted in the disk group at that time. So that's fairly conclusive but my questions are:

Is there a view in ASM that lists all the disks that belong in a disk group? Including ones asm can't see at present.

Or is this information stored in the file headers? If so how could I read it?

Or does ASM just probe all the disks (as defined in asm_diskstring) to find all the disks in a disk group? And if it does how does it know how many disks to expect? I don't see the "numbers of disks" displayed anywhere in V$ASK_DISKGROUP or in the output of kfed (shown below). Where would I find the number of disks in each disk group?

> kfed read /dev/oradsk/ASMDA1

kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483657 ; 0x008: TYPE=0x8 NUMB=0x9
kfbh.check:                  3604583326 ; 0x00c: 0xd6d9939e
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                168820736 ; 0x020: 0x0a100000
kfdhdb.dsknum:                        9 ; 0x024: 0x0009
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:            ORADATA_0009 ; 0x028: length=12
kfdhdb.grpname:                 ORADATA ; 0x048: length=7
kfdhdb.fgname:             ORADATA_0009 ; 0x068: length=12
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32975350 ; 0x0a8: HOUR=0x16 DAYS=0xf
MNTH=0xa YEAR=0x7dc
kfdhdb.crestmp.lo:           1831326720 ; 0x0ac: USEC=0x0 MSEC=0x1f5
SECS=0x12 MINS=0x1b
kfdhdb.mntstmp.hi:             32975859 ; 0x0b0: HOUR=0x13 DAYS=0x1f
MNTH=0xa YEAR=0x7dc
kfdhdb.mntstmp.lo:           3870520320 ; 0x0b4: USEC=0x0 MSEC=0xdd
SECS=0x2b MINS=0x39
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                  204760 ; 0x0c4: 0x00031fd8
kfdhdb.pmcnt:                         3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      0 ; 0x0d4: 0x00000000
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000
kfdhdb.dbcompat:              168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi:             32942731 ; 0x0e4: HOUR=0xb DAYS=0x14
MNTH=0xa YEAR=0x7da
kfdhdb.grpstmp.lo:           3678353408 ; 0x0e8: USEC=0x0 MSEC=0x3ce
SECS=0x33 MINS=0x36
kfdhdb.ub4spare[0]:                   0 ; 0x0ec: 0x00000000
kfdhdb.ub4spare[1]:                   0 ; 0x0f0: 0x00000000
kfdhdb.ub4spare[2]:                   0 ; 0x0f4: 0x00000000
kfdhdb.ub4spare[3]:                   0 ; 0x0f8: 0x00000000
kfdhdb.ub4spare[4]:                   0 ; 0x0fc: 0x00000000
kfdhdb.ub4spare[5]:                   0 ; 0x100: 0x00000000
kfdhdb.ub4spare[6]:                   0 ; 0x104: 0x00000000
kfdhdb.ub4spare[7]:                   0 ; 0x108: 0x00000000
kfdhdb.ub4spare[8]:                   0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[9]:                   0 ; 0x110: 0x00000000
kfdhdb.ub4spare[10]:                  0 ; 0x114: 0x00000000
kfdhdb.ub4spare[11]:                  0 ; 0x118: 0x00000000
kfdhdb.ub4spare[12]:                  0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[13]:                  0 ; 0x120: 0x00000000
kfdhdb.ub4spare[14]:                  0 ; 0x124: 0x00000000
kfdhdb.ub4spare[15]:                  0 ; 0x128: 0x00000000
kfdhdb.ub4spare[16]:                  0 ; 0x12c: 0x00000000
kfdhdb.ub4spare[17]:                  0 ; 0x130: 0x00000000
kfdhdb.ub4spare[18]:                  0 ; 0x134: 0x00000000
kfdhdb.ub4spare[19]:                  0 ; 0x138: 0x00000000
kfdhdb.ub4spare[20]:                  0 ; 0x13c: 0x00000000
kfdhdb.ub4spare[21]:                  0 ; 0x140: 0x00000000
kfdhdb.ub4spare[22]:                  0 ; 0x144: 0x00000000
kfdhdb.ub4spare[23]:                  0 ; 0x148: 0x00000000
kfdhdb.ub4spare[24]:                  0 ; 0x14c: 0x00000000
kfdhdb.ub4spare[25]:                  0 ; 0x150: 0x00000000
kfdhdb.ub4spare[26]:                  0 ; 0x154: 0x00000000
kfdhdb.ub4spare[27]:                  0 ; 0x158: 0x00000000
kfdhdb.ub4spare[28]:                  0 ; 0x15c: 0x00000000
kfdhdb.ub4spare[29]:                  0 ; 0x160: 0x00000000
kfdhdb.ub4spare[30]:                  0 ; 0x164: 0x00000000
kfdhdb.ub4spare[31]:                  0 ; 0x168: 0x00000000
kfdhdb.ub4spare[32]:                  0 ; 0x16c: 0x00000000
kfdhdb.ub4spare[33]:                  0 ; 0x170: 0x00000000
kfdhdb.ub4spare[34]:                  0 ; 0x174: 0x00000000
kfdhdb.ub4spare[35]:                  0 ; 0x178: 0x00000000
kfdhdb.ub4spare[36]:                  0 ; 0x17c: 0x00000000
kfdhdb.ub4spare[37]:                  0 ; 0x180: 0x00000000
kfdhdb.ub4spare[38]:                  0 ; 0x184: 0x00000000
kfdhdb.ub4spare[39]:                  0 ; 0x188: 0x00000000
kfdhdb.ub4spare[40]:                  0 ; 0x18c: 0x00000000
kfdhdb.ub4spare[41]:                  0 ; 0x190: 0x00000000
kfdhdb.ub4spare[42]:                  0 ; 0x194: 0x00000000
kfdhdb.ub4spare[43]:                  0 ; 0x198: 0x00000000
kfdhdb.ub4spare[44]:                  0 ; 0x19c: 0x00000000
kfdhdb.ub4spare[45]:                  0 ; 0x1a0: 0x00000000
kfdhdb.ub4spare[46]:                  0 ; 0x1a4: 0x00000000
kfdhdb.ub4spare[47]:                  0 ; 0x1a8: 0x00000000
kfdhdb.ub4spare[48]:                  0 ; 0x1ac: 0x00000000
kfdhdb.ub4spare[49]:                  0 ; 0x1b0: 0x00000000
kfdhdb.ub4spare[50]:                  0 ; 0x1b4: 0x00000000
kfdhdb.ub4spare[51]:                  0 ; 0x1b8: 0x00000000
kfdhdb.ub4spare[52]:                  0 ; 0x1bc: 0x00000000
kfdhdb.ub4spare[53]:                  0 ; 0x1c0: 0x00000000
kfdhdb.ub4spare[54]:                  0 ; 0x1c4: 0x00000000
kfdhdb.ub4spare[55]:                  0 ; 0x1c8: 0x00000000
kfdhdb.ub4spare[56]:                  0 ; 0x1cc: 0x00000000
kfdhdb.ub4spare[57]:                  0 ; 0x1d0: 0x00000000
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000

Thanks,
Fergal

--
http://www.freelists.org/webpage/oracle-l

Received on Wed Jan 02 2013 - 18:45:18 CET