Re: ASM disk path in exadata

From: Gaurav Mehta <gaurav_mehta77_at_hotmail.com>
Date: Wed, 22 Feb 2012 23:10:54 +0530
Message-ID: <SNT111-DS838547B933513D0268929F3640_at_phx.gbl>

Thanks.

-----Original Message-----
From: Andy Colvin
Sent: Tuesday, February 21, 2012 7:16 PM To: frits.hoogland_at_gmail.com
Cc: oracle-l
Subject: Re: ASM disk path in exadata

Right…the ASM disks on Exadata do not exist on the database servers as physical devices. It's not like a typical system that uses block or raw devices for the ASM disks. They're connected using the iDB protocol over RDS or UDP.
[enkdb01:root] /root
> fdisk -l

Disk /dev/sda: 291.9 GB, 291999055872 bytes 255 heads, 63 sectors/track, 35500 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        7832    62910508+  83  Linux
/dev/sda2            7833        9920    16771860   82  Linux swap / Solaris
/dev/sda3            9921       35500   205471350   83  Linux

[enkdb01:root] /root
> kfod disks=all

Disk          Size Path                                     User     Group
================================================================================
   1:    1501184 Mb o/192.168.12.3/DATA_CD_00_enkcel01       <unknown> 


<unknown>

   2:    1501184 Mb o/192.168.12.3/DATA_CD_01_enkcel01       <unknown> 


<unknown>

   3:    1501184 Mb o/192.168.12.3/DATA_CD_02_enkcel01       <unknown> 


<unknown>

   4:    1501184 Mb o/192.168.12.3/DATA_CD_03_enkcel01       <unknown> 


<unknown>

   5:    1501184 Mb o/192.168.12.3/DATA_CD_04_enkcel01       <unknown> 


<unknown>

   6:    1501184 Mb o/192.168.12.3/DATA_CD_05_enkcel01       <unknown> 


<unknown>

   7:    1501184 Mb o/192.168.12.3/DATA_CD_06_enkcel01       <unknown> 


<unknown>

   8:    1501184 Mb o/192.168.12.3/DATA_CD_07_enkcel01       <unknown> 


<unknown>

   9:    1501184 Mb o/192.168.12.3/DATA_CD_08_enkcel01       <unknown> 


<unknown>

  10:    1501184 Mb o/192.168.12.3/DATA_CD_09_enkcel01       <unknown> 


<unknown>

  11:    1501184 Mb o/192.168.12.3/DATA_CD_10_enkcel01       <unknown> 


<unknown>

  12:    1501184 Mb o/192.168.12.3/DATA_CD_11_enkcel01       <unknown> 


<unknown>

  13:      29808 Mb o/192.168.12.3/DBFS_DG_CD_02_enkcel01    <unknown> 


<unknown>

  14:      29808 Mb o/192.168.12.3/DBFS_DG_CD_03_enkcel01    <unknown> 


<unknown>

  15:      29808 Mb o/192.168.12.3/DBFS_DG_CD_04_enkcel01    <unknown> 


<unknown>

  16:      29808 Mb o/192.168.12.3/DBFS_DG_CD_05_enkcel01    <unknown> 


<unknown>

  17:      29808 Mb o/192.168.12.3/DBFS_DG_CD_06_enkcel01    <unknown> 


<unknown>

  18:      29808 Mb o/192.168.12.3/DBFS_DG_CD_07_enkcel01    <unknown> 


<unknown>

  19:      29808 Mb o/192.168.12.3/DBFS_DG_CD_08_enkcel01    <unknown> 


<unknown>

  20:      29808 Mb o/192.168.12.3/DBFS_DG_CD_09_enkcel01    <unknown> 


<unknown>

  21:      29808 Mb o/192.168.12.3/DBFS_DG_CD_10_enkcel01    <unknown> 


<unknown>

  22:      29808 Mb o/192.168.12.3/DBFS_DG_CD_11_enkcel01    <unknown> 


<unknown>

  23:     375344 Mb o/192.168.12.3/RECO_CD_00_enkcel01       <unknown> 


<unknown>

  24:     375344 Mb o/192.168.12.3/RECO_CD_01_enkcel01       <unknown> 


<unknown>

  25:     375344 Mb o/192.168.12.3/RECO_CD_02_enkcel01       <unknown> 


<unknown>

  26:     375344 Mb o/192.168.12.3/RECO_CD_03_enkcel01       <unknown> 


<unknown>

  27:     375344 Mb o/192.168.12.3/RECO_CD_04_enkcel01       <unknown> 


<unknown>

  28:     375344 Mb o/192.168.12.3/RECO_CD_05_enkcel01       <unknown> 


<unknown>

  29:     375344 Mb o/192.168.12.3/RECO_CD_06_enkcel01       <unknown> 


<unknown>

  30:     375344 Mb o/192.168.12.3/RECO_CD_07_enkcel01       <unknown> 


<unknown>

  31:     375344 Mb o/192.168.12.3/RECO_CD_08_enkcel01       <unknown> 


<unknown>

  32:     375344 Mb o/192.168.12.3/RECO_CD_09_enkcel01       <unknown> 


<unknown>

  33:     375344 Mb o/192.168.12.3/RECO_CD_10_enkcel01       <unknown> 


<unknown>

  34:     375344 Mb o/192.168.12.3/RECO_CD_11_enkcel01       <unknown> 


<unknown>

  35:    1501184 Mb o/192.168.12.4/DATA_CD_00_enkcel02       <unknown> 


<unknown>

  36:    1501184 Mb o/192.168.12.4/DATA_CD_01_enkcel02       <unknown> 


<unknown>

  37:    1501184 Mb o/192.168.12.4/DATA_CD_02_enkcel02       <unknown> 


<unknown>

  38:    1501184 Mb o/192.168.12.4/DATA_CD_03_enkcel02       <unknown> 


<unknown>

  39:    1501184 Mb o/192.168.12.4/DATA_CD_04_enkcel02       <unknown> 


<unknown>

  40:    1501184 Mb o/192.168.12.4/DATA_CD_05_enkcel02       <unknown> 


<unknown>

  41:    1501184 Mb o/192.168.12.4/DATA_CD_06_enkcel02       <unknown> 


<unknown>

  42:    1501184 Mb o/192.168.12.4/DATA_CD_07_enkcel02       <unknown> 


<unknown>

  43:    1501184 Mb o/192.168.12.4/DATA_CD_08_enkcel02       <unknown> 


<unknown>

  44:    1501184 Mb o/192.168.12.4/DATA_CD_09_enkcel02       <unknown> 


<unknown>

  45:    1501184 Mb o/192.168.12.4/DATA_CD_10_enkcel02       <unknown> 


<unknown>

  46:    1501184 Mb o/192.168.12.4/DATA_CD_11_enkcel02       <unknown> 


<unknown>

  47:      29808 Mb o/192.168.12.4/DBFS_DG_CD_02_enkcel02    <unknown> 


<unknown>

  48:      29808 Mb o/192.168.12.4/DBFS_DG_CD_03_enkcel02    <unknown> 


<unknown>

  49:      29808 Mb o/192.168.12.4/DBFS_DG_CD_04_enkcel02    <unknown> 


<unknown>

  50:      29808 Mb o/192.168.12.4/DBFS_DG_CD_05_enkcel02    <unknown> 


<unknown>

  51:      29808 Mb o/192.168.12.4/DBFS_DG_CD_06_enkcel02    <unknown> 


<unknown>

  52:      29808 Mb o/192.168.12.4/DBFS_DG_CD_07_enkcel02    <unknown> 


<unknown>

  53:      29808 Mb o/192.168.12.4/DBFS_DG_CD_08_enkcel02    <unknown> 


<unknown>

  54:      29808 Mb o/192.168.12.4/DBFS_DG_CD_09_enkcel02    <unknown> 


<unknown>

  55:      29808 Mb o/192.168.12.4/DBFS_DG_CD_10_enkcel02    <unknown> 


<unknown>

  56:      29808 Mb o/192.168.12.4/DBFS_DG_CD_11_enkcel02    <unknown> 


<unknown>

  57:     375344 Mb o/192.168.12.4/RECO_CD_00_enkcel02       <unknown> 


<unknown>

  58:     375344 Mb o/192.168.12.4/RECO_CD_01_enkcel02       <unknown> 


<unknown>

  59:     375344 Mb o/192.168.12.4/RECO_CD_02_enkcel02       <unknown> 


<unknown>

  60:     375344 Mb o/192.168.12.4/RECO_CD_03_enkcel02       <unknown> 


<unknown>

  61:     375344 Mb o/192.168.12.4/RECO_CD_04_enkcel02       <unknown> 


<unknown>

  62:     375344 Mb o/192.168.12.4/RECO_CD_05_enkcel02       <unknown> 


<unknown>

  63:     375344 Mb o/192.168.12.4/RECO_CD_06_enkcel02       <unknown> 


<unknown>

  64:     375344 Mb o/192.168.12.4/RECO_CD_07_enkcel02       <unknown> 


<unknown>

  65:     375344 Mb o/192.168.12.4/RECO_CD_08_enkcel02       <unknown> 


<unknown>

  66:     375344 Mb o/192.168.12.4/RECO_CD_09_enkcel02       <unknown> 


<unknown>

  67:     375344 Mb o/192.168.12.4/RECO_CD_10_enkcel02       <unknown> 


<unknown>

  68:     375344 Mb o/192.168.12.4/RECO_CD_11_enkcel02       <unknown> 


<unknown>

  69:    1501184 Mb o/192.168.12.5/DATA_CD_00_enkcel03       <unknown> 


<unknown>

  70:    1501184 Mb o/192.168.12.5/DATA_CD_01_enkcel03       <unknown> 


<unknown>

  71:    1501184 Mb o/192.168.12.5/DATA_CD_02_enkcel03       <unknown> 


<unknown>

  72:    1501184 Mb o/192.168.12.5/DATA_CD_03_enkcel03       <unknown> 


<unknown>

  73:    1501184 Mb o/192.168.12.5/DATA_CD_04_enkcel03       <unknown> 


<unknown>

  74:    1501184 Mb o/192.168.12.5/DATA_CD_05_enkcel03       <unknown> 


<unknown>

  75:    1501184 Mb o/192.168.12.5/DATA_CD_06_enkcel03       <unknown> 


<unknown>

  76:    1501184 Mb o/192.168.12.5/DATA_CD_07_enkcel03       <unknown> 


<unknown>

  77:    1501184 Mb o/192.168.12.5/DATA_CD_08_enkcel03       <unknown> 


<unknown>

  78:    1501184 Mb o/192.168.12.5/DATA_CD_09_enkcel03       <unknown> 


<unknown>

  79:    1501184 Mb o/192.168.12.5/DATA_CD_10_enkcel03       <unknown> 


<unknown>

  80:    1501184 Mb o/192.168.12.5/DATA_CD_11_enkcel03       <unknown> 


<unknown>

  81:      29808 Mb o/192.168.12.5/DBFS_DG_CD_02_enkcel03    <unknown> 


<unknown>

  82:      29808 Mb o/192.168.12.5/DBFS_DG_CD_03_enkcel03    <unknown> 


<unknown>

  83:      29808 Mb o/192.168.12.5/DBFS_DG_CD_04_enkcel03    <unknown> 


<unknown>

  84:      29808 Mb o/192.168.12.5/DBFS_DG_CD_05_enkcel03    <unknown> 


<unknown>

  85:      29808 Mb o/192.168.12.5/DBFS_DG_CD_06_enkcel03    <unknown> 


<unknown>

  86:      29808 Mb o/192.168.12.5/DBFS_DG_CD_07_enkcel03    <unknown> 


<unknown>

  87:      29808 Mb o/192.168.12.5/DBFS_DG_CD_08_enkcel03    <unknown> 


<unknown>

  88:      29808 Mb o/192.168.12.5/DBFS_DG_CD_09_enkcel03    <unknown> 


<unknown>

  89:      29808 Mb o/192.168.12.5/DBFS_DG_CD_10_enkcel03    <unknown> 


<unknown>

  90:      29808 Mb o/192.168.12.5/DBFS_DG_CD_11_enkcel03    <unknown> 


<unknown>

  91:     375344 Mb o/192.168.12.5/RECO_CD_00_enkcel03       <unknown> 


<unknown>

  92:     375344 Mb o/192.168.12.5/RECO_CD_01_enkcel03       <unknown> 


<unknown>

  93:     375344 Mb o/192.168.12.5/RECO_CD_02_enkcel03       <unknown> 


<unknown>

  94:     375344 Mb o/192.168.12.5/RECO_CD_03_enkcel03       <unknown> 


<unknown>

  95:     375344 Mb o/192.168.12.5/RECO_CD_04_enkcel03       <unknown> 


<unknown>

  96:     375344 Mb o/192.168.12.5/RECO_CD_05_enkcel03       <unknown> 


<unknown>

  97:     375344 Mb o/192.168.12.5/RECO_CD_06_enkcel03       <unknown> 


<unknown>

  98:     375344 Mb o/192.168.12.5/RECO_CD_07_enkcel03       <unknown> 


<unknown>

  99:     375344 Mb o/192.168.12.5/RECO_CD_08_enkcel03       <unknown> 


<unknown>

100:     375344 Mb o/192.168.12.5/RECO_CD_09_enkcel03       <unknown> 


<unknown>

101:     375344 Mb o/192.168.12.5/RECO_CD_10_enkcel03       <unknown> 


<unknown>

102:     375344 Mb o/192.168.12.5/RECO_CD_11_enkcel03       <unknown>

<unknown>

ORACLE_SID ORACLE_HOME

     +ASM1 /u01/app/11.2.0.3/grid
     +ASM2 /u01/app/11.2.0.3/grid

Andy Colvin

Principal Consultant
Enkitec
andy.colvin_at_enkitec.com
http://blog.oracle-ninja.com

On Feb 21, 2012, at 6:07 AM, Frits Hoogland wrote:

> Content-Transfer-Encoding: quoted-printable
> Content-Type: text/plain;
> charset=us-ascii
>
> There is no udev/path translation on the database/computing layer in > 
> exadata.
>
> During startup, ASM first needs to find the disks. It does that by > 
> picking up the cellserver information (addresses) in > 
> /etc/oracle/cell/network-config/cellip.ora
> Because the discovery string is set to "o/*/*", it picks up all > 
> griddisks available on the cellservers defined in cellip.ora
>
> Because of the entirely different path, ASM knows it needs to > 
> communicate over the network to get to the griddisks, instead of using a > 
> locally attached disk. This combination is RDS (infiniband) on Exadata, > 
> but can be UDP (ethernet).
>
> In fact, the discovery can manually be done by using kfod:
> kfod asm_diskstring="o/*/*", disks=all
> this is what the asmca uses behind the GUI to discover disks, also on > 
> normal platforms.
> When specifying the o/*/* diskstring, kfod too reads > 
> /etc/oracle/cell/network-config/cellip.ora to know where to look.
>
>
> Frits Hoogland
>
> http://fritshoogland.wordpress.com
> mailto:frits.hoogland_at_gmail.com
> cell: +31 6 53569942
>
> On Feb 21, 2012, at 10:32 AM, gaurav mehta wrote:
>
>> I have been trying to figure out how exadata griddisks are presented > as 
>> candidate asm disks.  We know that the disk discovery path for asm > 
>> instances using exadata is "o/<cell_ip_address>*/<griddisk_name>*".  I > 
>> was wondering what actual path / udev in the operating system on the db > 
>> machine does this translate to ?.
>> --
>> http://www.freelists.org/webpage/oracle-l
>>
>>
>
>
>
>
> --
> http://www.freelists.org/webpage/oracle-l
>
>

--
http://www.freelists.org/webpage/oracle-l


--
http://www.freelists.org/webpage/oracle-l

Received on Wed Feb 22 2012 - 11:40:54 CST

This message: [ Message body ]
Next message: Timo Raitalaakso: "Re: Flashback Schema ... any ideas"
Previous message: April Sims: "Re: Flashback Schema ... any ideas"
In reply to: Andy Colvin: "Re: ASM disk path in exadata"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

Original text of this message