Re: Database Will Not Mount: ORA-00205

From: David Barbour <david.barbour1_at_gmail.com>
Date: Sun, 11 Feb 2018 15:11:14 -0600
Message-ID: <CAFH+iffMBhwS4ocGouuFGhgfRF7WsDka0iBhkMwYocTGE01C5g_at_mail.gmail.com>



Not that I could see. I looked at the trace files and fed in some messages (as well as trying Google) but didn't find anything I thought might be useful.

What the asmb trace file shows is:

But these ARE all the disks.

The disk permissions look good:

[root_at_node1 mapper]# ls -al DATA*

lrwxrwxrwx. 1 root root 8 Feb 11 14:31 DATA01 -> ../dm-15
lrwxrwxrwx. 1 root root 8 Feb 11 14:39 DATA02 -> ../dm-11
lrwxrwxrwx. 1 root root 8 Feb 11 14:31 DATA03 -> ../dm-19
lrwxrwxrwx. 1 root root 7 Feb 11 14:31 DATA04 -> ../dm-7
[root_at_node1 mapper]# ls -al ../dm-15
brw-rw----. 1 grid asmadmin 253, 15 Feb 11 14:51 ../dm-15 [root_at_node1 mapper]# ls -al ../dm-11
brw-rw----. 1 grid asmadmin 253, 11 Feb 11 14:39 ../dm-11 [root_at_node1 mapper]# ls -al ../dm-19
brw-rw----. 1 grid asmadmin 253, 19 Feb 11 14:31 ../dm-19 [root_at_node1 mapper]# ls -al ../dm-7
brw-rw----. 1 grid asmadmin 253, 7 Feb 11 14:31 ../dm-7

I've stopped and started multipathd to no effect.

On Sun, Feb 11, 2018 at 2:15 PM, Andrew Kerber <andrew.kerber_at_gmail.com> wrote:

> Did MOS have any suggestions?
>
> On Sun, Feb 11, 2018 at 10:38 AM David Barbour <david.barbour1_at_gmail.com>
> wrote:
>
>> Both DBs have (some version?) of the patch installed. The one that
>> doesn't mount (orcl) has the recommended patchset bundle installed - "DATABASE
>> BUNDLE PATCH: 12.1.0.2.170418 (25397136)" . The one that does mount
>> (PRDREP) indicates the patchset is the critical patch update with the same
>> identifier - "OCW Patch Set Update : 12.1.0.2.170418 (25363740)"
>>
>> orcl1_at_node1:/u01/app/oracle/product/12.1.0/dbhome_1 $ opatch lsinventory
>> Oracle Interim Patch Installer version 12.2.0.1.8
>> Copyright (c) 2018, Oracle Corporation. All rights reserved.
>>
>>
>> Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
>> Central Inventory : /u01/app/oraInventory
>> from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
>> OPatch version : 12.2.0.1.8
>> OUI version : 12.1.0.2.0
>> Log file location : /u01/app/oracle/product/12.1.
>> 0/dbhome_1/cfgtoollogs/opatch/o
>> patch2018-02-11_10-28-28AM_1.log
>>
>> Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgto
>>
>> ollogs/opatch/lsinv/lsinventory2018-02-11_10-28-28AM.txt
>>
>> ------------------------------------------------------------
>> --------------------
>> Local Machine Information::
>> Hostname: node1.istation.com
>> ARU platform id: 226
>> ARU platform description:: Linux x86-64
>>
>> Installed Top-level Products (1):
>>
>> Oracle Database 12c
>> 12.1.0.2.0
>> There are 1 products installed in this Oracle Home.
>>
>>
>> Interim patches (2) :
>>
>> Patch 25397136 : applied on Fri Feb 09 17:51:30 CST 2018
>> Unique Patch ID: 21145057
>> Patch description: "DATABASE BUNDLE PATCH: 12.1.0.2.170418 (25397136)"
>>
>> PRDREP1_at_node1:/u01/app/oracle/product/12.1.0/dbhome_2 $ opatch
>> lsinventory
>> Oracle Interim Patch Installer version 12.2.0.1.8
>> Copyright (c) 2018, Oracle Corporation. All rights reserved.
>>
>>
>> Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_2
>> Central Inventory : /u01/app/oraInventory
>> from : /u01/app/oracle/product/12.1.0/dbhome_2/oraInst.loc
>> OPatch version : 12.2.0.1.8
>> OUI version : 12.1.0.2.0
>> Log file location : /u01/app/oracle/product/12.1.
>> 0/dbhome_2/cfgtoollogs/opatch/opatch2018-02-11_10-30-25AM_1.log
>>
>> Lsinventory Output file location : /u01/app/oracle/product/12.1.
>> 0/dbhome_2/cfgtoollogs/opatch/lsinv/lsinventory2018-02-11_10-30-25AM.txt
>>
>> ------------------------------------------------------------
>> --------------------
>> Local Machine Information::
>> Hostname: node1.istation.com
>> ARU platform id: 226
>> ARU platform description:: Linux x86-64
>>
>> Installed Top-level Products (1):
>>
>> Oracle Database 12c
>> 12.1.0.2.0
>> There are 1 products installed in this Oracle Home.
>>
>>
>> Interim patches (2) :
>>
>> Patch 25363740 : applied on Mon Jul 10 11:35:49 CDT 2017
>> Unique Patch ID: 20990699
>> Patch description: "OCW Patch Set Update : 12.1.0.2.170418 (25363740)"
>>
>>
>>
>> On Sun, Feb 11, 2018 at 9:45 AM, Chris Taylor <
>> christopherdtaylor1994_at_gmail.com> wrote:
>>
>>> David,
>>>
>>> Does this DB run out of a different DB_HOME (not GI_HOME) versus the
>>> other db that is working?
>>>
>>> *IF* it runs out of a different DB_HOME, verify that patch is installed
>>> to that home - maybe that home is missing the patch you mentioned?
>>>
>>> + Apply 12.1.0.2.170117 DB BP or above on RDBMS_HOME ( Doc ID 2247077.1
>>> )
>>>
>>> Chris
>>>
>>>
>>> On Sat, Feb 10, 2018 at 9:38 PM, David Barbour <david.barbour1_at_gmail.com
>>> > wrote:
>>>
>>>> What is interesting is that we have another 12.1.0.2 database on this
>>>> cluster and it's not having any issues.
>>>>
>>>> The control files are on ASM:
>>>>
>>>> SQL> show parameter control_files;
>>>>
>>>> NAME TYPE VALUE
>>>> ------------------------------------ -----------
>>>> ------------------------------
>>>> control_files string
>>>> +DATA/ORCL/CONTROLFILE/current
>>>> .263.881061111,
>>>> +DATA/ORCL/CON
>>>> TROLFILE/current.270.
>>>> 881061111
>>>>
>>>> The database alert log shows the following:
>>>>
>>>> NOTE: ASMB mounting group 1 (DATA)
>>>> NOTE: ASM background process initiating disk discovery for grp 1
>>>> (reqid:0)
>>>> WARNING: group 1 (DATA) has missing disks
>>>> ORA-15040: diskgroup is incomplete
>>>> WARNING: group 1 is being dismounted.
>>>> WARNING: ASMB force dismounting group 1 (DATA) due to missing disks
>>>> SUCCESS: diskgroup DATA was dismounted
>>>>
>>>> The database shows the diskgroup and associated disks are mounted and
>>>> present:
>>>>
>>>> 1 select a.disk_number, a.name, b.name, b.state from v$asm_disk a
>>>> 2 join v$asm_diskgroup b on a.group_number=b.group_number
>>>> 3* order by b.name, a.disk_number
>>>> SQL> /
>>>>
>>>> DISK_NUMBER NAME NAME
>>>> STATE
>>>> ----------- ------------------------------
>>>> ------------------------------ -----------
>>>> 0 DATA_0000 DATA
>>>> MOUNTED
>>>> 1 DATA_0001 DATA
>>>> MOUNTED
>>>> 2 DATA_0002 DATA
>>>> MOUNTED
>>>> 3 DATA_0003 DATA
>>>> MOUNTED
>>>> 0 FRA_0000 FRA
>>>> MOUNTED
>>>> 1 FRA_0001 FRA
>>>> MOUNTED
>>>> 0 GRID_0000 GRID
>>>> MOUNTED
>>>> 1 GRID_0001 GRID
>>>> MOUNTED
>>>> 2 GRID_0002 GRID
>>>> MOUNTED
>>>> 3 GRID_0003 GRID
>>>> MOUNTED
>>>> 0 PRDREPDATA01_0000 PRDREPDATA01
>>>> MOUNTED
>>>>
>>>> DISK_NUMBER NAME NAME
>>>> STATE
>>>> ----------- ------------------------------
>>>> ------------------------------ -----------
>>>> 1 PRDREPDATA01_0001 PRDREPDATA01
>>>> MOUNTED
>>>> 2 PRDREPDATA01_0002 PRDREPDATA01
>>>> MOUNTED
>>>> 3 PRDREPDATA01_0003 PRDREPDATA01
>>>> MOUNTED
>>>> 0 PRDREPFRA_0000 PRDREPFRA
>>>> MOUNTED
>>>> 0 REDO1_0000 REDO1
>>>> MOUNTED
>>>> 0 REDO2_0000 REDO2
>>>> MOUNTED
>>>>
>>>> 17 rows selected.
>>>>
>>>> The ASM alert log shows:
>>>>
>>>> NOTE: client orcl1:orcl:scan01-int no longer has group 1 (DATA) mounted
>>>>
>>>> CRS shows:
>>>>
>>>> [grid_at_node1 ~]$ crsctl stat res -t
>>>> ------------------------------------------------------------
>>>> --------------------
>>>> Name Target State Server State
>>>> details
>>>> ------------------------------------------------------------
>>>> --------------------
>>>> Local Resources
>>>> ------------------------------------------------------------
>>>> --------------------
>>>> ora.ASMNET1LSNR_ASM.lsnr
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.DATA.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.FRA.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.GRID.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.LISTENER.lsnr
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.PRDREPDATA01.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.PRDREPFRA.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.REDO1.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.REDO2.dg
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.net1.network
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.ons
>>>> ONLINE ONLINE node1 STABLE
>>>> ora.proxy_advm
>>>> OFFLINE OFFLINE node1 STABLE
>>>> ------------------------------------------------------------
>>>> --------------------
>>>> Cluster Resources
>>>> ------------------------------------------------------------
>>>> --------------------
>>>> ora.LISTENER_SCAN1.lsnr
>>>> 1 ONLINE ONLINE node1 STABLE
>>>> ora.LISTENER_SCAN2.lsnr
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.LISTENER_SCAN3.lsnr
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.MGMTLSNR
>>>> 1 ONLINE ONLINE node1
>>>> 169.254.112.28 1.1.1
>>>> .1,STABLE
>>>> ora.asm
>>>> 1 ONLINE ONLINE node1
>>>> Started,STABLE
>>>> 2 ONLINE OFFLINE STABLE
>>>> ora.cvu
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.mgmtdb
>>>> 1 ONLINE ONLINE node1 Open,STABLE
>>>> ora.node1.vip
>>>> 1 ONLINE ONLINE node1 STABLE
>>>> ora.node2.vip
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.orcl.db
>>>> 1 ONLINE OFFLINE STABLE
>>>> 2 OFFLINE OFFLINE STABLE
>>>> ora.orcl.test.svc
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.prdrep.db
>>>> 1 ONLINE ONLINE node1
>>>> Open,HOME=/u01/app/o
>>>>
>>>> racle/product/12.1.0
>>>>
>>>> /dbhome_2,STABLE
>>>> 2 ONLINE OFFLINE STABLE
>>>> ora.prdrep.prdrepsvc.svc
>>>> 1 ONLINE ONLINE node1 STABLE
>>>> 2 ONLINE OFFLINE STABLE
>>>> ora.qosmserver
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.scan1.vip
>>>> 1 ONLINE ONLINE node1 STABLE
>>>> ora.scan2.vip
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ora.scan3.vip
>>>> 1 OFFLINE OFFLINE STABLE
>>>> ------------------------------------------------------------
>>>> --------------------
>>>>
>>>>
>>>>
>>>> On Sat, Feb 10, 2018 at 7:14 PM, Vishnu <vishnukumarmp_at_gmail.com>
>>>> wrote:
>>>>
>>>>> Hi David, please check whether the file system which has the database
>>>>> control file is mounted and able to read ? check your asm alert log to get
>>>>> additional info.
>>>>>
>>>>> $crsctl stat res -t
>>>>>
>>>>> Above command will provide online/offline status of all of your
>>>>> resources including asm disk groups.
>>>>>
>>>>> Thanks,
>>>>> Vishnu
>>>>>
>>>>> On Sat, Feb 10, 2018 at 1:38 PM, David Barbour <
>>>>> david.barbour1_at_gmail.com> wrote:
>>>>>
>>>>>> Good Afternoon,
>>>>>>
>>>>>> I have upgraded the GI to 12.2 and the 12.1 databases are failing to
>>>>>> start with the error:
>>>>>>
>>>>>> ORA-00205: error in identifying control file, check alert log for
>>>>>> more info
>>>>>>
>>>>>> This behavior is described in Doc ID 2247077.1 but the solution is
>>>>>> to apply 12.1.0.2.170117 DB BP or above on RDBMS_HOME.
>>>>>> Unfortunately this doesn't seem to be the answer in this case as the
>>>>>> database is already at 12.1.0.2.170418.
>>>>>>
>>>>>> CRS shows the following:
>>>>>>
>>>>>> ora.orcl.db
>>>>>> 1 ONLINE OFFLINE STABLE
>>>>>> 2 OFFLINE OFFLINE STABLE
>>>>>>
>>>>>> I can't seem to get this database to go offline:offline.
>>>>>>
>>>>>> Has anyone had a similar issue?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>> --
> Andrew W. Kerber
>
> 'If at first you dont succeed, dont take up skydiving.'
>

--
http://www.freelists.org/webpage/oracle-l
Received on Sun Feb 11 2018 - 22:11:14 CET

Original text of this message