Recovering lost OCR & Voting disk in 11gR2

Recovering Lost OCR & Voting disk in 11gR2
If we lost both OCR & voting disks, we can recover then using following steps. Actually it is pretty straight forward. We need to run root.sh (after  rootcrs.pl -deconfig) on all the nodes. New OCR & Voting will be created in the location specified in $ORA_CRS_HOME/crs/install/crsconfig_params. Just in case, we want to change the location, we can edit following variables

CRS_VOTING_DISKS=/ocrvote/vote/vote.dat
CRS_OCR_LOCATIONS=/ocrvote/ocr/ocr.dat

STEP 1 – Run $ORA_CRS_HOME/root.sh from Node 1
=======================================

# uname –a
SunOS node1.mydomain.com 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise

# /oragrid/11.2.0>./root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
    ORACLE_OWNER= oragrid
    ORACLE_HOME=  /oragrid/11.2.0

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file “dbhome” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file “oraenv” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file “coraenv” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-06-22 07:06:12: Parsing the host name
2010-06-22 07:06:12: Checking for super user privileges
2010-06-22 07:06:12: User has super user privileges
Using configuration parameter file: /oragrid/11.2.0/crs/install/crsconfig_params
User oragrid has the required capabilities to run CSSD in realtime mode
LOCAL ADD MODE
Creating OCR keys for user ‘root’, privgrp ‘system’..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘node1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘node1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘node1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘node1’
CRS-2676: Start of ‘ora.diskmon’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘node1’
CRS-2676: Start of ‘ora.ctssd’ on ‘node1’ succeeded
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘system’..
Operation successful.
CRS-2672: Attempting to start ‘ora.crsd’ on ‘node1’
CRS-2676: Start of ‘ora.crsd’ on ‘node1’ succeeded
Now formatting voting disk: /ocrvote/vote/votedisk.dat.
CRS-4603: Successful addition of voting disk /ocrvote/vote/votedisk.dat.
##  STATE    File Universal Id                File Name Disk group
—  —–    —————–                ——— ———
 1. ONLINE   3ffb7f321cef4f80bf9ed6a76d4f2937 (/ocrvote/vote/votedisk.dat) []
Located 1 voting disk(s).
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘node1’
CRS-2677: Stop of ‘ora.crsd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘node1’
CRS-2677: Stop of ‘ora.ctssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘node1’
CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘node1’
CRS-2677: Stop of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘node1’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘node1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘node1’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘node1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘node1’
CRS-2676: Start of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘node1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘node1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘node1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘node1’
CRS-2676: Start of ‘ora.diskmon’ on ‘node1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘node1’
CRS-2676: Start of ‘ora.ctssd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘node1’
CRS-2676: Start of ‘ora.crsd’ on ‘node1’ succeeded
CRS-2672: Attempting to start ‘ora.evmd’ on ‘node1’
CRS-2676: Start of ‘ora.evmd’ on ‘node1’ succeeded

node1     2010/06/22 07:13:07     /oragrid/11.2.0/cdata/node1/backup_20100622_071307.olr
Configure Oracle Grid Infrastructure for a Cluster … succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB.   Actual 4096 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oraInventory
‘UpdateNodeList’ was successful.

STEP 2 – Run $ORA_CRS_HOME/root.sh from Node 2
=======================================

# uname -a
SunOS node2.mydomain.com 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise
# /oragrid/11.2.0> ./root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
    ORACLE_OWNER= oragrid
    ORACLE_HOME=  /oragrid/11.2.0

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file “dbhome” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file “oraenv” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:
The file “coraenv” already exists in /usr/local/bin.  Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-06-22 07:24:38: Parsing the host name
2010-06-22 07:24:38: Checking for super user privileges
2010-06-22 07:24:38: User has super user privileges
Using configuration parameter file: /oragrid/11.2.0/crs/install/crsconfig_params
User oragrid has the required capabilities to run CSSD in realtime mode
LOCAL ADD MODE
Creating OCR keys for user ‘root’, privgrp ‘system’..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating
CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘node2’
CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘node2’ succeeded
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘node2’
CRS-2676: Start of ‘ora.mdnsd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘node2’
CRS-2676: Start of ‘ora.gipcd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘node2’
CRS-2676: Start of ‘ora.gpnpd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘node2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘node2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘node2’
CRS-2676: Start of ‘ora.diskmon’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘node2’
CRS-2676: Start of ‘ora.ctssd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘node2’
CRS-2676: Start of ‘ora.crsd’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.evmd’ on ‘node2’
CRS-2676: Start of ‘ora.evmd’ on ‘node2’ succeeded

node2     2010/06/22 07:28:02     /oragrid/11.2.0/cdata/node2/backup_20100622_072802.olr
Configure Oracle Grid Infrastructure for a Cluster … succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer…

Checking swap space: must be greater than 500 MB.   Actual 5760 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oraInventory

STEP 3 – Verify the OCR location
========================

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2272
         Available space (kbytes) :     259848
         ID                       : 1219733138
         Device/File Name         : /ocrvote/ocr/ocr.dat
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded

STEP 4 – Verify the local OCR location
===========================

# ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2184
         Available space (kbytes) :     259936
         ID                       :  104492352
         Device/File Name         : /oragrid/11.2/cdata/node1.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded
         Logical corruption check succeeded

STEP 5  – Verify the Voting disk location
=============================
# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
—  —–    —————–                ——— ———
 1. ONLINE   19780dfd0ce64f36bff2c56b29ef94b9 (/ocrvote/vote/vote.dat) []
Located 1 voting disk(s).

NOTE:

After running rootcrs.pl -deconfig, default listeners are also removed. One may see errors like “Cluster database creation requires Default listener configured and running in Grid Infrastructure home” when creating the database.

 Simple solution to this problem is to execute netca as grid user and create default listener.

Advertisements
This entry was posted in Oracle Cluster Ready Services, Oracle Real Application Cluster and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s