Recovering lost OCR & Voting disk in 11gR1

I’ve recently faced this problem. In one of my test environment, some of the disks on which my OCR & voting disks were placed corrupted. I also didn’t had the backup of any of those. Normally this situation shouldn’t arise, but one has to pay for ignorance. I managed to recover both of them, using following steps (database version 11.1.0.7, but I guess should also work for 10gR2).

It was a 2 node SPARC cluster with Solaris 10 Update 8

STEP 1- Run $ORA_CRS_HOME/install/rootdelete.sh and Node 1
===============================================

# uname -a
SunOS node1.mydomain.com 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise

# /oracle/crs11gr1/install> ./rootdelete.sh
Getting local node name
NODE = node1.mydomain.com
PRKO-2006 : Invalid node name: node1.mydomain.com
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in ‘/var/opt/oracle/scls_scr’
Cleaning up Network socket directories

STEP 2 – Run $ORA_CRS_HOME/install/rootdelete.sh and Node 2
================================================

# uname -a
SunOS node2.mydomain.com 5.10 Generic_141444-07 sun4u sparc SUNW,SPARC-Enterprise

# /oracle/crs11gr1/install> ./rootdelete.sh
Getting local node name
NODE = node2.mydomain.com
PRKO-2006 : Invalid node name: node2.mydomain.com
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in ‘/var/opt/oracle/scls_scr’
Cleaning up Network socket directories

STEP 3 – Run $ORA_CRS_HOME/install/rootdeinstall.sh from last node of the cluster
=============================================================

– Run $ORA_CRS_HOME/install/rootdeinstall.sh from last node of the cluster

# /oracle/crs11gr1/install> ./rootdeinstall.sh (Need to run only from last node of the cluster)
Verifying existence of ocr.loc file

Removing contents from OCR device
2560+0 records in
2560+0 records out

STEP 4 – Check if any of the CRS process are running
======================================

# ps -e | grep -i ‘ocs[s]d’
# ps -e | grep -i ‘cr[s]d.bin’
# ps -e | grep -i ‘ev[m]d.bin’

STEP 5 – Run $ORA_CRS_HOME/root.sh from Node 1
=======================================

# uname -a
SunOS node1.mydomain.com 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise

# /oracle/crs11gr1> ./root.sh
Checking to see if Oracle CRS stack is already configured
Checking to see if any 9i GSD is up

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 0: node1 node1-priv node1
node 1: node2 node2-priv node2
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Now formatting voting device: /dev/vx/rdsk/ocrvotdg5000/votevol
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
        node1
Cluster Synchronization Services is inactive on these nodes.
        node2
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

STEP 6- Run $ORA_CRS_HOME/root.sh from Node 2
=======================================

# uname -a
SunOS node2.mydomain.com 5.10 Generic_141444-07 sun4u sparc SUNW,SPARC-Enterprise

# /oracle/crs11gr1> ./root.sh
Checking to see if Oracle CRS stack is already configured
Checking to see if any 9i GSD is up

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 0: node1 node1-priv node1
node 1: node2 node2-priv node2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
        node1
        node2
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (2) nodes…
Creating GSD application resource on (2) nodes…
Creating ONS application resource on (2) nodes…
Starting VIP application resource on (2) nodes…
Starting GSD application resource on (2) nodes…
Starting ONS application resource on (2) nodes…

STEP 7 – Manually configure the ONS
===========================

$ uname -a
SunOS node1.mydomain.com 5.10 Generic_139555-08 sun4u sparc SUNW,SPARC-Enterprise

$ /oracle/crs11gr1/install> onsconfig add_config node1:6251 node2:6251

The ONS configuration is created successfully
Stopping ONS resource ‘ora.node1.ons’
Attempting to stop `ora.node1.ons` on member `node1`
Stop of `ora.node1.ons` on member `node1` succeeded.
The resource ora.node1.ons stopped successfully for restart
Attempting to start `ora.node1.ons` on member `node1`
Start of `ora.node1.ons` on member `node1` succeeded.
The resource ora.node1.ons restarted successfully
Stopping ONS resource ‘ora.node2.ons’
Attempting to stop `ora.node2.ons` on member `node2`
Stop of `ora.node2.ons` on member `node2` succeeded.
The resource ora.node2.ons stopped successfully for restart
Attempting to start `ora.node2.ons` on member `node2`
Start of `ora.node2.ons` on member `node2` succeeded.
The resource ora.node2.ons restarted successfully

STEP 8  – Verify the public and private interfaces
===================================

$ /oracle/crs11gr1/bin> oifcfg setif -global  bge0/10.209.92.0:public
$ /oracle/crs11gr1/bin> oifcfg setif -global  bge1/10.20.28.0:cluster_interconnect

$ oifcfg getif
bge0  10.209.92.0  global  public
bge1  10.20.28.0  global  cluster_interconnect

STEP 9 – Verify the OCR location
========================

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :    2096812
         Used space (kbytes)      :       2012
         Available space (kbytes) :    2094800
         ID                       : 2110586102
         Device/File Name         : /dev/vx/rdsk/ocrvotdg5000/ocrvol
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

STEP 10  – Verify the Voting disk location
==============================

# crsctl query css votedisk
 0.     0    /dev/vx/rdsk/ocrvotdg5000/votevol
Located 1 voting disk(s).

Advertisements
This entry was posted in Oracle Cluster Ready Services, Oracle Real Application Cluster and tagged . Bookmark the permalink.

One Response to Recovering lost OCR & Voting disk in 11gR1

  1. Pingback: World Wide News Flash

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s