ASM & Database won’t start after server reboot

Recently I’ve seen a GRID/CRS behaviour, which I would like to share.

I was testing, if Oracle grid & database services start automatically after physical server reboot.

Tested two scenario’s

Scenario 1
==========

Server reboot without stopping the services (simulation of abnormal server shutdown)

When server came back, it started all grid (cssd, asm) & database (instance, services) without any problem.
 

Scenario 2
==========

Gracefully stopped the database & asm and then server reboot (simulation of planned downtime)

In this case, when server came back, but asm & db did not start automatically.

After checking the resource properties, found that AUTO_START was set to value ‘restore’ (which means, it will restore the resource to last known state)

$ crsctl stat res ora.asm -p
NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:ora11gr2:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
ASM_DISKSTRING=
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=600
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm)
ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SPFILE=/u01/app/ora11gr2/product/11.2.0/grid/dbs/spfile+ASM.ora
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.1.0

$ crsctl stat res ora.oradb.db -p
NAME=ora.oradb.db
TYPE=ora.database.type
ACL=owner:ora11gr2:--x,pgrp:oinstall:--x,other::r--,
group:oinstall:r-x,user:ora11gr2:rwx
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_INTERVAL=1
CHECK_TIMEOUT=600
CLUSTER_DATABASE=false
DB_UNIQUE_NAME=oradb
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=database)
PROPERTY(DB_UNIQUE_NAME= CONCAT(PARSE(%NAME%, ., 2), %USR_ORA_DOMAIN%, .))
ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle Database resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=60
FAILURE_THRESHOLD=1
GEN_AUDIT_FILE_DEST=/u01/app/ora11gr2/admin/oradb/adump
GEN_USR_ORA_INST_NAME=oradb
HOSTING_MEMBERS=
INSTANCE_FAILOVER=0
LOAD=1
LOGGING_LEVEL=1
MANAGEMENT_POLICY=AUTOMATIC
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
ORACLE_HOME=/u01/app/ora11gr2/product/11.2.0/database
PLACEMENT=balanced
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=2
ROLE=PRIMARY
SCRIPT_TIMEOUT=60
SERVER_POOLS=
SPFILE=/u01/app/ora11gr2/product/11.2.0/database/dbs/spfileoradb.ora
START_DEPENDENCIES=weak(type:ora.listener.type,uniform:ora.ons,uniform:
ora.eons) hard(ora.ORADATA1.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.ORADATA1.dg)
STOP_TIMEOUT=600
UPTIME_THRESHOLD=1h
USR_ORA_DB_NAME=
USR_ORA_DOMAIN=abc.com
USR_ORA_ENV=
USR_ORA_FLAGS=
USR_ORA_INST_NAME=oradb
USR_ORA_OPEN_MODE=open
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.1.0

But what if, we want to start a resource automatically in any case?

We need to change the auto_start attribute of a resource to value ‘always’ or ‘1’

crsctl modify resource “ora.asm” -attr “AUTO_START=always”
crsctl modify resource “ora.oradb.db” -attr “AUTO_START=always”

Possible auto_start values / constants are

-always (1)
 Causes the resource to restart when the node restarts regardless of the resource’s state when the node stopped.

-restore (0)
 Does not start the resource at restart time if it was in an offline state, such as STATE=OFFLINE, TARGET=OFFLINE, when the node stopped. The resource is restored to its state when the node went down. The resource is started only if it was online before and not otherwise.

-never (2)
 Oracle Clusterware never restarts the resource regardless of the resource’s state when the node is stopped.

Following script can be used to check the resources & their current auto_start values

for i in `crs_stat -p |  grep "NAME" | cut -f2 -d "="`; 
do echo "Resource=$i, `crsctl stat res $i -p | grep AUTO_START`"; done
Advertisements
This entry was posted in Oracle 11.2.0.x, Oracle Cluster Ready Services and tagged , . Bookmark the permalink.

2 Responses to ASM & Database won’t start after server reboot

  1. neil says:

    Thanks, I just had the same problem and this article described perfectly what I needed to know and to do in order to correct it!

  2. mikethodgson says:

    Thank you for posting this, I also had this issue, on a 11gR1 install; it was puzzling to why this occured like this.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s