Resolving ORA-15075 & ORA-15033 in RAC ASM environment

I wanted to add 2 disks to my NORMAL REDUNDANCY diskgroup with 2 failgroups in a RAC environment.

This is how my existing group look like

SELECT GROUP_NUMBER, NAME, TOTAL_MB, FREE_MB, USABLE_FILE_MB FROM V$ASM_DISKGROUP
/

GROUP_NUMBER NAME TOTAL_MB FREE_MB USABLE_FILE_MB
———— —— ———- ———- ————–
4 NDG 20456 20350 7618

Following disks are part of DG NDG

SELECT GROUP_NUMBER, PATH, STATE , HEADER_STATUS, MODE_STATUS FROM V$ASM_DISK WHERE HEADER_STATUS=’MEMBER’ ORDER BY GROUP_NUMBER
/
GROUP_NUMBER PATH STATE HEADER_STATU MODE_ST
———— —————————————- ——– ———— ——-
4 /dev/rdsk/c2t20120002AC000212d45s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d44s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d12s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d13s6 NORMAL MEMBER ONLINE

I use following command to add disk

ALTER DISKGROUP NDG ADD
FAILGROUP FG1 DISK ‘/dev/rdsk/c2t20120002AC000212d62s6’
FAILGROUP FG2 DISK ‘/dev/rdsk/c2t20120002AC000212d63s6’
/

ALTER DISKGROUP NDG ADD FAILGROUP FG1 DISK ‘/dev/rdsk/c2t20120002AC000212d62s6’ FAILGROUP FG2 DISK ‘/dev/rdsk/c2t20120002AC000212d63s6’
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15075: disk(s) are not visible cluster-wide

I forgot to change the permission of the disks on all nodes. So I change it now.

# chown ora10gr2:dba ../../devices/pci@0,0/pci1022,7450@b/pci1077,100@1/fp@0,0/disk@w20120002ac000212,3e:g,raw
# chown ora10gr2:dba ../../devices/pci@0,0/pci1022,7450@b/pci1077,100@1/fp@0,0/disk@w20120002ac000212,3f:g,raw

I again retry the same operation to add the disks. But now I get following error message

ALTER DISKGROUP NDG ADD FAILGROUP FG1 DISK ‘/dev/rdsk/c2t20120002AC000212d62s6’ FAILGROUP FG2 DISK ‘/dev/rdsk/c2t20120002AC000212d63s6’
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk ‘/dev/rdsk/c2t20120002AC000212d62s6’ belongs to diskgroup “NDG”
ORA-15033: disk ‘/dev/rdsk/c2t20120002AC000212d63s6’ belongs to diskgroup “NDG”

So what happened?

Disks are added in group ‘0’

SELECT GROUP_NUMBER, PATH, STATE , HEADER_STATUS, MODE_STATUS FROM V$ASM_DISK WHERE HEADER_STATUS=’MEMBER’
AND GROUP_NUMBER IN (0,4) ORDER BY GROUP_NUMBER
/

GROUP_NUMBER PATH STATE HEADER_STATU MODE_ST
———— ————————————– ——– ———— ——-
0 /dev/rdsk/c2t20120002AC000212d62s6 NORMAL MEMBER ONLINE
0 /dev/rdsk/c2t20120002AC000212d63s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d45s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d13s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d44s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d12s6 NORMAL MEMBER ONLINE

Also it stamped the disks with the diskgroup. This can be verified by kfed read

kfed read /dev/rdsk/c2t20120002AC000212d63s6 grep name
kfdhdb.dskname: NDG_0005 ; 0x028: length=8
kfdhdb.grpname: NDG ; 0x048: length=3
kfdhdb.fgname: FG2 ; 0x068: length=3
kfdhdb.capname: ; 0x088: length=0

kfed read /dev/rdsk/c2t20120002AC000212d62s6 grep name
kfdhdb.dskname: NDG_0004 ; 0x028: length=8
kfdhdb.grpname: NDG ; 0x048: length=3
kfdhdb.fgname: FG1 ; 0x068: length=3
kfdhdb.capname: ; 0x088: length=0

So what is the solution in this case?

I re-initialized the disks using

dd if=/dev/zero of=/dev/rdsk/c2t20120002AC000212d62s6 bs=8192 count=100
dd if=/dev/zero of=/dev/rdsk/c2t20120002AC000212d63s6 bs=8192 count=100

Verified the status of the disk. ASM metadata is removed from the disk(s)

-bash-3.00$ kfed read /dev/rdsk/c2t20120002AC000212d63s6
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000

-bash-3.00$ kfed read /dev/rdsk/c2t20120002AC000212d62s6
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000

Check the ASM view for the disk entries

SELECT GROUP_NUMBER, PATH, STATE , HEADER_STATUS, MODE_STATUS FROM V$ASM_DISK WHERE HEADER_STATUS=’MEMBER’
AND GROUP_NUMBER IN (0,4) ORDER BY GROUP_NUMBER
/

GROUP_NUMBER PATH STATE HEADER_STATU MODE_ST
———— —————————————- ——– ———— ——-
4 /dev/rdsk/c2t20120002AC000212d45s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d13s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d44s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d12s6 NORMAL MEMBER ONLINE

Now I again add the disk

ALTER DISKGROUP NDG ADD FAILGROUP FG1 DISK ‘/dev/rdsk/c2t20120002AC000212d62s6’ FAILGROUP FG2 DISK ‘/dev/rdsk/c2t20120002AC000212d63s6′
/

Diskgroup altered.

SELECT GROUP_NUMBER, PATH, STATE , HEADER_STATUS, MODE_STATUS FROM V$ASM_DISK WHERE HEADER_STATUS=’MEMBER’ ORDER BY GROUP_NUMBER
/

GROUP_NUMBER PATH STATE HEADER_STATU MODE_ST
———— —————————————- ——– ———— ——-
4 /dev/rdsk/c2t20120002AC000212d12s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d63s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d62s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d45s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d13s6 NORMAL MEMBER ONLINE
4 /dev/rdsk/c2t20120002AC000212d44s6 NORMAL MEMBER ONLINE

Note: kfed binary will be available in 11g. In 10g, we’ll have to build it

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ikfed

Advertisements
This entry was posted in Oracle Automatic Storage Management and tagged . Bookmark the permalink.

One Response to Resolving ORA-15075 & ORA-15033 in RAC ASM environment

  1. Michael says:

    Thanks!!! This totally resolved my issue! :-)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s