ASM – Removing disk with HUNG state

Diskgroup with different redundancies

HIGHDG – HIGH redundancy
NORMDG – NORMAL redundancy
DATA – EXTERNAL redundancy
FRA – EXTERNAL redundancy

NOTE: HIGHDG & NORMDG, both of these DG’s had only one disk in each failgroup

SQL> select group_number, path, name, header_status, state from v$asm_disk
/

GN PATH NAME HEADER_STATUS
— ———————————– ————– ————-
3 /dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER
3 /dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER
3 /dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER
1 /dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER
1 /dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER
1 /dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER
1 /dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER
1 /dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER
1 /dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER
4 /dev/rdsk/c2t200500A0B81843A2d2s6 NORMDG_0001 MEMBER
4 /dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER
1 /dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER
2 /dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER

Now I dropped one of the disks from NORMDG.

SQL> alter diskgroup NORMDG drop disk normdg_0001
/

But this resulted in following condition (NORMDG_0001 state changed to HUNG)

PATH NAME HEADER_STATU STATE
—————————————- —————————— ———— ——–
/dev/rdsk/c2t200400A0B81843A2d2s6 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
NORMDG_0001 UNKNOWN HUNG
/dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d10s6 FRA_0001 MEMBER NORMAL

Oracle Doc explains

HUNG – Disk drop operation cannot continue because there is insufficient space to relocate the data from the disk being dropped

Tried undropping disk using following command

SQL> alter diskgroup NORMDG undrop disks
/
Diskgroup altered.

But this resulted in following condition (NORMDG_0001 state changed to FORCING)

SQL> select path, name, header_status, state from v$asm_disk
/

PATH NAME HEADER_STATU STATE
—————————————- —————————— ———— ——–
/dev/rdsk/c2t200400A0B81843A2d2s6 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
NORMDG_0001 UNKNOWN FORCING
/dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
/dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL
/dev/rdsk/c2t200500A0B81843A2d10s6 FRA_0001 MEMBER NORMAL


Oracle Doc explains

FORCING – Disk is being removed from the disk group without attempting to offload its data. The data will be recovered from redundant copies, where possible.

For a while, STATE value was toggling between HUNG & FORCING.

Being last disk in failgroup, it was not able to perform the rebalance operation correctly. So the solution was to add new disk and remove existing disk (NORMDG_0001)

SQL> alter diskgroup NORMDG add failgroup NORMDG_0001 disk ‘/dev/rdsk/c2t200500a0b81843a2d8s6’ drop disk NORMDG_0001
/

But this command also failed, with error (DISK NORMDG_0001 is already being dropped. So in background disk drop operation was still active)

So just added

SQL> alter diskgroup NORMDG add failgroup NORMDG_0001 disk ‘/dev/rdsk/c2t200500a0b81843a2d8s6’
/
Diskgroup altered.

Changed rebalance power

SQL> alter diskgroup NORMDG rebalance power 10
/
Diskgroup altered.

After some time, NORMDG_0001 was removed from the NORMDG diskgroup (check GROUP_NUMBER for /dev/rdsk/c2t200400A0B81843A2d2s6), but ASM metadata was still present in the disk header (STATUS as MEMBER).

SQL> select group_number as gn, path, name, header_status, state from v$asm_disk
/

GN PATH NAME HEADER_STATU STATE
— ————————————— ——————– ———— ——–
0 /dev/rdsk/c2t200400A0B81843A2d2s6 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
4 /dev/rdsk/c2t200500A0B81843A2d8s6 NORMDG_0002 MEMBER NORMAL
4 /dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
2 /dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL


To clean up the header, created a dummy diskgroup & forcibly added this disk & then dropped the DG.

SQL> create diskgroup DUMMY external redundancy disk ‘/dev/rdsk/c2t200400a0b81843a2d2s6’ force
/

Diskgroup created.

SQL> select group_number, path, name, header_status, state from v$asm_disk
/

GN PATH NAME HEADER_STATU STATE
— —————————————- ——————– ———— ——–
3 /dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
5 /dev/rdsk/c2t200400A0B81843A2d2s6 DUMMY_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
4 /dev/rdsk/c2t200500A0B81843A2d8s6 NORMDG_0002 MEMBER NORMAL
4 /dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
2 /dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL

 

SQL> drop diskgroup DUMMY
/

Diskgroup dropped.

SQL> select GROUP_NUMBER, path, name, header_status, state from v$asm_disk
/

GN PATH NAME HEADER_STATU STATE
— —————————————- ——————– ———— ——–
0 /dev/rdsk/c2t200400A0B81843A2d2s6 FORMER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
4 /dev/rdsk/c2t200500A0B81843A2d8s6 NORMDG_0002 MEMBER NORMAL
4 /dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
2 /dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL


Cleaned the FORMER header by using dd

dd if=/dev/zero of=/dev/rdsk/c2t200400A0B81843A2d2s6 bs=8192 count=100

SQL> select GROUP_NUMBER, path, name, header_status, state from v$asm_disk
/

GN PATH NAME HEADER_STATU STATE
— —————————————- ——————– ———— ——–
0 /dev/rdsk/c2t200400A0B81843A2d2s6 CANDIDATE NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d7s6 HIGHDG_0002 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d6s6 HIGHDG_0001 MEMBER NORMAL
3 /dev/rdsk/c2t200400A0B81843A2d3s6 HIGHDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d0s6 DATA_0001 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d5s6 DATA_0004 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d4s6 DATA_0003 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d1s6 DATA_0002 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d12s6 DATA_0006 MEMBER NORMAL
1 /dev/rdsk/c2t200500A0B81843A2d11s6 DATA_0005 MEMBER NORMAL
4 /dev/rdsk/c2t200500A0B81843A2d8s6 NORMDG_0002 MEMBER NORMAL
4 /dev/rdsk/c2t200400A0B81843A2d14s6 NORMDG_0000 MEMBER NORMAL
1 /dev/rdsk/c2t200400A0B81843A2d13s6 DATA_0000 MEMBER NORMAL
2 /dev/rdsk/c2t200400A0B81843A2d9s6 FRA_0000 MEMBER NORMAL

 

This scenario can only be seen with NORMAL or HIGH redundancy diskgroups. Lesson learnt from this whole exercise was not to drop the last disk from the failgroup(s).

Advertisements
This entry was posted in Oracle Automatic Storage Management and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s