How to resolve ORA-1119 in ASM environment?

Scenario:

I was running out of space in one of the ASM diskgroup. I added new disks, same size of existing disks in diskgroup. Disks were added successfully. But when I tried creating the datafile using the the diskgroup (where the disks were recently added), I got “ORA-1119 signalled during: ALTER TABLESPACE ADD DATAFILE ‘+’ SIZE 10240M AUTOEXTEND OFF.

After seeing the complete stack, it also reported following errors

ORA-17502: ksfdcre:4 Failed to create file
ORA-15041: diskgroup space exhausted

But after some time, adding the same file worked fine.

There are two possibilities

1) ASM diskgroup genuinely does not have enough space to create or extend the datafile.

This possibility was ruled out, as I knew that I’ve added 100g of space & I was trying to create a datafile of size 10g.

2) Another possibility could be, that the disks are not rebalanced (this was the problem is my case)

Entries from RDBMS alert.log (when the disks were added to the diskgroup)

Fri Sep 5 15:05:01 2008
SUCCESS: disk DATA_DG_0002 (2.3992024182) added to diskgroup DATA_DG
SUCCESS: disk DATA_DG_0003 (3.3992024183) added to diskgroup DATA_DG

Entries from ASM alert log (when the disks were added to the diskgroup & ASM rebalance was ON)

NOTE: reconfiguration of group 1/0xe7a18492 (DATA_DG), full=1
Fri Sep 5 15:04:55 2008
NOTE: initializing header on grp 1 disk DATA_DG_0002
NOTE: initializing header on grp 1 disk DATA_DG_0003
NOTE: cache opening disk 2 of grp 1: DATA_DG_0002 path:/dev/rdsk/raw12
NOTE: cache opening disk 3 of grp 1: DATA_DG_0003 path:/dev/rdsk/raw13
NOTE: PST update: grp = 1
NOTE: requesting all-instance disk validation for group=1
Fri Sep 5 15:04:55 2008
NOTE: disk validation pending for group 1/0xe7a18492 (DATA_DG)
SUCCESS: validated disks for 1/0xe7a18492 (DATA_DG)
Fri Sep 5 15:04:58 2008
NOTE: PST update: grp = 1
NOTE: requesting all-instance membership refresh for group=1
Fri Sep 5 15:04:58 2008
NOTE: membership refresh pending for group 1/0xe7a18492 (DATA_DG)
SUCCESS: refreshed membership for 1/0xe7a18492 (DATA_DG)
Fri Sep 5 15:05:01 2008
NOTE: requesting all-instance membership refresh for group=1
Fri Sep 5 15:05:01 2008
NOTE: membership refresh pending for group 1/0xe7a18492 (DATA_DG)
SUCCESS: refreshed membership for 1/0xe7a18492 (DATA_DG)
Fri Sep 5 15:05:06 2008
NOTE: starting rebalance of group 1/0xe7a18492 (DATA_DG) at power 1
Starting background process ARB0
ARB0 started with pid=23, OS id=27493
Fri Sep 5 15:05:07 2008
NOTE: assigning ARB0 to group 1/0xe7a18492 (DATA_DG)
Fri Sep 5 15:05:11 2008

NOTE: X->S down convert bast on F1B3 bastCount=2
NOTE: X->S down convert bast on F1B3 bastCount=3
NOTE: X->S down convert bast on F1B3 bastCount=4
NOTE: X->S down convert bast on F1B3 bastCount=5
NOTE: X->S down convert bast on F1B3 bastCount=6
NOTE: X->S down convert bast on F1B3 bastCount=7
NOTE: X->S down convert bast on F1B3 bastCount=8
NOTE: X->S down convert bast on F1B3 bastCount=9
NOTE: X->S down convert bast on F1B3 bastCount=10
…………………………………………..
…………………………………………..
…………………………………………..
Fri Sep 5 19:50:49 2008
SUCCESS: rebalance completed for group 1/0xe7a18492 (DATA_DG)
Fri Sep 5 19:50:49 2008
SUCCESS: rebalance completed for group 1/0xe7a18492 (DATA_DG)
NOTE: PST update: grp = 1

So the rebalancing was active, when I tried creating the file. This could sometime, throw ORA-1119 error.

Solution is to rebalance the disk manually OR wait rebalance to complete (SUCCESS: rebalance completed for group in ASM alert.log for specific diskgroup)

We can also speed up the disk rebalance process by changing the value of ASM_POWER_LIMIT parameter (default is 1)

SYS@+ASM1> show parameter asm_power_limit

NAME TYPE VALUE
——– ——— ——-
asm_power_limit integer 1

SYS@+ASM1> alter system set asm_power_limit=11;

Changing the value of this parameter is only recommended, when there is not much of activity in the database. As rebalance using value 11, will incur high amount of I/O, which may impact day-to-day operations.

We can monitor the rebalance activity by selecting data from v$asm_operation view. We can also check the column “UNBALANCED” in v$asm_diskgroup view. If the disks are correctly balanced, we’ll see default value of N in this column.

One the rebalance is done; we can add the datafile using the same command used before.

Advertisements
This entry was posted in Oracle Automatic Storage Management and tagged . Bookmark the permalink.

2 Responses to How to resolve ORA-1119 in ASM environment?

  1. vayruz says:

    mil gracias viejo, su articulo me ayudo mucho, la bd no me permitia crear datafiles y era por ese error ora-1119. mil gracias.

  2. ybhandarkar says:

    I tried translating this (spanish to english).. I hope it helped :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s