RMAN block recovery & ORA-600

Recently database alertlog started complaining about the data block corruption.

To check the corruption, I’ve used

rman> validate database;
rman> backup validate check logical database;

Then to check the corrupted block queried the database_block_corruption view & check FILE#, BLOCK# columns

SQL> select * from v$database_block_corruption;

To fix the block corruption

-- This command will try to fix all corrupted blocks
RMAN> blockrecover corruption list; 
-- This command will try to fix specific blocks
RMAN> recover datafile <file#> block <block#> 
datafile <datafile#> block <block#>;

I used second option as only 1 block was reported as corrupt.

RMAN> recover datafile 2 block 2564924;

But after almost 30 minutes, rman came out with ora-600

finished flashback log search, restored 0 blocks
channel ORA_DISK_1: restoring block(s) from datafile copy 
/u02/testdb/datafile/o1_mf_undo_6rgx4696_.dbf
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 03/22/2011 03:52:49
ORA-00600: internal error code, arguments: [1883], [0x7F3E5BC440F0], 
[], [], [], [], [], [], [], [], [], []
ORA-19600: input file is datafile copy 0 
(/u02/testdb/datafile/o1_mf_undo_6rgx4696_.dbf)

IMAGECOPY backup was also configured for the database. So I tried another rman recover, without flashback logs

RMAN> recover datafile 2 block 2564924 exclude flashback log;

But this rman session also failed with ora-600 (different argument & both of the codes missing from metalink ora-600 search)

Starting recover at 03/22/2011 03:53:19
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 03/22/2011 03:53:19
RMAN-00600: internal error, arguments [10019] [krmxcr] [] [] []
RMAN-00600: internal error, arguments [10019] [krmxcr] [] [] []

As this was the undo tablespace, there was an option to recreate it. But prior to that, I wanted to make sure that the corrupted block is not needed for instance recovery. So I did shutdown immediate and startup, which worked. Now it was time to replace the undo datafile. Being a test box, I did not open a SR with Oracle support, but it is highly recommended.

SQL> startup mount 
SQL> alter system set undo_management = manual scope=spfile;
SQL> shutdown immediate
SQL> alter database datafile '+oradata1/oradata/testdb/undo_01.dbf' 
offline drop;
SQL> alter database open;
SQL> drop tablespace undo including contents and datafiles;

–Verified the datafile is dropped from asmcmd CLI

SQL> create undo tablespace undo datafile 
'+oradata1/oradata/testdb/undo_01.dbf' size 1G  autoextend on next 100M 
maxsize unlimited;
SQL> alter system set undo_management = auto scope=spfile;
SQL> shutdown immediate
SQL> startup

Although database was back online & I created fresh imagecopy, rman block recovery did not work in this case.

Advertisements
This entry was posted in Oracle Recovery Manager and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s