Recently database alertlog started complaining about the data block corruption.
To check the corruption, I’ve used
rman> validate database; rman> backup validate check logical database;
Then to check the corrupted block queried the database_block_corruption view & check FILE#, BLOCK# columns
SQL> select * from v$database_block_corruption;
To fix the block corruption
-- This command will try to fix all corrupted blocks RMAN> blockrecover corruption list; -- This command will try to fix specific blocks RMAN> recover datafile <file#> block <block#> datafile <datafile#> block <block#>;
I used second option as only 1 block was reported as corrupt.
RMAN> recover datafile 2 block 2564924;
But after almost 30 minutes, rman came out with ora-600
finished flashback log search, restored 0 blocks channel ORA_DISK_1: restoring block(s) from datafile copy /u02/testdb/datafile/o1_mf_undo_6rgx4696_.dbf RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 03/22/2011 03:52:49 ORA-00600: internal error code, arguments: [1883], [0x7F3E5BC440F0], [], [], [], [], [], [], [], [], [], [] ORA-19600: input file is datafile copy 0 (/u02/testdb/datafile/o1_mf_undo_6rgx4696_.dbf)
IMAGECOPY backup was also configured for the database. So I tried another rman recover, without flashback logs
RMAN> recover datafile 2 block 2564924 exclude flashback log;
But this rman session also failed with ora-600 (different argument & both of the codes missing from metalink ora-600 search)
Starting recover at 03/22/2011 03:53:19 using channel ORA_DISK_1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 03/22/2011 03:53:19 RMAN-00600: internal error, arguments [10019] [krmxcr] [] [] [] RMAN-00600: internal error, arguments [10019] [krmxcr] [] [] []
As this was the undo tablespace, there was an option to recreate it. But prior to that, I wanted to make sure that the corrupted block is not needed for instance recovery. So I did shutdown immediate and startup, which worked. Now it was time to replace the undo datafile. Being a test box, I did not open a SR with Oracle support, but it is highly recommended.
SQL> startup mount SQL> alter system set undo_management = manual scope=spfile; SQL> shutdown immediate SQL> alter database datafile '+oradata1/oradata/testdb/undo_01.dbf' offline drop; SQL> alter database open; SQL> drop tablespace undo including contents and datafiles;
–Verified the datafile is dropped from asmcmd CLI
SQL> create undo tablespace undo datafile '+oradata1/oradata/testdb/undo_01.dbf' size 1G autoextend on next 100M maxsize unlimited; SQL> alter system set undo_management = auto scope=spfile; SQL> shutdown immediate SQL> startup
Although database was back online & I created fresh imagecopy, rman block recovery did not work in this case.