ORA-00600: internal error code, arguments: [ksprcvsp:ksfdread_resilver]

On an Oracle 11204 restart environment, ASM spfile creation failed with

create spfile='+DATA5' from pfile='/u03/asmpfile.ora'
*
ERROR at line 1:
ORA-01078: failure in processing system parameters
ORA-00600: internal error code, arguments: 
[ksprcvsp:ksfdread_resilver],[0x0A2435D48], [3], 
[0x7F710EE38000], [512], [], [], [], [], [], [], []

This was caused due to a ASM disk with UNKNOWN HEADER_STATUS.

SQL> select name, path, header_status, failgroup from v$asm_disk;

NAME                   PATH           HEADER_STATU FAILGROUP
---------------------- -------------- ------------ ----------
                       /dev/raw/raw1   MEMBER
_DROPPED_0000_DATA5                    UNKNOWN     DATA5_FG1
DATA5_0001             /dev/raw/raw2   MEMBER      DATA5_FG2

Fixed the problem, by adding the disk in ASM diskgroup

SQL> alter diskgroup  DATA5 add FAILGROUP DATA5_FG1 disk  
'/dev/raw/raw1' force ;

Post this change, I was able to create spfile without any issue.

Metalink Doc ID 2096227.1

 

 

Posted in Oracle Administration, Oracle ORA-600 errors | Tagged , , , | Leave a comment

ORA-00600: internal error code, arguments: [kzsrgpw]

Recently I started getting ORA-00600 error, while I was trying to connect  as

$ sqlplus sys as sysdba
SQL*Plus:Release 11.2.0.3.0 Production on Tue Jan 17 06:13:16 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Enter password:
ERROR:
ORA-00600: internal error code, arguments: [kzsrgpw], [], [], [], 
[], [], [], [], [], [], [], []

This was caused due to a corrupt password file. Re-creating password file fixed the issue.

orapwd file=$ORACLE_HOME/dbs/orapw$ORACLE_SID 
password=syspassword entries=<#>
Metalink Doc ID 1073283.1
Posted in Oracle Administration, Oracle ORA-600 errors | Tagged , , , | Leave a comment

ORA-00704: bootstrap process failure

Bootstarp error indicates version mismatch between datafile headers and oracle binaries, which are being used to start the database.

I’m starting the database using Oracle 12.1.0.2 binaries, where as datafiles are from 11.2.0.4.

$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Thu Oct 27 07:18:50 2016
Copyright (c) 1982, 2014, Oracle. All rights reserved.
SQL> startup
ORACLE instance started.
Total System Global Area 1.8388E+10 bytes
Fixed Size 5366360 bytes
Variable Size 4898950568 bytes
Database Buffers 1.3422E+10 bytes
Redo Buffers 61739008 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 2
ORA-00904: “I”.”UNUSABLEBEGINNING#”: invalid identifier
Process ID: 5177412
Session ID: 1704 Serial number: 59059

Now how to check if there is really a mismatch? First we create a trace file with datafile header dump.

SQL> alter session set tracefile_identifier=’datafile_hdr’;
SQL> alter session set events ‘immediate trace name file_hdrs level 10’;

From trace file, extract section “V10 STYLE FILE HEADER” & check the value “Compatibility Vsn”.

V10 STYLE FILE HEADER:
Compatibility Vsn = 123456789=0xb200400

Current value b200400 <———> b 2 0 0 4 0 0 (Hex) – 11 2 0 4 0 0 (Decimal). Expected value was c100200.

Posted in Oracle Administration, Oracle Database Upgrade, Startup issues | Tagged , , , | Leave a comment

PRVG-2027 : Owner of file “filename” is inconsistent across nodes.

CVU (Cluster Verification Utility) logged multiple PRVG-2027 errors in CRS alret log. This environment was recently patched with latest PSU, which failed while applying changes to GMIR.

$GRID_HOME/bin/crsctl binary ownership should be root:dba.

db01:+ASM3:/u01/app/oracle>ls -al /u01/product/12.1.0.2/grid/bin/crsctl
-rwxr-xr-x 1 oracle dba 9545 Oct 27 07:05 /u01/product/12.1.0.2/grid/bin/crsctl
db02:+ASM4:/u01/app/oracle>ls -al /u01/product/12.1.0.2/grid/bin/crsctl
-rwxr-xr-x 1 root dba 9545 Oct 25 11:29 /u01/product/12.1.0.2/grid/bin/crsctl

List of patches and patch level on both nodes are same.

db01:+ASM3:>kfod op=patches
—————
List of Patches
===============
19769480
20299023
20831110
21359755
21436941
21948354
22291127
23054246
23854735
24006101
24007012
db01:+ASM3:>kfod op=patchlvl
——————-
Current Patch level
===================
1505651481
db02:+ASM4:>kfod op=patches
—————
List of Patches
===============
19769480
20299023
20831110
21359755
21436941
21948354
22291127
23054246
23854735
24006101
24007012
db02:+ASM4:>kfod op=patchlvl
——————-
Current Patch level
===================
1505651481

Node db01, I could see the enrty related to PREPATCH, but no entry for POSTPATCH.

Invoking “/u01/product/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state SUCCESS”

Node db02

Invoking “/u01/product/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_PREPATCH -state SUCCESS”
Invoking “/u01/product/12.1.0.2/grid/bin/cluutil -ckpt -oraclebase /u01/app/oracle -writeckpt -name ROOTCRS_POSTPATCH -state SUCCESS”

So ROOTCRS_POSTPATCH never got executed on db01, which explains difference in permissions

On db01

# crsctl stop crs
# $GRID_HOME/crs/install/rootcrs.sh -patch
db01:+ASM3:/u01/app/oracle>ls -al /u01/product/12.1.0.2/grid/bin/crsctl
-rwxr-xr-x 1 root dba 9545 Oct 27 07:05 /u01/product/12.1.0.2/grid/bin/crsctl
Posted in Oracle Administration, Oracle OPatch | Tagged , , , , | Leave a comment

PRVG-2029 : Octal permissions of file “filename” are inconsistent across nodes

Following entries were logged in CRS/GRID alert.log

2016-10-23 16:24:49.048 [SRVM(4849748)]CRS-10051: CVU found following errors with Clusterware setup :
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_asym.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_base.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_base_non_fips.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_ecc.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_ecc_accel_fips.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_ecc_accel_non_fips.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libccme_ecc_non_fips.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libcryptocme.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libldapjclnt12.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]
PRVG-2029 : Octal permissions of file “/u01/product/12.1.0.2/grid/lib/libowm2.so” are inconsistent across nodes. [Found = “{0644=[db01], 0755=[db02]}”]

This would typically happen, when we are doing opatch operation. One of the node is patched and another one is in progress or not patched. Some file permissions may change as part of PSU patching.

Check patch levels using opatch lsinventory. In the last section “Patching Level” column would show current state.

Patch level status of Cluster nodes :

Patching Level Nodes
————– —–
0 db01
1505651481 db02

This means db02 is patched and on higher level. We can ignore these messages, if they are logged when patching is in progress.

But if they are logged post patching i.e. “Patching Level” are same, we should contact Oracle support.

Posted in Oracle OPatch | Tagged , | Leave a comment

datapatch failing with ORA-20006: Number of RAC active instances and opatch jobs configured are not same

datapatch failing with ORA-20006: Number of RAC active instances and opatch jobs configured are not same

db02:orcl01:/u01/product/12.1.0.2/database/OPatch>./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Tue Oct 25 13:01:10 2016
Copyright (c) 2016, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_invocation.log
SQL Patching arguments:
verbose: 1
force: 0
prereq: 0
upgrade_mode_only:
oh:
bundle_series:
ignorable_errors:
bootstrap:
skip_upgrade_check:
userid:
pdbs:
Connecting to database…OK
catcon: ALL catcon-related output will be written to /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_catcon__catcon_6619568.lst
catcon: See /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_catcon_*.log files for output generated by scripts
catcon: See /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_catcon__*.lst files for spool files, if any
Bootstrapping registry and package to current versions…done
verify_queryable_inventory returned ORA-20006: Number of RAC active instances and opatch jobs configured are not same
Queryable inventory could not determine the current opatch status.
Execute ‘select dbms_sqlpatch.verify_queryable_inventory from dual’
and/or check the invocation log
/u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.
Please refer to MOS Note 1609718.1 and/or the invocation log
/u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_6619568_2016_10_25_13_02_22/sqlpatch_invocation.log
for information on how to resolve the above errors.
SQL Patching tool complete on Tue Oct 25 13:02:29 2016
SQL> select dbms_sqlpatch.verify_queryable_inventory from dual;
VERIFY_QUERYABLE_INVENTORY
———————————————————————————
ORA-20006: Number of RAC active instances and opatch jobs configured are not same
SQL> select dbms_qopatch.get_pending_activity from dual;
ERROR:
ORA-20006: Number of RAC active instances and opatch jobs configured are not same
ORA-06512: at “SYS.DBMS_QOPATCH”, line 1222
no rows selected
SQL> select NODE_NAME, INST_ID, INST_JOB from opatch_inst_job;
NODE_NAME INST_ID INST_JOB
—————– ———- ————————-
db01 2 Load_opatch_inventory_2
db02 1 Load_opatch_inventory_1
SQL> select job_name ,state from dba_scheduler_jobs where job_name like ‘%OPATCH%’ ;
JOB_NAME STATE
———————— —————
LOAD_OPATCH_INVENTORY_2 DISABLED

Delete all existing job entries

SQL> exec DBMS_SCHEDULER.DROP_JOB(‘LOAD_OPATCH_INVENTORY_2’);
PL/SQL procedure successfully completed.

Delete rows from opatch_inst_job

SQL> delete from opatch_inst_job;
2 rows deleted.
SQL> commit;
Commit complete.

Re-run datapatch

db02:orcl02:/u01/product/12.1.0.2/database/OPatch>./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Tue Oct 25 13:06:59 2016
Copyright (c) 2016, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_5506020_2016_10_25_13_06_59/sqlpatch_invocation.log
Connecting to database…OK
Bootstrapping registry and package to current versions…done
Determining current state…done
Current state of SQL patches:
Bundle series PSU:
ID 161018 in the binary registry and not installed in the SQL registry
Adding patches to installation queue and performing prereq checks…
Installation queue:
Nothing to roll back
The following patches will be applied:
24006101 (Database Patch Set Update : 12.1.0.2.161018 (24006101))
Installing patches…
Patch installation complete. Total patches installed: 1
Validating logfiles…
Patch 24006101 apply: SUCCESS
logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/24006101/20648640/24006101_apply_ORCL_2016Oct25_13_11_23.log (no errors)
SQL Patching tool complete on Tue Oct 25 13:18:34 2016
Posted in datapatch issues, Oracle Administration, Oracle OPatch | Tagged , , | Leave a comment

datapatch failing with ORA-27477: “SYS”.”LOAD_OPATCH_INVENTORY_X” already exists

datapatch is failing with ORA-27477, due to existing job_name in dba_scheduler_jobs

db02:orcl01:/u01/product/12.1.0.2/database/OPatch>./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Tue Oct 25 12:55:45 2016
Copyright (c) 2016, Oracle. All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_3997884_2016_10_25_12_55_45/sqlpatch_invocation.log
Connecting to database…OK
Bootstrapping registry and package to current versions…done
Queryable inventory could not determine the current opatch status.
Execute ‘select dbms_sqlpatch.verify_queryable_inventory from dual’
and/or check the invocation log
/u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_3997884_2016_10_25_12_55_45/sqlpatch_invocation.log
for the complete error.
Prereq check failed, exiting without installing any patches.
SQL> select dbms_sqlpatch.verify_queryable_inventory from dual;
VERIFY_QUERYABLE_INVENTORY
——————————————————————————–
ORA-27477: “SYS”.”LOAD_OPATCH_INVENTORY_1″ already exists
SQL> select job_name,state, start_date from dba_scheduler_jobs where job_name like ‘LOAD_OPATCH%’;
JOB_NAME STATE START_DATE
————————— ————— ————————————
LOAD_OPATCH_INVENTORY DISABLED 25-OCT-16 06.00.50.791058 AM +00:00
LOAD_OPATCH_INVENTORY_1 STOPPED 25-OCT-16 07.11.49.606570 AM +00:00
LOAD_OPATCH_INVENTORY_2 DISABLED 25-OCT-16 07.11.50.543851 AM +00:00

Clear all existing job entries

SQL> exec DBMS_SCHEDULER.DROP_JOB(‘LOAD_OPATCH_INVENTORY’);
PL/SQL procedure successfully completed.
SQL> exec DBMS_SCHEDULER.DROP_JOB(‘LOAD_OPATCH_INVENTORY_1’);
PL/SQL procedure successfully completed.
SQL> exec DBMS_SCHEDULER.DROP_JOB(‘LOAD_OPATCH_INVENTORY_2’);
PL/SQL procedure successfully completed.
SQL> select job_name,state, start_date from dba_scheduler_jobs where job_name like ‘LOAD_OPATCH%’;
no rows selected

Re-run datapatch

db02:orcl01:/u01/product/12.1.0.2/database/OPatch>./datapatch -verbose
Posted in datapatch issues, Oracle Administration, Oracle OPatch | Tagged , , | Leave a comment