Oracle clone failing with INS-40426

Recently I’ve seen this error in one of the clone operations.

— Copied the binary tar and executed

export ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid; export PATH=$ORACLE_HOME/bin:$PATH; export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32:$LD_LIBRARY_PATH; export LIBPATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32:$LIBPATH; export TNS_ADMIN=$ORACLE_HOME/network/admin;

runInstaller -silent -clone ORACLE_HOME=/u01/app/oracle/product/12.1.0/grid ORACLE_BASE=/u01/app/oracle ORACLE_HOME_NAME=Ora12102_Grid

This invoked runInstaller from  /u01/app/oracle/product/12.1.0/grid.

Oracle documentation was not really helpful. Metalink did not had any references to this.

INS-40426: Grid installation option has not been specified.

Cause: n/a
Action: n/a

But when executed from /u01/app/oracle/product/12.1.0/grid/oui/bin, it worked.

This error is caused due to “-formCluster” parameters which is passed in $ORACLE_HOME/runInstaller

This script defaults BUNDLE to value “crs”, which executes a CLI with a parameter -formCluster

BUNDLE=crs
….
….
….
case “$BUNDLE” in

crs)
$CMDDIR/install/.oui $* -formCluster -J-Doracle.install.setup.workDir=$CWDDIR -J-D${CVU_OS_SETTINGS}

Posted in 12c Upgrade issues, Oracle Database Cloning, Oracle Software Install / Deinstall | Tagged , , , , , , , , | Leave a comment

12.1.0.2 runcluvfy.sh – Check for I/O Completion Ports (IOCP) device status failed

On AIX platform, 12.1.0.2 runcluvfy.sh fails with following error

$ runcluvfy.sh stage -pre crsinst -upgrade -src_crshome /u01/app/oracle/product/11.2.0.4/grid -dest_crshome /u01/app/oracle/product/12.1.0.2/grid -dest_version 12.1.0.2.0

Starting check for I/O Completion Ports (IOCP) device status …
ERROR:
PRVE-10167 : I/O Completion Ports (IOCP) device status did not match the required value on node “node-01”. [Expected = “Available”; Found = “”]
PRVE-10167 : I/O Completion Ports (IOCP) device status did not match the required value on node “node-02”. [Expected = “Available”; Found = “”]
Check for I/O Completion Ports (IOCP) device status failed

As per metalink – The issue is being investigated in the following bug:
BUG 20225311 – PRVE-10167 : I/O COMPLETION PORTS (IOCP) [EXPECTED = “AVAILABLE”; FOUND = “”]

Problem is checkIOCPDeviceStatus.sh script (in /tmp/CVU_12.1.0.2.0_oracle). This script is using wrong path for lsdev command.

$ /tmp/CVU_12.1.0.2.0_oracle>cat checkIOCPDeviceStatus.sh | grep “LSDEV=”
LSDEV=”/etc/lsdev”

In AIX, actual path is /usr/sbin/lsdev

$ which lsdev
/usr/sbin/lsdev

Check the output of lsdev command

$ lsdev -c iocp
iocp0 Available I/O Completion Ports

If it is showing “Available”, this error can be ignored.

Posted in 12c Upgrade issues | Tagged , , , , | Leave a comment

TNS-00519: Operating system resource quota exceeded (Linux Error: 28: No space left on device)

Oracle restart stack failed to start completely (i.e. partially it was running) –

i101

Check at OS –

oradb:testdb:/usr/oracle>ps -ef | grep ora
oracle 23374 1 0 01:43 ? 00:00:10 /u01/app/oracle/product/11.2.0.3/grid/bin/ohasd.bin reboot
oracle 23534 1 0 01:43 ? 00:00:01 /u01/app/oracle/product/11.2.0.3/grid/bin/cssdagent
oracle 23566 1 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/ocssd.bin
oracle 23588 1 0 01:43 ? 00:00:03 /u01/app/oracle/product/11.2.0.3/grid/bin/oraagent.bin
oracle 23622 1 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/evmd.bin
oracle 23655 23622 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/evmlogger.bin -o /u01/app/oracle/product/11.2.0.3/grid/evm/log/evmlogger.info -l /u01/app/oracle/product/11.2.0.3/grid/evm/log/evmlogger.log

Manual LISTENER start failed with following error stack –

LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 21-JAN-2014 03:05:58
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Starting /u01/app/oracle/product/11.2.0.3/grid/bin/tnslsnr: please wait…
TNSLSNR for Linux: Version 11.2.0.3.0 – Production
System parameter file is /u01/app/oracle/product/11.2.0.3/grid/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oradb/listener/alert/log.xml
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
TNS-12549: TNS:operating system resource quota exceeded
TNS-12560: TNS:protocol adapter error
TNS-00519: Operating system resource quota exceeded
Linux Error: 28: No space left on device
Reference MOS note –
Listener / Database Startup fails with underlying OS error Linux Error: 28: No Space Left On Device (Doc ID 553709.1)

/var/log/message was indeed throwing errors related to /var filesystem –

Jan 21 03:11:32 oradb rpc.statd[3277]: Failed to insert: creating /var/lib/nfs/sm/st-1.net: No space left on device
Jan 21 03:11:32 oradb rpc.statd[3277]: STAT_FAIL to oradb for SM_MON of x.x.x.x
Jan 21 03:11:32 oradb kernel: [357911.542167] lockd: cannot monitor x.x.x.x

One of the files in /var/log was sized much bigger than the actual filesystem size

-rw-r–r– 1 root tty 321205298632 2013-01-21 03:14 lastlog

As described in the note, problem was due to exhausted inodes for /var filesystem. With help of system administrators, file recycled and now it’s of a reasonable size

-rw-r–r– 1 root tty 2628584 2013-01-21 00:28 lastlog

Post this change oracle restart stack could be started.

oradb:+ASM:/usr/oracle>crsctl start has

i102

 

Posted in Oracle Administration, Oracle Restart, Startup issues | Tagged , , , | Leave a comment

Enable flashback fails with ORA-01153

I was quickly running out of space in the flash recovery area & decided to turn off flashback logging.

SQL> alter database flashback off;
Database altered.

But later while enabling, it failed with following error

SQL> alter database flashback on;
alter database flashback on
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

When checked, an incremental RMAN datafilecopy merge backup was running

test

After backups finished, it allowed me to enable the flashback.

SQL> alter database flashback on;
Database altered.

 

Posted in Oracle Database Backup, Oracle flashback database, Oracle Recovery Manager | Leave a comment

ORA-27211: Failed to load Media Management Library

Recently after database out-of-place upgrade to a higher version (11.2.0.x to 11.2.0.x+1), TAPE backup started failing with following error

RMAN> ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE SBT;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ORA_MAINT_SBT_TAPE_2 channel at 01/02/2012 02:27:30
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library

As this was an out-of-place upgrade, ORACLE_HOME has changed. Due to ORACLE_HOME change, link between oracle MML & vendor libraries is broken.

1. Verify the link

Db-01:ORCL:/u01/app/oracle/product/11.2.0.x/database/lib>ls -al | grep obk | wc -l
0

2. Create the required link

$ ln -s /usr/lib/libobk64.a $ORACLE_HOME/lib/libobk.a

3. Verify the link again

Db-01:ORCL:/u01/app/oracle/product/11.2.0.x/database/lib>ls -al | grep obk
lrwxrwxrwx 1 oracle dba 19 Feb 01 05:45 libobk.a -> /usr/lib/libobk64.a
NOTE: I’m using tdpo (Tivoli/TDP for oracle) as my TAPE library solution. The steps I’ve mentioned are specific to tdpo. Location for other media vendor library files will be different.

Once this link was established, I was able to run the TAPE backups again

RMAN> ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE SBT;
using target database control file instead of recovery catalog
allocated channel: ORA_MAINT_SBT_TAPE_1
channel ORA_MAINT_SBT_TAPE_1: SID=25 device type=SBT_TAPE
channel ORA_MAINT_SBT_TAPE_1: Data Protection for Oracle: version 5.5.2.0
Posted in Oracle Database Backup, Oracle Database Upgrade, Oracle Recovery Manager | Tagged , , | Leave a comment

Oracle Restart upgrade fails with INS-40406 – The installer detects no existing Oracle Grid Infrastructure software

While upgrading 11203 Oracle restart environment to 11204, screen 2 – Selected option – “Upgrade oracle grid infra or oracle ASM”, fails with INS-40406 – The installer detects no existing Oracle Grid Infrastructure software Error.

Untitled

Following inventory entries were present, when upgrade started.

db-xxx-01:/u01/app/oraInventory/ContentsXML>cat inventory.xml
<?xml version=”1.0″ standalone=”yes” ?>
<!– Copyright (c) 1999, 2010, Oracle. All rights reserved. –>
<!– Do not modify the contents of this file by hand. –>
<INVENTORY>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
<VERSION_INFO>
<SAVED_WITH>11.1.0.8.0</SAVED_WITH>
<MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME=”Ora11gR2_DB” LOC=”/u01/app/oracle/product/11.2.0.3/database” TYPE=”O” IDX=”1″/>
<HOME NAME=”Ora11gR2_GI” LOC=”/u01/app/oracle/product/11.2.0.3/grid” TYPE=”O” IDX=”2″/>
</HOME_LIST>
</INVENTORY>

To fix the issue, use following CLI

$ runInstaller -updateNodeList ORACLE_HOME=”/u01/app/oracle/product/11.2.0.3/grid” CRS=true

Typically one would use updateNodeList flag when we are using RAC. But it also worked with Oracle restart environment.

Post fix, inventory contents were as following.

db-xxx-01:/u01/app/oraInventory/ContentsXML>cat inventory.xml
<?xml version=”1.0″ standalone=”yes” ?>
<!– Copyright (c) 1999, 2011, Oracle. All rights reserved. –>
<!– Do not modify the contents of this file by hand. –>
<INVENTORY>
<VERSION_INFO>
<SAVED_WITH>11.2.0.3.0</SAVED_WITH>
<MINIMUM_VER>2.1.0.6.0</MINIMUM_VER>
</VERSION_INFO>
<HOME_LIST>
<HOME NAME=”Ora11gR2_DB” LOC=”/u01/app/oracle/product/11.2.0.3/database” TYPE=”O” IDX=”1″/>
<HOME NAME=”Ora11gR2_GI” LOC=”/u01/app/oracle/product/11.2.0.3/grid” TYPE=”O” IDX=”2″ CRS=”true”/>
</HOME_LIST>
<COMPOSITEHOME_LIST>
</COMPOSITEHOME_LIST>
</INVENTORY>

Difference is CRS=”true” in the grid entry. Post this change GUI allowed me to proceed further and install/upgrade 11204 binaries.

Posted in Oracle Database Upgrade, Oracle Software Inventory | Tagged , , | Leave a comment

Database won’t start; Problem with controlfile on NFS

After physical server reboot, database fail to start with error ORA-27086: unable to lock file – already in use.

ORA-00210: cannot open the specified control file
ORA-00202: control file: ‘/u02/data/testdb/control_testdb_02.ctl’
ORA-27086: unable to lock file – already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8

This typically happens with NFS mounts. After rebooting server, for some reason file descriptor(s) are left open on the storage side, causing this error.

To verify the problem, check the type of filesystem where control file resides.

mysrv:testdb:/home/oracle>mount | grep u02
x.x.x.x:/fs_u02 on /u02 type nfs (rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp,addr=x.x.x.x)

There are couple of ways to resolve this problem.

1. Sysadmins / storage admins can release the lock from storage side. Details in Doc ID 145194.1 – ORA-1157 ORA-1110 ORA-27086 Starting up Database

2. We as a DBA can rename the existing file and create a copy

mv /u02/data/testdb/control_testdb_02.ctl /u02/data/testdb/control_testdb_02.ctl.orig
cp /u02/data/testdb/control_testdb_02.ctl.orig /u02/data/testdb/control_testdb_02.ctl

This creates a new file descriptor at storage side and will allow database to start.

Posted in Oracle Administration, Startup issues | Tagged , , , | Leave a comment