TNS-00519: Operating system resource quota exceeded (Linux Error: 28: No space left on device)

Oracle restart stack failed to start completely (i.e. partially it was running) –

i101

Check at OS –

oradb:testdb:/usr/oracle>ps -ef | grep ora
oracle 23374 1 0 01:43 ? 00:00:10 /u01/app/oracle/product/11.2.0.3/grid/bin/ohasd.bin reboot
oracle 23534 1 0 01:43 ? 00:00:01 /u01/app/oracle/product/11.2.0.3/grid/bin/cssdagent
oracle 23566 1 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/ocssd.bin
oracle 23588 1 0 01:43 ? 00:00:03 /u01/app/oracle/product/11.2.0.3/grid/bin/oraagent.bin
oracle 23622 1 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/evmd.bin
oracle 23655 23622 0 01:43 ? 00:00:00 /u01/app/oracle/product/11.2.0.3/grid/bin/evmlogger.bin -o /u01/app/oracle/product/11.2.0.3/grid/evm/log/evmlogger.info -l /u01/app/oracle/product/11.2.0.3/grid/evm/log/evmlogger.log

Manual LISTENER start failed with following error stack –

LSNRCTL for Linux: Version 11.2.0.3.0 – Production on 21-JAN-2014 03:05:58
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Starting /u01/app/oracle/product/11.2.0.3/grid/bin/tnslsnr: please wait…
TNSLSNR for Linux: Version 11.2.0.3.0 – Production
System parameter file is /u01/app/oracle/product/11.2.0.3/grid/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oradb/listener/alert/log.xml
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))
TNS-12549: TNS:operating system resource quota exceeded
TNS-12560: TNS:protocol adapter error
TNS-00519: Operating system resource quota exceeded
Linux Error: 28: No space left on device
Reference MOS note –
Listener / Database Startup fails with underlying OS error Linux Error: 28: No Space Left On Device (Doc ID 553709.1)

/var/log/message was indeed throwing errors related to /var filesystem –

Jan 21 03:11:32 oradb rpc.statd[3277]: Failed to insert: creating /var/lib/nfs/sm/st-1.net: No space left on device
Jan 21 03:11:32 oradb rpc.statd[3277]: STAT_FAIL to oradb for SM_MON of x.x.x.x
Jan 21 03:11:32 oradb kernel: [357911.542167] lockd: cannot monitor x.x.x.x

One of the files in /var/log was sized much bigger than the actual filesystem size

-rw-r–r– 1 root tty 321205298632 2013-01-21 03:14 lastlog

As described in the note, problem was due to exhausted inodes for /var filesystem. With help of system administrators, file recycled and now it’s of a reasonable size

-rw-r–r– 1 root tty 2628584 2013-01-21 00:28 lastlog

Post this change oracle restart stack could be started.

oradb:+ASM:/usr/oracle>crsctl start has

i102

 

Advertisements
This entry was posted in Oracle Administration, Oracle Restart, Startup issues and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s