Oracleasm configuration issues in RAC environment

I was configuring ASM on Linux with ASMLIB in RAC environment. Identified the disks (sdcm, sdcn, sdco, sdcq, sdcr, sdcs, sdct, sdcu, sdcv, sdcw, sdcx, sdcy) and created required partitions on it.

# fdisk  /dev/sdcq
Command (m for help): d
Partition number (1-8): 8
Command (m for help): d
Partition number (1-8): 3
If you want to maintain SunOS/Solaris compatibility, consider leaving this
partition as Whole disk (5), starting at 0, with 524271616 sectors
Command (m for help): n
Partition number (1-8): 1
First cylinder (0-63998): 1
Last cylinder or +size or +sizeM or +sizeK (1-63998, default 63998):
Using default value 63998
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

Now I’ve following disks, which will be used for creating ASMLIB disks

brw-rw---- 1 root disk 69, 161 Nov 23 17:53 sdcm1
brw-rw---- 1 root disk 69, 177 Nov 23 17:54 sdcn1
brw-rw---- 1 root disk 69, 193 Nov 23 17:54 sdco1
brw-rw---- 1 root disk 69, 225 Nov 23 17:55 sdcq1
brw-rw---- 1 root disk 69, 241 Nov 23 17:55 sdcr1
brw-rw---- 1 root disk 70,   1 Nov 23 17:56 sdcs1
brw-rw---- 1 root disk 70,  17 Nov 23 17:56 sdct1
brw-rw---- 1 root disk 70,  33 Nov 23 17:56 sdcu1
brw-rw---- 1 root disk 70,  49 Nov 23 17:56 sdcv1
brw-rw---- 1 root disk 70,  65 Nov 23 17:57 sdcw1
brw-rw---- 1 root disk 70,  81 Nov 23 17:57 sdcx1
brw-rw---- 1 root disk 70,  97 Nov 23 17:57 sdcy1

# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

Create ASMLIB disks

# oracleasm createdisk asmdsk1 /dev/sdcm1
# oracleasm createdisk asmdsk2 /dev/sdcn1
# oracleasm createdisk asmdsk3 /dev/sdco1
# oracleasm createdisk asmdsk4 /dev/sdcq1
# oracleasm createdisk asmdsk5 /dev/sdcr1
# oracleasm createdisk asmdsk6 /dev/sdcs1
# oracleasm createdisk asmdsk7 /dev/sdct1
# oracleasm createdisk asmdsk8 /dev/sdcu1
# oracleasm createdisk asmdsk9 /dev/sdcv1
# oracleasm createdisk asmdsk10 /dev/sdcw1
# oracleasm createdisk asmdsk11 /dev/sdcx1

Check created disks

# oracleasm listdisks
ASMDSK1
ASMDSK10
ASMDSK11
ASMDSK2
ASMDSK3
ASMDSK4
ASMDSK5
ASMDSK6
ASMDSK7
ASMDSK8
ASMDSK9

On the other RAC nodes, these disks will only be seen when we execute oracleasm scandisks

# oracleasm listdisks
# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

But the scandisks was running forever and not discovering the disks. When I checked for the partition device existence on other nodes, they were missing.

So to refresh the partition information (this is specific to Linux), I’ve used partprobe command on all other nodes. With this command all the disks are scanned and the partition information was refreshed, which also created the devices under /dev. I guess this can be achieved by bouncing the servers, as at boot time, disks will be scanned and device tree will be rebuilt. But this is not the viable option.

# partprobe /dev/sdcm
Warning: The disk CHS geometry (32635,255,63) reported by the operating 
system does not match the geometry stored on the disk label (10878,255,189).
# partprobe /dev/sdcn
Warning: The disk CHS geometry (32635,255,63) reported by the operating 
system does not match the geometry stored on the disk label (10878,255,189).

After partprobe, when I again executed oracleasm scandisks, it finished quickly and disks were now visible on other nodes.

# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMDSK1"
Instantiating disk "ASMDSK2"
Instantiating disk "ASMDSK3"
Instantiating disk "ASMDSK4"
Instantiating disk "ASMDSK5"
Instantiating disk "ASMDSK6"
Instantiating disk "ASMDSK7"
Instantiating disk "ASMDSK8"
Instantiating disk "ASMDSK9"
Instantiating disk "ASMDSK10"
Instantiating disk "ASMDSK11"

 

I tried another approach, which also worked

From the node, where I’ve created the devices I noted down the device major and minor number.

brw-rw---- 1 root disk 69, 161 Nov 23 17:53 sdcm1
brw-rw---- 1 root disk 69, 177 Nov 23 17:54 sdcn1

Now on other nodes, I’ve manually created the devices using mknod & same major, minor numbers.

# /dev/oracleasm/disks>  mknod ASMDSK1 b 69 161 
# /dev/oracleasm/disks>  mknod ASMDSK2 b 69 177

After this executed oracleasm scandisks, which identified the disks. But I did not wanted to follow this approach (as we may have different major, minor numbers across nodes).  We also need to be sure that, there is no existing device with same major, minor combination.

There is no risk as such, as ASM is intelligent enough to figure out valid device links. If we create a mknod link to a non-existing device, then oracleasm scandisks will clean up the devices.

About these ads
This entry was posted in Oracle Automatic Storage Management and tagged , , . Bookmark the permalink.

3 Responses to Oracleasm configuration issues in RAC environment

  1. arpit says:

    I’m trying to create Oracle 10g ASM instance on AIX 5.3. So is it necessary to create a volume group other than rootvg..
    I already having 10 disk extra in my storage.. so can I directly assign it to ASM in discovery path as ‘/dev/rhdisk1′,…../
    Can u please support in same..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s