I was configuring ASM on Linux with ASMLIB in RAC environment. Identified the disks (sdcm, sdcn, sdco, sdcq, sdcr, sdcs, sdct, sdcu, sdcv, sdcw, sdcx, sdcy) and created required partitions on it.
# fdisk /dev/sdcq Command (m for help): d Partition number (1-8): 8 Command (m for help): d Partition number (1-8): 3 If you want to maintain SunOS/Solaris compatibility, consider leaving this partition as Whole disk (5), starting at 0, with 524271616 sectors Command (m for help): n Partition number (1-8): 1 First cylinder (0-63998): 1 Last cylinder or +size or +sizeM or +sizeK (1-63998, default 63998): Using default value 63998 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
Now I’ve following disks, which will be used for creating ASMLIB disks
brw-rw---- 1 root disk 69, 161 Nov 23 17:53 sdcm1 brw-rw---- 1 root disk 69, 177 Nov 23 17:54 sdcn1 brw-rw---- 1 root disk 69, 193 Nov 23 17:54 sdco1 brw-rw---- 1 root disk 69, 225 Nov 23 17:55 sdcq1 brw-rw---- 1 root disk 69, 241 Nov 23 17:55 sdcr1 brw-rw---- 1 root disk 70, 1 Nov 23 17:56 sdcs1 brw-rw---- 1 root disk 70, 17 Nov 23 17:56 sdct1 brw-rw---- 1 root disk 70, 33 Nov 23 17:56 sdcu1 brw-rw---- 1 root disk 70, 49 Nov 23 17:56 sdcv1 brw-rw---- 1 root disk 70, 65 Nov 23 17:57 sdcw1 brw-rw---- 1 root disk 70, 81 Nov 23 17:57 sdcx1 brw-rw---- 1 root disk 70, 97 Nov 23 17:57 sdcy1
# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
Create ASMLIB disks
# oracleasm createdisk asmdsk1 /dev/sdcm1 # oracleasm createdisk asmdsk2 /dev/sdcn1 # oracleasm createdisk asmdsk3 /dev/sdco1 # oracleasm createdisk asmdsk4 /dev/sdcq1 # oracleasm createdisk asmdsk5 /dev/sdcr1 # oracleasm createdisk asmdsk6 /dev/sdcs1 # oracleasm createdisk asmdsk7 /dev/sdct1 # oracleasm createdisk asmdsk8 /dev/sdcu1 # oracleasm createdisk asmdsk9 /dev/sdcv1 # oracleasm createdisk asmdsk10 /dev/sdcw1 # oracleasm createdisk asmdsk11 /dev/sdcx1
Check created disks
# oracleasm listdisks ASMDSK1 ASMDSK10 ASMDSK11 ASMDSK2 ASMDSK3 ASMDSK4 ASMDSK5 ASMDSK6 ASMDSK7 ASMDSK8 ASMDSK9
On the other RAC nodes, these disks will only be seen when we execute oracleasm scandisks
# oracleasm listdisks # oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks...
But the scandisks was running forever and not discovering the disks. When I checked for the partition device existence on other nodes, they were missing.
So to refresh the partition information (this is specific to Linux), I’ve used partprobe command on all other nodes. With this command all the disks are scanned and the partition information was refreshed, which also created the devices under /dev. I guess this can be achieved by bouncing the servers, as at boot time, disks will be scanned and device tree will be rebuilt. But this is not the viable option.
# partprobe /dev/sdcm Warning: The disk CHS geometry (32635,255,63) reported by the operating system does not match the geometry stored on the disk label (10878,255,189). # partprobe /dev/sdcn Warning: The disk CHS geometry (32635,255,63) reported by the operating system does not match the geometry stored on the disk label (10878,255,189).
After partprobe, when I again executed oracleasm scandisks, it finished quickly and disks were now visible on other nodes.
# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "ASMDSK1" Instantiating disk "ASMDSK2" Instantiating disk "ASMDSK3" Instantiating disk "ASMDSK4" Instantiating disk "ASMDSK5" Instantiating disk "ASMDSK6" Instantiating disk "ASMDSK7" Instantiating disk "ASMDSK8" Instantiating disk "ASMDSK9" Instantiating disk "ASMDSK10" Instantiating disk "ASMDSK11"
I tried another approach, which also worked
From the node, where I’ve created the devices I noted down the device major and minor number.
brw-rw---- 1 root disk 69, 161 Nov 23 17:53 sdcm1 brw-rw---- 1 root disk 69, 177 Nov 23 17:54 sdcn1
Now on other nodes, I’ve manually created the devices using mknod & same major, minor numbers.
# /dev/oracleasm/disks> mknod ASMDSK1 b 69 161 # /dev/oracleasm/disks> mknod ASMDSK2 b 69 177
After this executed oracleasm scandisks, which identified the disks. But I did not wanted to follow this approach (as we may have different major, minor numbers across nodes). We also need to be sure that, there is no existing device with same major, minor combination.
There is no risk as such, as ASM is intelligent enough to figure out valid device links. If we create a mknod link to a non-existing device, then oracleasm scandisks will clean up the devices.