Storage Essentials for Oracle RAC DBA

Today DBA’s are suppose to understand many areas like, Storage, Network, Operating System etc. apart from database skills. One do not expect them to master these skills, but surely expect them to know basics and clearly specify the requirements in these areas. Earlier, I’ve written a blog, discussing few network basics.

In this blog, I’ll discuss few Storage basics.

 

What is WWN?

A World Wide Name or WWN, is a 64-bit address used in fiber channel networks to uniquely identify each element in a Fiber Channel network.

They are further categorized in following 2 categories

WWpN – World Wide port Name, a WWN assigned to a port on a Fabric.
WWnN – World Wide node Name, a WWN assigned to a node on a Fiber Channel fabric.

There are few commands, which will give some information on fiber  channel Eg. fcinfo, luxadm

# fcinfo hba-port
HBA Port WWN: 2100001b32xxxxxx
        OS Device Name: /dev/cfg/c1
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.01.00
        FCode/BIOS Version:  BIOS: 2.2; fcode: 2.1; EFI: 2.0;
        Serial Number: 0402H00-0951797289
        Driver Name: qlc
        Driver Version: 20090519-2.31
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 2000001b32xxxxxx
HBA Port WWN: 2101001b32xxxxxx
        OS Device Name: /dev/cfg/c2
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.01.00
        FCode/BIOS Version:  BIOS: 2.2; fcode: 2.1; EFI: 2.0;
        Serial Number: 0402H00-0951797289
        Driver Name: qlc
        Driver Version: 20090519-2.31
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 2001001b32xxxxxx
HBA Port WWN: 10000000c9xxxxxx
        OS Device Name: /dev/cfg/c3
        Manufacturer: Emulex
        Model: LPe11002-S
        Firmware Version: 2.80×7 (Z3F2.80X7)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999VM0-0849000P81
        Driver Name: emlxs
        Driver Version: 2.40r (2009.06.11.10.50)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000000c9xxxxxx
HBA Port WWN: 10000000c9xxxxxx
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LPe11002-S
        Firmware Version: 2.80×7 (Z3F2.80X7)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999VM0-0849000P81
        Driver Name: emlxs
        Driver Version: 2.40r (2009.06.11.10.50)
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000000c9xxxxxx
Here we have 4 FC channels, out of which 2 are being used(active).

# luxadm -e port
/devices/pci@2,600000/SUNW,qlc@0/fp@0,0:devctl                     NOT CONNECTED
/devices/pci@2,600000/SUNW,qlc@0,1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0:devctl       CONNECTED
/devices/pci@0,600000/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0:devctl     CONNECTED

What is a SAN fabric?

A storage area network (SAN) can be defined as group of storage devices and servers that are connected to a common communication and data transfer infrastructure such as a fiber channel. The common communication and data transfer mechanism for a given deployment is commonly known as the storage fabric.

What is a zone?

When storage is presented to a host or group of hosts, the host WWN and array WWN are logically grouped together which is called a zone. So we’ll have a port coming in from an Array and multiple port coming from different hosts (if the storage to be shared across multiple hosts). Zoning allows us to control the access of LUN’s at different levels. For example I’ve a cluster of 4 nodes.  I’ve two storage requirements

1. All the LUN’s to be shared across all the hosts in the cluster
2. A set of LUN’s to be shared across only 2 hosts

In this situation, we can create two zones. In first zone, we’ll have a WWN from Array and 4 WWN’s from hosts. In second zone, we’ll have a port from Array and only 2 WWN’s from the hosts, where this storage to be presented.
Zone creation is done at FC switch level.  The configuration of zone is normally stored at switch level.
To check the configuration at switch level, we can use following command.

admin> switchshow
switchName:     BRSW1 
switchType:     6.1
switchState:    Online  
switchRole:     Principal
switchDomain:   2       
switchId:       fffc02
switchWwn:      XX:XX:XX:XX:XX:XX:XX:XX
switchBeacon:   OFF                   
port  0: id  No_Light                                    
port  1: id  Online        F-Port  20:00:00:XX:XX:XX:XX:XX
port  2: id  Online        F-Port  20:00:00:XX:XX:XX:XX:XX
port  3: id  Online        F-Port  10:00:00:XX:XX:XX:XX:XX
port  4: id  No_Light     
port  5: id  No_Light     
port  6: id  No_Light     
port  7: id  No_Light     
port  8: id  No_Light     
port  9: id  No_Light     
port 10: id  No_Light     
port 11: id  No_Light     
port 12: id  No_Light                                    
port 13: id  No_Light 
port 14: id  Online        F-Port  50:06:04:XX:XX:XX:XX:XX
port 15: id  Online        F-Port  50:06:04:XX:XX:XX:XX:XX

This command will show the WWN’s of the hosts connected to this switch.
Normally WWN are hard to remember, so at switch level we can create a alias, so that we can easily remember them.

For example

admin> aliCreate “RACNODE1”, “20:00:00:XX:XX:XX:XX:XX”
admin> aliCreate “RACNODE2”, “20:00:00:XX:XX:XX:XX:YY”
admin> aliCreate “RACNODE3”, “20:00:00:XX:XX:XX:XX:ZZ”
admin> aliCreate “RACNODE4”, “20:00:00:XX:XX:XX:XX:AA”
admin> aliCreate “ARRAY-P1”, “50:06:04:XX:XX:XX:XX:OO”
admin> aliCreate “ARRAY-P2”, “50:06:04:XX:XX:XX:XX:KK”
admin> zoneCreate “RACZONE-ARRAY-XYZ”, “RACNODE1;RACNODE2;RACNODE3;RACNODE4; ARRAY-P1; ARRAY-P2”;
admin> zoneCreate “RACZONE-ARRAY-LMN”, “RACNODE1;RACNODE2; ARRAY-P1; ARRAY-P2”;
admin> cfgAdd “FABCONF1”, “RACZONE-ARRAY-XYZ; RACZONE-ARRAY-LMN”;
admin> cfgSave
Updating flash …
san1:admin> cfgEnable “FABCONF1”;
zone config “FABCONF1” is in effect
admin> zoneshow
Defined configuration:
 cfg:   FABCONF1      
                RACZONE-ARRAY-XYZ; RACZONE-ARRAY-LMN;
 zone:  RACZONE-ARRAY-XYZ            
                RACNODE1;RACNODE2;RACNODE3;RACNODE4; ARRAY-P1; ARRAY-P2
 zone:  RACZONE-ARRAY-LMN       
                RACNODE1;RACNODE2; ARRAY-P1; ARRAY-P2
 alias: RACNODE1                   
                20:00:00:XX:XX:XX:XX:XX
 alias: RACNODE2                  
                20:00:00:XX:XX:XX:XX:YY
 alias: RACNODE3                    
                20:00:00:XX:XX:XX:XX:ZZ
 alias: RACNODE4                    
                20:00:00:XX:XX:XX:XX:AA
 alias: ARRAY-P1                     
                50:06:04:XX:XX:XX:XX:OO
 alias: ARRAY-P2                     
                50:06:04:XX:XX:XX:XX:KK
                                      
Effective configuration:
 cfg:   FABCONF1     
 zone:  RACZONE-ARRAY-XYZ       
                20:00:00:XX:XX:XX:XX:XX
  20:00:00:XX:XX:XX:XX:YY
  20:00:00:XX:XX:XX:XX:ZZ
  20:00:00:XX:XX:XX:XX:AA
  50:06:04:XX:XX:XX:XX:OO
 zone:  RACZONE-ARRAY-LMN       
                20:00:00:XX:XX:XX:XX:XX
  20:00:00:XX:XX:XX:XX:YY
  50:06:04:XX:XX:XX:XX:KK

What are the type of Arrays?

Active-Active
=============

Active-Active means that IO being read from or written to a LUN can happen from either of two redundant FC storage controllers.
Active-Passive
==============

Active-Passive as name suggests, one of the FC interface will be active at any given time. If the active interface goes down, surviving FC storage controller will take over.

Active-active configuration is better most of the cases (except price).

What is a LUN?

Logical Unit Number is a device presented from SAN, which is accessed by SCSI or similar protocols such as Fiber Channel.  Normally Array enclosures have many physical disks and a RAID configuration. When a request is placed for specific amount of storage, LUN’s are created & presented to the host. This requested storage may span across multiple physical disks. Host will not know about actual physical disks, on which these LUN’s are created.

As part of LUN creation, we need to specify the OS of the target host.  Then we need to format the LUN’s and finally mask the LUN’s.

LUN Masking is an process, that makes a LUN available only to a host or set of hosts. This is achieved by, restricting access to LUN’s based on the host WWN. Here we need to use array management software, to create a server group which has zone information (host WWN and ARRAY WWN) and LUN’s. Now Specific hosts can see these LUN’s and not all zones where ARRAY WWN is shared.

On the host size, we can run command to re-discover the devices or reboot the server (which does device discovery).
for example in AIX

# cfgmgr -v
# lsdev -Cc disk

hdisk2  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk3  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk4  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk5  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk6  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk7  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk8  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk9  Available 0C-09-02     MPIO Other DS4K Array Disk
hdisk10 Available 0C-09-02     MPIO Other DS4K Array Disk

On Solaris

# devfsadm -Cv
# format

Here disk names are something like c3t2000002037CD9F72d0s0

where

c-part: controller ID of the host bus adapter,
t-part: target ID identifying the SCSI target on that bus,
d-part: disk ID identifying a LUN on that target,
s-part: slice ID identifying a specific slice on that disk.

What is Multipathing?

Multipath I/O is a mechanism to facilitate redundant paths for communication between HOST and the Storage system. This is normally achieved using controllers, switches.

For example,

We may have a HBA card on the host, which is paired with 2 FC ports of array in a zone. So if one of the array FC goes down, traffic will be routed through another array FC. But in this case, host HBA becomes a single point of failure.

So we can have 2 HBA cards on host which are paired with 2 FC ports from array.

We can also have multiple switches, which will provide greater level of resilience.

Few examples of multipathing Software’s

EMC PowerPath
Veritas Dynamic Multi Pathing (DMP)
Solaris Multiplexed I/O (MPxIO)

Advertisements
This entry was posted in Oracle Real Application Cluster and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s