Expert's Corner: June 2009

Wednesday, June 24, 2009

Removing the BootDisk from Veritas volume manager control

There are many circumstances in which it may be necessary to remove the operating system from Veritas Volume Manager control. For years, Veritas included the command vxunroot to accomplish this task; however, it had several restrictions, including the inability to execute if there were any partitions on the rootdisk other than the big-4 (/, /var, swap, and /opt). However, I just discovered that version VxVM 3.5 no longer includes this restriction. I've tested it successfully, and the following description may only be necessary for those systems running earlier versions of VxVM.

Typically, the administrator is often under pressure while performing these types of maintenance. Because simple mistakes at this stage can render the system unusable, it is important that the process be well documented and tested prior to using it in production.

Removing Veritas Volume Manager step-by-step

In the example below, the server pegasus has two internal disks (c0t0d0 and c0t1d0) under Veritas Volume Manager control. The operating system is mirrored between the two devices.

1. From the "ok" prompt, boot from the cdrom device into single-user mode:

ok boot cdrom -s

Boot device: /pci@1f,4000/scsi@3/disk@6,0:f  File and args: -s

SunOS Release 5.8 Version Generic_108528-07 64-bit

Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.

Configuring /dev and /devices

Using RPC Bootparams for network configuration information.

INIT: SINGLE USER MODE

# TERM=vt100; export TERM

2. Perform a filesystem check on the root filesystem:

# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

       0. c0t0d0

          /pci@1f,4000/scsi@3/sd@0,0

       1. c0t1d0

          /pci@1f,4000/scsi@3/sd@1,0

Specify disk (enter its number):

# fsck -y /dev/rdsk/c0t0d0s0

** /dev/rdsk/c0t0d0s0

** Last Mounted on /

** Phase 1 - Check Blocks and Sizes

** Phase 2 - Check Pathnames

** Phase 3 - Check Connectivity

** Phase 4 - Check Reference Counts

** Phase 5 - Check Cyl groups

41356 files, 1047059 used, 5553722 free (4802 frags, 693615 blocks,  0.0% fragmentation)

# mount /dev/dsk/c0t0d0s0 /a

3. Update the /a/etc/system file, removing references to the VxVM root device:

# cp /a/etc/system /a/etc/system.orig

# vi /a/etc/system

The system must not boot from the Veritas volumes. You can either delete the following two lines or change them from:

rootdev:/pseudo/vxio@0:0

set vxio:vol_rootdev_is_volume=1

to:

* rootdev:/pseudo/vxio@0:0

* set vxio:vol_rootdev_is_volume=1

4. Restore the pre-VxVM version of the vfstab file. Note that the vxinstall process makes a backup copy of the original vfstab file named vfstab.prevm. We'll make use of that backup now:

# cp /a/etc/vfstab /a/etc/vfstab.vm

# cp /a/etc/vfstab.prevm /a/etc/vfstab

# cat /a/etc/vfstab

#device         device          mount           FS      fsck    mount   mount

#to mount       to fsck         point           type    pass    at boot options

#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr          ufs     1       yes     -

fd                  -                  /dev/fd fd      -       no      -

/proc               -                  /proc   proc    -       no      -

/dev/dsk/c0t0d0s1   -                  -       swap    -       no      -

/dev/dsk/c0t0d0s0   /dev/rdsk/c0t0d0s0 /       ufs     1       no      logging

/dev/dsk/c0t0d0s4   /dev/rdsk/c0t0d0s4 /var    ufs     1       no      logging

swap                -                  /tmp    tmpfs   -       yes     -

5. If the install-db file exists, the VxVM daemons will not start on bootup:

# touch /a/etc/vx/reconfig.d/state.d/install-db

6. The root-done file indicates that the root disk has been encapsulated. Remove it now:

# rm /a/etc/vx/reconfig.d/state.d/root-done

7. Finally, unmount the root filesystem and reboot:

# cd /; umount /a

# stop-a

8. Boot from the root device

9.             ok boot

Resetting ...

screen not found.

Can't open input device.

Keyboard not present.  Using ttya for input and output.

Sun Ultra 30 UPA/PCI (UltraSPARC-II 296MHz), No Keyboard

OpenBoot 3.27, 512 MB memory installed, Serial #9377973.

Ethernet address 8:0:20:8f:18:b5, Host ID: 808f18b5.

Initializing Memory

Rebooting with command: boot

Boot device: disk  File and args:

SunOS Release 5.8 Version Generic_108528-16 64-bit

Copyright 1983-2001 Sun Microsystems, Inc.  All rights reserved.

NOTICE: VxVM not started

configuring IPv4 interfaces: hme0.

Hostname: pegasus

NOTICE: VxVM not started

NOTICE: VxVM not started

The system is coming up.  Please wait.

starting rpc services: rpcbind done.

Setting netmask of hme0 to 255.255.255.0

Setting default IPv4 interface for multicast: add net 224.0/4: gateway pegasus

Starting sshd...

This platform does not support both privilege separation and compression

Compression disabled

syslog service starting.

VxVM Provider initialization warning: vxinstall has not been run.

Print services started.

/dev/bd.off: not a serial device.

volume management starting.

  No VVR license installed on the system; vradmind not started.

  No VVR license installed on the system; in.vxrsyncd not started.

The system is ready.

pegasus console login: root

Password:

Oct 23 17:41:55 pegasus login: ROOT LOGIN /dev/console

Sun Microsystems Inc.   SunOS 5.8       Generic February 2000

# df -k

Filesystem            kbytes    used   avail capacity  Mounted on

/dev/dsk/c0t0d0s0    6607349 1053634 5487642    17%    /

/proc                      0       0       0     0%    /proc

fd                         0       0       0     0%    /dev/fd

mnttab                     0       0       0     0%    /etc/mnttab

/dev/dsk/c0t0d0s4    1016863  239898  715954    26%    /var

swap                 1401536      16 1401520     1%    /var/run

swap                 1401536      16 1401520     1%    /tmp

At this point, the operating system is no longer under Veritas Volume Manager control. The disk c0t1d0 is unused, and the system would not survive the failure of c0t0d0 if it were to die now.

Note that even though the operating system is no longer under system control, in some cases you might wish to continue running Veritas volume manager for non-os volumes. In this situation a few additional steps are required:

1. Manually start up volume manager processes:

# rm /etc/vx/reconfig.d/state.d/install-db
# vxiod set 10
# vxconfigd -m disable <- So that It will not try to reconfigure/recover anything
# vxdctl enable

2. Confirm that all previous root volumes are accessible:

# vxprint -ht

Disk group: rootdg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID

DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE

RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL

RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK

V  NAME         RVG          KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

dg rootdg       default      default  0        1035409531.1025.pegasus

dm rootdisk     c0t0d0s2     sliced   3359     17690400 -

dm rootmirror   c0t1d0s2     sliced   3359     17690400 -

v  rootvol      -            DISABLED ACTIVE   13423200 ROUND     -        root

pl rootvol-01   rootvol      DISABLED ACTIVE   13423200 CONCAT    -        RW

sd rootdisk-B0  rootvol-01   rootdisk 17690399 1        0         c0t0d0   ENA

sd rootdisk-02  rootvol-01   rootdisk 0        13423199 1         c0t0d0   ENA

pl rootvol-02   rootvol      DISABLED ACTIVE   13423200 CONCAT    -        RW

sd rootmirror-01 rootvol-02  rootmirror 0      13423200 0         c0t1d0   ENA

v  swapvol      -            DISABLED ACTIVE   2096640  ROUND     -        swap

pl swapvol-01   swapvol      DISABLED ACTIVE   2096640  CONCAT    -        RW

sd rootdisk-01  swapvol-01   rootdisk 13426559 2096640  0         c0t0d0   ENA

pl swapvol-02   swapvol      DISABLED ACTIVE   2096640  CONCAT    -        RW

sd rootmirror-02 swapvol-02  rootmirror 13423200 2096640 0        c0t1d0   ENA

……………….. . . . . .

3. Remove the previous root volumes from VxVM:

# vxedit -rf rm rootvol swapvol var

4. Remove the root disk from VxVM:

# vxdg rmdisk rootdisk

5. Finally, restore the original vtoc to c0t0d0:

 /etc/vx/bin/vxedvtoc -f /etc/vx/reconfig.d/disk.d/c0t0d0/vtoc /dev/rdsk/c0t0d0s2

# THE ORIGINAL PARTITIONING IS AS FOLLOWS :

#SLICE     TAG  FLAGS    START     SIZE

 0         0x2  0x200        0 13423200

 1         0x3  0x201 13426560  2096640

 2         0x5  0x200        0 17693760

 3         0xe  0x201        0 17693760

 4         0x7  0x200 15523200  2100000

 5         0x0  0x200        0        0

 6         0x0  0x200        0        0

 7         0xf  0x201 17690400     3360

# THE NEW PARTITIONING WILL BE AS FOLLOWS :

#SLICE     TAG  FLAGS    START     SIZE

 0         0x2  0x200        0 13423200

 1         0x3  0x201 13426560  2096640

 2         0x5  0x200        0 17693760

 3         0x0  0x200        0        0

 4         0x7  0x200 15523200  2100000

 5         0x0  0x200        0        0

 6         0x0  0x200        0        0

 7         0x0  0x200        0        0

DO YOU WANT TO WRITE THIS TO THE DISK ? [Y/N] :y

WRITING THE NEW VTOC TO THE DISK

At this point the root disk c0t0d0 is completely free of VxVM, but any other Veritas volumes can be brought online and mounted if required.

How to test the VCS

Check Veritas Licenses - for FileSystem, Volume Manager AND Cluster

vxlicense -p

If any licenses are not valid or expired -- get them FIXED before continuing! All licenses should say "No expiration". If ANY license has an actual expiration date, the test failed. Permenant licenses do NOT have an expiration date. Non-essential licenses may be moved -- however, a senior admin should do this.

1. Hand check SystemList & AutoStartList

On either machine:

grep SystemList /etc/VRTSvcs/conf/config/main.cf

system1

system2

grep AutoStartList /etc/VRTSvcs/conf/config/main.cf

system1

system2

Each list should contain both machines. If not, many of the next tests will fail.

main.cf

system1

system2

system1

system2

2. Verify Cluster is Running

First verify that veritas is up & running:

hastatus -summary

vi /.profile

. /.profile

hastatus -summary

Here is the expected result (your SYSTEMs/GROUPs may vary):

One system should be OFFLINE and one system should be ONLINE ie:
# hastatus -summary

  -- SYSTEM STATE
 -- System               State                Frozen             

 A  e4500a               RUNNING              0                   
 A  e4500b               RUNNING              0                   

 -- GROUP STATE
 -- Group           System               Probed     AutoDisabled    State         

 B  oragrp          e4500a               Y          N               ONLINE        
 B  oragrp          e4500b               Y          N               OFFLINE

If your systems do not show the above status, try these debugging steps:

If NO systems are up, run hastart on both systems and run hastatus -summary again.
If only one system is shown, start other system with hastart. Note: one system should ALWAYS be OFFLINE for the way we configure systems here. (If we ran oracle parallel server, this could change -- but currently we run standard oracle server)

If both systems are up but are OFFLINE and hastart did NOT correct the problem and oracle filesystems are not running on either system, the cluster needs to be reset. (This happens under strange network situations with GE Access.) [You ran hastart and that wasn't enough to get full cluster to work.]

Verify that the systems have the following EXACT status (though your machine names will vary for other customers):

gedb002# hastatus -summary

-- SYSTEM STATE
-- System               State                Frozen             

A  gedb001              RUNNING              0                   
A  gedb002              RUNNING              0                   

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State       


B  oragrp          gedb001              Y          N               OFFLINE     

B  oragrp          gedb002              Y          N               OFFLINE     

gedb002#  hares -display | grep  ONLINE
nic-qfe3  State           gedb001   ONLINE
nic-qfe3  State           gedb002   ONLINE

gedb002# vxdg list
NAME         STATE           ID
rootdg       enabled  957265489.1025.gedb002

gedb001# vxdg list
NAME         STATE           ID
rootdg       enabled  957266358.1025.gedb001

Recovery Commands:

hastop -all

hastart

hastatus -summary

If none of these steps resolved the situation, contact Lorraine or Luke (possibly Russ Button or Jen Redman if they made it to Veritas Cluster class) or a Veritas Consultant.

3. Verify Services Can Switch Between Systems

Once, hastatus -summary works, note the GROUP name used. Usually, it will be "oragrp", but the installer can use any name, so please determine it's name.

First check if group can switch back and forth. On the system that is running (system1), switch veritas to other system (system2):

hagrp -switch groupname -to system2

Watch failover with hastatus -summary. Once it is failed over, switch it back:

hagrp -switch groupname -to system1

4. Verify OTHER System Can Go Up & Down Smoothly For Maintanence

On system that is OFFLINE (should be system 2 at this point), reboot the computer.

ssh system2

/usr/sbin/shutdown -i6 -g0 -y

Make sure that the when the system comes up & is running after the reboot. That is, when the reboot is finished, the second system should say it is offline using hastatus.

hastatus -summary

Once this is done, hagrp -switch groupname -to system2 and repeat reboot for the other system

hagrp -switch groupname -to system2

ssh system1

/usr/sbin/shutdown -i6 -g0 -y

Verify that system1 is in cluster once rebooted

hastatus -summary

5. Test Actual Failover For System 2 (and pray db is okay)

To do this, we will kill off the listener process, which should force a failover. This test SHOULD be okay for the db (that is why we choose LISTENER) but there is a very small chance things will go wrong .. hence the "pray" part :).

On system that is online (should be system2), kill off ORACLE LISTENER Process

ps -ef | grep LISTENER

Output should be like:

  root  1415   600  0 20:43:58 pts/0    0:00 grep LISTENER
 oracle   831     1  0 20:27:06 ?        0:00 /apps/oracle/product/8.1.5/bin/tnslsnr LISTENER -inherit

kill -9 process-id

Failover will take a few minutes

You will note that system 2 is faulted -- and system 1 is now online

You need to CLEAR the fault before trying to fail back over.

hares -display | grep FAULT

hares -clear resource-name -sys faulted-system

6. Test Actual Failover For System 1 (and pray db is okay)

Now we do same thing for the other system first verify that the other system is NOT faulted

hastatus -summary

Now do the same thing on this system... To do this, we will kill off the listener process, which should force a failover.

On system that is online (should be system2), kill off ORACLE LISTENER Process

ps -ef | grep LISTENER

Output should be like:

  oracle   987     1  0 20:49:19 ?        0:00 /apps/oracle/product/8.1.5/bin/tnslsnr LISTENER -inherit
 root  1330   631  0 20:58:29 pts/0    0:00 grep LISTENER

kill -9 process-id

Failover will take a few minutes

You will note that system 1 is faulted -- and system 1 is now online

You need to CLEAR the fault before trying to fail back over.

hares -display | grep FAULT

hares -clear resource-name -sys faulted-system

Run:

hastatus -summary

to make sure everything is okay.

Configure IPMP network with Multiple NICs in Solaris

IP Multipathing (IPMP) in Sun Solaris enables the load balancing capabilities and resilience for Network connections with multple Network Interface Cards (NIC).

IPMP is mostly used to provide resilience with network connections wherein a Sun Solaris Server with multiple NICs can be configured such that if the primary NIC fails it automatically failover to the secondary NIC on the system. Also, if it is a multi-switch environment then each Network Interface participating in IPMP can be connection to two different switches such that a network connection failure due to a NIC failure or a switch failure can be avoided.

To configure IPMP for redundancy we need to have

1. Virtual IP Address

2. Test IP Address for each NIC.

The Virtual IP is the actual Data IP and the test IPs are the ones used by IPMP to probe a remote target system to check connectivity. The test IP interfaces are marked as “deprecated” which indicates that this IP should be used by applications for any network data transfer. The NICs doesn’t have to be of the same make or have to be the same speed (10/100/1000Mbps). However, they should be of the same kind (Ethernet here)

In our configuration,

192.168.1.100 – Virtual IP

192.168.1.101 – Test IP for ce0 (NIC1)

192.168.1.102 – Test IP for ce1 (NIC2)

appserver – Hostname

appserver-ce0 – Hostname for IP on ce0 interface

appserver-ce1 – Hostname for IP on ce1 interface

Add Host Entries in /etc/hosts

Let’s start with adding the hosts entries for the IP addresses in the /etc/hosts file.

# IPMP group appserver-ipmp
127.0.0.1 localhost
192.168.1.100 appserver loghost
192.168.1.101 appserver-ce0 loghost
192.168.1.102 appserver-ce1 loghost

Create hostname.ce* files

For every interface on the system create a hostname.ce* file. For us, create the files

hostname.ce0 & hostname.ce1

Edit hostname.ce0

Add the following on the hostname.ce0 file. This is the primary or master interface of the IPMP Pair

appserver-ce0 netmask + broadcast + group appserver-ipmp deprecated -failover up\

addif appserver netmask + broadcast + failover up

Edit hostname.ce1

Add the following on the hostname.ce1 file. This is the secondary or slave interface of the IPMP Pair

appserver-ce1 netmask + broadcast + group appserver-ipmp deprecated \
-failover standby up

where

netmask – sets default netmask values

broadcast – sets default broadcast address

-failover – indicates that the test IPs should not be failed over

deprecated – indicates that the interface should not be used for data transfer

Now, the configuration is complete and an ifconfig output should look like as follows:

root@ appserver:/$
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=9040843 mtu 1500 index 2
inet 192.168.1.101 netmask ffffff00 broadcast 192.168.1.255
groupname appserver-ipmp
ether 0:xx:xx:xx:xx:x
ce0:1: flags=1000843 mtu 1500 index 2
inet 192.168.1.100 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=69040843 mtu 1500 index 3
inet 192.168.1.102 netmask ffffff00 broadcast 192.168.1.255
groupname appserver-ipmp
ether 0:xx:xx:xx:xx:x

You can see a new virtual interface for “ce0″ is being created as “ce0:1″ and the “ce1″ interface status is UP but INACTIVE. Indicating it is up and participating in the IPMP but is as a standby. Test by pinging the device continously from a remote PC and unplugging the active NIC and you should see no packets drop. Reverse the tests and check if it works OK. Also, if you are on the console, you can see the live link messages

How to Find number of Physical & Virtual Processors in Solaris 10

In Solaris 10, if you want to find the processor type and the number of physical processors installed on the system and the number of Virtual Processors available on the system then the psrinfo command does job for you.

To simply display the number of Physical processors, simply run the command with the -p option as follows:

root@sunserver # psrinfo -p
2

where 2 implies that there are 2 physical processors installed on the system.

If you would like to check the number of Virtual Processors on each of these Physical processors then type the command with the “-pv” arguement as follows:

root@ server:/root$ uname -a

SunOS server 5.10 Generic_137111-02 sun4v sparc SUNW,SPARC-Enterprise-T5120

root@server:/root$ psrinfo -pv

The physical processor has 32 virtual processors (0-31)

UltraSPARC-T2 (cpuid 0 clock 1165 MHz)

The above indicates that there was only one physical processor (UltraSPARC-T2) on the T5120 server which has 32 Virtual processors. Each virtual processor is an entity with its own interrupt ID, capable of executing independent threads.

In simple terms, the number of Virtual Processors supported by a physical CPU is

“Number of Core” x “Number of threads”

For instance, the above is on a T5120 server with the UltraSPARC-T2. This CPU has 4 cores and each core can support 8 threads and that gives us 32 Virtual processors.

The number of Virtual processors on a Server is simply the total Virtual processors supported on each of the physical processor.

In the following T5140 server there are 2 Physical UltraSPARC-T2+ processors with 6 cores, each supporting 8 threads which means we get 48 Virtual processors per Physical processor and hence a total of 96 Virtual processors (sum of VPs on individual processors) for the server:

root@ bserver:/root$ uname -a

SunOS bserver 5.10 Generic_137111-02 sun4v sparc SUNW,T5140

root@ bserver:/root$ psrinfo -pv

The physical processor has 48 virtual processors (0-23 32-55)

UltraSPARC-T2+ (cpuid 0 clock 1167 MHz)

The physical processor has 48 virtual processors (64-71 80-119)

UltraSPARC-T2+ (cpuid 64 clock 1167 MHz)

In earlier versions of Solaris, the -p arguement is not supported and hence wouldn’t provide this summarised output on the counts of the physical and Virtual CPUs.

How to reset root password in Sun solaris

Boot the server with a Sun Solaris Operating System CD (I’m using a Solaris 10 CD but doesn’t matter really) or a network boot with a JumpStart server from the OBP OK prompt.

OK boot cdrom -s

OK boot net -s

This will boot the server from the CD or Jumpstart server and launch a single user mode (No Password).

Mount the root file system (assume /dev/dsk/c0t0d0s0 here) onto /a

solaris# mount /dev/dsk/c0t0d0s0 /a

NOTE: /a is a temporary mount point that is available when you boot from CD or a JumpStart server

Now, with the root file system mounted on /a. All you need to do is to edit the shadow file and remove the encrypted password for root.

solaris# vi /a/etc/shadow

Now, exit the mounted filesystem, unmount the root filesystem and reboot the system to single-user mode booting of the disk.

solaris# cd /

solaris# umount /a

solaris# init s

This should boot of the disk and take you to the single-user mode. Press enter at the prompt to enter a password for root.

This should allow you to login to the system. Once in, set the password and change to multi-user mode.

NOTE: Single-User mode is only to ensure that the root user without password is not exposed to others if started in multi-user mode before being set with a new password.

solaris# passwd root

solaris# reboot

This should do. You should now be able to logon with the new password set for root

How to find the WWN in Solaris

World Wide Name (WWN) are unique 8 byte (64-bit) identifiers in SCSI or fibre channel similar to that of MAC Addresses on a Network Interface Card (NIC).

Talking about the WWN names, there are also

World Wide port Name (WWpN), a WWN assigned to a port on a Fabric which is what you would be looking for most of the time.

World Wide node Name (WWnN), a WWN assigned to a node/device on a Fibre Channel fabric.

To find the WWN numbers of your HBA card in Sun Solaris, you can use one the following procedures

Using fcinfo (Solaris 10 only)

This is probably the easiest way to find the WWN numbers on your HBA card. Here you can see the HBA Port WWN (WWpN) and the Node WWN (WWnN) of the two ports on the installed Qlogic HAB card.

This is also useful in finding the Model number, Firmwar version FCode, supported and current speeds and the port status of the HBA card/port.

root@ sunserver:/root# fcinfo hba-port | grep WWN
HBA Port WWN: 2100001b32xxxxxx
Node WWN: 2000001b32xxxxxx
HBA Port WWN: 2101001b32yyyyyy
Node WWN: 2001001b32yyyyyy

For detailed info including Make & model number, Firmware, Fcode and current status and supported/current speeds then

root@ sunserver:/root# fcinfo hba-port
HBA Port WWN: 2100001b32xxxxxx
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3356-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b32xxxxxx
HBA Port WWN: 2101001b32yyyyyy
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3356-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: not established
Node WWN: 2001001b32yyyyyy

Using scli

root@ sunserver:/root# scli -i | egrep “Node Name|Port Name”
Node Name : 20-00-00-1B-32-XX-XX-XX
Port Name : 21-00-00-1B-32-XX-XX-XX
Node Name : 20-01-00-1B-32-YY-YY-YY
Port Name : 21-01-00-1B-32-YY-YY-YY

For more detailed info on the HBA Cards run as follows: Similar to fcinfo but also provides Model Name and serial number.

root@ sunserver:/root# scli -i
——————————————————————————
Host Name : sunserver
HBA Model : QLE2462
HBA Alias :
Port : 1
Port Alias :
Node Name : 20-00-00-1B-32-XX-XX-XX
Port Name : 21-00-00-1B-32-XX-XX-XX
Port ID : 11-22-33
Serial Number : AAAAAAA-bbbbbbbbbb
Driver Version : qlc-20080514-2.28
FCode Version : 1.24
Firmware Version : 4.04.01
HBA Instance : 2
OS Instance : 2
HBA ID : 2-QLE2462
OptionROM BIOS Version : 1.24
OptionROM FCode Version : 1.24
OptionROM EFI Version : 1.08
OptionROM Firmware Version : 4.00.26
Actual Connection Mode : Point to Point
Actual Data Rate : 2 Gbps
PortType (Topology) : NPort
Total Number of Devices : 2
HBA Status : Online
——————————————————————————
Host Name : sunserver
HBA Model : QLE2462
HBA Alias :
Port : 2
Port Alias :
Node Name : 20-01-00-1B-32-YY-YY-YY
Port Name : 21-01-00-1B-32-YY-YY-YY
Port ID : 00-00-00
Serial Number : AAAAAAA-bbbbbbbbbb
Driver Version : qlc-20080514-2.28
FCode Version : 1.24
Firmware Version : 4.04.01
HBA Instance : 3
OS Instance : 3
HBA ID : 3-QLE2462
OptionROM BIOS Version : 1.24
OptionROM FCode Version : 1.24
OptionROM EFI Version : 1.08
OptionROM Firmware Version : 4.00.26
Actual Connection Mode : Unknown
Actual Data Rate : Unknown
PortType (Topology) : Unidentified
Total Number of Devices : 0
HBA Status : Loop down

Using prtconf

root@ sunserver:/root# prtconf -vp | grep -i wwn
port-wwn: 2100001b.32xxxxxx
node-wwn: 2000001b.32xxxxxx
port-wwn: 2101001b.32yyyyyy
node-wwn: 2001001b.32yyyyyy

Using prtpicl

root@ sunserver:/root# prtpicl -v | grep wwn
:node-wwn 20 00 00 1b 32 xx xx xx
:port-wwn 21 00 00 1b 32 xx xx xx
:node-wwn 20 01 00 1b 32 yy yy yy
:port-wwn 21 01 00 1b 32 yy yy yy

Using luxadm

Run the following command to obtain the physical path to the HBA Ports

root@ sunserver:/root$ luxadm -e port
/devices/pci@400/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl CONNECTED
/devices/pci@400/pci@0/pci@9/SUNW,qlc@0,1/fp@0,0:devctl NOT CONNECTED

With the physical path obtained from the above command, we can trace the WWN numbers as follows. here I use the physical path to the one that is connected:

root@ sunserver:/root$ luxadm -e dump_map /devices/pci@400/pci@0/pci@9/SUNW,qlc@0/fp@0,0:devctl
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 123456 0 1111111111111111 2222222222222222 0×0 (Disk device)
1 789123 0 1111111111111111 2222222222222222 0×0 (Disk device)
2 453789 0 2100001b32xxxxxx 2000001b32xxxxxx 0×1f (Unknown Type,Host Bus Adapter)

Enable/Disable IP Forwarding in Solaris 10 without reboot

IP packet forwarding is the process of routing packets between network interfaces on one system. A packet arriving on one network interface and addressed to a host on a different network is forwarded to the appropriate interface.

In Solaris 10, IP Forwarding can be enabled or disabled using the routeadm & ifconfig commands as against the ndd commands in Solaris 9 and earlier. The advantage is the change dynamic and real-time and the change persist across reboot unlike the ndd command.

Enable/Disable IP Forwarding globally

To globally enable IP Forwarding in Solaris 10 use the routeadm command as follows:

In IPv4

solaris10# routeadm -e ipv4-forwarding

In IPv6

solaris10# routeadm -e ipv6-forwarding

The switches “-e” enables IP Forwarding.

To disable IP Forwarding

In IPv4

solaris10# routeadm -d ipv4-forwarding

In IPv6

solaris10# routeadm -d ipv6-forwarding

The switches “-d” enables IP Forwarding.

After the change run the following command for the changes to take effect.

solaris10# routeadm -u

Enable/Disable IP Forwarding per interface

To enable IP Forwarding on a specific interface (say bge0) using the ifconfig command

In IPv4

solaris10# ifconfig bge0 router

In IPv6

solaris10# ifconfig bge0 inet6 router

To disable IP Forwarding for an interface (say bge0)

In IPv4

solaris10# ifconfig bge0 -router

In IPv6

solaris10# ifconfig bge0 inet6 -router

How to add a Static Route in Sun Solaris

To add a Static Route in Sun Solaris operating system, you can use the route command. This will dynamically update the Kernel IP Routing table. However, when a server is restarted, these routes will be lost. To prevent this from happening, add a startup script S76static-routes with all the route commands for the static route that needs to persist. This will ensure that the route gets added at boot time.

To use the route command,

Syntax:

# route add [net|host] netmask [GatewayAddr|-interface ]

Example:

Add a network

# route add net 10.10.10.0 netmask 255.255.255.0 192.168.1.1 1

same as

# route add 10.10.10.0/24 192.168.1.1 1

Add a host

# route add host 1.1.1.1 netmask 255.255.255.0 192.168.1.1 1

same as

# route add 1.1.1.1/24 192.168.1.1 1

To route the traffic through an interface instead of an IP Gateway

# route add 1.1.1.1/24 -interface hme0

To check that the roots are added to Kernel IP Routing table,

# netstat -rn

Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
——————– ——————– —– —– —— ———
192.168.1.0 192.168.1.1 U 1 273 hme0
224.0.0.0 192.168.1.1 U 1 0 hme0
default 192.168.1.1 UG 1 196

Static Routes at boot time

To make the routes available at boot time so the next time when the server reboots, the routes are still available. Add a startup script named as

/etc/rc2.d/S76static-routes

and add the required route commands as above.

Change the permissions for the file so that the file is executable by root.

# chmod 744 /etc/rc2.d/S76static-routes

mount an ISO image in Linux

You can mount an ISO image onto your operating system instead of burning it onto a CD or a DVD. This can come handy and easy in situations and there you have the advantage of not wasting a media. A classic example would be when setting up a Jumpstart setup.

Let’s say that we have a ISO image of a CD named opensuse.iso in the directory /software

To mount it

opensuse11:~ # mount -o loop /software/opensuse.iso /mnt

where /mnt is the mount point.

This is equivalent to

opensuse11:~ # mount -t iso9660 -o loop /software/opensuse.iso /mnt

Now you can browse the contents of the ISO image by changing directory to /mnt

opensuse11:~ # cd /mnt

This should mount the ISO image. While this article is focussed on SUSE Linux and openSUSE, should work on most of the linux distributions including Ubuntu Fedora.

How to mount an ISO image in SUN Solaris

In Sun Solaris, we can use the Loopback File driver to mount an ISO image without having to write the ISO image onto a CD or DVD.

Following procedure should help you mount an ISO image in Sun Solaris

Attach a Block Device

sunsolaris# lofiadm -a /export/software/iso_image.iso
/dev/lofi/1

Mount the ISO Image block device

sunsolaris# mount -F hsfs -o ro /dev/lofi/1 /mnt

Where /mnt is the mount point.

This should mount the ISO image.

To confirm, change directory to /mnt and do a “ls” to lis the files

sunsolaris# cd /mnt

sunsolaris# ls

if at anytime, you want to look at these block devices simply type “lofiadm” command with no arguement.

sunsolaris# lofiadm
Block Device File
/dev/lofi/1 /export/software/iso_image.iso

When we are done with the files on the mounted ISO, we can unmount and detach the Block device we attached earlier:

sunsolaris# umount /mnt

sunsolaris# lofiadm -d /dev/lofi/1

SSH login without password on Solaris

If you want to log onto Server A, and then ssh to Server B without using a password, do the following...

1) Log onto Server A

2) Type: cd .ssh

3) Type: ssh-keygen -t rsa

Hit return for all prompts. Use default file name. Do NOT supply a passphrase. This will create both private (id_rsa) and public (id_rsa.pub) key files.

4) Type: cat id_rsa.pub | ssh userB@machineB "cat - >> .ssh/authorized_keys"

You will be prompted for the password of userB, go ahead and type it.

5) At this point, you are done. You will be able to connect to userB on server B, FROM userA on severA without supplying a password.

6) Test: ssh userB@serverB

This should log you onto Server B with no password.

Do the same from Server B to Server A, if you want to be able to do it from either direction

Test the failover of a NIC within a MultiNICA resource in Veritas Cluster

The MultiNICA agent represents a set of network interfaces and provides failover capabilities between them.

Each interface in a MultiNICA resource has a base IP address, which can be the same or different. The MultiNICA agent configures one interface at a time. If it does not detect activity on the configured interface, it configures a new interface and migrates IP aliases to it.

It's always a good idea to test the resource after configuring it and before putting the cluster into a production environment.

To illustrate the correct and incorrect way to test a MultiNICA resource, let's consider the following example with a simple service group:

group test_multiNIC (

SystemList = { node1 = 1, node2 }

)

IPMultiNIC IPMNIC (

Address = "192.200.99.5"

MultiNICResName = MNIC

)

MultiNICA MNIC (

Device @node1 = { hme0 = "192.200.100.1", qfe0 = "192.200.100.1" }

Device @node2 = { hme0 = "192.200.100.2", qfe0 = "192.200.100.2" }

)

IPMNIC requires MNIC

// resource dependency tree

// group test_multiNIC

// {

// IPMultiNIC IPMNIC

// {

// MultiNICA MNIC

// }

Correct way to test the NIC failover:

1. Bring the service group ONLINE:

# hagrp -online test_multiNIC -sys node1

2. Verify that the primary NIC (first NIC in the Device attribute) is properly set up:

# ifconfig -a (on node1)

The output should include 2 lines for hme0 (physical and virtual IP addresses), looking like:

hme0: flags=1000843 mtu 1500 index 12

inet 192.200.100.1 netmask ffffff00 broadcast 192.200.100.255

ether 8:0:20:b0:a8:1f

hme0:1: flags=1000843 mtu 1500 index 12

inet 192.200.99.5 netmask ffffff00 broadcast 192.200.99.255

3. In a shell window, launch a command to monitor the state of the nodes, service groups and resources:

# hastatus

4. In another shell window, launch a command to monitor the main VCS log file:

# tail -f /var/VRTSvcs/log/engine_A.log

5. Pull the cable from hme0 or switch off the hub or switch this NIC is attached to (as long as the other NIC in the Device attribute is not connected to this same network component).

The MultiNICA agent will perform the NIC failover (from hme0 to qfe0) after a 2-3 minute interval. This delay occurs because the MultiNICA agent tests the failed NIC several times before doing the NIC failover.

6. Check the engine_A.log log file to see the failover occurring. You should see lines like:

TAG_C 2002/01/31 09:26:27 (node1) VCS:136502:monitor:MNIC:MultiNICA: Device hme0 FAILED

TAG_C 2002/01/31 09:26:27 (node1) VCS:136503:monitor:MNIC:MultiNICA: Acquired a WRITE Lock

TAG_C 2002/01/31 09:26:27 (node1) VCS:136504:monitor:MNIC:MultiNICA: Bringing down IP addresses

TAG_C 2002/01/31 09:26:27 (node1) VCS:136505:monitor:MNIC:MultiNICA: Trying to online Device qfe0

TAG_C 2002/01/31 09:26:29 (node1) VCS:136506:monitor:MNIC:MultiNICA: Sleeping 5 seconds

TAG_C 2002/01/31 09:26:34 (node1) VCS:136507:monitor:MNIC:MultiNICA: Pinging Broadcast address 192.200.100.255 on Device qfe0, iteration 1

TAG_C 2002/01/31 09:26:34 (node1) VCS:136514:monitor:MNIC:MultiNICA: Migrated to Device qfe0

TAG_C 2002/01/31 09:26:34 (node1) VCS:136515:monitor:MNIC:MultiNICA: Releasing Lock

7. In the meantime, verify the hastatus output hasn't changed. The test_multiNIC should still be ONLINE on node1; no resources were affected. That is the expected behavior.

Incorrect way to test the NIC failover:

Some people can be tempted to unplumb the NIC via a command line to test the MultiNICA failover.

If you unplumb a NIC with a command line (for example: "ifconfig hme0 down unplumb"), VCS will notice that it hasn't put both MultiNICA and IPMultiNIC resources down itself. In other words, these resources will become OFFLINE, not being initiated by the agent monitor procedures.

The engine_A.log log files shows:

TAG_D 2002/01/31 09:32:53 (node1) VCS:13067:Agent is calling clean for resource(IPMNIC) because the resource became OFFLINE unexpectedly, on its own.

TAG_D 2002/01/31 09:32:54 (node1) VCS:13068:Resource(IPMNIC) - clean completed successfully.

TAG_E 2002/01/31 09:32:55 VCS:10307:Resource IPMNIC (Owner: unknown Group: test_multiNIC) is offline on node1

(Not initiated by VCS.)

Then the IPMultiNIC resource gets faulted on node1 and, as it is a critical resource, the whole service group will failover. That is obviously not the expected behavior.

service group in main.cf containing MultiNICA and IPMultiNIC resources is shown below:

group mnic_test ( SystemList = { csvcs3 = 1, csvcs4 } )

IPMultiNIC mIP (

Address = "166.98.21.173"

MultiNICResName = mnic

)

MultiNICA mnic (

Device @csvcs3 = { qfe2 = "166.98.21.197", qfe3 = "166.98.21.197" }

Device @csvcs4 = { qfe2 = "166.98.21.198", qfe3 = "166.98.21.198" }

ArpDelay = 5

IfconfigTwice = 1

PingOptimize = 0

Handshake-Interval = 10

)

mIP requires mnic

// resource dependency tree

// group mnic_test

// {

// IPMultiNIC mIP

// MultiNICA mnic

// }

Plumb qfe2 on each machine with its respective base IPs. In the example above, the base IP on csvcs3 is 166.98.21.197, while that on csvcs4 is 166.98.21.198. The virtual IP is 166.98.21.173 as shown in the IPMultiNIC resource. Then create the mnic_test group as shown above.

In the sample configuration given above, the following additional attributes are set for MultiNICA resource on a very active network. ArpDelay set to 5 secs, to induce 5 second sleep between configuring an interface and sending out a broadcast to inform routers about base IP address. Default is 1 second. IfconfigTwice is set to cause the IP address to be plumbed up twice, using an ifconfig up-down-up sequence. This increases the probability of gratuitous arps (local broadcast) reaching the clients. Default is 0 (not set).

The following attributes for MultiNICA can be set to decrease the agent detection/failover time :

PingOptimize set to 0 to perform a broadcast ping each monitor cycle and detect the inactive interface within the cycle. Default value of 1 requires 2 monitor cycles.

Handshake-Interval set to the least value of 10, from the default value of 90. This makes the agent attempt 1 time (as opposed to 9 times from default), either to ping a host (from the NetworkHosts attribute) or to ping the default broadcast address depending on the attribute configured, when it fails over to a new NIC.

Also it is to be noted that setting PingOptimize and Handshake-Interval to the above values would certainly improve the response time, but also would increase the chance for spurious failovers. So, essentially it is a tradeoff between performance and reliability.

To test the configuration pull the cable from qfe2 on csvcs3. It will fail over to qfe3 along with the virtual IP on the first node. Then pull the cable off qfe3. After a 2-3 minute interval, the mIP resource on csvcs3 will become faulted and the whole mnic_test group will go online on csvcs4. This delay occurs because the MultiNICA agent tests the NIC several times before marking the resource offline.

Adding a new service group to Veritas Cluster Server

Make sure the diskgroup can be deported, and imported without errors.

# vxdg deport

# vxdg -t import

How to configure a basic service group.

VERITAS Cluster Server logically groups together a set of Resources (such as Mount points, Applications, Oracle instances, etc) into Service Groups. Generally speaking, when one resource in a service group is no longer available, VCS will mark the resource as FAULTED, and migrate the entire service group to which it belongs to another node in the cluster. This is referred to as 'failover'. This document assumes that you have successfully completed the installation process, and that 'had' is currently running on your system, and that the command "/opt/VRTSvcs/bin/hastatus -summary" produces output similar to the following:

# /opt/VRTSvcs/bin/hastatus -summary

-- SYSTEM STATE

-- System State Frozen

A node1 RUNNING 0

A node2 RUNNING 0

The basic outline of creating a new service group is:

1. Create a list of related resources. For example, some typical resources are VXVM DiskGroups and Mount points.

2. Create a one-to-one map of the resources defined in step (1) with the Agents provided by VERITAS. For more information on Agents, see the VCS Bundled Agent Guide. Looking in the Bundled Agent Guide, we see that VCS comes with a DiskGroup agent, and a Mount agent.

3. In order to configure a service group, you must first make the configuration writeable and add it to the configuration. As always, when you add the group you should choose a meaningful name:

# haconf -makerw

# hagrp -add NEW_SG_NAME

VCS:10136:group added; populating SystemList and setting the Parallel attribute recommended before adding resources

4. Now you must, as the message says, populate the SystemList attribute (the Parallel attribute is beyond the scope of this document). The system list is the list of systems where it is possible for this service group to come online.

# hagrp -modify SystemList node1 0 node2 1

5. Configure the "AutoStartList" attribute - this is the system where this service group will automatically start at boot time:

# hagrp -modify AutoStartList

6. Once this is done, you may begin adding resources to the group. For more information on the types of resources available, and their individual requirements, refer to the VCS Bundled Agent Guide (or appropriate documentation that shipped with your agent). We will add a Disk Group and a Mount resource in this example. Again, be sure to choose meaningful names:

# hares -add MY_DG_RESOURCE DiskGroup

VCS:10245:Resource added

NameRule and Enabled attributes must be set before agent monitors

# hares -modify DiskGroup

# hares -add MY_MOUNTPOINT_RESOURCE Mount

# hares -modify BlockDevice

# hares -modify MountPoint

# hares -modify FsckOpt [%-y|%-n]

7. Once all of the resources have been added to the service group, VCS must be told in which order to bring them online. Otherwise, it will try to bring all resources on line simultaneously. The command 'hares -link res2 res1' will create a dependency such that "res1" MUST be online before VCS will attempt to start "res2":

# hares -link

8. Once all resources have been added, and all dependencies created, execute the following commands to enable all resources and write the configuration to disk:

# hagrp -enableresources

# haconf -dump -makero

9. Finally, to online your new service group, execute the command:

# hagrp -online -sys

On my test systems, I used the examples to run the following commands:

# hagrp -add MyMountGrp

VCS:10136:group added; populating SystemList and setting the Parallel attribute recommended before adding resources

# hagrp -modify MyMountGrp SystemList camcs12 0 camcs9 1

# hagrp -modify MyMountGrp AutoStartList camcs12

# hares -add MyDgResource DiskGroup MyMountGrp

VCS:10245:Resource added

NameRule and Enabled attributes must be set before agent monitors

# hares -modify MyDgResource DiskGroup vxvm_mydg

# hares -add MyMountResource Mount MyMountGrp

VCS:10245:Resource added

NameRule and Enabled attributes must be set before agent monitors

# hares -modify MyMountResource BlockDevice /dev/vx/dsk/vxvm_mydg/vol01

# hares -modify MyMountResource MountPoint /my/mount/point

# hares -modify MyMountResource FsckOpt %-y

# hares -link MyMountResource MyDgResource

# hagrp -enableresources MyMountGrp

# haconf -dump -makerw

# hagrp -online MyMountGrp -sys camcs12

# hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A camcs12 RUNNING 0

A camcs9 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B MyMountGrp camcs12 Y N ONLINE

B MyMountGrp camcs9 Y N OFFLINE

Looking in the /etc/VRTSvcs/conf/config/main.cf file, we can see the new service group configuration has been added:

# cat /etc/VRTSvcs/conf/config/main.cf

include "types.cf"

cluster vcs01 (

)

system camcs12 (

)

system camcs9 (

)

group MyMountGrp (

SystemList = { camcs12 = 0, camcs9 = 1 }

AutoStartList = { camcs12 }

)

DiskGroup MyDgResource (

DiskGroup = vxvm_mydg

)

Mount MyMountResource (

MountPoint = "/my/mount/point"

BlockDevice = "/dev/vx/dsk/vxvm_mydg/vol01"

FsckOpt = "-y"

)

MyMountResource requires MyDgResource

Expert's Corner

Search This Blog

Blog Archive

Sugumar

Wednesday, June 24, 2009

Removing the BootDisk from Veritas volume manager control

Removing Veritas Volume Manager step-by-step

How to test the VCS

Check Veritas Licenses - for FileSystem, Volume Manager AND Cluster

1. Hand check SystemList & AutoStartList

2. Verify Cluster is Running

3. Verify Services Can Switch Between Systems

4. Verify OTHER System Can Go Up & Down Smoothly For Maintanence

5. Test Actual Failover For System 2 (and pray db is okay)

6. Test Actual Failover For System 1 (and pray db is okay)

Configure IPMP network with Multiple NICs in Solaris

How to Find number of Physical & Virtual Processors in Solaris 10

How to reset root password in Sun solaris

How to find the WWN in Solaris

Enable/Disable IP Forwarding in Solaris 10 without reboot

How to add a Static Route in Sun Solaris

mount an ISO image in Linux

How to mount an ISO image in SUN Solaris

SSH login without password on Solaris

Test the failover of a NIC within a MultiNICA resource in Veritas Cluster

Adding a new service group to Veritas Cluster Server