VERITAS VOLUME MANAGEMENT

VERITAS VXVM CMDS (4.0) FOR SOLARIS BOOT DISK

MANAGEMENT
VxVM Manual ch 4, 5
The correct procedure for remirroring bootdisk from bootmirror so that
they are nearly identical in paritioning scheme should be:
(after many tries):

/etc/vx/bin/vxdisksetup -i c1t0d0 format=sliced

Add disk c1t0d0 to vertias control, using sliced format for
boot-able di
sk.

vxdg -g rootdg adddisk bootdisk=c1t0d0

vxmirror -g rootdg bootmirror bootdisk

This mirror partition and cyl in right seq.

/etc/vx/bin/vxbootsetup -g rootdg rootdisk

just in case vtoc or boot sector info is missing.

----

Some more notes from the try, vxassist command used as per book
suggestion.

fix later, from try 2 on oaprod2

vxplex -g rootdg dis rootvol-01

vxplex -g rootdg dis var-01 u01-01 swapvol-01
vxedit -g rootdg -fr rm rootvol-01
vxedit -g rootdg -fr rm swapvol-01 var-01 u01-01
vxdg -g rootdg rmdisk bootdisk
vxdisk rm c1t0d0

/etc/vx/bin/vxdisksetup -i c1t0d0 format=sliced

Add disk c1t0d0 to vertias control, using sliced format for
boot-able disk.

vxdg -g rootdg adddisk bootdisk=c1t0d0

vxassist -g rootdg mirror swapvol bootdisk &

Mirror swap slide
try mirror swap first so that it will start at cyl 2
It won't put any partition info in vtoc yet.

/etc/vx/bin/vxrootmir bootdisk
mirror the root partition, it will need slide 0 be avail.
This would probably work, except somehow var was placed in disk
earlier
than root, so will need to do that first.
 vxassist -g rootdg mirror var bootdisk &
vxassist -g rootdg mirror u01 bootdisk &

So instead, use vxmirror...

vxmirror -g rootdg bootmirror bootdisk
This mirror partition and cyl in right seq.
oaprod2 seems to be fine.

vxmirror : mirror root disk

Missing!! Try in test "4"

/etc/vx/bin/vxbootsetup -g rootdg rootdisk
Configure bootable info on vxvm boot disk in rootdg disk group.
This include boot sector and solaris VTOC info that match
location of root, swap, /usr, /var.
Optional media name(s) at end (rootdisk) specify action to be
carried out
on specified disk only (if omitted, done for all disk in dg).

More info:
http://www.sun.com/blueprints/0800/vxvmref.pdf

vxbootsetup would be needed for vxassist, and slide indeed show

up in vtoc/format.
but vxassist method req more work to copy partition in the right
cyl
manner. Maybe if bootmirror was created using vxassist, then
that
would have been good. But right now, vxmirror works great
and don't really need vxbootsetup (at least in 4.0)

---

VxVM commands on oaprod1, clean tba

544 vxmirror -g rootdg c1t1d0 c1t0d0

545 vxmirror c1t1d0s2 c1t0d0s2
546 vxdg -g rootdg c1t1d0s2 c1t0d0s2
547 vxmirror -g rootdg c1t1d0s2 c1t0d0s2
548 vxmirror -g rootdg c1t1d0s2 c1t0d0s2
549 vxmirror -g rootdg c1t1d0s0 c1t0d0s0
550 vxprint -hrt
556 vxdisk list
564 vxdmpadm -h
565 vxtask list
566 vxprint -hrt
567 vxdisk list
576 vxdisk list
577 man vxdisk
578

vxdisk list
display all disk managed by Veritas
vxdisk -s list
much more detailed version than above
vxprint
print disk slice/mirroring info, plex,vol, etc
vxprint -hrt
diff version than above, useful to see boot disk mirroring
location
on disk.

vxvol -g oracledg stopall

stop all veritas volume on the specified disk group oracledg.
all volume must be unmounted first

vxdg deport oracledg

export the disk group oracledg
this would make all the disk/volume of the said disk group
to be available for import by other hosts.
optional -h parameter tell who is expected to do import

vxdg list
list all disk group
vxdisk -o alldgs list
scan all disk, incl those not currently managed by veritas.
those with disk group name inside () are exported dg, ready for
import.

vxdg import oracledg

import the said disk group oracledg

vxvol -g oracledg startall

start all disk in said disk group oracledg
This will make all vol in said disk group to be avail for
mounting
and general use.
Veritas may have some locking mechanism based in scsi3 to fence
off
multiple host from importing a disk group, even if forceful.
(as maybe problem on SAN disks shared in cluster).
Check this first though.

----

vxddladm addjbod VID=DGC pagecode=0x83 offset=8 length=16

disable Veritas DMP (dynamic MultiPath) and use EMC PowerPath
driver instead
(EMCpower pkg need to be installed).

vxddladm listjbod
Display current settings

vxddladm rmjbod vid=DGC

Undo use of EMC PowerPath, and let Veritas DMP do its thing
again.

Veritas DMP will continue to be used, PowerPath sits at lower layer and
intercept the calls
to the different disks. Veritas will know multiple path to the LUN, and
it will know they are
the same. If fiber is removed, DMP won't know as power path work behind
the scene. Syslog
will log error. After using vxddladm addjbod, reboot machine w/
reconfigure to ensure all devices are seen.

vxdmpadm getdmpnode enclosure=Disk

list paths to various disks/luns

May 3 15:05:15 oaprod1 vxio: WARNING: VxVM vxio V-5-0-181 Illegal

vminor encountered

The error is said due to vm disk starting up before vxconfigd, but my

experience is that vxconfigd is already running.
Other things to check:
There is no files named:
/VXVM-UPGRADE/.start_runed
/etc/vx/reconfig.d/state.d/install-db

ls -l /dev/vx/dsk/
reminor disk group whose import is in conflict w/ existing disk group.

ls -l /dev/vx/dsk/oracledg
brw------- 1 root root 259,52001 May 4 17:10 u02
brw------- 1 root root 259,52000 May 4 17:10 u03
^^^^^
^^^^^ 52000 is the minor number,
shown also in vxprint -l oracledg

vxdg -g oracledg -f reminor 53011 # change the minor number of a

disk group,
# must be done while disk group
is mounted
# but changes take effect only in
the next import run
# and this minor number belongs
to the disk group, which get carried to
# another node when it is
imported there.

Other commands
vxassist

/usr/sbin/modinfo | grep vx : look for loaded veritas modules (to see

if veritas was running)

vxlicense -p : list licenses available in the system

vxlicense -c : interactive program to enter veritas license
key
: license files are stored in /etc/vx/elm/
/usr/sbin/vxdisk list : list veritas disks, status

/usr/sbin/vxprint -ht : show some long config,

/usr/sbin/vxprint -m :
: veritas setup, etc

vxdiskadm : encaptulate root disk

vxmirror : mirror root disk
vxassistmirror

vxrecover : rebuild? failed vx slide

vsedvtoc : edit the veritas "fdisk/format" info

vxvol stop : stop all veritas volumes on root disk

vxedit -r : remove volume

VERITAS CLUSTER SERVER

VOLUME MANAGEMENT CONSIDERATION UNDER VCS
Disabling io fencing:

/etc/rc2.d/S97vxfen stop
echo "vxfen_mode=disabled" > /etc/vxfenmode
/etc/rc2.d/S97vxfen start
Above will still load driver, which allows Veritas Cluster VM to work
w/o io fencing disks,
but driver need to be loaded. This is supported for certain CVM config,
but not for RAC.

/var/adm/messages that kernel driver is loaded.

AND error on console that it is disabled (where command was started).

Best way is still to turn iofencing off completely in the init script.

modinfo | egrep vx\|gab\|llt

31 12490f9 26abc 258 1 vxdmp (VxVM 4.1z: DMP Driver)
32 78216000 20f869 259 1 vxio (VxVM 4.1z I/O driver)
34 126d9f0 1499 260 1 vxspec (VxVM 4.1z control/status driver)
277 7859f205 c7b 261 1 vxportal (VxFS 4.1_REV-4.1B18_sol_GA_s10b)
278 78a24000 170105 8 1 vxfs (VxFS 4.1_REV-4.1B18_sol_GA_s10b)
282 78bc2000 22467 266 1 llt (LLT 4.1)
283 78be6000 46a88 267 1 gab (GAB device 4.1)
284 78c2e000 39491 268 1 vxfen (VRTS Fence 4.1)

IOFencing.
driver get loaded by kernel.
it search for vxfendg to find which disk group to use, then
run vx... cmd to generate /etc/vxfentab, which has list of device path
for LUN beloging to io fencing dg.
4.0 use /dev/rdsk/cXtXdXsX, 4.1 use the multipath devices, such as
/dev/rdsk/emcpowerXc
the driver is much more robust than the vxfentsthdw script.
4.0 vxfentsthdw -g vxfencoorddg fails, though driver should be all good.
4.1 vxfentsthdw -g will work correctly using emcpowerX devices and know
that they may not be the
same device path on different nodes. 4.1 resolved all io fencing issues
found in 4.0.

4.1 vxdisk -o alldgs list will also show the emcpowerX as device name,
instead of generic DISK_X.
The naming now is at the mercy of PowerPath, veritas see them just as
solaris format command see them.
It may still not be persistent binding, but at least easy corss ref b/w
veritas, solaris format, and info
presented in EMC Navisphere. No ASL (Array Support Lib) was needed.

VCS CLUSTER CONFIG

gabconfig -a : display link config info. a = ??gab port, ie loaded ok
by kernel. b = iofencing port. h = cluster port.

GAB Port Memberships

===============================================================
Port a gen 1ea001 membership 01
Port b gen 1ea00f membership 01
Port h gen 1ea012 membership 01

vxfenadm -i /dev/rdsk/emcpower0c
Display serial number of the device (LUN, disk)
vxfenadm -g /dev/rdsk/emcpower0c
Show IO Fencing info

graceful shutdown of cluster

hastop -all # stop vcs for the whole cluster, ready for both machine
to shutdown.
hastop -local # stop vcs on local machine only, it will stop the
services, no migration by default.
hastop -local -evacuate # stop vcs, migrate (evacuate) service to
another node
# evacuate a single node, just single node clean
exit out of cluster.

hastatus # monitor cluster status, no arg act like tail -f

-sum # display summary and exit.

lltstat # general summary

-nvv # see cluster interconnect link info (heartbeat).
hares -online Mount_u02 -sys oaprod1 # online the give resource
at the specified system
# resource name is as per
config (Main.cf)
# resource and group name
are listed by hastatus cmd.
hares -offline Oracle_oaprod -sys oaprod2 # offline the whole
resrouce group on the specified system
# migration to another
node will NOT happen for -offline.
hagrp -switch oracle_group -to oaprod1 # switch a service group
to the specified system
hares -modify Oracle_oaprod Owner oracle # change
resouce=Oracle_oaprod attribute=Owner new_value=oracle
haconf -dump # save vcs config to
Main.cf (edited via special command)
# do not edit Main.cf
while cluster is up, it will be ignored.
haconf -dump -makero # equiv of "close config"
of hagui, config still kept by conf editor

seq of offline commands:

hares -offline Netlsnr_oaprod -sys oaprod2
hares -offline Oracle_oaprod -sys oaprod2
hares -offline Mount_u02 -sys oaprod2
hares -offline Mount_u03 -sys oaprod2
hares -offline Volume_u02 -sys oaprod2
hares -offline Volume_u03 -sys oaprod2
hares -offline DiskGroup_oracledg -sys oaprod2 # some sort of
high level container wrapper.
hares -offline IPMultiNICB_oaprod -sys oaprod2

hares -clear Oracle_oaprod -sys oaprod1

hares -online Oracle_oaprod -sys oaprod1 # bring up oracle service
group, w/ all deps
hagrp -switch oracle_group -to oaprod2

---
config eg, for adding oracle test group.

haconf -makerw
hagrp -freeze oracle_group
hares -modify Oracle_oaprod User veritas_monitor
hares -modify Oracle_oaprod Pword veritas_password
hares -modify Oracle_oaprod Table monitor
hares -modify Oracle_oaprod MonScript "./bin/Oracle/SqlTest.pl"
hares -modify Oracle_oaprod DetailMonitor 1
haconf -dump -makero
hagrp -unfreeze oracle_group

---
config commands (typically located in /opt/VRTS/bin):

hacf -verify /etc/VRTSvcs/conf/config/

verify that the main.cf config file is correct, parseable.

haconf -makerw
turn config to be read write, so that changes can be made via
haclus
haconf -dump -makero
save and close config from rw, must remember to do this, or else
reboot will have issues!
haclus ...
change cluster config param. (CLI change instead of gui).

hauser -add vcsuser

Add a new user that can use hagui, it will prompt for the new
password of the new user.

hauser -modify Administrators -add vcsuser

The new user is placed in admin group so that full control is
granted.
Best way to add admin when password is forgotten :)

hagui GUI, java, for monitor and making changes to cluster

VCS config files

/etc/VRTSvcs/conf/config/main.cf
config file for vcs, usually changed using hagui or haclus
command.
Once cluster is live, config is in memory, and this file is only
backup.
Changes to it will be ignored if cluster is up.
Cluster start does read this file, so easy manual chage of
config if cluster is down.

/etc/init.d/vx*
vxvm-relocover
starts several deamon, which also take argument and email root
at local machines.
change these!

3 files in /etc need to be copied to each of the node in the cluster

(rsh of install should create these if doing multinode install w/
install script).

/etc/llttab ::
set-node oaprod1 # diff for each node, reflect
local node name
set-cluster 1
link ce1 /dev/ce:1 - ether - -
link ce3 /dev/ce:3 - ether - -
link-lowpri ce0 /dev/ce:0 - ether - -
/etc/llthosts ::
0 oaprod1
1 oaprod2

/etc/gabtab ::
/sbin/gabconfig -c -n2

Try this

change the grep to use the ID on your devices...

HOSTNAME=`uname -n`
echo ""
echo "Hostname Logical CU:LDEV Volumes"
echo "-------- ------- ------- -------"
USEDDISKS=`vxprint -ht|grep "sd " |grep -v root|grep -v swap|awk '{print
$8}'|sort|uniq`
for DISK in $USEDDISKS;do
CULDEV=`vxdisk list $DISK | grep 7D86 | cut -b32-35`
CU=`echo $CULDEV|cut -b1-2`
LDEV=`echo $CULDEV|cut -b3-4`
VOLLIST=`vxprint -ht|grep $DISK|grep -v "dm " | awk {'print $3'}|cut -d"-"
-f1`
for VOL in $VOLLIST;do
VFSLINE=`egrep "$VOL |$VOL " /etc/vfstab`
if [ $? -eq 0 ];then
MNTPT=`echo $VFSLINE |awk '{print $3}'`
echo "$HOSTNAME $DISK $CU:$LDEV $MNTPT "
else
echo "$HOSTNAME $DISK $CU:$LDEV $VOL "
fi
done
done

Me <me@hotmail.com> wrote:
>NO way to do that.
>
>vxdisk list will list the Disk Access (da) name (or the name of the
>disk in Volume Manager) to a c#t#d# (or to an enclosure-based name)
>
>
>What I suspect is that you want to know how the enclosure-based name
>(something like EMC0_1 or EMC0_12) convetrs back to a LUN.
>
>
>The reason why enclosure-based names are used, is to make it easy for
>you. The "real" name of the disk always has the form c#t#d# Biggest
>problem is that you will see that the t# will include the 20 character
>WWNN (World Wide Node Number) of the disk. So EMC0_12 might have a
>"real" name like c5t234ed852ab23c423961d0
>
>This is a bit difficult to manage, but if you want to see the "real"
>names next to the enclosure-based name, do "vxdisk -e list".
>
>Once you've got the c#t#d#, you will have to use EMC tools to get the
>LUN numbers.