Find it

Sunday, March 24, 2013

Change mount point under VCS.

It's my B'day today!!! On this occasion I can’t miss an opportunity to write a blog entry as I think, being a tech savvy  guy I should write something which would be the nice gift from me to myself..

On this special day, today I would like to share a blog entry with you all. So let's start...

Past week I was requested to change mount point of /opt/vdf to /data along with applicable alteration in VCS configuration to make sure VCS functionality should work as expected as before for this volume & mount resource.

The mount point needs to be renamed to /data is as follows –

# df -kh /opt/vdf/
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
            400G 165M 375G 1% /opt/vdf

Hence in order to do so, I’ll have to –

Unmount the /opt/vdf filesystem and re-mount it on /data as mount point. Before we do so, check if any data resides under filesystem and if any application active processes running within the filesystem which needs to be renamed. In case if any application is running and using the filesystem which needs renamed then at first place you need to stop the application and verify if no application process is hooked with the filesystem & If data resides on filesystem then simply create a temporary filesystem with temporary mountpoint and then copy over the data from original filesystem needs to be renamed. I trust this is simple one and any experienced SA can certainly do it.

Now in order to remount the filesystem with new mountpoint, first you need to modify your VCS configuration a bit so that you can un-mount the filesystem successfully.

Obliviously, the first basic thing to verify that VCS cluster configuration is currently read-only mode or write mode. To verify that you can use below command.

# haclus -display | grep -i 'readonly'
ReadOnly 1

Where,

0 = write mode
1 = read only mode

Well, currently VCS configuration is in Read-only mode so let’s make it write mode.

# haconf -makerw

# haclus -display | grep -i 'readonly'
ReadOnly 0

Good, now we have VCS configuration in write mode so we can make appropriate changes to the configuration and save them.

Now change the appropriate attributes in VCS configuration for filesystem/mount point resource that we are supposed to rename. This may vary, in my case I’m not changing the volume name and changing just a mountpoint name so I’m modifying only “MountPoint” attribute for resource “optvdf_mnt”

# hares -modify optvdf_mnt MountPoint "/data"

To verify.

# hares -display optvdf_mnt | grep -i MountPoint
optvdf_mnt ArgListValues adevdf01s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt ArgListValues adevdf02s MountPoint 1 /data BlockDevice 1 /dev/vx/dsk/vdfdg/opt_vdf_vol FSType 1 vxfs MountOpt 1 "" FsckOpt 1 -n SnapUmount 1 0 CkptUmount 1 1 SecondLevelMonitor 1 0 SecondLevelTimeout 1 30 OptCheck 1 0 CreateMntPt 1 0 MntPtPermission 1 "" MntPtOwner 1 "" MntPtGroup 1 "" AccessPermissionChk 1 0 RecursiveMnt 1 0 VxFSMountLock 1 1
optvdf_mnt MountPoint global /data

At this point we are good to un-mount the filesystem with mountpoint named /opt/vdf.

# umount /opt/vdf
UX:vxfs umount: ERROR: V-3-26388: file system /opt/vdf has been mount locked

Error!!! Whenever I get any error I feel very happy as every error teaches something new & especially new errors which I never came across!

The Solaris native OS command "umount" may through below error -

Well this error occurs if the mount point is locked by VCS. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control. The purpose of the VERITAS File System (VxFS) Mount Lock is to prevent accidental unmounting of a VxFS file system. The feature is enabled for the VCS Mount resource by default. This feature can be disabled by setting the Mount resource attribute VxFSMountLock to 0.

# hagrp -resources vdfapp_sg
vdfapp_dg
vdfappIP
optcdvdf_mnt
optvdf_mnt
optcdvdf_vol
optvdf_vol

# hares -display vdfapp_dg | grep -i UmountVolumes
vdfapp_dg ArgListValues adevdf01s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg ArgListValues adevdf02s DiskGroup 1 vdfdg StartVolumes 1 1 StopVolumes 1 1 MonitorOnly 1 0 MonitorReservation 1 0 tempUseFence 1 SCSI3 PanicSystemOnDGLoss 1 0 DiskGroupType 1 private UmountVolumes 1 0 vdfapp_dg UmountVolumes global 0

To get rid of this error use the VxFS umount command for Solaris to manually unmount the file system.

# /opt/VRTS/bin/umount -o mntunlock=VCS /opt/vdf

To mount filesystem/volume on new mountpoint use –

# mount -F vxfs -o mntlock=VCS /dev/vx/dsk/vdfdg/opt_vdf_vol /data

# df -kh /data
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/vdfdg/opt_vdf_vol
              400G  165M  375G  1%   /data

Make sure to save and make cluster configuration read-only.

# haconf -dump -makero

That's it! Done, isn't this easy! :)

Saturday, March 23, 2013

Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS)

Hi there! busy.. busy.. busy... stuck with routine work.. lots of work! :)

Anyways, today I finally decided to write a blog entry on a bit unusual task I did recently - Migrate VERITAS (VxVM) to Solaris Disk Suite (SDS). In past have done several migrations, like UFS to ZFS, UFS/SDS to VxVM, VxVM to ZFS however this time I been contacted for some different and backward migration which is VxVM to SDS. When SDS is getting obsolete but on other hand still there are some requirement comes, in which demanding for such things… anyways, good to perform anything which you feels interesting and uncommon. Then let’s do it…

Point To Ponder - Make sure you have full backup of the system on which you're going to operate.

Let's first see which disks are part of bootdg.

Disk_0 auto rootdisk rootdg online c3t0d0s2
Disk_5 auto rootmirror rootdg online c0t0d0s2

Which all volumes needs to be converted to the SVM -

# df -kh | grep bootdg
/dev/vx/dsk/bootdg/rootvol 5.9G 4.3G 1.5G 75% /
/dev/vx/dsk/bootdg/var 5.9G 4.7G 1.1G 81% /var
/dev/vx/dsk/bootdg/opt 5.9G 1.8G 4.0G 32% /opt

Take the pre-veritas vfstab copy.

root@XXXXXX# cat /etc/vfstab.prevm

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s1 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no nologging
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /var ufs 1 no nologging
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /opt ufs 2 yes nologging
/dev/dsk/c0t2d0s0 /dev/rdsk/c0t2d0s0 /var/crash ufs 2 yes nologging
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes -

#/dev/dsk/c3t0d0s0 is currently mounted on /.
#/dev/dsk/c3t0d0s1 is currently used by swap.
#/dev/dsk/c3t0d0s5 is currently mounted on /opt.
#/dev/dsk/c3t0d0s6 is currently mounted on /var.

Let's first unencapsulate the rootdisk using the vxunroot.

Detach all the plexes associated with the 'rootmirror' disk if applicable & Verify the rootmirror plexes have been detached.

# vxprint -qhtg rootdg -s | grep -i rootmirror | awk '{print $3}' > /var/tmp/subs.plex && cat /var/tmp/subs.plex
rootvol-02
swapvol-02
opt-02
var-02

# for x in `cat /var/tmp/subs.plex`

> do
> vxplex -g rootdg dis $x
> vxprint -qhtg rootdg -p $x
> done
pl rootvol-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-01 rootvol-02 rootmirror 0 12584484 0 Disk_5 ENA
pl swapvol-02 - DISABLED - 31458321 CONCAT - RW
sd rootmirror-02 swapvol-02 rootmirror 12584484 31458321 0 Disk_5 ENA
pl opt-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-03 opt-02 rootmirror 44042805 12584484 0 Disk_5 ENA
pl var-02 - DISABLED - 12584484 CONCAT - RW
sd rootmirror-04 var-02 rootmirror 56627289 12584484 0 Disk_5 ENA

# /etc/vx/bin/vxunroot

VxVM vxunroot NOTICE V-5-2-1564
This operation will convert the following file systems from
   volumes to regular partitions:
opt rootvol swapvol var

   VxVM vxunroot INFO V-5-2-2011
Replacing volumes in root disk to partitions will require a system
  reboot. If you choose to continue with this operation, system
  configuration will be updated to discontinue use of the volume
  manager for your root and swap devices.

Do you wish to do this now [y,n,q,?] (default: y) y
VxVM vxunroot INFO V-5-2-287 Restoring kernel configuration...
VxVM vxunroot INFO V-5-2-78
A shutdown is now required to install the new kernel.
You can choose to shutdown now, or you can shutdown later, at your
   convenience.

Do you wish to shutdown now [y,n,q,?] (default: n) n

VxVM vxunroot INFO V-5-2-258
Please shutdown before you perform any additional volume manager
   or disk reconfiguration. To shutdown your system cd to / and type

          shutdown -g0 -y -i6

# sync;sync;sync;shutdown -g0 -y -i6

Well, after 2-3 reboots server came back online and now I've UFS filesystems for OS volumes.

# df -kh / /var /opt
Filesystem size used avail capacity Mounted on
/dev/dsk/c3t0d0s0 5.9G 4.3G 1.5G 74% /
/dev/dsk/c3t0d0s6 5.9G 4.7G 1.1G 81% /var
/dev/dsk/c3t0d0s5 5.9G 1.8G 4.0G 32% /opt

Just to be sure, once just try booting from mirror disk too.

Now let's create a partition slice for metadb.

# format c3t0d0

selecting c3t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c3t0d0s0 is currently mounted on /. Please see umount(1M).
/dev/dsk/c3t0d0s1 is currently used by swap. Please see swap(1M).
/dev/dsk/c3t0d0s5 is currently mounted on /opt. Please see umount(1M).
/dev/dsk/c3t0d0s6 is currently mounted on /var. Please see umount(1M).

FORMAT MENU:
                disk - select a disk
                type - select (define) a disk type
                partition - select (define) a partition table
                current - describe the current disk
                format - format and analyze the disk
                repair - repair a defective sector
                label - write label to the disk
                analyze - surface analysis
                defect - defect list management
                backup - search for backup labels
                verify - read and display labels
                save - save new disk/partition definitions
                inquiry - show vendor, product and revision
                volname - set 8-character volume name
                ![cmd] - execute [cmd], then return
                quit
format> p

PARTITION MENU:
        0 - change `0' partition
        1 - change `1' partition
        2 - change `2' partition
        3 - change `3' partition
        4 - change `4' partition
        5 - change `5' partition
        6 - change `6' partition
        7 - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name - name the current table
        print - display the current table
        label - write partition map and label to the disk
        ![cmd] - execute [cmd], then return
        quit
partition> p
Current partition table (original):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 0 0 (0/0/0) 0

partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 23960
Enter partition size[0b, 0c, 23960e, 0.00mb, 0.00gb]: 128mb
partition> p
Current partition table (unnamed):
Total disk cylinders available: 24620 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 3 - 4358 6.00GB (4356/0/0) 12584484
1 swap wu 4359 - 15247 15.00GB (10889/0/0) 31458321
2 backup wu 0 - 24619 33.92GB (24620/0/0) 71127180
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 15248 - 19603 6.00GB (4356/0/0) 12584484
6 var wm 19604 - 23959 6.00GB (4356/0/0) 12584484
7 unassigned wm 23960 - 24050 128.37MB (91/0/0) 262899

partition> l
Ready to label disk, continue? yes

Good enough, now let's create metadb.

# metadb -a -f -c 3 c3t0d0s7

# metadb -i
       flags first blk block count
         a  u     16      8192     /dev/dsk/c3t0d0s7
         a  u     8208    8192     /dev/dsk/c3t0d0s7
         a  u     16400   8192     /dev/dsk/c3t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Set SDS on root file system.

# metainit -f d11 1 1 c3t0d0s0
d11: Concat/Stripe is setup

# metainit d10 -m d11
d10: Mirror is setup
# metaroot d10

After executing metaroot, check the changes in /etc/vfstab and /etc/system.

#live-upgrade: updated boot environment

#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap -no nologging
/dev/dsk/c3t0d0s1 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/dsk/c3t0d0s6 /dev/rdsk/c3t0d0s6 /var ufs 1 no nologging,nosuid
/dev/dsk/c3t0d0s5 /dev/rdsk/c3t0d0s5 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,10,blk
* End MDD root info (do not edit)

Good enough.

# metastat -ac
d10           m 6.0GB d11
   d11        s 6.0GB c3t0d0s0

# metainit -f d31 1 1 c3t0d0s6
d31: Concat/Stripe is setup
# metainit d30 -m d31
d30: Mirror is setup

# metainit -f d51 1 1 c3t0d0s5
d51: Concat/Stripe is setup
# metainit d50 -m d51
d50: Mirror is setup

# metainit -f d1 1 1 c3t0d0s1
d1: Concat/Stripe is setup
# metainit d0 -m d1
d0: Mirror is setup

# metastat -ac
d0      m 15GB d1
   d1   s 15GB c3t0d0s1
d50     m 6.0GB d51
   d51  s 6.0GB c3t0d0s5
d30     m 6.0GB d31
   d31  s 6.0GB c3t0d0s6
d10     m 6.0GB d11
   d11  s 6.0GB c3t0d0s0

So in above,

d0 - swap
d10 - /
d30 - /var
d50 - /opt

Now it's time to change the vfstab to change slices into metadb devices.

# vi /etc/vfstab

# cat /etc/vfstab
#live-upgrade: updated boot environment
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
fd - /dev/fd fd - no -
/proc - /proc proc - no -
#live-upgrade::# /dev/vx/dsk/bootdg/swapvol - - swap - no nologging
/dev/md/dsk/d0 - - swap - no -
/dev/md/dsk/d10 /dev/md/rdsk/d10 / ufs 1 no nologging
/dev/md/dsk/d30 /dev/md/rdsk/d30 /var ufs 1 no nologging,nosuid
/dev/md/dsk/d50 /dev/md/rdsk/d50 /opt ufs 2 yes nologging
/dev/vx/dsk/crashdg/crashvol /dev/dsk/crashdg/crashvol /var/crash vxfs 2 yes -
/devices - /devices devfs - no -
ctfs - /system/contract ctfs - no -
objfs - /system/object objfs - no -
swap - /tmp tmpfs - yes nosuid

Reboot the box once.

Now it's time to add mirror disk to existing metadevices.

At this stage the mirror disk is part of rootdg and for rootdg it's the last disk in the disk group hence we need to destroy rootdg.

# vxdg destroy rootdg

Unsetup the disk so it will be out of VERITAS control.

# vxdiskunsetup -C Disk_5

Cool, now I'm all set to create/copy partition table on mirror disk as exisitng disk part of metadevices.

# prtvtoc /dev/rdsk/c3t0d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Fine, Set redundant database on mirror disk.

# metadb -a -f -c 3 c0t0d0s7

# metadb -i

        flags    first blk    block count
     a m p luo   16           8192         /dev/dsk/c3t0d0s7
     a   p luo   8208         8192         /dev/dsk/c3t0d0s7
     a   p luo   16400        8192         /dev/dsk/c3t0d0s7
     a      u    16           8192         /dev/dsk/c0t0d0s7
     a      u    8208         8192         /dev/dsk/c0t0d0s7
     a      u    16400        8192         /dev/dsk/c0t0d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input
W - replica has device write errors
a - replica is active, commits are occurring to this replica
M - replica had problem with master blocks
D - replica had problem with data blocks
F - replica had format problems
S - replica is too small to hold current data base
R - replica had device read errors

Create metadevices on mirror.

# metainit -f d12 1 1 c0t0d0s0
d12: Concat/Stripe is setup

# metainit -f d2 1 1 c0t0d0s1
d2: Concat/Stripe is setup

# metainit -f d52 1 1 c0t0d0s5
d52: Concat/Stripe is setup

# metainit -f d32 1 1 c0t0d0s6
d32: Concat/Stripe is setup

Attach metadevices.

# metattach d10 d12
d10: submirror d12 is attached

# metattach d30 d32
d30: submirror d32 is attached

# metattach d50 d52
d50: submirror d52 is attached

# metattach d0 d2
d0: submirror d2 is attached

# metastat -ac
d0              m 15GB d1 d2 (resync-76%)
     d1         s 15GB c3t0d0s1
d2              s 15GB c0t0d0s1
     d50        m 6.0GB d51 d52
d51             s 6.0GB c3t0d0s5
     d52        s 6.0GB c0t0d0s5
d30             m 6.0GB d31 d32
     d31        s 6.0GB c3t0d0s6
d32             s 6.0GB c0t0d0s6
     d10        m 6.0GB d11 d12
d11             s 6.0GB c3t0d0s0
     d12        s 6.0GB c0t0d0s0

Install boot block on both disks.

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c3t0d0s0

Well, that completes VxVM to SDS migration.  I'm sure hardly anyone need to do such backward migration but just in case then this method will certainly help you... :)

Have a good weekend!