Wednesday, July 26, 2017

# hp-ux cluster fs # unix cluster fs new fs

UNIX: Creating a File System on a Cluster server with New Disks Allocation

Yesterday I had blogged about extending an AIX file system and today we will do a new FS creation in a cluster server

As usual, we start with a user request

Hi Unix Adm

Pls create a new filesystem - /oracle/PRD/sapdata11 of size 80GB.

Thanks

First things first, we make sure the FS isn't created, and by the looks of the FS name request, there may be a /oracle/PRD/sapdata10 existing on the server, so look for it and find the VG name from it. We will need the VG name to add the new disks to, in order to create a FS

1. Look for the LV and VG to use to create the new FS

root@hedkandi:(root)> bdf /oracle/PRD/sapdata10
Filesystem          kbytes    used   avail %used Mounted on
/dev/vgprddata/lvol10
                   52428800 25665564 25090534   51% /oracle/PRD/sapdata10


2. So the current LV name is lvol10 and the VG name is vgprddata. We also check the size of available FREE PE on this VG which is about 38 GB

root@hedkandi:(root)> vgdisplay vgprddata
--- Volume groups ---
VG Name                     /dev/vgprddata
VG Write Access             read/write     
VG Status                   available, exclusive      
Max LV                      255    
Cur LV                      10     
Open LV                     10     
Max PV                      16     
Cur PV                      10     
Act PV                      10     
Max PE per PV               21247        
VGDA                        20  
PE Size (Mbytes)            4               
Total PE                    212440  
Alloc PE                    202468  
Free PE                     9972    
Total PVG                   0        
Total Spare PVs             0              
Total Spare PVs in use      0

The math I did to get this is

root@hedkandi:(root)> bc
9972*4
39888
39888/1024
38 GB

In order for me to raise a Change Request for this, I need to know the existing disk size so I can get the SAN team to allocate a new disk of the same size for my FS creation. I run the vgdisplay -v command to get a complete list of disks for VG vgprddata and check a few disks' sizes

From a truncated vgdisplay -v output:

   --- Physical volumes ---
   PV Name                     /dev/dsk/c10t0d3
   PV Name                     /dev/dsk/c12t0d3 Alternate Link
   PV Name                     /dev/dsk/c14t0d3 Alternate Link
   PV Name                     /dev/dsk/c8t0d3  Alternate Link
   PV Status                   available                
   Total PE                    21244   
   Free PE                     0       
   Autoswitch                  On        

   Proactive Polling           On 

Here, I picked disk c8t2d7 to check its size. Make sure you add the r in the /dsk when you run the diskinfo command because VG output doesn't show the raw character "r" of the SCSI disk


root@hedkandi:(root)> diskinfo /dev/rdsk/c8t2d7
SCSI describe of /dev/rdsk/c8t2d7:
             vendor: HP      
         product id: HSV200          
               type: direct access
               size: 87031808 Kbytes
   bytes per sector: 51

How do I know what size disk to ask for? Lets transform the size in kb to GB

root@hedkandi:(root)> bc
87031808/1024/1024
83

After SAN team has allocated the LUN, you do a hardware scan to detect the disks

ioscan -fn

root@hedkandi:(root)> insf -e
insf: Installing special files for asio0 instance 0 address 0/0/1/0
insf: Installing special files for asio0 instance 1 address 0/0/1/1
insf: Installing special files for sdisk instance 0 address 0/0/3/0.0.0.0
insf: Installing special files for sctl instance 0 address 0/0/3/0.0.7.0

I have truncated the output because it will show a pretty long list of all the hardware on the server

What I like to do is check the syslog at this stage, because it will show the new disks added

Jan 21 15:34:37 hedkandi vmunix: 0/5/1/0/4/0.1.0.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/5/1/0/4/0.1.1.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/2/1/0/4/0.1.0.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/2/1/0/4/0.1.1.0.0.3.0 sdisk

And what's left to do is to do an I/O scan on the disks individually and make sure their status is CLAIMED

root@hedkandi:(root)> ioscan -fnH 0/5/1/0/4/0.1.0.0.0.3.0
Class     I  H/W Path       Driver   S/W State   H/W Type     Description
==========================================================================
disk     63  0/5/1/0/4/0.1.0.0.0.3.0  sdisk    CLAIMED     DEVICE       HP      HSV200
                           /dev/dsk/c14t3d0   /dev/rdsk/c14t3d0

root@hedkandi:(root)> ioscan -fnH 0/5/1/0/4/0.1.1.0.0.3.0
Class     I  H/W Path       Driver   S/W State   H/W Type     Description
==========================================================================
disk     64  0/5/1/0/4/0.1.1.0.0.3.0  sdisk    CLAIMED     DEVICE       HP      HSV200
                           /dev/dsk/c8t3d0   /dev/rdsk/c8t3d0

root@hedkandi:(root)> ioscan -fnH 0/2/1/0/4/0.1.0.0.0.3.0
Class     I  H/W Path       Driver   S/W State   H/W Type     Description
==========================================================================
disk     65  0/2/1/0/4/0.1.0.0.0.3.0  sdisk    CLAIMED     DEVICE       HP      HSV200
                           /dev/dsk/c10t3d0   /dev/rdsk/c10t3d0

root@hedkandi:(root)> ioscan -fnH 0/2/1/0/4/0.1.1.0.0.3.0
Class     I  H/W Path       Driver   S/W State   H/W Type     Description
==========================================================================
disk     66  0/2/1/0/4/0.1.1.0.0.3.0  sdisk    CLAIMED     DEVICE       HP      HSV200
                           /dev/dsk/c12t3d0   /dev/rdsk/c12t3d0


From now on, it will be the LVM management, and we start off by adding the disk to the VG vgprddata

1. Create the PV first

pvcreate /dev/rdsk/c14t3d0

2. Extend the VG of vgprddata

vgextend vgprddata /dev/dsk/c14t3d0

3. Check the extended VG size

vgdisplay vgprddata

4. Create the LV

lvcreate -L 84992 -n lvol11 -s y /dev/vgprddata

5. Check the lvdisplay, LV changes must be added

lvdisplay /dev/vgprddata/lvol11

6. Make the directory

mkdir /oracle/PRD/sapdata11

7. Make the FS

newfs -F vxfs -o largefiles /dev/vgprddata/rlvol10

8. Mount the LV to the FS

mount /dev/vgprddata/lvol11 /oracle/PRD/sapdata11

9. Change the ownership of the FS

chown -R oraprd:dba /oracle/PRD/sapdata11

10. Edit the Bad Block Allocation to ON if it is set to NONE. Our VG has the setting to ON

lvchange -r N /dev/VG/LV----------this is only to edit if the Bad Block is set to None, change it to ON

11. We then make changes to the control file of the cluster server hedkandi to add the new FS into it. To know which files to edit, run the cmviewcl command 

root@hedkandi:(root)> cmviewcl

CLUSTER        STATUS       
kandi            up           
  
  NODE           STATUS       STATE        
  hedkandi        up           running      

    PACKAGE        STATUS       STATE        AUTO_RUN     NODE        
    prdms          up           running      enabled      hedkandi--------->this one to edit   
  
  NODE           STATUS       STATE        
  bluekandi        up           running


Therefore...

root@hedkandi:(root)> find / -name *.cntl
/etc/cmcluster/PRD/sapdbci.cntl
/etc/cmcluster/PRD/prdms.cntl-------------->this one to edit
/etc/cmcluster/prdms.old/prdms.cntl
/opt/samba/HA/samba.cntl
/opt/cmcluster/sap/LC/saplc.cntl
/opt/cmcluster/sap/SID/sapwas.cntl
/opt/cmcluster/sap/contrib/J2EE620/sap620Engine.cntl

12. Go to the FS

cd /etc/cmcluster/PRD/prdms.cntl

13. Make a backup copy of the file

cp -p prdms.cntl prdms.cntl.21012010

14. Run the more command to have a look at the VGs and the LVs in the server hedkandi

root@hedkandi:(PRD)> more /etc/cmcluster/PRD/prdms.cntl | grep vgprddata
VG[0]="vgprddata"
LV[13]="/dev/vgprddata/lvol1";  FS[13]="/oracle/PRD/sapdata1";  FS_MOUNT_OPT[13]="-o delaylog,largefiles"; FS_FSCK_OPT[13]=""; FS_TYPE[13]="vxfs"
LV[14]="/dev/vgprddata/lvol2";  FS[14]="/oracle/PRD/sapdata2";  FS_MOUNT_OPT[14]="-o delaylog,largefiles"; FS_FSCK_OPT[14]=""; FS_TYPE[14]="vxfs"
LV[15]="/dev/vgprddata/lvol3";  FS[15]="/oracle/PRD/sapdata3";  FS_MOUNT_OPT[15]="-o delaylog,largefiles"; FS_FSCK_OPT[15]=""; FS_TYPE[15]="vxfs"
LV[16]="/dev/vgprddata/lvol4";  FS[16]="/oracle/PRD/sapdata4";  FS_MOUNT_OPT[16]="-o delaylog,largefiles"; FS_FSCK_OPT[16]=""; FS_TYPE[16]="vxfs"
LV[17]="/dev/vgprddata/lvol5";  FS[17]="/oracle/PRD/sapdata5";  FS_MOUNT_OPT[17]="-o delaylog,largefiles"; FS_FSCK_OPT[17]=""; FS_TYPE[17]="vxfs"
LV[18]="/dev/vgprddata/lvol6";  FS[18]="/oracle/PRD/sapdata6";  FS_MOUNT_OPT[18]="-o delaylog,largefiles"; FS_FSCK_OPT[18]=""; FS_TYPE[18]="vxfs"
LV[20]="/dev/vgprddata/lvol7";  FS[20]="/oracle/PRD/sapdata7";  FS_MOUNT_OPT[20]="-o delaylog,largefiles"; FS_FSCK_OPT[20]=""; FS_TYPE[20]="vxfs"
LV[22]="/dev/vgprddata/lvol8";  FS[22]="/oracle/PRD/sapdata8";  FS_MOUNT_OPT[22]="-o delaylog,largefiles"; FS_FSCK_OPT[22]=""; FS_TYPE[22]="vxfs"
LV[23]="/dev/vgprddata/lvol9";  FS[23]="/oracle/PRD/sapdata9";  FS_MOUNT_OPT[23]="-o delaylog,largefiles"; FS_FSCK_OPT[23]=""; FS_TYPE[23]="vxfs"
LV[24]="/dev/vgprddata/lvol10";  FS[24]="/oracle/PRD/sapdata10";  FS_MOUNT_OPT[24]="-o delaylog,largefiles"; FS_FSCK_OPT[24]=""; FS_TYPE[24]="vxfs"


15. From here, we can see that the last LV runs at number 24, so you will need to add another line under it for the FS recently created

LV[25]="/dev/vgprddata/lvol11";  FS[25]="/oracle/PRD/sapdata11";  FS_MOUNT_OPT[25]="-o delaylog,largefiles"; FS_FSCK_OPT[25]=""; FS_TYPE[24]="vxfs"

16. Run cmviewcl command again, and make sure prdms package is UP

# cmviewcl 

17. Disable autorun

# cmmodpkg -d prdms

18.Run cmviewcl command again, and make sure the prdms package status is DOWN and autorun is disabled

# cmviewcl 

18. Now, we make a note of the ownership and permissions for VG map file of VG vgprddata

root@hedkandi:(PRD)> ll /etc/lvmconf/vgprddata.mapfile
-rw-r--r--   1 root       sys            104 Jan 21 10:30 /etc/lvmconf/vgprddata.mapfile

19. Do a tail on the file

root@hedkandi:(PRD)> tail  /etc/lvmconf/vgprddata.mapfile
1 lvol1
2 lvol2
3 lvol3
4 lvol4
5 lvol5
6 lvol6
7 lvol7
8 lvol8
9 lvol9
10 lvol10

20. Run vgexport to export the LV into the VG mapping

# vgexport -m /etc/lvmconf/vgprddata.mapfile -pvs /dev/vgprddata

21. Tail the file again, and will see the new LV added into the VG mapping of VG vgprddata

22. Now FTP the mapfile and the cntl files to the cluster server bluekandi

# scp /etc/lvmconf/vgprddata.mapfile root@bluekandi:/tmp

# scp /etc/cmcluster/PRD/prdms.cntl root@bluekandi:/tmp

23. Before we go about on server bluekandi, we make sure that the new LV has been added to the LVM conf file

root@hedkandi:(lvmconf)> ls -l /dev/vgprddata | grep lvol1*
brw-r-----   1 root       sys         64 0x040001 Nov 16  2008 lvol1
brw-r-----   1 root       sys         64 0x04000a Sep  8 12:58 lvol10
brw-r-----   1 root       sys         64 0x040002 Nov 16  2008 lvol2
brw-r-----   1 root       sys         64 0x040003 Nov 16  2008 lvol3
brw-r-----   1 root       sys         64 0x040004 Nov 16  2008 lvol4
brw-r-----   1 root       sys         64 0x040005 Nov 16  2008 lvol5
brw-r-----   1 root       sys         64 0x040006 Nov 16  2008 lvol6
brw-r-----   1 root       sys         64 0x040007 Nov 16  2008 lvol7
brw-r-----   1 root       sys         64 0x040008 Nov 16  2008 lvol8
brw-r-----   1 root       sys         64 0x040009 Dec 22  2008 lvol9
crw-r-----   1 root       sys         64 0x040001 Nov 16  2008 rlvol1
crw-r-----   1 root       sys         64 0x04000a Sep  8 12:58 rlvol10
crw-r-----   1 root       sys         64 0x040002 Nov 16  2008 rlvol2
crw-r-----   1 root       sys         64 0x040003 Nov 16  2008 rlvol3
crw-r-----   1 root       sys         64 0x040004 Nov 16  2008 rlvol4
crw-r-----   1 root       sys         64 0x040005 Nov 16  2008 rlvol5
crw-r-----   1 root       sys         64 0x040006 Nov 16  2008 rlvol6
crw-r-----   1 root       sys         64 0x040007 Nov 16  2008 rlvol7
crw-r-----   1 root       sys         64 0x040008 Nov 16  2008 rlvol8
crw-r-----   1 root       sys         64 0x040009 Dec 22  2008 rlvol9

Now, we can go to server bluekandi and import the updated VG, LV and FS from server hedkandi. Because both servers are clustered, you will need to run the hardware scan commands

# ioscan -fn

# insf -e


1. We make a backup of the the cntl file on server bluekandi

root@bluekandi:(root)> cd /etc/cmcluster/PRD/

root@bluekandi:(PRD)> ls -l 
total 1600
-rwxr-xr-x   1 root       sys          13018 Oct 18  2006 hanfs.sh
-rwx------   1 root       sys          60199 Dec 14 16:17 prdms.cntl-------------bkp this one
-rwx------   1 root       sys          59574 Oct 28  2006 prdms.cntl-28Nov2006
-rwx------   1 root       sys          59692 Mar 27  2007 prdms.cntl.20070609
-rwx------   1 root       sys          59758 Mar 20  2008 prdms.cntl.20080320
-rwx------   1 root       sys          59904 Mar 20  2008 prdms.cntl.20081222
-rwx------   1 root       sys          60050 Dec 22  2008 prdms.cntl.20090908
-rw-r--r--   1 root       sys          56557 Nov 16  2008 prdms.cntl.log
-rw-r--r--   1 root       root         18727 Oct 28  2006 prdms.cntl.log_1
-rw-r--r--   1 root       root         80408 Jun 17  2007 prdms.cntl.log_2
-rwx------   1 root       sys          60199 Jan 20 20:31 prdms.cntl_new
-rwx------   1 root       sys          59779 Jun  9  2007 prdms.cntl_pJun07
-rwx------   1 root       sys          59571 Oct 27  2006 prdms.cntl_save
-rwx------   1 root       sys          16325 Jun 18  2007 prdms.conf
-rwxr-xr-x   1 root       sys          17481 Jun 14  2007 sap.conf
-rwxr-xr-x   1 root       sys           3839 Dec 17  2002 sapdbci.cntl

2. Remember the cntl and mapfile files in tmp earlier that we SCP from hedkandi to bluekandi? We move them to current directory and change it's ownership and permissions if we have to

# cp -p /tmp/prdms.cntl /etc/cmcluster/PRD/prdms.cntl

3. On cluster server bluekandi, we make a backup of the VG map file

cd /etc/lvmconf/

root@bluekandi:(lvmconf)> ls -lrt vgprddata.mapfile*
-rw-r--r--   1 root       sys            104 Jan 21 11:02 vgprddata.mapfile
root@bluekandi:(lvmconf)>

cp -p vgprddata.mapfile vgprddata.mapfile.date

4. Then, we move the files from /tmp and replace them with the VG map file

# cp /tmp/vgprddata.mapfile vgprddata.mapfile

# chown root:sys vgprddata.mapfile

5. We do one final check of the map and control files

# ll /etc/lvmconf/vgprddata.mapfile
# ll /etc/cmcluster/PRD/prdms.cntl

6. Finally, we can export the VG into the cluster server bluekandi

# vgexport -v vgprddata
# ls -l /dev/vgprddata

We go back to cluster server hedkandi to import the VG

# vgimport -m /etc/lvmconf/vgprddata.mapfile -vs /dev/vgprddata

# tail /etc/lvmconf/vgprddata.mapfile

7. We make sure the minor number is same on both servers

hedkandi

ls -l /dev/*/group | grep vgprddata

bluekandi

ls -l /dev/*/group | grep vgprddata

We go to cluster server bluekandi and set the autopath of the disk to SST

# autopath set-l "LUN_WWN" -b SST

We then enable the cluster package prdms

# cmmodpkg-e prdms

# cmviewcl

Done!

No comments:

Post a Comment