As usual, we start with a user request
Hi Unix Adm
Pls create a new filesystem - /oracle/PRD/sapdata11 of size 80GB.
Thanks
First things first, we make sure the FS isn't created, and by the looks of the FS name request, there may be a /oracle/PRD/sapdata10 existing on the server, so look for it and find the VG name from it. We will need the VG name to add the new disks to, in order to create a FS
1. Look for the LV and VG to use to create the new FS
root@hedkandi:(root)> bdf /oracle/PRD/sapdata10
Filesystem kbytes used avail %used Mounted on
/dev/vgprddata/lvol10
52428800 25665564 25090534 51% /oracle/PRD/sapdata10
2. So the current LV name is lvol10 and the VG name is vgprddata. We also check the size of available FREE PE on this VG which is about 38 GB
root@hedkandi:(root)> vgdisplay vgprddata
--- Volume groups ---
VG Name /dev/vgprddata
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 10
Act PV 10
Max PE per PV 21247
VGDA 20
PE Size (Mbytes) 4
Total PE 212440
Alloc PE 202468
Free PE 9972
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
The math I did to get this is
root@hedkandi:(root)> bc
9972*4
39888
39888/1024
38 GB
In order for me to raise a Change Request for this, I need to know the existing disk size so I can get the SAN team to allocate a new disk of the same size for my FS creation. I run the vgdisplay -v command to get a complete list of disks for VG vgprddata and check a few disks' sizes
From a truncated vgdisplay -v output:
--- Physical volumes ---
PV Name /dev/dsk/c10t0d3
PV Name /dev/dsk/c12t0d3 Alternate Link
PV Name /dev/dsk/c14t0d3 Alternate Link
PV Name /dev/dsk/c8t0d3 Alternate Link
PV Status available
Total PE 21244
Free PE 0
Autoswitch On
Proactive Polling On
Here, I picked disk c8t2d7 to check its size. Make sure you add the r in the /dsk when you run the diskinfo command because VG output doesn't show the raw character "r" of the SCSI disk
root@hedkandi:(root)> diskinfo /dev/rdsk/c8t2d7
SCSI describe of /dev/rdsk/c8t2d7:
vendor: HP
product id: HSV200
type: direct access
size: 87031808 Kbytes
bytes per sector: 51
How do I know what size disk to ask for? Lets transform the size in kb to GB
root@hedkandi:(root)> bc
87031808/1024/1024
83
After SAN team has allocated the LUN, you do a hardware scan to detect the disks
ioscan -fn
root@hedkandi:(root)> insf -e
insf: Installing special files for asio0 instance 0 address 0/0/1/0
insf: Installing special files for asio0 instance 1 address 0/0/1/1
insf: Installing special files for sdisk instance 0 address 0/0/3/0.0.0.0
insf: Installing special files for sctl instance 0 address 0/0/3/0.0.7.0
I have truncated the output because it will show a pretty long list of all the hardware on the server
What I like to do is check the syslog at this stage, because it will show the new disks added
Jan 21 15:34:37 hedkandi vmunix: 0/5/1/0/4/0.1.0.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/5/1/0/4/0.1.1.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/2/1/0/4/0.1.0.0.0.3.0 sdisk
Jan 21 15:34:37 hedkandi vmunix: 0/2/1/0/4/0.1.1.0.0.3.0 sdisk
And what's left to do is to do an I/O scan on the disks individually and make sure their status is CLAIMED
root@hedkandi:(root)> ioscan -fnH 0/5/1/0/4/0.1.0.0.0.3.0
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 63 0/5/1/0/4/0.1.0.0.0.3.0 sdisk CLAIMED DEVICE HP HSV200
/dev/dsk/c14t3d0 /dev/rdsk/c14t3d0
root@hedkandi:(root)> ioscan -fnH 0/5/1/0/4/0.1.1.0.0.3.0
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 64 0/5/1/0/4/0.1.1.0.0.3.0 sdisk CLAIMED DEVICE HP HSV200
/dev/dsk/c8t3d0 /dev/rdsk/c8t3d0
root@hedkandi:(root)> ioscan -fnH 0/2/1/0/4/0.1.0.0.0.3.0
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 65 0/2/1/0/4/0.1.0.0.0.3.0 sdisk CLAIMED DEVICE HP HSV200
/dev/dsk/c10t3d0 /dev/rdsk/c10t3d0
root@hedkandi:(root)> ioscan -fnH 0/2/1/0/4/0.1.1.0.0.3.0
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 66 0/2/1/0/4/0.1.1.0.0.3.0 sdisk CLAIMED DEVICE HP HSV200
/dev/dsk/c12t3d0 /dev/rdsk/c12t3d0
From now on, it will be the LVM management, and we start off by adding the disk to the VG vgprddata
1. Create the PV first
pvcreate /dev/rdsk/c14t3d0
2. Extend the VG of vgprddata
vgextend vgprddata /dev/dsk/c14t3d0
3. Check the extended VG size
vgdisplay vgprddata
4. Create the LV
lvcreate -L 84992 -n lvol11 -s y /dev/vgprddata
5. Check the lvdisplay, LV changes must be added
lvdisplay /dev/vgprddata/lvol11
6. Make the directory
mkdir /oracle/PRD/sapdata11
7. Make the FS
newfs -F vxfs -o largefiles /dev/vgprddata/rlvol10
8. Mount the LV to the FS
mount /dev/vgprddata/lvol11 /oracle/PRD/sapdata11
9. Change the ownership of the FS
chown -R oraprd:dba /oracle/PRD/sapdata11
10. Edit the Bad Block Allocation to ON if it is set to NONE. Our VG has the setting to ON
lvchange -r N /dev/VG/LV----------this is only to edit if the Bad Block is set to None, change it to ON
11. We then make changes to the control file of the cluster server hedkandi to add the new FS into it. To know which files to edit, run the cmviewcl command
root@hedkandi:(root)> cmviewcl
CLUSTER STATUS
kandi up
NODE STATUS STATE
hedkandi up running
PACKAGE STATUS STATE AUTO_RUN NODE
prdms up running enabled hedkandi--------->this one to edit
NODE STATUS STATE
bluekandi up running
Therefore...
root@hedkandi:(root)> find / -name *.cntl
/etc/cmcluster/PRD/sapdbci.cntl
/etc/cmcluster/PRD/prdms.cntl-------------->this one to edit
/etc/cmcluster/prdms.old/prdms.cntl
/opt/samba/HA/samba.cntl
/opt/cmcluster/sap/LC/saplc.cntl
/opt/cmcluster/sap/SID/sapwas.cntl
/opt/cmcluster/sap/contrib/J2EE620/sap620Engine.cntl
12. Go to the FS
cd /etc/cmcluster/PRD/prdms.cntl
13. Make a backup copy of the file
cp -p prdms.cntl prdms.cntl.21012010
14. Run the more command to have a look at the VGs and the LVs in the server hedkandi
root@hedkandi:(PRD)> more /etc/cmcluster/PRD/prdms.cntl | grep vgprddata
VG[0]="vgprddata"
LV[13]="/dev/vgprddata/lvol1"; FS[13]="/oracle/PRD/sapdata1"; FS_MOUNT_OPT[13]="-o delaylog,largefiles"; FS_FSCK_OPT[13]=""; FS_TYPE[13]="vxfs"
LV[14]="/dev/vgprddata/lvol2"; FS[14]="/oracle/PRD/sapdata2"; FS_MOUNT_OPT[14]="-o delaylog,largefiles"; FS_FSCK_OPT[14]=""; FS_TYPE[14]="vxfs"
LV[15]="/dev/vgprddata/lvol3"; FS[15]="/oracle/PRD/sapdata3"; FS_MOUNT_OPT[15]="-o delaylog,largefiles"; FS_FSCK_OPT[15]=""; FS_TYPE[15]="vxfs"
LV[16]="/dev/vgprddata/lvol4"; FS[16]="/oracle/PRD/sapdata4"; FS_MOUNT_OPT[16]="-o delaylog,largefiles"; FS_FSCK_OPT[16]=""; FS_TYPE[16]="vxfs"
LV[17]="/dev/vgprddata/lvol5"; FS[17]="/oracle/PRD/sapdata5"; FS_MOUNT_OPT[17]="-o delaylog,largefiles"; FS_FSCK_OPT[17]=""; FS_TYPE[17]="vxfs"
LV[18]="/dev/vgprddata/lvol6"; FS[18]="/oracle/PRD/sapdata6"; FS_MOUNT_OPT[18]="-o delaylog,largefiles"; FS_FSCK_OPT[18]=""; FS_TYPE[18]="vxfs"
LV[20]="/dev/vgprddata/lvol7"; FS[20]="/oracle/PRD/sapdata7"; FS_MOUNT_OPT[20]="-o delaylog,largefiles"; FS_FSCK_OPT[20]=""; FS_TYPE[20]="vxfs"
LV[22]="/dev/vgprddata/lvol8"; FS[22]="/oracle/PRD/sapdata8"; FS_MOUNT_OPT[22]="-o delaylog,largefiles"; FS_FSCK_OPT[22]=""; FS_TYPE[22]="vxfs"
LV[23]="/dev/vgprddata/lvol9"; FS[23]="/oracle/PRD/sapdata9"; FS_MOUNT_OPT[23]="-o delaylog,largefiles"; FS_FSCK_OPT[23]=""; FS_TYPE[23]="vxfs"
LV[24]="/dev/vgprddata/lvol10"; FS[24]="/oracle/PRD/sapdata10"; FS_MOUNT_OPT[24]="-o delaylog,largefiles"; FS_FSCK_OPT[24]=""; FS_TYPE[24]="vxfs"
15. From here, we can see that the last LV runs at number 24, so you will need to add another line under it for the FS recently created
LV[25]="/dev/vgprddata/lvol11"; FS[25]="/oracle/PRD/sapdata11"; FS_MOUNT_OPT[25]="-o delaylog,largefiles"; FS_FSCK_OPT[25]=""; FS_TYPE[24]="vxfs"
16. Run cmviewcl command again, and make sure prdms package is UP
# cmviewcl
17. Disable autorun
# cmmodpkg -d prdms
18.Run cmviewcl command again, and make sure the prdms package status is DOWN and autorun is disabled
# cmviewcl
18. Now, we make a note of the ownership and permissions for VG map file of VG vgprddata
root@hedkandi:(PRD)> ll /etc/lvmconf/vgprddata.mapfile
-rw-r--r-- 1 root sys 104 Jan 21 10:30 /etc/lvmconf/vgprddata.mapfile
19. Do a tail on the file
root@hedkandi:(PRD)> tail /etc/lvmconf/vgprddata.mapfile
1 lvol1
2 lvol2
3 lvol3
4 lvol4
5 lvol5
6 lvol6
7 lvol7
8 lvol8
9 lvol9
10 lvol10
20. Run vgexport to export the LV into the VG mapping
# vgexport -m /etc/lvmconf/vgprddata.mapfile -pvs /dev/vgprddata
21. Tail the file again, and will see the new LV added into the VG mapping of VG vgprddata
22. Now FTP the mapfile and the cntl files to the cluster server bluekandi
# scp /etc/lvmconf/vgprddata.mapfile root@bluekandi:/tmp
# scp /etc/cmcluster/PRD/prdms.cntl root@bluekandi:/tmp
23. Before we go about on server bluekandi, we make sure that the new LV has been added to the LVM conf file
root@hedkandi:(lvmconf)> ls -l /dev/vgprddata | grep lvol1*
brw-r----- 1 root sys 64 0x040001 Nov 16 2008 lvol1
brw-r----- 1 root sys 64 0x04000a Sep 8 12:58 lvol10
brw-r----- 1 root sys 64 0x040002 Nov 16 2008 lvol2
brw-r----- 1 root sys 64 0x040003 Nov 16 2008 lvol3
brw-r----- 1 root sys 64 0x040004 Nov 16 2008 lvol4
brw-r----- 1 root sys 64 0x040005 Nov 16 2008 lvol5
brw-r----- 1 root sys 64 0x040006 Nov 16 2008 lvol6
brw-r----- 1 root sys 64 0x040007 Nov 16 2008 lvol7
brw-r----- 1 root sys 64 0x040008 Nov 16 2008 lvol8
brw-r----- 1 root sys 64 0x040009 Dec 22 2008 lvol9
crw-r----- 1 root sys 64 0x040001 Nov 16 2008 rlvol1
crw-r----- 1 root sys 64 0x04000a Sep 8 12:58 rlvol10
crw-r----- 1 root sys 64 0x040002 Nov 16 2008 rlvol2
crw-r----- 1 root sys 64 0x040003 Nov 16 2008 rlvol3
crw-r----- 1 root sys 64 0x040004 Nov 16 2008 rlvol4
crw-r----- 1 root sys 64 0x040005 Nov 16 2008 rlvol5
crw-r----- 1 root sys 64 0x040006 Nov 16 2008 rlvol6
crw-r----- 1 root sys 64 0x040007 Nov 16 2008 rlvol7
crw-r----- 1 root sys 64 0x040008 Nov 16 2008 rlvol8
crw-r----- 1 root sys 64 0x040009 Dec 22 2008 rlvol9
Now, we can go to server bluekandi and import the updated VG, LV and FS from server hedkandi. Because both servers are clustered, you will need to run the hardware scan commands
# ioscan -fn
# insf -e
1. We make a backup of the the cntl file on server bluekandi
root@bluekandi:(root)> cd /etc/cmcluster/PRD/
root@bluekandi:(PRD)> ls -l
total 1600
-rwxr-xr-x 1 root sys 13018 Oct 18 2006 hanfs.sh
-rwx------ 1 root sys 60199 Dec 14 16:17 prdms.cntl-------------bkp this one
-rwx------ 1 root sys 59574 Oct 28 2006 prdms.cntl-28Nov2006
-rwx------ 1 root sys 59692 Mar 27 2007 prdms.cntl.20070609
-rwx------ 1 root sys 59758 Mar 20 2008 prdms.cntl.20080320
-rwx------ 1 root sys 59904 Mar 20 2008 prdms.cntl.20081222
-rwx------ 1 root sys 60050 Dec 22 2008 prdms.cntl.20090908
-rw-r--r-- 1 root sys 56557 Nov 16 2008 prdms.cntl.log
-rw-r--r-- 1 root root 18727 Oct 28 2006 prdms.cntl.log_1
-rw-r--r-- 1 root root 80408 Jun 17 2007 prdms.cntl.log_2
-rwx------ 1 root sys 60199 Jan 20 20:31 prdms.cntl_new
-rwx------ 1 root sys 59779 Jun 9 2007 prdms.cntl_pJun07
-rwx------ 1 root sys 59571 Oct 27 2006 prdms.cntl_save
-rwx------ 1 root sys 16325 Jun 18 2007 prdms.conf
-rwxr-xr-x 1 root sys 17481 Jun 14 2007 sap.conf
-rwxr-xr-x 1 root sys 3839 Dec 17 2002 sapdbci.cntl
2. Remember the cntl and mapfile files in tmp earlier that we SCP from hedkandi to bluekandi? We move them to current directory and change it's ownership and permissions if we have to
# cp -p /tmp/prdms.cntl /etc/cmcluster/PRD/prdms.cntl
3. On cluster server bluekandi, we make a backup of the VG map file
cd /etc/lvmconf/
root@bluekandi:(lvmconf)> ls -lrt vgprddata.mapfile*
-rw-r--r-- 1 root sys 104 Jan 21 11:02 vgprddata.mapfile
root@bluekandi:(lvmconf)>
cp -p vgprddata.mapfile vgprddata.mapfile.date
4. Then, we move the files from /tmp and replace them with the VG map file
# cp /tmp/vgprddata.mapfile vgprddata.mapfile
# chown root:sys vgprddata.mapfile
5. We do one final check of the map and control files
# ll /etc/lvmconf/vgprddata.mapfile
# ll /etc/cmcluster/PRD/prdms.cntl
6. Finally, we can export the VG into the cluster server bluekandi
# vgexport -v vgprddata
# ls -l /dev/vgprddata
We go back to cluster server hedkandi to import the VG
# vgimport -m /etc/lvmconf/vgprddata.mapfile -vs /dev/vgprddata
# tail /etc/lvmconf/vgprddata.mapfile
7. We make sure the minor number is same on both servers
hedkandi
ls -l /dev/*/group | grep vgprddata
bluekandi
ls -l /dev/*/group | grep vgprddata
We go to cluster server bluekandi and set the autopath of the disk to SST
# autopath set-l "LUN_WWN" -b SST
# autopath set-l "LUN_WWN" -b SST
We then enable the cluster package prdms
# cmmodpkg-e prdms
# cmviewcl
No comments:
Post a Comment