GPFS配置手册.docx

1、GPFS配置手册江西省农信社GPFS安装手册V1.01.江西省农信社GPFS规划1.1江西省农信社GPFS集群规划IP地址节点名操作系统集群角色10.0.147.98AIX 7100-02-02-1316Quorum-manager10.0.147.97AIX 7100-02-02-1316Quorum-manager10.0.147.49Redhat 6.4 for Powerclient10.0.147.50Redhat 6.4 for Powerclient10.0.147.51Redhat 6.4 for Powerclient10.0.147.52Redhat 6.4 for Pow

2、erclient10.0.147.53Redhat 6.4 for Powerclient10.0.147.54Redhat 6.4 for Powerclient10.0.147.55Redhat 6.4 for Powerclient10.0.147.56Redhat 6.4 for Powerclient10.0.147.57Redhat 6.4 for Powerclient10.0.147.58Redhat 6.4 for Powerclient10.0.147.59Redhat 6.4 for Powerclient10.0.147.60Redhat 6.4 for Powercl

3、ient10.0.147.61Redhat 6.4 for Powerclient10.0.147.62Redhat 6.4 for Powerclient10.0.147.63Redhat 6.4 for Powerclient10.0.147.64Redhat 6.4 for Powerclient10.0.147.65Redhat 6.4 for Powerclient10.0.147.66Redhat 6.4 for Powerclient10.0.147.67Redhat 6.4 for Powerclient10.0.147.68Redhat 6.4 for Powerclient

4、10.0.147.69Redhat 6.4 for Powerclient10.0.147.70Redhat 6.4 for Powerclient10.0.147.71Redhat 6.4 for Powerclient10.0.147.72Redhat 6.4 for Powerclient10.0.147.73Redhat 6.4 for Powerclient10.0.147.74Redhat 6.4 for Powerclient10.0.147.75Redhat 6.4 for Powerclient10.0.147.76Redhat 6.4 for Powerclient10.0

5、.147.77Redhat 6.4 for Powerclient10.0.147.78Redhat 6.4 for Powerclient10.0.147.79Redhat 6.4 for Powerclient10.0.147.80Redhat 6.4 for Powerclient10.0.147.81Redhat 6.4 for Powerclient10.0.147.82Redhat 6.4 for Powerclient10.0.147.83Redhat 6.4 for Powerclient10.0.147.84Redhat 6.4 for Powerclient10.0.147

6、.85Redhat 6.4 for Powerclient10.0.147.86Redhat 6.4 for Powerclient10.0.147.87Redhat 6.4 for Powerclient10.0.147.88Redhat 6.4 for Powerclient10.0.147.89Redhat 6.4 for Powerclient10.0.147.90Redhat 6.4 for Powerclient10.0.147.91Redhat 6.4 for Powerclient10.0.147.92Redhat 6.4 for Powerclient10.0.147.93R

7、edhat 6.4 for Powerclient10.0.147.94Redhat 6.4 for Powerclient10.0.147.95Redhat 6.4 for Powerclient10.0.147.96Redhat 6.4 for Powerclient1.2集群Quorum方式集群Quorum方式采用Node Quorum + Tiebreaker disks,两个AIX节点加上一块专用的SAN磁盘,目前只有一个AIX节点,第二个AIX节点和SAN磁盘需要后续添加。1.3NSD磁盘组成PV_NAMESIZENSD_NAMEFS_NAMENSD SERVERS LISThdi

8、sk2500Gbjxrccb_nsd1gpfshdisk3500Gbjxrccb_nsd2gpfs1.4文件系统基本信息1.5GPFS系统环境示意图GPFS Server: 2个AIX分区连接V7000SAN Storage: SVCGPFS Client: 48个Power Linux分区GPFS管理网络: 千兆交换机2. GPFS安装前准备2.1安装介质及补丁准备安装包:GPFS v3.5.0.0补丁：GPFS-3.5.0.112.2Openssl及Openssh确认AIX上openssl及openssh是否已安装,Linux默认已经安装;lslpp -l |grep open2.3Lin

9、ux节点安装ksh#rpm -ivh ksh-20100621-19.el6.ppc64.rpm2.4Linux节点修改网络参数修改/etc/sysctl.conf文件,需修改的网络参数如下:# increase Linux TCP buffer limitsnet.core.rmem_max = 8388608net.core.wmem_max = 8388608# increase default and maximum Linux TCP buffer sizesnet.ipv4.tcp_rmem = 4096 262144 8388608net.ipv4.tcp_wmem = 4096

10、 262144 8388608# increase max backlog to avoid dropped packetsdev_max_backlog = 2500修改完毕后执行sysctl -p /etc/sysctl.conf让配置生效3.GPFS安装配置3.1配置ssh信任关系在节点上生成并配置ssh密钥对#ssh-keygen -t rsa# cd .ssh# lsid_rsa id_rsa.pub# cat id_rsa.pub authorized_keys# chmod 600 authorized_keys# echo StrictHostKeyChecking no co

11、nfig# chmod 600 config修改各节点的/etc/hosts文件,加入对集群中各节点的别名和ip地址10.0.147.49 10.0.147.50 10.0.147.51 10.0.147.52 10.0.147.53 10.0.147.54 10.0.147.55 10.0.147.56 10.0.147.57 10.0.147.58 10.0.147.59 10.0.147.60 10.0.147.61 10.0.147.62 10.0.147.63 10.0.147.64 10.0.147.65 10.0.147.66 10.0.147.67 10.0.147.68 10

12、.0.147.69 10.0.147.70 10.0.147.71 10.0.147.72 10.0.147.73 10.0.147.74 10.0.147.75 10.0.147.76 10.0.147.77 10.0.147.78 10.0.147.79 10.0.147.80 10.0.147.81 10.0.147.82 10.0.147.83 10.0.147.84 10.0.147.85 10.0.147.86 10.0.147.87 10.0.147.88 10.0.147.89 10.0.147.90 10.0.147.91 10.0.147.92 10.0.147.93 10

13、.0.147.94 10.0.147.95 10.0.147.96 10.0.147.97 10.0.147.98 验证本机无需密码的ssh登录#ssh hostname将.ssh目录复制到其他各节点#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/验证各节点间无密码的ssh登录3.2AIX安装GPFS软件1）gpfs3.5.0安装包放在/home/gpfsaix/iso目录下2）切换/home/gpfsaix/iso目录下,通过smitty installp安装GPFS软件3

14、)安装补丁gpfs3.5版本，需要升级到3.5.0.11以上，补丁下载地址：下载后上传到服务器后解压：gzip -d -c filename.tar.gz |tar -xvf -使用smitty update_all安装补丁4）安装后检查#lslpp -l|grep gpfs如果之前已经创建了cluster，在所有node都down的状态下，加载GPFS版本到最新版本：mmchconfig release=LATEST;如果是新装的GPFS,则可以忽略此步。3.3Linux安装GPFS软件1)上传安装介质安装文件位于/home/gpfsrpm/，补丁文件位于/home/gpfsrpm/updt

15、e；2)安装GPFS软件在安装介质所在目录下执行：#rpm -ivh gpfs*.rpm3)升级GPFS软件版本在补丁文件所在目录下执行：#rpm -Uvh gpfs*.rpm4)检查安装结果#rpm -qa |grep gpfs5)Building the GPFS portability layer#cd /usr/lpp/mmfs/src#make Autoconfig#make World#make InstallImages3.4GPFS集群配置修改系统环境变量AIX平台下修改/etc/environment在PATH后增加/usr/lpp/mmfs/bin#vi /etc/envi

16、ronmentLinux平台下修改/root/.bashprofile,在PATH后增加/usr/lpp/mmfs/bin创建集群按照规划，本次集群中包含1个AIX节点和48个Linux client；首先在1个AIX节点平台创建GPFS集群创建集群定义文件#cd /home#mkdir gpfs#cd gpfs#vi jxrcc_cluster10.0.147.98:quorum-manager a)创建GPFS集群#mmcrcluster N jxrccb_cluster p 10.0.147.98 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_clu

17、ster A# mmcrcluster -N jxrccb_cluster -p 10.0.147.98 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_cluster -AWarning: Permanently added (RSA) to the list of known hosts.Tue Nov 19 20:01:38 CST 2013: 6027-1664 mmcrcluster: Processing node mmcrcluster: Command successfully completedmmcrcluster: 6027-1254

18、Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed.b)配置许可# mmchlicense server -accept -N 10.0.147.98# mmchlicense server -accept -N 10.0.147.98The following nodes will be designated as possessing GPFS server licenses: mmchlicense

19、: Command successfully completedc)添加Linux节点通过如下命令依次添加48个Linux节点：# mmaddnode N # mmchlicense server -accept -N # mmaddnode N # mmchlicense server -accept -N 查看集群状态#mmlsclusterd)修改集群参数#mmchconfig prefetchThreads=1000 N 10.0.147.98# mmchconfig pagepool=4G N 10.0.147.98# mmchconfig failureDetectionTime=

20、15# mmchconfig MaxMBps=4096e)启动集群#mmstartup -a启动后，查看集群状态#mmgetstate -a确保所有节点的状态为Active创建NSD磁盘a)修改fscsi参数在所有直接访问SAN盘的节点上执行#lsdev |grep fscsi|awk print chdev -l,$1,-a dyntrk=yes -a fc_err_recov=fast_fail|shb)识别外置存储磁盘在所有直接访问SAN盘的节点上执行#cfgmgrc)修改hdisk参数在所有直接访问SAN盘的节点上执行#lsdev -Cc disk|grep MPIO |awk pri

21、nt chdev -l,$1,-a algorithm=round_robin -a queue_depth=8 -a reserve_policy=no_reserve|shd)创建NSD描述文件#cd /gpfs#vi nsd.confhdisk2: 10.0.147.98:jxrccb_nsd1:hdisk3: 10.0.147.98:jxrccb_nsd2:备注文件格式：DiskName:ServerList:DiskUsage:FailureGroup:DesiredName:StoragePool在本集群中，将AIX节点定义为NSD Server，如果NSD较多可以将每块盘的首选NSD Server设置为不同，以均衡NSD Sever的负载；e)生成NSD盘#mmcrnsd -F nsd.conff)查看NSD盘#mmlsnsd# mmlsnsd File system Disk name NSD servers -