CDH514部署手册.docx

1、 CDH514 部署手册部署手册 CDH大数据集群部署手册 cdh5.14 版本 一、概述 针对海尔大数据应用的需求，本文完成了 CDH5.12大数据集群的部署，并记录了详细步骤。二、集群部署准备 1.集群硬件信息 1.1集群各主机 IP与主机名映射表 1.2主机 qdcdh00-01 内存情况：主机 qdcdh02-09内存情况：1.3主机逻辑 cpu 个数：$cat/proc/cpuinfo|grep processor|wc-l 主机物理 cpu 个数：$cat/proc/cpuinfo|grep physical id|sort-u|wc-l 主机单个 cpu 核心数：$cat/pro

2、c/cpuinfo|grep cpu cores|sort-u 主机单个 cpu core 线程数：$lscpu|grep Thread 主机 qdcdh00-01的 cpu 信息：主机 qdcdh02-09的 cpu 信息：1.4主机硬盘分区情况查看命令：$df-h 主机 qdcdh00-04硬盘分区情况：主机 qdcdh05-09硬盘分区情况：2.集群服务部署规划 2.1集群各主机角色分为四类：边缘节点一台(qdcdh00)、管理节点一台(qdcdh01)、主节点三台(qdcdh02-04)和数据节点五台(qdcdh05-09);2.2各服务在集群主机上的分配：服务主机 qdcdh00 q

3、dcdh01 qdcdh02 qdcdh03 qdcdh04 qdcdh05-9 CM服务 Server Server(未启用)Agen Agen Agen Agen Database Mysql(M)Mysql(S)kerberos KDC(M)KDC(S)openldap Server(M)Server(M)HAProxy Server HDFS Gateway Namenode Namenode Datanode YARN Gateway History Resource manager Resource manager Node Manager server Hive hiveserv

4、er2 hiveserver2 metastore Impala catalog statestore impalad HBase Gateway master master master Region Spark gateway history server gateway Oozie server Sqoop Gateway server Zookeeper server server server Kafka Kudu master master master tablet Sentry Gateway server Hue server/LB server 3.管理节点到其他节点免密钥

5、配置 3.1配置 qdcdh01 到其他主机的免密钥登录：$ssh-keygen-t rsa$ssh-copy-id qdcdh00-9 4.系统优化 4.1调整 swap分区的使用：$echo vm.swappiness=1 /etc/sysctl.conf$sysctl-p 4.2关闭透明大页：$echo never /sys/kernel/mm/transparent_hugepage/defrag$echo never /sys/kernel/mm/transparent_hugepage/enabled 将这两句添加到/etc/rc.local 中。5.时钟同步 5.1设置 qdcd

6、h01 为 ntp server,修改/etc/ntp.conf文件，将红色部分内容注释，添加蓝色内容：rootqdcdh01 cdh_install#cat/etc/ntp.conf#For more information about this file,see the man pages#ntp.conf(5),ntp_acc(5),ntp_auth(5),ntp_clock(5),ntp_misc(5),ntp_mon(5).driftfile/var/lib/ntp/drift#Permit time synchronization with our time source,but

7、do not#permit the source to query or modify the service on this system.restrict default kod nomodify notrap nopeer noquery restrict-6 default kod nomodify notrap nopeer noquery#Permit all access over the loopback interface.This could#be tightened as well,but to do so would effect some of#the adminis

8、trative functions.restrict 127.0.0.1 restrict-6:1#Hosts on local network are less restricted.#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap#Use public servers from the pool.ntp.org project.#Please consider joining the pool(http:/www.pool.ntp.org/join.html).#server 0.centos.pool.ntp.org ibu

9、rst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.org iburst#server 3.centos.pool.ntp.org iburst server 127.127.1.0 fudge 127.127.1.0 stratum 8 restrict 10.138.225.0 mask 255.255.255.0 nomodify notrap#broadcast 192.168.1.255 autokey#broadcast server#broadcastclient#broadcast client#br

10、oadcast 224.0.1.1 autokey#multicast server#multicastclient 224.0.1.1#multicast client#manycastserver 239.255.254.254#manycast server#manycastclient 239.255.254.254 autokey#manycast client#Enable public key cryptography.#crypto includefile/etc/ntp/crypto/pw#Key file containing the keys and key identi

11、fiers used when operating#with symmetric key cryptography.keys/etc/ntp/keys#Specify the key identifiers which are trusted.#trustedkey 4 8 42#Specify the key identifier to use with the ntpdc utility.#requestkey 8#Specify the key identifier to use with the ntpq utility.#controlkey 8#Enable writing of

12、statistics records.#statistics clockstats cryptostats loopstats peerstats 5.2启动 qdcdh01 上的 ntpd 服务，并设置开机启动：$service ntpd start$chkconfig ntpd on 5.3在集群的其他主机上，修改/etc/ntp.conf文件，修改 server 配置字段内容问qdcdh01 主机，注释红色部分，添加蓝色部分：rootqdcdh00#cat/etc/ntp.conf#For more information about this file,see the man page

13、s#ntp.conf(5),ntp_acc(5),ntp_auth(5),ntp_clock(5),ntp_misc(5),ntp_mon(5).driftfile/var/lib/ntp/drift#Permit time synchronization with our time source,but do not#permit the source to query or modify the service on this system.restrict default kod nomodify notrap nopeer noquery restrict-6 default kod

14、nomodify notrap nopeer noquery#Permit all access over the loopback interface.This could#be tightened as well,but to do so would effect some of#the administrative functions.restrict 127.0.0.1 restrict-6:1#Hosts on local network are less restricted.#restrict 192.168.1.0 mask 255.255.255.0 nomodify not

15、rap#Use public servers from the pool.ntp.org project.#Please consider joining the pool(http:/www.pool.ntp.org/join.html).#server 0.centos.pool.ntp.org iburst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.org iburst#server 3.centos.pool.ntp.org iburst server #broadcast 192.168.1.255 au

16、tokey#broadcast server#broadcastclient#broadcast client#broadcast 224.0.1.1 autokey#multicast server#multicastclient 224.0.1.1#multicast client#manycastserver 239.255.254.254#manycast server#manycastclient 239.255.254.254 autokey#manycast client#Enable public key cryptography.#crypto includefile/etc

17、/ntp/crypto/pw#Key file containing the keys and key identifiers used when operating#with symmetric key cryptography.keys/etc/ntp/keys#Specify the key identifiers which are trusted.#trustedkey 4 8 42#Specify the key identifier to use with the ntpdc utility.#requestkey 8#Specify the key identifier to

18、use with the ntpq utility.#controlkey 8#Enable writing of statistics records.#statistics clockstats cryptostats loopstats peerstats 5.4如 5.2 操作，重启各主机的 ntpd 服务，并添加开机启动。6.CDH本地 yum源搭建 6.1在 qdcdh01 主机上安装 httpd服务，并将本次安装的 cdh5.12 相关安装文件放到/var/www/html 下：$yum install httpd 6.2在集群的各主机/etc/yum.repos.d 下添加自建

19、的 cdh_5.12.repo 文件：cm_5.12 name=cloudera manager 5.12.1 baseurl=http:/qdcdh01/cdh_5.12_files/cm_5.12.1 gpgcheck=0 mysql_5.7 name=mysql 5.7 baseurl=http:/qdcdh01/cdh_5.12_files/other_rpms/mysql_5.7 gpgcheck=0 cdh_other_paks name=cdh other packages baseurl=http:/qdcdh01/cdh_5.12_files/other_rpms/other

20、_deps gpgcheck=0 6.3检查 yum源中是否存在 cdh5.12 的相关包：7.MySQL 主从安装配置 7.1在 qdcdh01 和 qdcdh00 上安装 MySQL，其中 qdcdh01 为 master，qdcdh00 为slave；7.2卸载系统中原先安装的 mysql-libs：$rpm-e mysql-libs-nodeps 7.3在 qdcdh01 上安装 MySQL server：$yum install mysql mysql-server mysql-community-libs-compat 7.4初始化 MySQL数据库：$mysqld-user=my

21、sql-datadir=/var/lib/mysql-initialize 7.5创建 MySQL日志目录：$mkdir/var/log/mysql$chown mysql:mysql-R/var/log/mysql 7.6修改配置文件/etc/f:rootqdcdh01 yum.repos.d#cat/etc/f#For advice on how to change settings please see#http:/ mysqld#Remove leading#and set to the amount of RAM for the most important data#cache i

22、n MySQL.Start at 70%of total RAM for dedicated server,else 10%.#innodb_buffer_pool_size=128M#Remove leading#to turn on a very important data integrity option:logging#changes to the binary log between backups.#log_bin#Remove leading#to set options mainly useful for reporting servers.#The server defau

23、lts are faster for transactions and fast SELECTs.#Adjust sizes as needed,experiment to find the optimal values.#join_buffer_size=128M#sort_buffer_size=2M#read_rnd_buffer_size=2M datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock server-id=38 transaction-isolation=READ-COMMITTED key_buffer_size=

24、32M max_allowed_packet=32M thread_stack=256K thread_cache_size=64 query_cache_limit=8M query_cache_size=64M query_cache_type=1 max_connections=550 log_bin=/var/lib/mysql/mysql_binary_log binlog_format=mixed relay-log=/var/lib/mysql/mysql-relay-bin relay-log-index=/var/lib/mysql/mysql-relay-bin.index

25、 master-info-file=/var/lib/mysql/mysql-master.info relay-log-info-file=/var/lib/mysql/mysql-relay-log.info log-bin=/var/lib/mysql/mysql-bin read_buffer_size=2M read_rnd_buffer_size=16M sort_buffer_size=8M join_buffer_size=8M innodb_file_per_table=1 innodb_flush_log_at_trx_commit=2 innodb_log_buffer_

26、size=64M innodb_buffer_pool_size=4G innodb_thread_concurrency=8 innodb_flush_method=O_DIRECT innodb_log_file_size=512M bind-address=0.0.0.0 default-storage-engine=innodb sql_mode=STRICT_ALL_TABLES#Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 log-error=/

27、var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid 7.7启动 MySQL服务，并设置开机启动：$service mysqld start$chkconfig mysqld on 7.8通过 mysql_secure_installation 命令设置 mysql，需要输入的 mysql 初始化密码位于/var/log/mysqld.log 中：$mysql_secure_installation 设置 mysql 密码为Pass_haier,并设置不允许 root 用户远程登录，然后用新密码登录测试：$mysql-uroot-hloc

28、alhost-p 7.9将 mysql-connector-java.jar 放到 Java 共享目录下：$cp mysql-connector-java-5.1.38-bin.jar/usr/share/java/mysql-connector-java.jar 7.10以上步骤同样在 qdcdh00 上操作一遍；7.11在 qdcdh00 和 qdcdh01 两台机器上同步执行如下命令：$mysql-uroot-p 7.12在 master（qdcdh01）上设置用来同步数据的用户 rep、slave 的 host（qdcdh00）及密码repmysql:mysqluse mysql;my

29、sqlGRANT ALL PRIVILEGES ON*.*TO root%IDENTIFIED BY mysql WITH GRANT OPTION;mysqlGRANT REPLICATION SLAVE ON*.*TO IDENTIFIED BY repmysql;mysqlFLUSH TABLES WITH READ LOCK;mysqlSHOW MASTER STATUS;+-+-+-+-+-+|File|Position|Binlog_Do_DB|Binlog_Ignore_DB|Executed_Gtid_Set|+-+-+-+-+-+|mysql-bin.000001|1393|

30、+-+-+-+-+-+1 row in set(0.00 sec)此时 master上不要做任何操作；7.13在 slave（qdcdh00）上设置进行设置 master 的主机名，master 的同步用户名及同步用户密码，以及在 master 上执行的 SHOW MASTER STATUS 的属性：mysqluse mysql;mysqlstop slave;mysqlCHANGE MASTER TO MASTER_HOST=,MASTER_USER=rep,MASTER_PASSWORD=repmysql,mysqlMASTER_LOG_FILE=mysql-bin.000001,MAST

31、ER_LOG_POS=1393;mysqlstart slave;7.14在 master MySQL上进行测试：mysqlUNLOCK TABLES;mysqlcreate database test1;mysqluse test1;mysqlcreate table tb1(id INT(4),text(varchar(10);7.15在 slave mysql 上查看数据库是否同步过去，同步则 MySQL 主从配置完成。8.创建集群各服务的数据库 create database scm default character set utf8;grant all on scm.*to scm

32、%identified by Pass_scm;create database amon default character set utf8;grant all on amon.*to amon%identified by Pass_amon;create database rman default character set utf8;grant all on rman.*to rman%identified by Pass_rman;create database metastore default character set utf8;grant all on metastore.*t

33、o hive%identified by Pass_hive;create database sentry default character set utf8;grant all on sentry.*to sentry%identified by Pass_sentry;create database nav default character set utf8;grant all on nav.*to nav%identified by Pass_nav;create database navms default character set utf8;grant all on navms.*to navms%identified by Pass_navms;create database hue default character set utf8 default collate u