Ubuntu服务器安装HadoopHBase Hive.docx

1、Ubuntu服务器安装HadoopHBase HiveIP规划：192.168.88.111 ubuntuserver1192.168.88.112 ubuntuserver2192.168.88.113 ubuntuserver3192.168.88.114 ubuntuserver4192.168.88.115 ubuntuserver5Hadoop规划：ubuntuserver1 NameNode、JobTracker、HadoopThriftServerubuntuserver2 DataNode、TaskTracker、SecondaryNameNodeubuntuserver3 D

2、ataNode、TaskTrackerubuntuserver4 DataNode、TaskTrackerubuntuserver5 DataNode、TaskTrackerHBase规划：ubuntuserver3 HRegionServer、HQuorumPeerubuntuserver4 HMaster、HQuorumPeerubuntuserver5 HRegionServer、HQuorumPeerHive规划：ubuntuserver5 MySQL服务器、Hive服务器1.基础工作为了加快apt下载速度，修改源（每台机器必做）：$sudo cp /etc/apt/sources.l

3、ist /etc/apt/sources.list.baksudo vi /etc/apt/sources.list:%s/:wqsudo apt-get updatesudo apt-get upgrade修改机器名（每台机器必做）：sudo vi /etc/hostname修改IP（每台机器必做）：sudo vi /etc/network/interfacessudo /etc/init.d/networking restart修改Hosts（每台机器必做）sudo vi /etc/hosts127.0.0.1 localhost192.168.88.111 ubuntuserver119

4、2.168.88.112 ubuntuserver2192.168.88.113 ubuntuserver3192.168.88.114 ubuntuserver4192.168.88.115 ubuntuserver5安装OpenJDK（每台机器必做）：sudo apt-get install openjdk-6-jdk安装OpenSSH（每台机器必做）：sudo apt-get install openssh-server安装编译相关支持包（每台机器必做）：sudo apt-get install ant libboost-dev libboost-test-dev libboost-pr

5、ogram-options-dev libevent-dev automake libtool flex bison pkg-config g+ libssl-devubuntuserver1、2、3、4、5机器创建hadoop用户组、用户及程序目录并授权：sudo addgroup hadoopsudo adduser -ingroup hadoop hadoopsudo mkdir /opt/hadoopsudo mkdir /opt/hadoopdatasudo chown -R hadoop:hadoop /opt/hadoopsudo chown -R hadoop:hadoop /

6、opt/hadoopdataubuntuserver3、4、5机器创建hbase程序目录并授权：sudo mkdir /opt/hbasesudo chown -R hadoop:hadoop /opt/hbaseubuntuserver5机器创建hive程序目录并授权：sudo mkdir /opt/hivesudo chown -R hadoop:hadoop /opt/hiveubuntuserver1、2、3、4、5机器建立hadoop用户ssh无密码登录：su hadoopssh-keygen -t rsa -P cd /.sshcat id_rsa.pub authorized_k

7、eysssh localhostyesexitexit完成hadoop用户ssh无密码登录配置后，需从ubuntuserver1以hadoop用户ssh依次登录ubuntuserver2、3、4、5，确认登录成功（首次登录需要确认）。2.安装（安装配置只需在一台机器上操作，完成后用SCP指令复制到其他机器即可，以下安装配置均在ubuntuserver3上进行）切换到hadoop用户：su hadoop上传或下载hadoop、hbase安装包到/home/hadoop：解压（需保证在各自程序根目录，即进入程序目录后不用再次进入下一层目录）：tar -zxf /home/ubuntu/hadoop

8、-1.1.1.tar.gz -C /opt/hadooptar -zxf /home/ubuntu/hbase-0.94.4-security.tar.gz -C /opt/hbase3.配置Hadoop编译hadoop：cd /opt/hadoopant compile删除hadoop/build目录：rm -r /opt/hadoop/build修改start_thrift_server.sh权限：chmod 775 /opt/hadoop/src/contrib/thriftfs/scripts/start_thrift_server.sh配置HadoopThriftServer：vi

9、/opt/hadoop/src/contrib/thriftfs/scripts/start_thrift_server.shTOP=/opt/hadoopCLASSPATH=$CLASSPATH:$TOP/build/contrib/thriftfs/classes/:$TOP/build/classes/:$TOP/conf/配置hadoop-env.sh（找到#export JAVA_HOME=,去掉#，然后加上本机jdk的路径，3264位系统路径有别，请查看真实路径）：vi /opt/hadoop/conf/hadoop-env.shexport JAVA_HOME=/usr/lib/

10、jvm/java-6-openjdk-i386配置core-site.xml：vi /opt/hadoop/conf/core-site.xml fs.default.name hdfs:/ubuntuserver1:9000 hadoop.tmp.dir /home/hadoop/tmp 配置mapred-site.xml：vi /opt/hadoop/conf/core-site.xml mapred.job.tracker ubuntuserver1:9001 mapred.local.dir /opt/hadoopdata/mapred/local true mapred.system

11、.dir /opt/hadoopdata/mapred/system true配置hdfs-site.xml：vi /opt/hadoop/conf/hdfs-site.xml dfs.name.dir /opt/hadoopdata/name dfs.data.dir /opt/hadoopdata/data dfs.replication 3 fs.checkpoint.dir /opt/hadoopdata/secondary fs.checkpoint.period 1800 fs.checkpoint.size 33554432 fs.trash.interval 1440 dfs.

12、datanode.du.reserved 1073741824 dfs.block.size 134217728 dfs.permissions false配置masters（添加SecondaryNameNode主机）：vi /opt/hadoop/conf/mastersubuntuserver2配置slaves（添加slave主机，一行一个）：vi /opt/hadoop/conf/slavesubuntuserver2ubuntuserver3ubuntuserver4ubuntuserver54.配置HBase配置hbase-env.sh：vi /opt/hbase/conf/hba

13、se-env.sh#本机jdk的路径，3264位系统路径有别export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386#运动自带的zookeeperexport HBASE_MANAGES_ZK=trueexport HADOOP_HOME=/opt/hadoopexport HBASE_HOME=/opt/hbase配置hbase-site.xml（hbase.zookeeper.quorum必须是奇数个）：vi /opt/hbase/conf/hbase-site.xml hbase.rootdir hdfs:/ubuntuserver1:9000/

14、hbase hbase.cluster.distributed true hbase.master ubuntuserver4:6000 hbase.zookeeper.quorum ubuntuserver3,ubuntuserver4,ubuntuserver5配置regionservers：vi /opt/hbase/conf/regionserversubuntuserver3ubuntuserver55.SCP复制将ubuntuserver3服务器/opt/hadoop目录复制到ubuntuserver1、2、4、5服务器相同目录：scp -r /opt/hadoop hadoop:

15、hadoopubuntuserver1:/optscp -r /opt/hadoop hadoop:hadoopubuntuserver2:/optscp -r /opt/hadoop hadoop:hadoopubuntuserver4:/optscp -r /opt/hadoop hadoop:hadoopubuntuserver5:/opt将ubuntuserver3服务器/opt/hbase目录复制到ubuntuserver3、5服务器相同目录：scp -r /opt/hbase hadoop:hadoopubuntuserver3:/optscp -r /opt/hbase hado

16、op:hadoopubuntuserver5:/opt6.格式化namenode，hadoop用户登录到ubuntuserver1运行如下指令：/opt/hadoop/bin/hadoop namenode -format7.启停服务器集群启动Hadoop集群，hadoop用户登录到ubuntuserver1运行如下指令：/opt/hadoop/bin/start-all.sh运行jps指令，查看各机器进程，若与Hadoop规划一节中描述的进程一致，则成功。停止Hadoop集群，hadoop用户登录到ubuntuserver1运行如下指令：/opt/hadoop/bin/stop-all.sh

17、停止Hadoop集群时，应先停止HBase集群。启动HadoopThriftServer（指定了9606端口，不指定则使用随机端口），hadoop用户登录到ubuntuserver1运行如下指令：/opt/hadoop/src/contrib/thriftfs/scripts/start_thrift_server.sh 9606 &启动HBase集群，hbase用户登录到ubuntuserver4运行如下指令：/opt/hbase/bin/start-hbase.sh启动HBase集群时，应先启动Hadoop集群。运行jps指令，查看各机器进程，若与HBase规划一节中描述的进程一致，则成功

18、。停止HBase集群，hbase用户登录到ubuntuserver4运行如下指令：/opt/hbase/bin/stop-hbase.sh启动HBaseThriftServer（默认端口：9090），hbase用户登录到ubuntuserver4运行如下指令：/opt/hbase/bin/hbase thrift start &进入控制台：/opt/hbase/bin/hbase shellhbaselistHBase shell语法（http:/wiki.apache.org/hadoop/Hbase/Shell）：建表：create Article,NAME = Data,VERSIONS

19、 = 1列出表： list8. Hive安装配置（ubuntuserver5）：安装MySQL服务器sudo apt-get install mysql-server配置MySQL：sudo vi /etc/mysql/f#注释bind-address#bind-address = 127.0.0.1sudo service mysql restart创建Hive使用的账号（用户名：hive，密码：hivepwd，允许从任意机器远程登录）：mysql -uroot -pmysqlinsert into mysql.user(Host,User,Password) values(%,hive,p

20、assword(hivepwd);mysqlgrant all privileges on *.* to hive% identified by hivepwd with grant option;mysqlflush privileges;mysqlexit;切换到hadoop用户：su - hadoop上传或下载hive安装包到/home/hadoop：解压：tar -zxf /home/ubuntu/hive-0.10.0-bin.tar.gz -C /opt/hive下载MySQL Connector/J 5.1.24到/home/hadoop，解压后将mysql-connector-

21、java-5.1.24-bin.jar复制到/opt/hive/lib：wget http:/mysql.ntu.edu.tw/Downloads/Connector-J/mysql-connector-java-5.1.24.tar.gztar -zxf mysql-connector-java-5.1.24.tar.gzcp mysql-connector-java-5.1.24/mysql-connector-java-5.1.24-bin.jar /opt/hive/lib删除/opt/hive/lib下的旧版本hbase-xxxxx.jar、hbase-xxxxx-tests.jar

22、、zookeeper-xxxx.jarrm /opt/hive/lib/hbase-0.92.0.jarrm /opt/hive/lib/hbase-0.92.0-tests.jarrm /opt/hive/lib/zookeeper-3.4.3.jar复制/opt/hbase下的hbase-xxxxx.jar、hbase-xxxxx-tests.jar,/opt/hbase/lib下的zookeeper-xxxx.jar、protobuf-java-xxxxx.jar复制到/opt/hive/libcp /opt/hbase/hbase-0.94.4-security.jar /opt/hi

23、ve/libcp /opt/hbase/hbase-0.94.4-security-tests.jar /opt/hive/libcp /opt/hbase/lib/zookeeper-3.4.5.jar /opt/hive/libcp /opt/hbase/lib/protobuf-java-2.4.0a.jar /opt/hive/lib/配置hive-env.shcp /opt/hive/conf/hive-env.sh.template /opt/hive/conf/hive-env.shvi /opt/hive/conf/hive-env.shexport JAVA_HOME=/us

24、r/lib/jvm/java-6-openjdk-i386export HADOOP_HEAPSIZE=64export HADOOP_HOME=/opt/hadoopexport HBASE_HOME=/opt/hbaseexport HIVE_HOME=/opt/hive配置hive-site.xml，添加或修改以下节点（注意：原始文档中有的description标记没有关闭，请根据错误信息等位到错误行进行校正）：cp /opt/hive/conf/hive-default.xml.template /opt/hive/conf/hive-site.xmlvi /opt/hive/conf

25、/hive-site.xml hbase.zookeeper.quorum ubuntuserver2,ubuntuserver3,ubuntuserver4 hive.metastore.local true javax.jdo.option.ConnectionURL jdbc:mysql:/localhost:3306/HiveDB?createDatabaseIfNotExist=true javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName hive javax.jdo.option.ConnectionPassword hivepwd配置hive-log4j.properties：针对0.10.0版本的特别处理。cp /opt/hive/con