hadoop大数据应用开发环境配置.docx

1、hadoop大数据应用开发环境配置Hadoop大数据应用开发环境配置一、 运行环境计算机名IP角色备注Kingo-mn192.168.0.230主节点(namenode), masters负责总管分布式数据的调度。Kingo-ms192.168.0.231副节点(JobTracker), masters分解任务的执行。资源控制器Kingo-d1192.168.0.232数据节点1(Datanode1)Slaves/TaskTracker，负责分布式数据存储以及任务的执行。Kingo-d2192.168.0.238数据节点2(Datanode2) Kingo-d3192.168.0.239数据节

2、点3(Datanode3) 说明修改 /etc/sysconfig/network文件可以修改主机名；主机名不能含“_”。Ip主机名程序进程192.168.0.230Kingo-mnJdkHadoophbaseNamenodeDFSZKFailoverControllerHamsterresourceManager192.168.0.231Kingo-msJdkHadoophbaseNamenodeSecondaryNamenodeJobTrackerDFSZKFailoverControllerHamsterresourceManager192.168.0.232Kingo-d1JdkHad

3、oopZookeeperhbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.238Kingo-d2JdkHadoopZooKeeper hbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerTaskTracker192.168.0.239Kingo-d3JdkHadoopZookeeperHbaseDatanodenodeManagerJournalNodeQuorumPeerMainHregionServerT

4、askTracker二、 创建帐号四台机器均创建kingo用户组和bigdata帐号，密码统一为Kingo123。#groupadd kingo# useradd -d /home/bigdata -g kingo bigdata#passwd bigdata /设置密码为Kingo123#cd /home #chgrp -R kingo bigdata /设置目录所属组#chmod -R g=rwx bigdata /设置目录所属组的权限为读、写、执行三、 绑定hostname与IP#vi /etc/hosts，增加内容如下：192.168.0.230 kingo-mn192.168.0.2

5、31 kingo-ms192.168.0.238 kingo-rm192.168.0.232 kingo-d1192.168.0.239 kingo-d2四、 设置节点之间的免密码通信1、 向各节点复制Hosts文件scp /etc/hosts kingo-ms:/etc/hostsscp /etc/hosts kingo-d1:/etc/hostsscp /etc/hosts kingo-d2:/etc/hostsscp /etc/hosts kingo-d3:/etc/hosts2、 在各节点上生成各自SSH秘钥对#ssh-keygen -t dsa -P -f /.ssh/id_dsa#

6、cd /.ssh#cat id_dsa.pubauthorized_keys#chmod 600 authorized_keys3、复制公钥到某一节点#ssh-copy-id -i /.ssh/id_dsa.pub kingo-mn4、复制密钥到各个节点。在kingo-mn上执行如下命令：#scp /root/.ssh/authorized_keys kingo-ms:/root/.ssh/;scp /root/.ssh/authorized_keys kingo-d1:/root/.ssh/;scp /root/.ssh/authorized_keys kingo-d2:/root/.ssh

7、/; scp /root/.ssh/authorized_keys kingo-d3:/root/.ssh/五、 安装JDK把jdk-7u80-linux-x64.rpm安装包复制到/bigdata/setup文件夹下，执行：#rpm ivh jdk-7u80-linux-x64.rpmJdk安装到默认目录/usr/java/jdk1.7.0_80下。#vi /etc/profile加入如下配置信息：export JAVA_HOME=/usr/java/jdk1.7.0_80export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/d

8、t.jarexport PATH=.:$JAVA_HOME/bin:$PATH#source profile /执行profile内容，确认是否配置成功#java version /查看是否JDK安装成功六、 设置hadoop环境变量#vi /etc/profileexport HADOOP_HOME=/home/bigdata/hadoop-2.7.1export PATH=.:$HADOOPHOME/sbin:$HADOOP_HOME/bin:$JAVAHOME/bin:$PATHexport HADOOP_LOG_DIR=/home/bigdata/hadoop-2.7.1/logsex

9、port YARN_LOG_DIR=$HADOOP_LOG_DIRexportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexportHADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib#hadoop version /查看hadoop是否安装成功七、 配置hadoop1、配置hadoop2.7.1/etc/hadoop下的hadoop-env.sh、yarn-env.sh、mapred-env.sh修改JAVA_HOME值：export JAVA_HOME=/usr/java/jdk1.7

10、.0_802、配置hadoop2.7.1/etc/hadoop/slaveskingo-d1kingo-d2kingo-d33、配置hadoop2.7.1/etc/hadoop/core-site.xmlfs.defaultFShdfs:/kingo-mn:9000/ 设定 namenode 的 主机名 及 端口 hadoop.tmp.dir/home/bigdata/tmp/hadoop-$user.name 存放临时文件的目录 hadoop.proxyuser.hadoop.hosts*hadoop.proxyuser.hadoop.groups*4、配置hadoop2.7.1/etc/h

11、adoop/hdfs-site.xml dfs.namenode.http-address kingo-mn:50070 NameNode获得fsimage和edits dfs.namenode.secondary.http-address kingo-ms:50090 SecondNameNode获得最新的fsimage dfs.namenode.rpc-address kingo-mn:9000 kingo-mn dfs.replication3 设定 HDFS 存储文件的副本个数，默认为3 dfs.namenode.name.dir file:/home/bigdata/hadoop-2

12、.7.1/hdfs/name namenode 用来持续存放命名空间和交换日志的本地文件系统路径 dfs.datanode.data.dirfile:/home/bigdata/hadoop-2.7.1/hdfs/data DataNode 在本地存放块文件的目录列表，用逗号分隔 dfs.namenode.checkpoint.dir file:/home/bigdata/hadoop-2.7.1/hdfs/namesecondary 设置 secondary 存放临时镜像的本地文件系统路径，如果这是一个用逗号分隔的文件列表，则镜像将会冗余复制到所有目录，只对 secondary 有效 dfs

13、.webhdfs.enabledtruedfs.stream-buffer-size131072 默认是4KB，作为hadoop缓冲区，用于hadoop读hdfs的文件和写hdfs的文件，还有map的输出都用到了这个缓冲区容量，对于现在的硬件很保守，可以设置为128k(131072),甚至是1M(太大了map和reduce任务可能会内存溢出) dfs.namenode.checkpoint.period 3600 两次 checkpoints 之间的间隔，单位为秒，只对 secondary 有效 dfs.permissions.enabled false If true, enable per

14、mission checking in HDFS. If false, permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. 5、配置hadoop2.7.1/etc/hadoop/mapred-site.xml mapreduce.framework.nameyarn mapred

15、uce.jobtracker.addresshdfs:/kingo-mn:9001mapreduce.jobhistory.addresskingo-mn:10020MapReduce JobHistory Server host:port, defaultport is10020.mapreduce.jobhistory.webapp.addresskingo-mn:19888MapReduce JobHistory Server Web UI host:port, defaultport is19888.6、配置hadoop2.7.1/etc/hadoop/yarn-site.xmlyar

16、n.resourcemanager.hostnamekingo-mnyarn.nodemanager.aux-servicesmapreduce_shuffleyarn.nodemanager.aux-services.mapreduce.shuffle.classorg.apache.hadoop.mapred.ShuffleHandleryarn.resourcemanager.addresskingo-mn:8032yarn.resourcemanager.scheduler.addresskingo-mn:8030yarn.resourcemanager.resource-tracke

17、r.addresskingo-mn:8031yarn.resourcemanager.admin.addresskingo-mn:8033yarn.resourcemanager.webapp.addresskingo-mn:8088八、 启动hadoop#hdfs namenode -format /格式化hadoop将集群信息复制到kingo-ms上#hdfs namenode bootstrapstandby#start-all.sh九、 管理hadoop主节点：http:/192.168.0.230:50070/集群：http:/192.168.0.230:8088/节点一：http:

18、/192.168.0.231:8042/节点二：http:/192.168.0.232:8042/节点三：http:/192.168.0.239:8042/注意：如果启动正常，无法访问主节点web页面，可以关闭防火墙。#service iptables stop 或#chkconfig iptables off一十、 集群验证1、 在HDFS创建数据目录hadoop fs -mkdir -p /data/wordcount /存放数据文件hadoop fs -mkdir -p /output /存放MapReduce任务的输出结果2、 上传本地文件到HDFS中：hadoop fs -put /

19、home/bigdata/hadoop-2.7.1/etc/hadoop/*.xml data/wordcount/3、 查看上传后的文件：hadoop fs -ls /data/wordcount4、 运行WordCount例子hadoopjar/home/bigdata/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /data/wordcount /output/wordcount1.先启动zookeeper在h4,h5,h6上zkServer.sh start查看状态zk

20、Server.sh status（会发现有一个leader，两个follower）2.启动journalnode(在h1上启动)hadoop-daemons.sh start journalnode3格式化HDFS(在h1上启动)hadoop namenode format此时会在hadoop目录里面产生tmp文件夹，将这个文件夹拷贝到h2上3.格式化ZK(在h1上启动)hdfs zkfc formatZK4.启动hadoop(在h1上启动)start-all.shHBase大数据应用开发环境配置一、 设置Zookeeper环境变量解压zookeeper-3.4.6，配置如下环境变量：#vi

21、/etc/profile# Set zookeeper environment variablesexport ZOOKEEPER_HOME=/home/zookeeper-3.4.6# Add zookeeper bin/ directory to PATHexport PATH=$PATH:$ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf二、 配置Zookeeper配置文件#mkdir data#mkdir datalog将conf/zoo_sample.cfg改成conf/zoo.cfg。#vi zoo.cfgdataDir=/home/zookeeper-3.4.6/datadataLogDir=/home/zookeeper-3.4.6/datalogserver.1=kingo-mn:2888:3888server.2=kingo-ms:2888:3888server.3=kingo-d1:2888:3888server.4=kingo-d2:2888:3888在data文件夹下创建myid文件#vi myid输入：1，保存。其它服务输入zoo.cfg中对应的数字。三、 设置HBase环境变量解压hbase-1.0.1.1，配置如下环境变量：#vi /etc/profile# Set