hadoop学习笔记.docx

1、hadoop学习笔记一、Hadoop的结构：NameNodeDataNodeSecondaryNameNodeLobTrackerTaskTracker二、Hadoop安装流程:1、 新建用户和组2、 配置服务器域名和主机名3、 SSH无密码登陆4、 JDK的安装5、 Hadoop配置6、 格式化与启动7、 WEB监控Hadoop的运行状态三、下载：VirtualBoxCenterOSJDKHadoopHBasePutty四、Hadoop FS Shellhadoop fs command parameterhelpls 、 lsrdu、 duscountcopyFromLocal、copyT

2、oLocal、cpmoveFromLocal、mv、putcatchgrp、chmod、chownexpungeget、getmergemkdirrm、rmrsetrep、stat、tail、test、text、touchzLinux的安装在安装时内存大小如果小于652MB时默认会进入text model(文本安装模式)。因此在此处需要将内存调大些网络：VirtualBox有4种网络连接方式1、 NAT2、 Bridged Adapter3、 Internal4、 Host-only AdapterVMWare有三种连接方式（上面的除去Internal）1、 NATNAT:Network Ad

3、dress Translation:网络地址转换NAT模式是最简单的实现虚拟机上网的方式4、 Host-only Adapter(主机模型)主机模型是一种比较复杂的模式，需要有比较扎实的网络基础知识才能玩转，可以说前面的几种模式所实现的功能在这种模式下，通过虚拟机及网卡的设置都可以被实现网络配置：DEVICE=eth0BOOTPROTO=staticHWADDR=08:00:27:34:F4:69NM_CONTROLLED=yesONBOOT=yesTYPE=EthernetUUID=35b4fc2a-f79b-4013-a465-0a6aa793b487IPADDR=192.168.137.

4、10NETMASK=255.255.255.0GATEWAY=192.168.137.1DNS1=101.226.4.6DNS2=192.168.0.1Hadoop的配置步骤：添加用户和组rootlocalhost # groupadd hadoop #添加组rootlocalhost # useradd hadoop g hadoop #添加用户rootlocalhost # su hadoop #切换hadooplocalhost root$ exit #退出exitHosts的配置：rootlocalhost #vi /etc/hosts追加192.168.137.10 master #

5、本机IP地址 主机名rootlocalhost #vi /etc/sysconfig/network修改HOSTNAME=masterrootlocalhost #hostname #查看主机名（需要重启才能修改主机名）rootlocalhost #hostname master #临时把主机名修改为masterrootlocalhost #logout #登出SSH无密码登陆rootmaster #yum y install openssh-clients #安装SSH客户端#ssh默认是需要输入密码的：root master #ssh localhostyes#接下来需要输入密码，我们按C

6、trl+C结束，接下来我们设置ssh的无密码登陆root master #su hadoop #切换到hadoop用户hadoopmaster root$ cdhadoopmaster $ ssh-keygen t rsa P f /.ssh/id_rsa #设置无密码登陆公钥和私钥对(除了rsa格式的公私钥还有dsa格式的：ssh-keygen t dsa P f /.ssh/id_dsa)hadoopmaster $ pwd/home/hadoophadoopmaster $ cd .ssh/hadoopmaster .ssh$ ls alh #查看生成的公钥(id_rsa.pub)和私钥

7、(id_rsa)hadoopmaster .ssh$ cat id_rsa.pub authorized_keyshadoopmaster .ssh$ ls -alhhadoopmaster .ssh$ chmod 600 authorized_keyshadoopmaster .ssh$ cd .hadoopmaster $ chmod 700 .ssh/ssh无密码登陆设置成功。测试：hadoopmaster $ ssh masterexhadoopmaster $ exit #退出sshJDK的安装：hadoopmaster $ exit #退出hadoop用户，进入root用户root

8、master $ cd software/ #却换到JDK安装包目录rootmaster software$ mkdir /usr/java #创建解压目录rootmaster software$ tar -zxvf jdk-7u25-linux-i586.tar.gz C /usr/java #解压（rpm包的解压：rpm ivh jdkxxx.rpm）rootmaster software$ cd /usr/javarootmaster java$ ls alhrootmaster $ vi /etc/profile #配置PATHexport JAVA_HOME=/usr/java/jd

9、kexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarrootmaster $ source /etc/profile #使配置生效hadoop的配置rootmaster # cd /software/rootmaster software# ls -alhrootmaster software#mkdir /software_jyrootmaster software# tar -zxvf hadoop-1.2.1.tar.gz C /software

10、_jy rootmaster software# cd /software_jy/rootmaster software_jy# ls -alh总用量 12Kdrwxr-xr-x. 3 root root 4.0K 4月 22 11:19 .dr-xr-xr-x. 27 root root 4.0K 4月 22 11:19 .drwxr-xr-x. 15 root root 4.0K 7月 23 2013 hadoop-1.2.1rootmaster software_jy# chown -R hadoop:hadoop hadoop-1.2.1/rootmaster software_jy#

11、 mv hadoop-1.2.1/ hadooprootmaster software_jy# ls -alhrootmaster software_jy# su hadoophadoopmaster software_jy$ cd hadoop/confhadoopmaster hadoop$ ls -alhhadoopmaster conf$ vi hadoop-env.sh #配置环境变量export JAVA_HOME=/usr/java/jdkexport HADOOP_HEAPSIZE=256export HADOOP_PID_DIR=/home/$USER/pidshadoopm

12、aster conf$ vi core-site.xml fs.default.name hdfs:/master:9000 hadoop.tmp.dir /home/$user.name/tmp hadoopmaster conf$ vi hdfs-site.xml #配置hdfs需要配置namenode的数据目录、datanode的数据目录、集群个数（dfs.replication） dfs.replication 1 dfs.name.dir /home/$user.name/dfs_name dfs.data.dir /home/$user.name/dfs_data hadoopma

13、ster conf$ vi mapred-site.xml mapred.job.tracker master:9001 mapred.system.dir /home/$user.name/mapred_system mapred.local.dir /home/$user.name/mapred_local hadoopmaster conf$ vi masters #配置辅助名称节点hadoopmaster conf$ vi slaves #配置slaves到此已配置完成。验证一下是否真正的配置成功：hadoopmaster conf$ cd ./binhadoopmaster bin$

14、 ./hadoop namenode -formathadoopmaster bin$ ./start-all.shhadoopmaster bin$ jps查看命令是否能正常执行。hadoopmaster bin$ ./hadoop dfsadmin report #执行dfsadmin命令，-report作用：打印一个报告查看hdfs中有什么内容：hadoopmaster bin$ ./hadoop fs -ls /hadoopmaster bin$ ./hadoop fs -ls /home/hadoop/mapred_system查看日志文件：hadoopmaster bin$ cd

15、./logshadoopmaster logs$ ls alhHbase的配置：hadoopmaster logs$ cd ./confhadoopmaster conf$ vi hdfs-site.xml dfs.replication 1 dfs.name.dir /home/$user.name/dfs_name dfs.data.dir /home/$user.name/dfs_data dfs.support.append true dfs.datanode.max.xcievers 4096 重启hadoophadoopmaster conf$ cd ./binhadoopmast

16、er bin$ ./stop-all.shhadoopmaster bin$ ./start-all.shhadoopmaster bin$ ./hadoop dfsadmin report #查看启动状态http:/192.168.137.10:50030/http:/192.168.137.10:50070/开始安装Hbase解压Hbasehadoopmaster bin$ exitexitrootmaster # cd /softwarerootmaster software# tar -zxvf hbase-0.94.10.tar.gz -C /software_jy/rootmast

17、er software# cd /software_jy/rootmaster software_jy# ls alhrootmaster software_jy# mv hbase-0.94.10/ hbaserootmaster software_jy# chown -R hadoop:hadoop hbase/rootmaster software_jy# cd hbase/rootmaster hbase# ls alhrootmaster hbase# cd conf/rootmaster conf# ls -alhrootmaster conf# vi hbase-env.shex

18、port JAVA_HOME=/usr/java/jdkexport HBASE_MANAGES_ZK=truerootmaster conf# vi hbase-site.xml hbase.rootdir hdfs:/master:9000/hbase hbase.cluster.distributed true hbase.master hdfs:/master:60000 hbase.tmp.dir /home/$user.name/tmp hbase.zookeeper.quorum master hbase.zookeeper.property.clientPort 2181 hb

19、ase.zookeeper.property.dataDir /home/$user.name/zookeeper rootmaster conf# vi regionservers覆盖hadoop类库：与hadoop集成时需要，hadoop类库需要与Hbase类库相同rootmaster conf# cd ./lib/ #进入hbase的lib目录rootmaster lib# ls hadoop-core-1.0.4.jar #查看hbase使用的hadooprootmaster lib# rm -rf hadoop-core-1.0.4.jar #删除hbase中使用的hadoop类库r

20、ootmaster lib# cp /software_jy/hadoop/hadoop-core-1.2.1.jar . #将hadoop中的hadoop类库复制到hbase中的lib目录下rootmaster lib# chown -R hadoop:hadoop hadoop-core-1.2.1.jar #授权hadooprootmaster lib# su hadoop #切换hadoop用户hadoopmaster lib$ cd ./bin/ #进入hbase的bin目录hadoopmaster bin$ ./start-hbase.sh #启动hbasehadoopmaster

21、 bin$ jpshttp:/192.168.137.10:60010/http:/192.168.137.10:60030/http:/192.168.137.10:60010/zk.jsphadoopmaster bin$ ./hbase shell #执行shell命令hbase(main):001:0 list #查看表hbase(main):003:0 create user,f1,f2,f3 #创建表hbase(main):004:0 list #查看表hbase(main):005:0 put user,1,f1:name,suhbase(main):007:0 scan use

22、rhbase(main):007:0 exit #退出shell命令hadoopmaster bin$ ./stop-hbase.sh #停止hbaseMapReduce配置：hadoopmaster bin$ cd ././hadoop/conf #进入hadoop的配置目录hadoopmaster conf$ vi hadoop-env.sh #进入hadoop环境变量配置文件export HBASE_HOME=/software_jy/hbaseexport HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.94.10.jar:$HBASE_HOME/hbase-0.94.10-tests.jar:$HBASE_HOME/conf:$HBASE_HOME/lib/zookeeper-3.4.5.jar:$HBASE_HOME/lib/protobuf-java-2.4.0a.jarhadoopmaster c