1、hadoop安装配置笔记Hadoop安装配置教程一、安装JDK1.8.0_401. 网上下载好压缩包jdk-8u40-linux-x64.gz后,手动安装JDK1.8.0_40sudo mkdir /usr/lib/jvmsudo tar zxvf jdk-8u40-linux-x64.gz -C /usr/lib/jvm2. 配准环境变量sudo gedit /etc/profile在profile文件最下面输入:#set Java Evironmentexport JAVA_HOME=/usr/lib/jvm/jdk1.8.0_40export CLASSPATH=.:$JAVA_HOME
2、/lib:$CLASSPATHexport PATH=$JAVA_HOME/bin:$PATH3. 验证JDK是否安装成功java version如果验证成功,会出现如下信息:否则会出现如下信息:依次输入如下命令:sudo update-alternatives -install /usr/bin/java java /usr/lib/jvm/jdk1.8.0_40/bin/java 300sudo update-alternatives -install /usr/bin/javac javac /usr/lib/jvm/jdk1.8.0_40/bin/javac 300sudo updat
3、e-alternatives -config java此时再输入命令java version即可看到所安装的JDK版本信息了。二、配置SSH免密码登陆假设电脑的当前目录是/home/hadoop,其中hadoop是用户名,主机名字是master,从机名字是slave1和slave21. 确认已经连接网络,输入如下命令安装SSH:sudo apt-get install ssh2. 三台电脑下输入如下命令:sudo gedit /etc/hosts在打开的hosts文件中添加如下信息:172.16.132.37 slave1172.16.132.51 slave2172.16.132.54 ma
4、ster3. 三台电脑都输入如下命令,以实现三台电脑免密码登录它们自身:ssh-keygen -t rsa -P (注意空格,最后那个是两个单引号而不是双引号)此时出现如下信息:按回车键,出现如下信息:此时会在目录/home/hadoop下自动生成一个隐藏文件夹.ssh,.ssh文件夹下生成了两个文件:id_rsa(私钥)和id_rsa.pub(公钥)输入如下命令,把公钥追加到authorized_keys中:cat /.ssh/id_rsa.pub /.ssh/authorized_keys输入如下命令:ls /.ssh可看到.ssh文件夹下有三个文件:输入如下命令,可以免密码登录它们自身:
5、ssh localhost4. master免密码登录slave1的实现:在master下面输入如下命令,把master上的公钥id_dsa.pub复制到slave1中,在slave1中存储为id_dsa.pub.master:sudo scp /.ssh/id_rsa.pub slave1:/home/hadoop/.ssh/id_rsa.pub.master在hadoop-slave1下输入如下命令,把id_rsa.pub.master追加到authorized_keys中cat /.ssh/id_rsa.pub.master /.ssh/authorized_keys5. master免
6、密码登录slave2的实现:跟上一步类似。三、安装hadoop2.6.0在master机器中执行以下1-10步:1. 解压hadoop-2.6.0.tar.gz到路径/home/hadoop下,其中hadoop是用户名tar xzfv hadoop-2.6.0.tar.gz2. 用命令sudo gedit hadoop-2.6.0/etc/hadoop/hadoop-env.sh打开hadoop-env.sh,用命令echo $JAVA_HOME查看JDK路径,往hadoop-env.sh中添加如下内容:export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_403. 修
7、改hadoop-2.6.0/etc/hadoop/core-site.xml:sudo gedit hadoop-2.6.0/etc/hadoop/core-site.xml注意:必须加在节点内 hadoop.tmp.dir /home/hadoop/hadoop-2.6.0/tmp Abase for other temporary directories. fs.default.name hdfs:/master:8020 4. 修改hadoop-2.6.0/etc/hadoop/hdfs-site.xmlsudo gedit hadoop-2.6.0/etc/hadoop/hdfs-si
8、te.xml dfs.name.dir /home/hadoop/hadoop-2.6.0/dfs/name Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently. dfs.permissions false dfs.data.dir /home/hadoop/hadoop-2.6.0/dfs/data Comma separated list of paths on the local filesystem of a DataNode wh
9、ere it should store its blocks. dfs.replication 2 5. 修改hadoop-2.6.0/etc/hadoop/mapred-site.xmlsudo gedit hadoop-2.6.0/etc/hadoop/mapred-site.xml mapred.job.tracker master:8021 Host or IP and port of JobTracker. 6. 修改hadoop-2.6.0/etc/hadoop/masterssudo gedit hadoop-2.6.0/etc/hadoop/masters列出namenode机
10、器的名字:master7. 修改hadoop-2.6.0/etc/hadoop/slavessudo gedit hadoop-2.6.0/etc/hadoop/slaves列出所有datanode的机器名字:slave1slave28. 把文件夹hadoop-2.6.0拷贝到slave1和slave2机器中:scp r /hadoop-2.6.0 slave1:/scp r /hadoop-2.6.0 slave2:/9. Hadoop的启动进入hadoop安装目录:cd /home/hadoop/hadoop-2.6.0格式化HDFS文件系统的namenode:bin/hdfs namen
11、ode format启动hadoop集群:sbin/start-dfs.sh成功的话输入jps会出现如下界面:关闭集群:sbin/stop-dfs.sh10. 运行wordcount程序启动集群:sbin/start-dfs.sh在hadoop2.6.0目录下创建input目录:mkdir input打开文件夹input,新建两个文档f1和f2:f1中输入如下:f2中输入如下:在hdfs创建Example目录,命令如下:bin/hadoop fs -mkdir /Example将f1、f2文件copy到Exmaple目录,命令如下:bin/hadoop fs-put input/ /Examp
12、le执行wordcount程序:bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /Example/input/ /output查看执行结果:bin/hadoop fs -cat /output/*四、安装和配置eclipse。1. 下载好eclipse-java-luna-SR2-linux-gtk-x86_64.tar.gz后解压到文件夹/home/hadoop下tar zxvf eclipse-java-luna-SR2-linux-gtk-x86_64.tar.gz2.
13、 把插件hadoop-eclipse-plugin-2.6.0.jar放到目录/home/hadoop/eclipse/plugins下3. 打开软件eclipse,点击window/open Perspective/other/Map/Reduce点击下图右下角的大象:下图的Location name随便取个名字,左侧的Host输入主机master的IP,Port输入文件mapred-site.xml中的端口号,右侧的Port输入文件core-site.xml中的端口号:之后会出现如下界面:点击window/Preferences/Hadoop Map/Reduce,在Hadoop inst
14、allation directory:中选择hadoop的安装路径。五、用Eclipse编译、运行hadoop程序。1. 新建工程点击FileNewProjectMap/Reduce Project,工程名字写wordcount。2. 新建WordCount类。右击工程wordcount,NewClass,类名写wordcount:3. 把hadoop安装目录下的WordCount源码复制到wordcount.java文件中,程序代码如下:package wordcount;import java.io.IOException;import java.util.StringTokenizer;
15、import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.m
16、apreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class wordcount public static class TokenizerMapper extends Mapper private final static IntWritable one = new IntWritable(1); private Text word
17、= new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException StringTokenizer itr = new StringTokenizer(value.toString(); while (itr.hasMoreTokens() word.set(itr.nextToken(); context.write(word, one); public static class IntSumReducer extends Reduce
18、r private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable values, Context context ) throws IOException, InterruptedException int sum = 0; for (IntWritable val : values) sum += val.get(); result.set(sum); context.write(key, result); public static void main(String args) t
19、hrows Exception Configuration conf = new Configuration(); String otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length 2) System.err.println(Usage: wordcount . ); System.exit(2); Job job = new Job(conf, word count); job.setJarByClass(wordcount.class); job.setMappe
20、rClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); for (int i = 0; i otherArgs.length - 1; +i) FileInputFormat.addInputPath(job, new Path(otherArgsi); FileO
21、utputFormat.setOutputPath(job, new Path(otherArgsotherArgs.length - 1); System.exit(job.waitForCompletion(true) ? 0 : 1); 4. 在hadoop安装目录(/home/hadoop/hadoop-2.6.0)下新建文件夹input,input中新建两个文件f1和f2,f1和f2中输入如下内容:5. 把input文件夹上传到分布式文件系统中:在终端执行如下命令:cd /home/hadoop/hadoop-2.6.0 (进入hadoop安装目录)bin/hadoop fs -mk
22、dir /Example (在分布式文件系统中创建文件夹Example)bin/hadoop fs -put input /Example (把input文件夹上传到分布式文件系统的/Example/目录下)bin/hadoop fs -ls /Example/input (查看分布式文件系统/Example/input下的文件)由上图可看到,在dfs的/Example/input目录下,有四个文件夹,其中f1、f2是我们刚刚建立的input文件夹下的两个文件,而f1和f2不知道是什么来的,内容是上一次写在f1和f2上的内容,可以用如下命令查看这四个文件中的内容:bin/hadoop fs -
23、cat /Example/input/f1bin/hadoop fs -cat /Example/input/f2bin/hadoop fs -cat /Example/input/f1bin/hadoop fs -cat /Example/input/f2删除f1和f2这两个文件:bin/hadoop fs -rm -r -f /Example/input/f1bin/hadoop fs -rm -r -f /Example/input/f26. 回到Eclipse界面,设置输入输出参数。点击Run Configurations:设置Program arguments如下图所示,其中第一行表示输入参数,第二行表示输出参数:点击上图的Run,运行程序。7. 查看输出结果。在终端输入如下命令,即可查看输出结果:bin/hadoop fs -cat /Example/output/*
copyright@ 2008-2022 冰豆网网站版权所有
经营许可证编号:鄂ICP备2022015515号-1