hadoop220在windows上安装.docx

资源描述

hadoop220在windows上安装.docx

《hadoop220在windows上安装.docx》由会员分享，可在线阅读，更多相关《hadoop220在windows上安装.docx（16页珍藏版）》请在冰豆网上搜索。

hadoop220在windows上安装.docx

hadoop220在windows上安装

hadoop2.2.0在windows上安装

分类：

hapoop2013-12-3020:

28 2480人阅读评论（5）收藏举报

目录（?

）[+]

GoodnewsforHadoopdeveloperswhowanttouse MicrosoftWindowsOS fortheirdevelopmentactivities.Finally ApacheHadoop2.2.0 releaseofficiallysupportsforrunningHadooponMicrosoftWindowsaswell.Butthe bin distributionof ApacheHadoop2.2.0 releasedoesnotcontainsomewindowsnativecomponents（likewinutils.exe,hadoop.dll etc）.Asaresult,ifwetrytorunHadoopinwindows,we'llencounter ERRORutil.Shell:

Failedtolocatethewinutilsbinaryinthehadoopbinarypath

Inthisarticle,I'lldescribehowtobuild bin nativedistributionfromsourcecodes,install,configureandrunHadoopinWindowsPlatform.

ToolsandTechnologiesusedinthisarticle:

1.ApacheHadoop2.2.0Sourcecodes

2.Windows7OS

3.MicrosoftWindowsSDKv7.1

4.Maven3.1.1

5.ProtocolBuffers2.5.0

6.Cygwin

7.JDK1.6

BuildHadoopbindistributionforWindows

1.Downloadandinstall MicrosoftWindowsSDKv7.1 .

2.DownloadandinstallUnixcommand-linetool Cygwin .

3.Downloadandinstall Maven3.1.1 .

4.Download ProtocolBuffers2.5.0 andextracttoafolder（say c:

\protobuf ）.

5.AddEnvironmentVariables JAVA_HOME , M2_HOME and Platform ifnotaddedalready.

AddEnvironmentVariables:

Note:

Variablename Platform iscasesensitive.Andvaluewillbeeither x64 orWin32 forbuildingona 64-bit or 32-bit system.

Edit Path Variabletoadd bin directoryof Cygwin （say C:

\cygwin64\bin ）, bindirectoryof Maven （say C:

\maven\bin ）andinstallationpathof ProtocolBuffers（say c:

\protobuf ）.

EditPathVariable:

6.Download hadoop-2.2.0-src.tar.gz andextracttoafolderhavingshortpath（sayc:

\hdfs ）toavoidruntimeproblemduetomaximumpathlengthlimitationinWindows.

7.Select Start-->AllPrograms-->MicrosoftWindowsSDKv7.1 andopen WindowsSDK7.1CommandPrompt .ChangedirectorytoHadoopsourcecodefolder（c:

\hdfs ）.Execute mvnpackage withoptions -Pdist,native-win-DskipTests-DtartocreateWindowsbinarytardistribution.

WindowsSDK7.1CommandPrompt

SettingSDKenvironmentrelativetoC:

\ProgramFiles\MicrosoftSDKs\Windows\v7.1\.

TargetingWindows7x64Debug

\ProgramFiles\MicrosoftSDKs\Windows\v7.1>cdc:

\hdfs

\hdfs>mvnpackage-Pdist,native-win-DskipTests-Dtar

[INFO]Scanningforprojects...

[INFO]------------------------------------------------------------------------

[INFO]ReactorBuildOrder:

[INFO]

[INFO]ApacheHadoopMain

[INFO]ApacheHadoopProjectPOM

[INFO]ApacheHadoopAnnotations

[INFO]ApacheHadoopAssemblies

[INFO]ApacheHadoopProjectDistPOM

[INFO]ApacheHadoopMavenPlugins

[INFO]ApacheHadoopAuth

[INFO]ApacheHadoopAuthExamples

[INFO]ApacheHadoopCommon

[INFO]ApacheHadoopNFS

[INFO]ApacheHadoopCommonProject

[INFO]ApacheHadoopHDFS

[INFO]ApacheHadoopHttpFS

[INFO]ApacheHadoopHDFSBookKeeperJournal

[INFO]ApacheHadoopHDFS-NFS

[INFO]ApacheHadoopHDFSProject

[INFO]hadoop-yarn

[INFO]hadoop-yarn-api

[INFO]hadoop-yarn-common

[INFO]hadoop-yarn-server

[INFO]hadoop-yarn-server-common

[INFO]hadoop-yarn-server-nodemanager

[INFO]hadoop-yarn-server-web-proxy

[INFO]hadoop-yarn-server-resourcemanager

[INFO]hadoop-yarn-server-tests

[INFO]hadoop-yarn-client

[INFO]hadoop-yarn-applications

[INFO]hadoop-yarn-applications-distributedshell

[INFO]hadoop-mapreduce-client

[INFO]hadoop-mapreduce-client-core

[INFO]hadoop-yarn-applications-unmanaged-am-launcher

[INFO]hadoop-yarn-site

[INFO]hadoop-yarn-project

[INFO]hadoop-mapreduce-client-common

[INFO]hadoop-mapreduce-client-shuffle

[INFO]hadoop-mapreduce-client-app

[INFO]hadoop-mapreduce-client-hs

[INFO]hadoop-mapreduce-client-jobclient

[INFO]hadoop-mapreduce-client-hs-plugins

[INFO]ApacheHadoopMapReduceExamples

[INFO]hadoop-mapreduce

[INFO]ApacheHadoopMapReduceStreaming

[INFO]ApacheHadoopDistributedCopy

[INFO]ApacheHadoopArchives

[INFO]ApacheHadoopRumen

[INFO]ApacheHadoopGridmix

[INFO]ApacheHadoopDataJoin

[INFO]ApacheHadoopExtras

[INFO]ApacheHadoopPipes

[INFO]ApacheHadoopToolsDist

[INFO]ApacheHadoopTools

[INFO]ApacheHadoopDistribution

[INFO]ApacheHadoopClient

[INFO]ApacheHadoopMini-

[INFO]

[INFO]------------------------------------------------------------------------

[INFO]BuildingApacheHadoopMain2.2.0

[INFO]------------------------------------------------------------------------

[INFO]

[INFO]---maven-enforcer-plugin:

1.3.1:

enforce（default）@hadoop-main---

[INFO]

[INFO]---maven-site-plugin:

3.0:

attach-descriptor（attach-descriptor）@hadoop-main---

Note:

Ihavepastedonlythestartingfewlinesofhugelogsgeneratedbymaven.ThisbuildingsteprequiresInternetconnectionasMavenwilldownloadalltherequireddependencies.

8.Ifeverythinggoeswellinthepreviousstep,thennativedistribution hadoop-2.2.0.tar.gz willbecreatedinside C:

\hdfs\hadoop-dist\target\hadoop-2.2.0directory.

InstallHadoop

1.Extract hadoop-2.2.0.tar.gz toafolder（say c:

\hadoop ）.

2.AddEnvironmentVariable HADOOP_HOME andedit Path Variabletoadd bindirectoryof HADOOP_HOME （say C:

\hadoop\bin ）.

AddEnvironmentVariables:

ConfigureHadoop

MakefollowingchangestoconfigureHadoop

∙File:

\hadoop\etc\hadoop\core-site.xml

∙

xmlversion="1.0"encoding="UTF-8"?

∙

xml-stylesheettype="text/xsl"href="configuration.xsl"?

∙

∙LicensedundertheApacheLicense,Version2.0（the"License"）;

∙youmaynotusethisfileexceptincompliancewiththeLicense.

∙YoumayobtainacopyoftheLicenseat

∙

∙http:

//www.apache.org/licenses/LICENSE-2.0

∙

∙Unlessrequiredbyapplicablelaworagreedtoinwriting,software

∙distributedundertheLicenseisdistributedonan"ASIS"BASIS,

∙WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.

∙SeetheLicenseforthespecificlanguagegoverningpermissionsand

∙limitationsundertheLicense.SeeaccompanyingLICENSEfile.

∙-->

∙

--Putsite-specificpropertyoverridesinthisfile.-->

∙

∙fs.defaultFS

∙hdfs:

//localhost:

9000

∙

fs.defaultFS:

Thenameofthedefaultfilesystem.AURIwhoseschemeandauthoritydeterminetheFileSystemimplementation.Theuri'sschemedeterminestheconfigproperty（fs.SCHEME.impl）namingtheFileSystemimplementationclass.Theuri'sauthorityisusedtodeterminethehost,port,etc.forafilesystem.

∙File:

\hadoop\etc\hadoop\hdfs-site.xml

∙

xmlversion="1.0"encoding="UTF-8"?

∙

xml-stylesheettype="text/xsl"href="configuration.xsl"?

∙

∙LicensedundertheApacheLicense,Version2.0（the"License"）;

∙youmaynotusethisfileexceptincompliancewiththeLicense.

∙YoumayobtainacopyoftheLicenseat

∙

∙http:

//www.apache.org/licenses/LICENSE-2.0

∙

∙Unlessrequiredbyapplicablelaworagreedtoinwriting,software

∙distributedundertheLicenseisdistributedonan"ASIS"BASIS,

∙WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.

∙SeetheLicenseforthespecificlanguagegoverningpermissionsand

∙limitationsundertheLicense.SeeaccompanyingLICENSEfile.

∙-->

∙

--Putsite-specificpropertyoverridesinthisfile.-->

∙

∙dfs.replication

∙1

∙

∙dfs.namenode.name.dir

∙file:

/hadoop/data/dfs/namenode

∙

∙dfs.datanode.data.dir

∙file:

/hadoop/data/dfs/datanode

∙

dfs.replication:

Defaultblockreplication.Theactualnumberofreplicationscanbespecifiedwhenthefileiscreated.Thedefaultisusedifreplicationisnotspecifiedincreatetime.

dfs.namenode.name.dir:

DetermineswhereonthelocalfilesystemtheDFSnamenodeshouldstorethenametable（fsimage）.Ifthisisacomma-delimitedlistofdirectoriesthenthenametableisreplicatedinallofthedirectories,forredundancy.

dfs.datanode.data.dir:

DetermineswhereonthelocalfilesystemanDFSdatanodeshouldstoreitsblocks.Ifthisisacomma-delimitedlistofdirectories,thendatawillbestoredinallnameddirectories,typicallyondifferentdevices.Directoriesthatdonotexistareignored.

Note:

Create namenode and datanode directoryunder c:

/hadoop/data/dfs/ .

∙File:

\hadoop\etc\hadoop\yarn-site.xml

∙

xmlversion="1.0"?

∙

∙LicensedundertheApacheLicense,Version2.0（the"License"）;

∙youmaynotusethisfileexceptincompliancewiththeLicense.

∙YoumayobtainacopyoftheLicenseat

∙

∙http:

//www.apache.org/licenses/LICENSE-2.0

∙

∙Unlessrequiredbyapplicablelaworagreedtoinwriting,software

∙distributedundertheLicenseisdistributedonan"ASIS"BASIS,

∙WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.

∙SeetheLicenseforthespecificlanguagegoverningpermissionsand

∙limitationsundertheLicense.SeeaccompanyingLICENSEfile.

∙-->

∙

∙yarn.nodemanager.aux-services

∙mapreduce_shuffle

∙

∙yarn.nodemanager.aux-services.mapreduce.shuffle.class

∙org.apache.hadoop.mapred.ShuffleHandler

∙

yarn.nodemanager.aux-services:

Theauxiliaryservicename.Defaultvalueis omapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class:

Theauxiliaryserviceclasstouse.Defaultvalueisorg.apache.hadoop.mapred.ShuffleHandler

∙File:

\hadoop\etc\hadoop\mapred-site.xml

∙

xmlversion="1.0"?

∙

xml-stylesheettype="text/xsl"href="configurati

展开阅读全文