hadoop220在windows上安装文档格式.docx
《hadoop220在windows上安装文档格式.docx》由会员分享，可在线阅读，更多相关《hadoop220在windows上安装文档格式.docx（16页珍藏版）》请在冰豆网上搜索。
nativedistributionfromsourcecodes,install,configureandrunHadoopinWindowsPlatform.
ToolsandTechnologiesusedinthisarticle:
1.ApacheHadoop2.2.0Sourcecodes
2.Windows7OS
3.MicrosoftWindowsSDKv7.1
4.Maven3.1.1
5.ProtocolBuffers2.5.0
6.Cygwin
7.JDK1.6
BuildHadoopbindistributionforWindows
1.Downloadandinstall
MicrosoftWindowsSDKv7.1
.
2.DownloadandinstallUnixcommand-linetool
Cygwin
3.Downloadandinstall
Maven3.1.1
4.Download
ProtocolBuffers2.5.0
andextracttoafolder（say
c:
\protobuf
）.
5.AddEnvironmentVariables
JAVA_HOME
M2_HOME
and
Platform
ifnotaddedalready.
AddEnvironmentVariables:
Note:
Variablename
iscasesensitive.Andvaluewillbeeither
x64
orWin32
forbuildingona
64-bit
or
32-bit
system.
Edit
Path
Variabletoadd
directoryof
（say
C:
\cygwin64\bin
）,
bindirectoryof
Maven
\maven\bin
）andinstallationpathof
ProtocolBuffers（say
EditPathVariable:
6.Download
hadoop-2.2.0-src.tar.gz
andextracttoafolderhavingshortpath（sayc:
\hdfs
）toavoidruntimeproblemduetomaximumpathlengthlimitationinWindows.
7.Select
Start-->
AllPrograms-->
MicrosoftWindowsSDKv7.1
andopen
WindowsSDK7.1CommandPrompt
.ChangedirectorytoHadoopsourcecodefolder（c:
）.Execute
mvnpackage
withoptions
-Pdist,native-win-DskipTests-DtartocreateWindowsbinarytardistribution.
WindowsSDK7.1CommandPrompt
SettingSDKenvironmentrelativetoC:
\ProgramFiles\MicrosoftSDKs\Windows\v7.1\.
TargetingWindows7x64Debug
\ProgramFiles\MicrosoftSDKs\Windows\v7.1>
cdc:
\hdfs
\hdfs>
mvnpackage-Pdist,native-win-DskipTests-Dtar
[INFO]Scanningforprojects...
[INFO]------------------------------------------------------------------------
[INFO]ReactorBuildOrder:
[INFO]
[INFO]ApacheHadoopMain
[INFO]ApacheHadoopProjectPOM
[INFO]ApacheHadoopAnnotations
[INFO]ApacheHadoopAssemblies
[INFO]ApacheHadoopProjectDistPOM
[INFO]ApacheHadoopMavenPlugins
[INFO]ApacheHadoopAuth
[INFO]ApacheHadoopAuthExamples
[INFO]ApacheHadoopCommon
[INFO]ApacheHadoopNFS
[INFO]ApacheHadoopCommonProject
[INFO]ApacheHadoopHDFS
[INFO]ApacheHadoopHttpFS
[INFO]ApacheHadoopHDFSBookKeeperJournal
[INFO]ApacheHadoopHDFS-NFS
[INFO]ApacheHadoopHDFSProject
[INFO]hadoop-yarn
[INFO]hadoop-yarn-api
[INFO]hadoop-yarn-common
[INFO]hadoop-yarn-server
[INFO]hadoop-yarn-server-common
[INFO]hadoop-yarn-server-nodemanager
[INFO]hadoop-yarn-server-web-proxy
[INFO]hadoop-yarn-server-resourcemanager
[INFO]hadoop-yarn-server-tests
[INFO]hadoop-yarn-client
[INFO]hadoop-yarn-applications
[INFO]hadoop-yarn-applications-distributedshell
[INFO]hadoop-mapreduce-client
[INFO]hadoop-mapreduce-client-core
[INFO]hadoop-yarn-applications-unmanaged-am-launcher
[INFO]hadoop-yarn-site
[INFO]hadoop-yarn-project
[INFO]hadoop-mapreduce-client-common
[INFO]hadoop-mapreduce-client-shuffle
[INFO]hadoop-mapreduce-client-app
[INFO]hadoop-mapreduce-client-hs
[INFO]hadoop-mapreduce-client-jobclient
[INFO]hadoop-mapreduce-client-hs-plugins
[INFO]ApacheHadoopMapReduceExamples
[INFO]hadoop-mapreduce
[INFO]ApacheHadoopMapReduceStreaming
[INFO]ApacheHadoopDistributedCopy
[INFO]ApacheHadoopArchives
[INFO]ApacheHadoopRumen
[INFO]ApacheHadoopGridmix
[INFO]ApacheHadoopDataJoin
[INFO]ApacheHadoopExtras
[INFO]ApacheHadoopPipes
[INFO]ApacheHadoopToolsDist
[INFO]ApacheHadoopTools
[INFO]ApacheHadoopDistribution
[INFO]ApacheHadoopClient
[INFO]ApacheHadoopMini-
[INFO]BuildingApacheHadoopMain2.2.0
[INFO]---maven-enforcer-plugin:
1.3.1:
enforce（default）@hadoop-main---
[INFO]---maven-site-plugin:
3.0:
attach-descriptor（attach-descriptor）@hadoop-main---
Ihavepastedonlythestartingfewlinesofhugelogsgeneratedbymaven.ThisbuildingsteprequiresInternetconnectionasMavenwilldownloadalltherequireddependencies.
8.Ifeverythinggoeswellinthepreviousstep,thennativedistribution
hadoop-2.2.0.tar.gz
willbecreatedinside
\hdfs\hadoop-dist\target\hadoop-2.2.0directory.
InstallHadoop
1.Extract
toafolder（say
\hadoop
2.AddEnvironmentVariable
HADOOP_HOME
andedit
\hadoop\bin
ConfigureHadoop
MakefollowingchangestoconfigureHadoop
∙File:
C:
\hadoop\etc\hadoop\core-site.xml
∙<
?
xmlversion="
1.0"
encoding="
UTF-8"
>
xml-stylesheettype="
text/xsl"
href="
configuration.xsl"
!
--
∙LicensedundertheApacheLicense,Version2.0（the"
License"
）;
∙youmaynotusethisfileexceptincompliancewiththeLicense.
∙YoumayobtainacopyoftheLicenseat
∙
∙http:
//www.apache.org/licenses/LICENSE-2.0
∙Unlessrequiredbyapplicablelaworagreedtoinwriting,software
∙distributedundertheLicenseisdistributedonan"
ASIS"
BASIS,
∙WITHOUTWARRANTIESORCONDITIONSOFANYKIND,eitherexpressorimplied.
∙SeetheLicenseforthespecificlanguagegoverningpermissionsand
∙limitationsundertheLicense.SeeaccompanyingLICENSEfile.
∙-->
--Putsite-specificpropertyoverridesinthisfile.-->
configuration>
∙<
property>
name>
fs.defaultFS<
/name>
value>
hdfs:
//localhost:
9000<
/value>
/property>
<
/configuration>
fs.defaultFS:
Thenameofthedefaultfilesystem.AURIwhoseschemeandauthoritydeterminetheFileSystemimplementation.Theuri'
sschemedeterminestheconfigproperty（fs.SCHEME.impl）namingtheFileSystemimplementationclass.Theuri'
sauthorityisusedtodeterminethehost,port,etc.forafilesystem.
\hadoop\etc\hadoop\hdfs-site.xml
dfs.replication<
1<
dfs.namenode.name.dir<
file:
/hadoop/data/dfs/namenode<
dfs.datanode.data.dir<
/hadoop/data/dfs/datanode<
dfs.replication:
Defaultblockreplication.Theactualnumberofreplicationscanbespecifiedwhenthefileiscreated.Thedefaultisusedifreplicationisnotspecifiedincreatetime.
dfs.namenode.name.dir:
DetermineswhereonthelocalfilesystemtheDFSnamenodeshouldstorethenametable（fsimage）.Ifthisisacomma-delimitedlistofdirectoriesthenthenametableisreplicatedinallofthedirectories,forredundancy.
dfs.datanode.data.dir:
DetermineswhereonthelocalfilesystemanDFSdatanodeshouldstoreitsblocks.Ifthisisacomma-delimitedlistofdirectories,thendatawillbestoredinallnameddirectories,typicallyondifferentdevices.Directoriesthatdonotexistareignored.
Create
namenode
datanode
directoryunder
/hadoop/data/dfs/
\hadoop\etc\hadoop\yarn-site.xml
yarn.nodemanager.aux-services<
mapreduce_shuffle<
yarn.nodemanager.aux-services.mapreduce.shuffle.class<
org.apache.hadoop.mapred.ShuffleHandler<
yarn.nodemanager.aux-services:
Theauxiliaryservicename.Defaultvalueis
omapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class:
Theauxiliaryserviceclasstouse.Defaultvalueisorg.apache.hadoop.mapred.ShuffleHandler
\hadoop\etc\hadoop\mapred-site.xml
configurati
升级会员 