如何利用MapReduce的分治策略提高KNN算法的运行速度.docx
《如何利用MapReduce的分治策略提高KNN算法的运行速度.docx》由会员分享，可在线阅读，更多相关《如何利用MapReduce的分治策略提高KNN算法的运行速度.docx（22页珍藏版）》请在冰豆网上搜索。
如何利用MapReduce的分治策略提高KNN算法的运行速度
如何利用MapReduce的分治策略提高KNN算法的运行速度
集群环境介绍：
hadoop2.4.164位
6台服务器：
hadoop11NameNode、SecondaryNameNode
hadoop22ResourceManager
hadoop33DataNode、NodeManager
hadoop44DataNode、NodeManager
hadoop55DataNode、NodeManager
hadoop66DataNode、NodeManager
实验1：
训练集train.txt样例个数为245057（3.24M）测试集test.txt样例个数为51444（640kb），并将全部测试集都存放在test.txt中
[root@hadoop11local]#hadoopfs-lsr/dir6/
-rw-r--r--3rootsupergroup34008162016-07-1719:
28/dir6/test.txt
注意：
此时所有的测试集都在一个文本中（test.txt）存放，作为输入路径
ＫＮＮ算法运行日志：
16/07/1719:
32:
24INFOclient.RMProxy:
ConnectingtoResourceManagerathadoop22/10.187.84.51:
8032
16/07/1719:
32:
25WARNmapreduce.JobSubmitter:
Hadoopcommand-lineoptionparsingnotperformed.ImplementtheToolinterfaceandexecuteyourapplicationwithToolRunnertoremedythis.
16/07/1719:
32:
25INFOinput.FileInputFormat:
Totalinputpathstoprocess:
1
16/07/1719:
32:
25INFOmapreduce.JobSubmitter:
numberofsplits:
1
16/07/1719:
32:
26INFOmapreduce.JobSubmitter:
Submittingtokensforjob:
job_1468752229715_0016
16/07/1719:
32:
26INFOimpl.YarnClientImpl:
Submittedapplicationapplication_1468752229715_0016
16/07/1719:
32:
26INFOmapreduce.Job:
Theurltotrackthejob:
http:
//hadoop22:
8088/proxy/application_1468752229715_0016/
16/07/1719:
32:
26INFOmapreduce.Job:
Runningjob:
job_1468752229715_0016
16/07/1719:
32:
32INFOmapreduce.Job:
Jobjob_1468752229715_0016runninginubermode:
false
16/07/1719:
32:
32INFOmapreduce.Job:
map0%reduce0%
16/07/1719:
32:
49INFOmapreduce.Job:
map1%reduce0%
16/07/1719:
33:
05INFOmapreduce.Job:
map2%reduce0%
16/07/1719:
33:
20INFOmapreduce.Job:
map3%reduce0%
16/07/1719:
33:
35INFOmapreduce.Job:
map4%reduce0%
16/07/1719:
33:
50INFOmapreduce.Job:
map5%reduce0%
16/07/1719:
34:
02INFOmapreduce.Job:
map6%reduce0%
16/07/1719:
34:
17INFOmapreduce.Job:
map7%reduce0%
16/07/1719:
34:
32INFOmapreduce.Job:
map8%reduce0%
16/07/1719:
34:
47INFOmapreduce.Job:
map9%reduce0%
16/07/1719:
35:
02INFOmapreduce.Job:
map10%reduce0%
16/07/1719:
35:
14INFOmapreduce.Job:
map11%reduce0%
16/07/1719:
35:
29INFOmapreduce.Job:
map12%reduce0%
16/07/1719:
35:
44INFOmapreduce.Job:
map13%reduce0%
16/07/1719:
35:
59INFOmapreduce.Job:
map14%reduce0%
16/07/1719:
36:
12INFOmapreduce.Job:
map15%reduce0%
16/07/1719:
36:
27INFOmapreduce.Job:
map16%reduce0%
16/07/1719:
36:
42INFOmapreduce.Job:
map17%reduce0%
16/07/1719:
36:
57INFOmapreduce.Job:
map18%reduce0%
16/07/1719:
37:
12INFOmapreduce.Job:
map19%reduce0%
16/07/1719:
37:
27INFOmapreduce.Job:
map20%reduce0%
16/07/1719:
37:
39INFOmapreduce.Job:
map21%reduce0%
16/07/1719:
37:
54INFOmapreduce.Job:
map22%reduce0%
16/07/1719:
38:
09INFOmapreduce.Job:
map23%reduce0%
16/07/1719:
38:
24INFOmapreduce.Job:
map24%reduce0%
16/07/1719:
38:
39INFOmapreduce.Job:
map25%reduce0%
16/07/1719:
38:
51INFOmapreduce.Job:
map26%reduce0%
16/07/1719:
39:
06INFOmapreduce.Job:
map27%reduce0%
16/07/1719:
39:
22INFOmapreduce.Job:
map28%reduce0%
16/07/1719:
39:
37INFOmapreduce.Job:
map29%reduce0%
16/07/1719:
39:
52INFOmapreduce.Job:
map30%reduce0%
16/07/1719:
40:
07INFOmapreduce.Job:
map31%reduce0%
16/07/1719:
40:
22INFOmapreduce.Job:
map32%reduce0%
16/07/1719:
40:
37INFOmapreduce.Job:
map33%reduce0%
16/07/1719:
40:
52INFOmapreduce.Job:
map34%reduce0%
16/07/1719:
41:
04INFOmapreduce.Job:
map35%reduce0%
16/07/1719:
41:
22INFOmapreduce.Job:
map36%reduce0%
16/07/1719:
41:
37INFOmapreduce.Job:
map37%reduce0%
16/07/1719:
41:
52INFOmapreduce.Job:
map38%reduce0%
16/07/1719:
42:
07INFOmapreduce.Job:
map39%reduce0%
16/07/1719:
42:
22INFOmapreduce.Job:
map40%reduce0%
16/07/1719:
42:
37INFOmapreduce.Job:
map41%reduce0%
16/07/1719:
42:
53INFOmapreduce.Job:
map42%reduce0%
16/07/1719:
43:
08INFOmapreduce.Job:
map43%reduce0%
16/07/1719:
43:
23INFOmapreduce.Job:
map44%reduce0%
16/07/1719:
43:
41INFOmapreduce.Job:
map45%reduce0%
16/07/1719:
43:
56INFOmapreduce.Job:
map46%reduce0%
16/07/1719:
44:
12INFOmapreduce.Job:
map47%reduce0%
16/07/1719:
44:
30INFOmapreduce.Job:
map48%reduce0%
16/07/1719:
44:
45INFOmapreduce.Job:
map49%reduce0%
16/07/1719:
45:
00INFOmapreduce.Job:
map50%reduce0%
16/07/1719:
45:
15INFOmapreduce.Job:
map51%reduce0%
16/07/1719:
45:
30INFOmapreduce.Job:
map52%reduce0%
16/07/1719:
45:
48INFOmapreduce.Job:
map53%reduce0%
16/07/1719:
46:
03INFOmapreduce.Job:
map54%reduce0%
16/07/1719:
46:
18INFOmapreduce.Job:
map55%reduce0%
16/07/1719:
46:
33INFOmapreduce.Job:
map56%reduce0%
16/07/1719:
46:
49INFOmapreduce.Job:
map57%reduce0%
16/07/1719:
47:
07INFOmapreduce.Job:
map58%reduce0%
16/07/1719:
47:
22INFOmapreduce.Job:
map59%reduce0%
16/07/1719:
47:
37INFOmapreduce.Job:
map60%reduce0%
16/07/1719:
47:
55INFOmapreduce.Job:
map61%reduce0%
16/07/17
升级会员 