5Nutch代码阅读.docx

1、5Nutch代码阅读1 Nutch工作流程1) 建立初始URL集2) 将URL集注入crawldb数据库-inject 3) 根据crawldb数据库创建抓取列表-generate 4) 执行抓取，获取网页信息-fetch 5) 更新数据库，把获取到的页面信息存入数据库中-updatedb 6) 重复进行35的步骤，直到预先设定的抓取深度。-这个循环过程被称为“产生/抓取/更新”循环7) 根据sengments的内容更新linkdb数据库-LinkDb8) 建立索引-Indexer9) 用户通过用户接口进行查询操作10) 将用户查询转化为lucene查询11) 返回结果其中，16属于爬虫部分；

2、7、8属于索引部分；911属于查询部分2 Nutch数据集的基本组成2.1 crawldb: 爬行数据库，用来存储所要爬行的网址2.2 segments: 抓取的网址被作为一个单元，而一个segment就是一个单元。一个segment包括以下几个子目录: 2.2.1 crawl_generate:包含所抓取的网址列表2.2.2 crawl_fetch:包含每个抓取页面的状态2.2.3 content:包含每个抓取页面的内容2.2.4 parse_text:包含每个抓取页面的解析文本2.2.5 parse_data:包含每个页面的外部链接和元数据2.2.6 crawl_parse:包含网址的外部

3、链接地址，用于更新crawldb数据库2.3 linkdb: 链接数据库，用来存储每个网址的链接地址，包括源地址和链接地址 2.4 indexes: 采用Lucene的格式建立索引集2.5 index：？3 Nutch分析方法和工具以下分析都是执行urls -dir 1crawled -depth 2 -threads 4 -topN 20后的结果分析3.1 Crawdb分析3.1.1 查看概要heyiheyi-PC /cygdrive/d/nutch-1.2_new$ bin/nutch readdb 1crawled/crawldb/ -statsCrawlDb statistics st

4、art: 1crawled/crawldb/Statistics for CrawlDb: 1crawled/crawldb/TOTAL urls: 24retry 0: 24min score: 0.026avg score: 0.118875max score: 1.333status 1 (db_unfetched): 3status 2 (db_fetched): 21CrawlDb statistics: done3.1.2 把内容dump出来看看$ bin/nutch readdb 1crawled/crawldb/ -dump crawldb_dumpCrawlDb dump:

5、startingCrawlDb db: 1crawled/crawldb/CrawlDb dump: done$ more * Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:22 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 1.3333334Signature: 3dbbb785e082201813d53ec4ba7be28

6、aMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:00 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 3c453e109c74e605284d61e5e2fc70adMetadata: _pst_: s

7、uccess(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:13 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 28fb7fbac69a66a0bceb2674f377ef90Metadata: _pst_: success(1), lastModi

8、fied=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:26 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 70995f0eb00bc8e153765fe5bae4dc19Metadata: _pst_: success(1), lastModified=0 Version: 7St

9、atus: 2 (db_fetched)Fetch time: Fri May 24 17:21:31 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 378627e7517e264ea27d7d0e74c950b3Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched

10、)Fetch time: Fri May 24 17:21:42 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 02438da3a457b38a80229e64d8c27909Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri Ma

11、y 24 17:22:04 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 6fc20c5aafa485a65e7a74a4a19b01f8Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:21 CST 2

12、013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 8a0a290038d07706c8bfd2bc63f488ddMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:39 CST 2013Modified time: T

13、hu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.048245616Signature: 4b3f7e8effd5740de2f5ee1ff8ffdd4fMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:23 CST 2013Modified time: Thu Jan 01 08:00:0

14、0 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.04385965Signature: 2075a99894392384877e38f93fcd3d5dMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:48 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries

15、since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: 396a1d8b06f6264897e98f55271887f4Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:09 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry

16、 interval: 2592000 seconds (30 days)Score: 0.0877193Signature: c01ea2329fafce6bbf60b0d2473a62f9Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:19 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000

17、seconds (30 days)Score: 0.0877193Signature: 8cd4147b713a809079fce1b89743a5e9Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 1 (db_unfetched)Fetch time: Wed Apr 24 17:21:25 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)

18、Score: 0.02631579Signature: nullMetadata: Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:11 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.04385965Signature: 57791dc2e777119b78db53eef98453eeMetadata: _pst_: suc

19、cess(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:50 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.04385965Signature: e3bc420cd8218add38992391adda0cfcMetadata: _pst_: success(1), lastModif

20、ied=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:58 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.04385965Signature: 3c3a64ae8e48162ebbb3aa06488828deMetadata: _pst_: success(1), lastModified=0 Version: 7St

21、atus: 2 (db_fetched)Fetch time: Fri May 24 17:21:37 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.04385965Signature: 48db854d816c5e843d7b3e21b028727dMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetche

22、d)Fetch time: Fri May 24 17:21:40 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: b0df676f1a3756c0f08d26f0b1af2f38Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 1 (db_unfetched)Fetch time: Wed

23、 Apr 24 17:21:25 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.02631579Signature: nullMetadata: Version: 7Status: 1 (db_unfetched)Fetch time: Wed Apr 24 17:21:25 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries si

24、nce fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.02631579Signature: nullMetadata: Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:03 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature:

25、41869be3ce4600f2bf698ff167433413Metadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:22:06 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.0877193Signature: afefd0c30c512eb38fc

26、31f0fdd2b226aMetadata: _pst_: success(1), lastModified=0 Version: 7Status: 2 (db_fetched)Fetch time: Fri May 24 17:21:57 CST 2013Modified time: Thu Jan 01 08:00:00 CST 1970Retries since fetch: 0Retry interval: 2592000 seconds (30 days)Score: 0.048245616Signature: 406fdfeb8e14100d3b286347b0a1918dMeta

27、data: _pst_: success(1), lastModified=03.2 Segments分析3.2.1 查看概要$ bin/nutch readseg -list -dir 1crawled/segments/NAME GENERATED FETCHER START FETCHER END FETCHED PARSED20130424172116 1 2013-04-24T17:21:22 2013-04-24T17:21:22 1 120130424172128 20 2013-04-24T17:21:31 2013-04-24T17:22:26 20 203.2.2 把内容dump出来看