Lemur索引文件格式分析.docx

1、Lemur索引文件格式分析Lemur索引文件格式分析 WORD文档使用说明：Lemur索引文件格式分析 来源于PDFWORD 本WOED文件是采用在线转换功能下载而来，因此在排版和显示效果方面可能不能满足您的应用需求。如果需要查看原版WOED文件，请访问这里Lemur索引文件格式分析 文件原版地址:Lemur索引文件格式分析|PDF转换成WROD_PDF阅读器下载IndriBuildIndex C 索引文件格式IndriBuildIndex C 索引文件格式 . 1 I. Repository . 1 A. 文件列表及描述 . 1 II. DiskIndex . 1 A. 文件列表及描述 .

2、1 B. 文件格式（参见 IndexWriter:write()） . 2 III. CompressedCollection . 3 A. 文件列表及描述 . 4 B. 文件格式（参见 CompressedCollection:addDocument()） . 4I. Repository A. 文件列表及描述FileName Index (Directory) collection (Directory) deleted manifest Description Contains zero or more DiskIndex instances, in numbered subdirect

3、ories. Contains a CompressedCollection instance. Bitmap containing a list of all deleted documents. XML file storing configuration information about the collection, including index counts, stemmer and stopword information.II. DiskIndex A. 文件列表及描述FileName directFile documentLengths documentStatistics

4、 fieldsFile frequentID frequentString frequentTerms Description Numeric representation of each document in the collection, useful for query expansion Length of each document in words, 4 bytes per document Offset of each document in the directFile, document length in the directFile, document term len

5、gth, number of unique terms in each document Inverted extent list for fields BulkTree storing the mapping from termID to termString and term statistics for frequent terms BulkTree storing the mapping from termString to termID and term statistics for frequent terms List of termID, termString pairs, n

6、ot stored in a tree, used at index merge timeinfrequentID infrequentString invertedFile manifestBulkTree storing the mapping from termID to termString and term statistics for infrequent terms BulkTree storing the mapping from termString to termID and term statistics for infrequent terms Inverted lis

7、ts for each term in the collection XML file storing important collection statistics and configuration information文件格式（ IndexWriter:write() ()） B. 文件格式（参见 IndexWriter:write()）invertedFile invertedFile TermStatistic * RVLDataLength(UINT32), TermString(CString), TermData TermDocCount(UInt), TermMaxDocL

8、ength(UInt),TermData TermTotalCount(UINT64), TermMinDocLength(UInt), TermFieldStatistics TermFieldStatistics TermInvList TopDocs BatchData DocEntryFiledCountControlByte(Byte), ?, * TopDocsCount(Int), TopDocsCount NextBatchDocID(DOCID_T), RVLDataLength(int), * DocID(Int), PositionCount(Int), Position

9、CountfrequentTerms, frequentID, frequentString, infrequentID, frequentString frequentTerms frequentID frequentString DistTermData1 DiskTermData2 DistTermData3 * Map Map TermData, TermID(TERMID_T), TermString(CString), TermFilePointer TermData, TermString(CString), TermFilePointer TermData, TermID(TE

10、RMID_T), TermFilePointerTermFilePointer infrequentID infrequentString fieldsFile fieldsFile BatchDataTermInvDataOffset(UINT64), TermInvDataLength(UINT64) Map MapFieldDataOffsetControlByte(UINT8),*NextBatchDocID(DOCID_T), RVLDataLength(Int), * ExtentLength(Int),ExtentCount(Int), ExtentCount directFil

11、e, documentLengths, documentStatistics directFile * TermCount(Int), FieldCount(Int),DocDirectData FieldCount FiledExtent FieldNumber(UINT64) documentLengths documentStatisticsTermCount,FiledID(Int),FilesdParentOrdinal(Int),FieldBegin(Int),FieldEnd(Int),* *DocumentData DocDirectDataOffset(UINT64), Do

12、cDirectDataLength(Int), DocIndexedLength(Int), DocTotalLength(Int), DocUniqTermCount(Int) manifest manifest IndexType(CString), IndexBuildDate(CString), IndriDistribution(CString), CorpusStatistics, FiledCount CorpusStatistics TotalDocCount(UINT64), TotalTermCount(UINT64), UniqTermCount(UINT64), Doc

13、Base(DOCID_T), FrequentTermCount(Int), MaxDocument(DOCID_T) FieldDescription IsNumeric(Bool), IsOrdinal(Bool), IsParental(Bool), FieldName(CString), ParseName(String)?, TotalDocCount(UInt), TotalTermCount(UINT64), FieldDataOffset(UINT64)III.CompressedCollection CompressedCollectionA. 文件列表及描述FileName

14、 lookup manifest storage Description Keyfile (B-Tree) that stores the mapping between documented and offset into storage XML file storing configuration information for this CompressedCollection Stores compressed version of each document (here using zlib compression library) in the collection, along

15、with byte offsets for each word in each document, and various document metadata added at index time. Keyfile (B-Tree) that stores the mapping between a documentID and a metadata string Keyfile (B-Tree) that stores the mapping between a metadata string and one or more documentIDsforwardLookupn revers

16、eLookupn文件格式（ CompressedCollection:addDocument() B. 文件格式（参见 CompressedCollection:addDocument()）lookup lookup Mapmanifest manifest ForwardMetadataList, ReverseMetadataList MetadataName(String) MetadataName(String)ForwardMetadataList ReverseMetadataList storage storage* PairCount, PairCount,StorageDoc

17、Data PairCount(UINT32) KeyValuePair MetadataPairMetadataPair | TermPositionPair | TextPair | ContentPair | ContentLengthPair MetadataKey(CString), MetadataValue(Void*) TermPositionFlag(CString), *TermPositionPair TextPairTextFlag(CString), TextData(Void*)ContentPairContentFlag(CString), ContentOffset(Int) ContentLengthFlag(CString), ContentLength(Int)ContentLengthPairforwardLookupn, reverseLookupn forwardLookupn reverseLookupn Map Map尊重他人劳动,转载请注明来自 PDF转换成WROD_PDF阅读器下载:本文【Lemur索引文件格式分析】网址: