- java.lang.Object
-
- org.apache.lucene.index.IndexingChain
-
- All Implemented Interfaces:
Accountable
final class IndexingChain extends java.lang.Object implements Accountable
Default general purpose indexing chain, which handles indexing all types of fields.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
IndexingChain.FieldSchema
A schema of the field in the current document.private static class
IndexingChain.IntBlockAllocator
private class
IndexingChain.PerField
NOTE: not static: accesses at least docState, termsHash.(package private) static class
IndexingChain.ReservedField<T extends IndexableField>
-
Field Summary
Fields Modifier and Type Field Description private java.util.function.Consumer<java.lang.Throwable>
abortingExceptionConsumer
private ByteBlockPool.Allocator
byteBlockAllocator
(package private) Counter
bytesUsed
private IndexingChain.PerField[]
docFields
(package private) ByteBlockPool
docValuesBytePool
private IndexingChain.PerField[]
fieldHash
(package private) FieldInfos.Builder
fieldInfos
private IndexingChain.PerField[]
fields
private boolean
hasHitAbortingException
private int
hashMask
private int
indexCreatedVersionMajor
private LiveIndexWriterConfig
indexWriterConfig
private InfoStream
infoStream
private long
nextFieldGen
(package private) StoredFieldsConsumer
storedFieldsConsumer
(package private) TermsHash
termsHash
(package private) TermVectorsConsumer
termVectorsWriter
private int
totalFieldCount
(package private) VectorValuesConsumer
vectorValuesConsumer
-
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
-
Constructor Summary
Constructors Constructor Description IndexingChain(int indexCreatedVersionMajor, SegmentInfo segmentInfo, Directory directory, FieldInfos.Builder fieldInfos, LiveIndexWriterConfig indexWriterConfig, java.util.function.Consumer<java.lang.Throwable> abortingExceptionConsumer)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) void
abort()
private void
finishStoredFields()
Calls StoredFieldsWriter.finishDocument, aborting the segment if it hits any exception.(package private) Sorter.DocMap
flush(SegmentWriteState state)
java.util.Collection<Accountable>
getChildResources()
Returns nested resources of this class.private LeafReader
getDocValuesLeafReader()
(package private) DocIdSetIterator
getHasDocValues(java.lang.String field)
private IndexingChain.PerField
getOrAddPerField(java.lang.String fieldName, boolean reserved)
Returns a previously createdIndexingChain.PerField
, absorbing the type information fromFieldType
, and creates a newIndexingChain.PerField
if this field name wasn't seen yet.private IndexingChain.PerField
getPerField(java.lang.String name)
Returns a previously createdIndexingChain.PerField
, or null if this field name wasn't seen yet.private void
indexDocValue(int docID, IndexingChain.PerField fp, DocValuesType dvType, IndexableField field)
Called from processDocument to index one field's doc valueprivate void
indexVectorValue(int docID, IndexingChain.PerField pf, VectorEncoding vectorEncoding, IndexableField field)
private void
initializeFieldInfo(IndexingChain.PerField pf)
(package private) <T extends IndexableField>
IndexingChain.ReservedField<T>markAsReserved(T field)
Wraps the given field in a reserved field and registers it as reserved.private Sorter.DocMap
maybeSortSegment(SegmentWriteState state)
private void
onAbortingException(java.lang.Throwable th)
private void
oversizeDocFields()
(package private) void
processDocument(int docID, java.lang.Iterable<? extends IndexableField> document)
private boolean
processField(int docID, IndexableField field, IndexingChain.PerField pf)
Index each field Returnstrue
, if we are indexing a unique field with postingslong
ramBytesUsed()
Return the memory usage of this object in bytes.private void
rehash()
private void
startStoredFields(int docID)
Calls StoredFieldsWriter.startDocument, aborting the segment if it hits any exception.private static void
updateDocFieldSchema(java.lang.String fieldName, IndexingChain.FieldSchema schema, IndexableFieldType fieldType)
private void
validateIndexSortDVType(Sort indexSort, java.lang.String fieldToValidate, DocValuesType dvType)
private static void
validateMaxVectorDimension(java.lang.String fieldName, int vectorDim, int maxVectorDim)
private static void
verifyUnIndexedFieldType(java.lang.String name, IndexableFieldType ft)
private void
writeDocValues(SegmentWriteState state, Sorter.DocMap sortMap)
Writes all buffered doc values (called fromflush(org.apache.lucene.index.SegmentWriteState)
).private void
writeNorms(SegmentWriteState state, Sorter.DocMap sortMap)
private void
writePoints(SegmentWriteState state, Sorter.DocMap sortMap)
Writes all buffered points.
-
-
-
Field Detail
-
bytesUsed
final Counter bytesUsed
-
fieldInfos
final FieldInfos.Builder fieldInfos
-
termsHash
final TermsHash termsHash
-
docValuesBytePool
final ByteBlockPool docValuesBytePool
-
storedFieldsConsumer
final StoredFieldsConsumer storedFieldsConsumer
-
vectorValuesConsumer
final VectorValuesConsumer vectorValuesConsumer
-
termVectorsWriter
final TermVectorsConsumer termVectorsWriter
-
fieldHash
private IndexingChain.PerField[] fieldHash
-
hashMask
private int hashMask
-
totalFieldCount
private int totalFieldCount
-
nextFieldGen
private long nextFieldGen
-
fields
private IndexingChain.PerField[] fields
-
docFields
private IndexingChain.PerField[] docFields
-
infoStream
private final InfoStream infoStream
-
byteBlockAllocator
private final ByteBlockPool.Allocator byteBlockAllocator
-
indexWriterConfig
private final LiveIndexWriterConfig indexWriterConfig
-
indexCreatedVersionMajor
private final int indexCreatedVersionMajor
-
abortingExceptionConsumer
private final java.util.function.Consumer<java.lang.Throwable> abortingExceptionConsumer
-
hasHitAbortingException
private boolean hasHitAbortingException
-
-
Constructor Detail
-
IndexingChain
IndexingChain(int indexCreatedVersionMajor, SegmentInfo segmentInfo, Directory directory, FieldInfos.Builder fieldInfos, LiveIndexWriterConfig indexWriterConfig, java.util.function.Consumer<java.lang.Throwable> abortingExceptionConsumer)
-
-
Method Detail
-
onAbortingException
private void onAbortingException(java.lang.Throwable th)
-
getDocValuesLeafReader
private LeafReader getDocValuesLeafReader()
-
maybeSortSegment
private Sorter.DocMap maybeSortSegment(SegmentWriteState state) throws java.io.IOException
- Throws:
java.io.IOException
-
flush
Sorter.DocMap flush(SegmentWriteState state) throws java.io.IOException
- Throws:
java.io.IOException
-
writePoints
private void writePoints(SegmentWriteState state, Sorter.DocMap sortMap) throws java.io.IOException
Writes all buffered points.- Throws:
java.io.IOException
-
writeDocValues
private void writeDocValues(SegmentWriteState state, Sorter.DocMap sortMap) throws java.io.IOException
Writes all buffered doc values (called fromflush(org.apache.lucene.index.SegmentWriteState)
).- Throws:
java.io.IOException
-
writeNorms
private void writeNorms(SegmentWriteState state, Sorter.DocMap sortMap) throws java.io.IOException
- Throws:
java.io.IOException
-
abort
void abort() throws java.io.IOException
- Throws:
java.io.IOException
-
rehash
private void rehash()
-
startStoredFields
private void startStoredFields(int docID) throws java.io.IOException
Calls StoredFieldsWriter.startDocument, aborting the segment if it hits any exception.- Throws:
java.io.IOException
-
finishStoredFields
private void finishStoredFields() throws java.io.IOException
Calls StoredFieldsWriter.finishDocument, aborting the segment if it hits any exception.- Throws:
java.io.IOException
-
processDocument
void processDocument(int docID, java.lang.Iterable<? extends IndexableField> document) throws java.io.IOException
- Throws:
java.io.IOException
-
oversizeDocFields
private void oversizeDocFields()
-
initializeFieldInfo
private void initializeFieldInfo(IndexingChain.PerField pf) throws java.io.IOException
- Throws:
java.io.IOException
-
processField
private boolean processField(int docID, IndexableField field, IndexingChain.PerField pf) throws java.io.IOException
Index each field Returnstrue
, if we are indexing a unique field with postings- Throws:
java.io.IOException
-
getOrAddPerField
private IndexingChain.PerField getOrAddPerField(java.lang.String fieldName, boolean reserved)
Returns a previously createdIndexingChain.PerField
, absorbing the type information fromFieldType
, and creates a newIndexingChain.PerField
if this field name wasn't seen yet.
-
updateDocFieldSchema
private static void updateDocFieldSchema(java.lang.String fieldName, IndexingChain.FieldSchema schema, IndexableFieldType fieldType)
-
verifyUnIndexedFieldType
private static void verifyUnIndexedFieldType(java.lang.String name, IndexableFieldType ft)
-
validateMaxVectorDimension
private static void validateMaxVectorDimension(java.lang.String fieldName, int vectorDim, int maxVectorDim)
-
validateIndexSortDVType
private void validateIndexSortDVType(Sort indexSort, java.lang.String fieldToValidate, DocValuesType dvType) throws java.io.IOException
- Throws:
java.io.IOException
-
indexDocValue
private void indexDocValue(int docID, IndexingChain.PerField fp, DocValuesType dvType, IndexableField field)
Called from processDocument to index one field's doc value
-
indexVectorValue
private void indexVectorValue(int docID, IndexingChain.PerField pf, VectorEncoding vectorEncoding, IndexableField field) throws java.io.IOException
- Throws:
java.io.IOException
-
getPerField
private IndexingChain.PerField getPerField(java.lang.String name)
Returns a previously createdIndexingChain.PerField
, or null if this field name wasn't seen yet.
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
getChildResources
public java.util.Collection<Accountable> getChildResources()
Description copied from interface:Accountable
Returns nested resources of this class. The result should be a point-in-time snapshot (to avoid race conditions).- Specified by:
getChildResources
in interfaceAccountable
- See Also:
Accountables
-
getHasDocValues
DocIdSetIterator getHasDocValues(java.lang.String field)
-
markAsReserved
<T extends IndexableField> IndexingChain.ReservedField<T> markAsReserved(T field)
Wraps the given field in a reserved field and registers it as reserved. Only DWPT should do this to mark fields as private / reserved to prevent this fieldname to be used from the outside of the IW / DWPT eco-system
-
-