Module org.apache.lucene.core
Class Lucene99HnswVectorsWriter
- java.lang.Object
-
- org.apache.lucene.codecs.KnnVectorsWriter
-
- org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,Accountable
public final class Lucene99HnswVectorsWriter extends KnnVectorsWriter
Writes vector values and knn graphs to index segments.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
Lucene99HnswVectorsWriter.FieldWriter<T>
-
Nested classes/interfaces inherited from class org.apache.lucene.codecs.KnnVectorsWriter
KnnVectorsWriter.MergedVectorValues
-
-
Field Summary
Fields Modifier and Type Field Description private int
beamWidth
private java.util.List<Lucene99HnswVectorsWriter.FieldWriter<?>>
fields
private boolean
finished
private FlatVectorsWriter
flatVectorWriter
private int
M
private TaskExecutor
mergeExec
private IndexOutput
meta
private int
numMergeWorkers
private SegmentWriteState
segmentWriteState
private static long
SHALLOW_RAM_BYTES_USED
private IndexOutput
vectorIndex
-
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
-
Constructor Summary
Constructors Constructor Description Lucene99HnswVectorsWriter(SegmentWriteState state, int M, int beamWidth, FlatVectorsWriter flatVectorWriter, int numMergeWorkers, TaskExecutor mergeExec)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description KnnFieldVectorsWriter<?>
addField(FieldInfo fieldInfo)
Add new field for indexingvoid
close()
private HnswGraphMerger
createGraphMerger(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, TaskExecutor parallelMergeTaskExecutor, int numParallelMergeWorkers)
(package private) static int
distFuncToOrd(VectorSimilarityFunction func)
void
finish()
Called once at the end before closevoid
flush(int maxDoc, Sorter.DocMap sortMap)
Flush all buffered data on disk *void
mergeOneField(FieldInfo fieldInfo, MergeState mergeState)
Write field for merginglong
ramBytesUsed()
Return the memory usage of this object in bytes.private HnswGraph
reconstructAndWriteGraph(OnHeapHnswGraph graph, int[] newToOldMap, int[] oldToNewMap, int[][] levelNodeOffsets)
Reconstructs the graph given the old and new node ids.private void
reconstructAndWriteNeighbours(NeighborArray neighbors, int[] oldToNewMap, int maxOrd)
private void
writeField(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData)
private int[][]
writeGraph(OnHeapHnswGraph graph)
private void
writeMeta(FieldInfo field, long vectorIndexOffset, long vectorIndexLength, int count, HnswGraph graph, int[][] graphLevelNodeOffsets)
private void
writeSortingField(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData, Sorter.DocMap sortMap)
-
Methods inherited from class org.apache.lucene.codecs.KnnVectorsWriter
merge
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
SHALLOW_RAM_BYTES_USED
private static final long SHALLOW_RAM_BYTES_USED
-
segmentWriteState
private final SegmentWriteState segmentWriteState
-
meta
private final IndexOutput meta
-
vectorIndex
private final IndexOutput vectorIndex
-
M
private final int M
-
beamWidth
private final int beamWidth
-
flatVectorWriter
private final FlatVectorsWriter flatVectorWriter
-
numMergeWorkers
private final int numMergeWorkers
-
mergeExec
private final TaskExecutor mergeExec
-
fields
private final java.util.List<Lucene99HnswVectorsWriter.FieldWriter<?>> fields
-
finished
private boolean finished
-
-
Constructor Detail
-
Lucene99HnswVectorsWriter
public Lucene99HnswVectorsWriter(SegmentWriteState state, int M, int beamWidth, FlatVectorsWriter flatVectorWriter, int numMergeWorkers, TaskExecutor mergeExec) throws java.io.IOException
- Throws:
java.io.IOException
-
-
Method Detail
-
addField
public KnnFieldVectorsWriter<?> addField(FieldInfo fieldInfo) throws java.io.IOException
Description copied from class:KnnVectorsWriter
Add new field for indexing- Specified by:
addField
in classKnnVectorsWriter
- Throws:
java.io.IOException
-
flush
public void flush(int maxDoc, Sorter.DocMap sortMap) throws java.io.IOException
Description copied from class:KnnVectorsWriter
Flush all buffered data on disk *- Specified by:
flush
in classKnnVectorsWriter
- Throws:
java.io.IOException
-
finish
public void finish() throws java.io.IOException
Description copied from class:KnnVectorsWriter
Called once at the end before close- Specified by:
finish
in classKnnVectorsWriter
- Throws:
java.io.IOException
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.
-
writeField
private void writeField(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData) throws java.io.IOException
- Throws:
java.io.IOException
-
writeSortingField
private void writeSortingField(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData, Sorter.DocMap sortMap) throws java.io.IOException
- Throws:
java.io.IOException
-
reconstructAndWriteGraph
private HnswGraph reconstructAndWriteGraph(OnHeapHnswGraph graph, int[] newToOldMap, int[] oldToNewMap, int[][] levelNodeOffsets) throws java.io.IOException
Reconstructs the graph given the old and new node ids.Additionally, the graph node connections are written to the vectorIndex.
- Parameters:
graph
- The current on heap graphnewToOldMap
- the new node ids indexed to the old node idsoldToNewMap
- the old node ids indexed to the new node idslevelNodeOffsets
- where to place the new offsets for the nodes in the vector index.- Returns:
- The graph
- Throws:
java.io.IOException
- if writing to vectorIndex fails
-
reconstructAndWriteNeighbours
private void reconstructAndWriteNeighbours(NeighborArray neighbors, int[] oldToNewMap, int maxOrd) throws java.io.IOException
- Throws:
java.io.IOException
-
mergeOneField
public void mergeOneField(FieldInfo fieldInfo, MergeState mergeState) throws java.io.IOException
Description copied from class:KnnVectorsWriter
Write field for merging- Overrides:
mergeOneField
in classKnnVectorsWriter
- Throws:
java.io.IOException
-
writeGraph
private int[][] writeGraph(OnHeapHnswGraph graph) throws java.io.IOException
- Parameters:
graph
- Write the graph in a compressed format- Returns:
- The non-cumulative offsets for the nodes. Should be used to create cumulative offsets.
- Throws:
java.io.IOException
- if writing to vectorIndex fails
-
writeMeta
private void writeMeta(FieldInfo field, long vectorIndexOffset, long vectorIndexLength, int count, HnswGraph graph, int[][] graphLevelNodeOffsets) throws java.io.IOException
- Throws:
java.io.IOException
-
createGraphMerger
private HnswGraphMerger createGraphMerger(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, TaskExecutor parallelMergeTaskExecutor, int numParallelMergeWorkers)
-
close
public void close() throws java.io.IOException
- Throws:
java.io.IOException
-
distFuncToOrd
static int distFuncToOrd(VectorSimilarityFunction func)
-
-