Module org.apache.lucene.core
Class Lucene99ScalarQuantizedVectorsFormat
- java.lang.Object
-
- org.apache.lucene.codecs.hnsw.FlatVectorsFormat
-
- org.apache.lucene.codecs.lucene99.Lucene99ScalarQuantizedVectorsFormat
-
public class Lucene99ScalarQuantizedVectorsFormat extends FlatVectorsFormat
Format supporting vector quantization, storage, and retrieval
-
-
Field Summary
Fields Modifier and Type Field Description private static int
ALLOWED_BITS
(package private) byte
bits
(package private) boolean
compress
(package private) java.lang.Float
confidenceInterval
Controls the confidence interval used to scalar quantize the vectors the default value is calculated as `1-1/(vector_dimensions + 1)`static float
DYNAMIC_CONFIDENCE_INTERVAL
Dynamic confidence interval(package private) Lucene99ScalarQuantizedVectorScorer
flatVectorScorer
private static float
MAXIMUM_CONFIDENCE_INTERVAL
The maximum confidence interval(package private) static java.lang.String
META_CODEC_NAME
(package private) static java.lang.String
META_EXTENSION
private static float
MINIMUM_CONFIDENCE_INTERVAL
The minimum confidence intervalstatic java.lang.String
NAME
static java.lang.String
QUANTIZED_VECTOR_COMPONENT
private static FlatVectorsFormat
rawVectorFormat
(package private) static java.lang.String
VECTOR_DATA_CODEC_NAME
(package private) static java.lang.String
VECTOR_DATA_EXTENSION
(package private) static int
VERSION_ADD_BITS
(package private) static int
VERSION_CURRENT
(package private) static int
VERSION_START
-
Constructor Summary
Constructors Constructor Description Lucene99ScalarQuantizedVectorsFormat()
Constructs a format using default graph construction parametersLucene99ScalarQuantizedVectorsFormat(java.lang.Float confidenceInterval, int bits, boolean compress)
Constructs a format using the given graph construction parameters.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static float
calculateDefaultConfidenceInterval(int vectorDimension)
FlatVectorsReader
fieldsReader(SegmentReadState state)
Returns aKnnVectorsReader
to read the vectors from the index.FlatVectorsWriter
fieldsWriter(SegmentWriteState state)
Returns aFlatVectorsWriter
to write the vectors to the index.java.lang.String
toString()
-
-
-
Field Detail
-
ALLOWED_BITS
private static final int ALLOWED_BITS
- See Also:
- Constant Field Values
-
QUANTIZED_VECTOR_COMPONENT
public static final java.lang.String QUANTIZED_VECTOR_COMPONENT
- See Also:
- Constant Field Values
-
NAME
public static final java.lang.String NAME
- See Also:
- Constant Field Values
-
VERSION_START
static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_ADD_BITS
static final int VERSION_ADD_BITS
- See Also:
- Constant Field Values
-
VERSION_CURRENT
static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
META_CODEC_NAME
static final java.lang.String META_CODEC_NAME
- See Also:
- Constant Field Values
-
VECTOR_DATA_CODEC_NAME
static final java.lang.String VECTOR_DATA_CODEC_NAME
- See Also:
- Constant Field Values
-
META_EXTENSION
static final java.lang.String META_EXTENSION
- See Also:
- Constant Field Values
-
VECTOR_DATA_EXTENSION
static final java.lang.String VECTOR_DATA_EXTENSION
- See Also:
- Constant Field Values
-
rawVectorFormat
private static final FlatVectorsFormat rawVectorFormat
-
MINIMUM_CONFIDENCE_INTERVAL
private static final float MINIMUM_CONFIDENCE_INTERVAL
The minimum confidence interval- See Also:
- Constant Field Values
-
MAXIMUM_CONFIDENCE_INTERVAL
private static final float MAXIMUM_CONFIDENCE_INTERVAL
The maximum confidence interval- See Also:
- Constant Field Values
-
DYNAMIC_CONFIDENCE_INTERVAL
public static final float DYNAMIC_CONFIDENCE_INTERVAL
Dynamic confidence interval- See Also:
- Constant Field Values
-
confidenceInterval
final java.lang.Float confidenceInterval
Controls the confidence interval used to scalar quantize the vectors the default value is calculated as `1-1/(vector_dimensions + 1)`
-
bits
final byte bits
-
compress
final boolean compress
-
flatVectorScorer
final Lucene99ScalarQuantizedVectorScorer flatVectorScorer
-
-
Constructor Detail
-
Lucene99ScalarQuantizedVectorsFormat
public Lucene99ScalarQuantizedVectorsFormat()
Constructs a format using default graph construction parameters
-
Lucene99ScalarQuantizedVectorsFormat
public Lucene99ScalarQuantizedVectorsFormat(java.lang.Float confidenceInterval, int bits, boolean compress)
Constructs a format using the given graph construction parameters.- Parameters:
confidenceInterval
- the confidenceInterval for scalar quantizing the vectors, when `null` it is calculated based on the vector dimension. When `0`, the quantiles are dynamically determined by sampling many confidence intervals and determining the most accurate pair.bits
- the number of bits to use for scalar quantization (must be between 1 and 8, inclusive)compress
- whether to compress the vectors, if true, the vectors that are quantized with lte 4 bits will be compressed into a single byte. If false, the vectors will be stored as is. This provides a trade-off of memory usage and speed.
-
-
Method Detail
-
calculateDefaultConfidenceInterval
public static float calculateDefaultConfidenceInterval(int vectorDimension)
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
fieldsWriter
public FlatVectorsWriter fieldsWriter(SegmentWriteState state) throws java.io.IOException
Description copied from class:FlatVectorsFormat
Returns aFlatVectorsWriter
to write the vectors to the index.- Specified by:
fieldsWriter
in classFlatVectorsFormat
- Throws:
java.io.IOException
-
fieldsReader
public FlatVectorsReader fieldsReader(SegmentReadState state) throws java.io.IOException
Description copied from class:FlatVectorsFormat
Returns aKnnVectorsReader
to read the vectors from the index.- Specified by:
fieldsReader
in classFlatVectorsFormat
- Throws:
java.io.IOException
-
-