- java.lang.Object
-
- org.apache.lucene.index.CheckIndex
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class CheckIndex extends java.lang.Object implements java.io.Closeable
Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.As this tool checks every byte in the index, on a large index it can take quite a long time to run.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
CheckIndex.CheckIndexException
The marker RuntimeException used by CheckIndex APIs when index integrity failure is detected.private static class
CheckIndex.ConstantRelationIntersectVisitor
private static interface
CheckIndex.DocValuesIteratorSupplier
static class
CheckIndex.Options
Run-time configuration options for CheckIndex commands.static class
CheckIndex.Status
Returned fromcheckIndex()
detailing the health and status of the index.static class
CheckIndex.VerifyPointsVisitor
Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.
-
Field Summary
Fields Modifier and Type Field Description private static boolean
assertsOn
private boolean
checksumsOnly
private boolean
closed
private Directory
dir
private boolean
doSlowChecks
private boolean
failFast
private java.io.PrintStream
infoStream
private java.text.NumberFormat
nf
private int
threadCount
private boolean
verbose
private Lock
writeLock
-
Constructor Summary
Constructors Constructor Description CheckIndex(Directory dir)
Create a new CheckIndex on the directory.CheckIndex(Directory dir, Lock writeLock)
Expert: create a directory with the specified lock.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static boolean
assertsOn()
Check whether asserts are enabled or not.private <T> java.util.function.Supplier<T>
callableToSupplier(java.util.concurrent.Callable<T> callable)
private static void
checkBinaryDocValues(java.lang.String fieldName, BinaryDocValues bdv, BinaryDocValues bdv2)
private static void
checkByteVectorValues(ByteVectorValues values, FieldInfo fieldInfo, CheckIndex.Status.VectorValuesStatus status, CodecReader codecReader)
private static void
checkDocValues(FieldInfo fi, DocValuesProducer dvReader, CheckIndex.Status.DocValuesStatus status)
private static void
checkDVIterator(FieldInfo fi, CheckIndex.DocValuesIteratorSupplier producer)
private static CheckIndex.Status.TermIndexStatus
checkFields(Fields fields, Bits liveDocs, int maxDoc, FieldInfos fieldInfos, NormsProducer normsProducer, boolean doPrint, boolean isVectors, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks)
checks Fields api is consistent with itself.private static void
checkFloatVectorValues(FloatVectorValues values, FieldInfo fieldInfo, CheckIndex.Status.VectorValuesStatus status, CodecReader codecReader)
(package private) static void
checkImpacts(Impacts impacts, int lastTarget)
For use in tests only.CheckIndex.Status
checkIndex()
Returns aCheckIndex.Status
instance detailing the state of the index.CheckIndex.Status
checkIndex(java.util.List<java.lang.String> onlySegments)
Returns aCheckIndex.Status
instance detailing the state of the index.CheckIndex.Status
checkIndex(java.util.List<java.lang.String> onlySegments, java.util.concurrent.ExecutorService executorService)
Returns aCheckIndex.Status
instance detailing the state of the index.private static void
checkNumericDocValues(java.lang.String fieldName, NumericDocValues ndv, NumericDocValues ndv2)
private static CheckIndex.Status.SoftDeletsStatus
checkSoftDeletes(java.lang.String softDeletesField, SegmentCommitInfo info, SegmentReader reader, java.io.PrintStream infoStream, boolean failFast)
private static void
checkSortedDocValues(java.lang.String fieldName, SortedDocValues dv, SortedDocValues dv2)
private static void
checkSortedNumericDocValues(java.lang.String fieldName, SortedNumericDocValues ndv, SortedNumericDocValues ndv2)
private static void
checkSortedSetDocValues(java.lang.String fieldName, SortedSetDocValues dv, SortedSetDocValues dv2)
private static void
checkTermsIntersect(Terms terms, Automaton automaton, BytesRef startTerm)
void
close()
int
doCheck(CheckIndex.Options opts)
Actually perform the index checkprivate static int
doMain(java.lang.String[] args)
boolean
doSlowChecks()
private void
ensureOpen()
void
exorciseIndex(CheckIndex.Status result)
Repairs the index using previously returned result fromcheckIndex()
.boolean
getChecksumsOnly()
SeegetChecksumsOnly()
.boolean
getFailFast()
SeesetFailFast(boolean)
.static void
main(java.lang.String[] args)
Command-line interface to check and exorcise corrupt segments from an index.private static void
msg(java.io.PrintStream out, java.io.ByteArrayOutputStream msg)
private static void
msg(java.io.PrintStream out, java.lang.String msg)
private static double
nsToSec(long ns)
static CheckIndex.Options
parseOptions(java.lang.String[] args)
Parse command line args into fieldsprivate void
processSegmentInfoStatusResult(CheckIndex.Status result, SegmentCommitInfo info, CheckIndex.Status.SegmentInfoStatus segmentInfoStatus)
private <R> java.util.concurrent.CompletableFuture<R>
runAsyncSegmentCheck(java.util.concurrent.Callable<R> asyncCallable, java.util.concurrent.ExecutorService executorService)
void
setChecksumsOnly(boolean v)
If true, only validate physical integrity for all files.void
setDoSlowChecks(boolean v)
If true, additional slow checks are performed.void
setFailFast(boolean v)
If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.void
setInfoStream(java.io.PrintStream out)
Set infoStream where messages should go.void
setInfoStream(java.io.PrintStream out, boolean verbose)
Set infoStream where messages should go.void
setThreadCount(int tc)
Set threadCount used for parallelizing index integrity checking.private static boolean
testAsserts()
static CheckIndex.Status.DocValuesStatus
testDocValues(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test docvalues.static CheckIndex.Status.FieldInfoStatus
testFieldInfos(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test field infos.static CheckIndex.Status.FieldNormStatus
testFieldNorms(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test field norms.static CheckIndex.Status.LiveDocStatus
testLiveDocs(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test live docs.static CheckIndex.Status.PointsStatus
testPoints(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test the points index.static CheckIndex.Status.TermIndexStatus
testPostings(CodecReader reader, java.io.PrintStream infoStream)
Test the term index.static CheckIndex.Status.TermIndexStatus
testPostings(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)
Test the term index.private CheckIndex.Status.SegmentInfoStatus
testSegment(SegmentInfos sis, SegmentCommitInfo info, java.io.PrintStream infoStream)
static CheckIndex.Status.IndexSortStatus
testSort(CodecReader reader, Sort sort, java.io.PrintStream infoStream, boolean failFast)
Tests index sort order.static CheckIndex.Status.StoredFieldStatus
testStoredFields(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test stored fields.static CheckIndex.Status.TermVectorStatus
testTermVectors(CodecReader reader, java.io.PrintStream infoStream)
Test term vectors.static CheckIndex.Status.TermVectorStatus
testTermVectors(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)
Test term vectors.static CheckIndex.Status.VectorValuesStatus
testVectors(CodecReader reader, java.io.PrintStream infoStream, boolean failFast)
Test the vectors index.private void
updateMaxSegmentName(CheckIndex.Status result, SegmentCommitInfo info)
-
-
-
Field Detail
-
dir
private final Directory dir
-
writeLock
private final Lock writeLock
-
nf
private final java.text.NumberFormat nf
-
infoStream
private java.io.PrintStream infoStream
-
closed
private volatile boolean closed
-
doSlowChecks
private boolean doSlowChecks
-
failFast
private boolean failFast
-
verbose
private boolean verbose
-
checksumsOnly
private boolean checksumsOnly
-
threadCount
private int threadCount
-
assertsOn
private static boolean assertsOn
-
-
Constructor Detail
-
CheckIndex
public CheckIndex(Directory dir) throws java.io.IOException
Create a new CheckIndex on the directory.- Throws:
java.io.IOException
-
CheckIndex
public CheckIndex(Directory dir, Lock writeLock)
Expert: create a directory with the specified lock. This should really not be used except for unit tests!!!! It exists only to support special tests (such as TestIndexWriterExceptions*), that would otherwise be more complicated to debug if they had to close the writer for each check.
-
-
Method Detail
-
ensureOpen
private void ensureOpen()
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
setDoSlowChecks
public void setDoSlowChecks(boolean v)
If true, additional slow checks are performed. This will likely drastically increase time it takes to run CheckIndex!
-
doSlowChecks
public boolean doSlowChecks()
-
setFailFast
public void setFailFast(boolean v)
If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.
-
getFailFast
public boolean getFailFast()
SeesetFailFast(boolean)
.
-
getChecksumsOnly
public boolean getChecksumsOnly()
SeegetChecksumsOnly()
.
-
setChecksumsOnly
public void setChecksumsOnly(boolean v)
If true, only validate physical integrity for all files. Note that the returned nested status objects (e.g. storedFieldStatus) will be null.
-
setThreadCount
public void setThreadCount(int tc)
Set threadCount used for parallelizing index integrity checking.
-
setInfoStream
public void setInfoStream(java.io.PrintStream out, boolean verbose)
Set infoStream where messages should go. If null, no messages are printed. If verbose is true then more details are printed.
-
setInfoStream
public void setInfoStream(java.io.PrintStream out)
Set infoStream where messages should go. SeesetInfoStream(PrintStream,boolean)
.
-
msg
private static void msg(java.io.PrintStream out, java.io.ByteArrayOutputStream msg)
-
msg
private static void msg(java.io.PrintStream out, java.lang.String msg)
-
checkIndex
public CheckIndex.Status checkIndex() throws java.io.IOException
Returns aCheckIndex.Status
instance detailing the state of the index.As this method checks every byte in the index, on a large index it can take quite a long time to run.
WARNING: make sure you only call this when the index is not opened by any writer.
- Throws:
java.io.IOException
-
checkIndex
public CheckIndex.Status checkIndex(java.util.List<java.lang.String> onlySegments) throws java.io.IOException
Returns aCheckIndex.Status
instance detailing the state of the index.- Parameters:
onlySegments
- list of specific segment names to checkAs this method checks every byte in the specified segments, on a large index it can take quite a long time to run.
- Throws:
java.io.IOException
-
checkIndex
public CheckIndex.Status checkIndex(java.util.List<java.lang.String> onlySegments, java.util.concurrent.ExecutorService executorService) throws java.io.IOException
Returns aCheckIndex.Status
instance detailing the state of the index.This method allows caller to pass in customized ExecutorService to speed up the check.
WARNING: make sure you only call this when the index is not opened by any writer.
- Throws:
java.io.IOException
-
updateMaxSegmentName
private void updateMaxSegmentName(CheckIndex.Status result, SegmentCommitInfo info)
-
processSegmentInfoStatusResult
private void processSegmentInfoStatusResult(CheckIndex.Status result, SegmentCommitInfo info, CheckIndex.Status.SegmentInfoStatus segmentInfoStatus)
-
runAsyncSegmentCheck
private <R> java.util.concurrent.CompletableFuture<R> runAsyncSegmentCheck(java.util.concurrent.Callable<R> asyncCallable, java.util.concurrent.ExecutorService executorService)
-
callableToSupplier
private <T> java.util.function.Supplier<T> callableToSupplier(java.util.concurrent.Callable<T> callable)
-
testSegment
private CheckIndex.Status.SegmentInfoStatus testSegment(SegmentInfos sis, SegmentCommitInfo info, java.io.PrintStream infoStream) throws java.io.IOException
- Throws:
java.io.IOException
-
testSort
public static CheckIndex.Status.IndexSortStatus testSort(CodecReader reader, Sort sort, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Tests index sort order.- Throws:
java.io.IOException
-
testLiveDocs
public static CheckIndex.Status.LiveDocStatus testLiveDocs(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test live docs.- Throws:
java.io.IOException
-
testFieldInfos
public static CheckIndex.Status.FieldInfoStatus testFieldInfos(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test field infos.- Throws:
java.io.IOException
-
testFieldNorms
public static CheckIndex.Status.FieldNormStatus testFieldNorms(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test field norms.- Throws:
java.io.IOException
-
checkFields
private static CheckIndex.Status.TermIndexStatus checkFields(Fields fields, Bits liveDocs, int maxDoc, FieldInfos fieldInfos, NormsProducer normsProducer, boolean doPrint, boolean isVectors, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks) throws java.io.IOException
checks Fields api is consistent with itself. searcher is optional, to verify with queries. Can be null.- Throws:
java.io.IOException
-
checkTermsIntersect
private static void checkTermsIntersect(Terms terms, Automaton automaton, BytesRef startTerm) throws java.io.IOException
- Throws:
java.io.IOException
-
checkImpacts
static void checkImpacts(Impacts impacts, int lastTarget)
For use in tests only.
-
testPostings
public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader, java.io.PrintStream infoStream) throws java.io.IOException
Test the term index.- Throws:
java.io.IOException
-
testPostings
public static CheckIndex.Status.TermIndexStatus testPostings(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast) throws java.io.IOException
Test the term index.- Throws:
java.io.IOException
-
testPoints
public static CheckIndex.Status.PointsStatus testPoints(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test the points index.- Throws:
java.io.IOException
-
testVectors
public static CheckIndex.Status.VectorValuesStatus testVectors(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test the vectors index.- Throws:
java.io.IOException
-
checkFloatVectorValues
private static void checkFloatVectorValues(FloatVectorValues values, FieldInfo fieldInfo, CheckIndex.Status.VectorValuesStatus status, CodecReader codecReader) throws java.io.IOException
- Throws:
java.io.IOException
-
checkByteVectorValues
private static void checkByteVectorValues(ByteVectorValues values, FieldInfo fieldInfo, CheckIndex.Status.VectorValuesStatus status, CodecReader codecReader) throws java.io.IOException
- Throws:
java.io.IOException
-
testStoredFields
public static CheckIndex.Status.StoredFieldStatus testStoredFields(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test stored fields.- Throws:
java.io.IOException
-
testDocValues
public static CheckIndex.Status.DocValuesStatus testDocValues(CodecReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
Test docvalues.- Throws:
java.io.IOException
-
checkDVIterator
private static void checkDVIterator(FieldInfo fi, CheckIndex.DocValuesIteratorSupplier producer) throws java.io.IOException
- Throws:
java.io.IOException
-
checkBinaryDocValues
private static void checkBinaryDocValues(java.lang.String fieldName, BinaryDocValues bdv, BinaryDocValues bdv2) throws java.io.IOException
- Throws:
java.io.IOException
-
checkSortedDocValues
private static void checkSortedDocValues(java.lang.String fieldName, SortedDocValues dv, SortedDocValues dv2) throws java.io.IOException
- Throws:
java.io.IOException
-
checkSortedSetDocValues
private static void checkSortedSetDocValues(java.lang.String fieldName, SortedSetDocValues dv, SortedSetDocValues dv2) throws java.io.IOException
- Throws:
java.io.IOException
-
checkSortedNumericDocValues
private static void checkSortedNumericDocValues(java.lang.String fieldName, SortedNumericDocValues ndv, SortedNumericDocValues ndv2) throws java.io.IOException
- Throws:
java.io.IOException
-
checkNumericDocValues
private static void checkNumericDocValues(java.lang.String fieldName, NumericDocValues ndv, NumericDocValues ndv2) throws java.io.IOException
- Throws:
java.io.IOException
-
checkDocValues
private static void checkDocValues(FieldInfo fi, DocValuesProducer dvReader, CheckIndex.Status.DocValuesStatus status) throws java.lang.Exception
- Throws:
java.lang.Exception
-
testTermVectors
public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader, java.io.PrintStream infoStream) throws java.io.IOException
Test term vectors.- Throws:
java.io.IOException
-
testTermVectors
public static CheckIndex.Status.TermVectorStatus testTermVectors(CodecReader reader, java.io.PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast) throws java.io.IOException
Test term vectors.- Throws:
java.io.IOException
-
exorciseIndex
public void exorciseIndex(CheckIndex.Status result) throws java.io.IOException
Repairs the index using previously returned result fromcheckIndex()
. Note that this does not remove any of the unreferenced files after it's done; you must separately open anIndexWriter
, which deletes unreferenced files when it's created.WARNING: this writes a new segments file into the index, effectively removing all documents in broken segments from the index. BE CAREFUL.
- Throws:
java.io.IOException
-
testAsserts
private static boolean testAsserts()
-
assertsOn
public static boolean assertsOn()
Check whether asserts are enabled or not.- Returns:
- true iff asserts are enabled
-
main
public static void main(java.lang.String[] args) throws java.io.IOException, java.lang.InterruptedException
Command-line interface to check and exorcise corrupt segments from an index.Run it like this:
java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]
-exorcise
: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*-segment X
: only check the specified segment(s). This can be specified multiple times, to check more than one segment:-segment _2 * -segment _a
. You can't use this with the -exorcise option.
WARNING:
-exorcise
should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.
This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.
- Throws:
java.io.IOException
java.lang.InterruptedException
-
doMain
private static int doMain(java.lang.String[] args) throws java.io.IOException, java.lang.InterruptedException
- Throws:
java.io.IOException
java.lang.InterruptedException
-
parseOptions
public static CheckIndex.Options parseOptions(java.lang.String[] args)
Parse command line args into fields- Parameters:
args
- The command line arguments- Returns:
- An Options struct
- Throws:
java.lang.IllegalArgumentException
- if any of the CLI args are invalid
-
doCheck
public int doCheck(CheckIndex.Options opts) throws java.io.IOException, java.lang.InterruptedException
Actually perform the index check- Parameters:
opts
- The options to use for this check- Returns:
- 0 iff the index is clean, 1 otherwise
- Throws:
java.io.IOException
java.lang.InterruptedException
-
checkSoftDeletes
private static CheckIndex.Status.SoftDeletsStatus checkSoftDeletes(java.lang.String softDeletesField, SegmentCommitInfo info, SegmentReader reader, java.io.PrintStream infoStream, boolean failFast) throws java.io.IOException
- Throws:
java.io.IOException
-
nsToSec
private static double nsToSec(long ns)
-
-