- java.lang.Object
-
- org.apache.lucene.index.IndexFileDeleter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
final class IndexFileDeleter extends java.lang.Object implements java.io.Closeable
This class keeps track of each SegmentInfos instance that is still "live", either because it corresponds to a segments_N file in the Directory (a "commit", i.e. a committed SegmentInfos) or because it's an in-memory SegmentInfos that a writer is actively updating but has not yet committed. This class uses simple reference counting to map the live SegmentInfos instances to individual files in the Directory.The same directory file may be referenced by more than one IndexCommit, i.e. more than one SegmentInfos. Therefore we count how many commits reference each file. When all the commits referencing a certain file have been deleted, the refcount for that file becomes zero, and the file is deleted.
A separate deletion policy interface (IndexDeletionPolicy) is consulted on creation (onInit) and once per commit (onCommit), to decide when a commit should be removed.
It is the business of the IndexDeletionPolicy to choose when to delete commit points. The actual mechanics of file deletion, retrying, etc, derived from the deletion of commit points is the business of the IndexFileDeleter.
The current default deletion policy is
KeepOnlyLastCommitDeletionPolicy
, which removes all prior commits when a new commit has completed. This matches the behavior before 2.2.Note that you must hold the write.lock before instantiating this class. It opens segments_N file(s) directly with no retry logic.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
IndexFileDeleter.CommitPoint
Holds details for each commit point.
-
Field Summary
Fields Modifier and Type Field Description private java.util.List<IndexFileDeleter.CommitPoint>
commits
private java.util.List<IndexFileDeleter.CommitPoint>
commitsToDelete
private Directory
directory
private Directory
directoryOrig
private FileDeleter
fileDeleter
private InfoStream
infoStream
private java.util.List<java.lang.String>
lastFiles
private SegmentInfos
lastSegmentInfos
private IndexDeletionPolicy
policy
(package private) boolean
startingCommitDeleted
static boolean
VERBOSE_REF_COUNTS
Change to true to see details of reference counts when infoStream is enabledprivate IndexWriter
writer
-
Constructor Summary
Constructors Constructor Description IndexFileDeleter(java.lang.String[] files, Directory directoryOrig, Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, boolean initialIndexExists, boolean isReaderInit)
Initialize the deleter: find all previous commits in the Directory, incref the files they reference, call the policy to let it delete commits.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) boolean
assertCommitsAreNotDeleted(java.util.List<IndexFileDeleter.CommitPoint> commits)
void
checkpoint(SegmentInfos segmentInfos, boolean isCommit)
For definition of "check point" see IndexWriter comments: "Clarification: Check Points (and commits)".void
close()
(package private) void
decRef(java.util.Collection<java.lang.String> files)
Decrefs all provided files, even on exception; throws first exception hit, if any.(package private) void
decRef(SegmentInfos segmentInfos)
private void
deleteCommits()
Remove the CommitPoints in the commitsToDelete List by DecRef'ing all files from each SegmentInfos.(package private) void
deleteNewFiles(java.util.Collection<java.lang.String> files)
Deletes the specified files, but only if they are new (have not yet been incref'd).(package private) void
ensureOpen()
boolean
exists(java.lang.String fileName)
(package private) void
incRef(java.util.Collection<java.lang.String> files)
(package private) void
incRef(SegmentInfos segmentInfos, boolean isCommit)
(package private) static void
inflateGens(SegmentInfos infos, java.util.Collection<java.lang.String> files, InfoStream infoStream)
Set all gens beyond what we currently see in the directory, to avoid double-write in cases where the previous IndexWriter did not gracefully close/rollback (e.g.(package private) boolean
isClosed()
private boolean
locked()
private void
logInfo(FileDeleter.MsgType msgType, java.lang.String msg)
(package private) void
refresh()
Writer calls this when it has hit an error and had to roll back, to tell us that there may now be unreferenced files in the filesystem.(package private) void
revisitPolicy()
Revisits theIndexDeletionPolicy
by calling itsIndexDeletionPolicy.onCommit(List)
again with the known commits.
-
-
-
Field Detail
-
commits
private java.util.List<IndexFileDeleter.CommitPoint> commits
-
lastFiles
private final java.util.List<java.lang.String> lastFiles
-
commitsToDelete
private java.util.List<IndexFileDeleter.CommitPoint> commitsToDelete
-
infoStream
private final InfoStream infoStream
-
directoryOrig
private final Directory directoryOrig
-
directory
private final Directory directory
-
policy
private final IndexDeletionPolicy policy
-
startingCommitDeleted
final boolean startingCommitDeleted
-
lastSegmentInfos
private SegmentInfos lastSegmentInfos
-
VERBOSE_REF_COUNTS
public static boolean VERBOSE_REF_COUNTS
Change to true to see details of reference counts when infoStream is enabled
-
fileDeleter
private final FileDeleter fileDeleter
-
writer
private final IndexWriter writer
-
-
Constructor Detail
-
IndexFileDeleter
public IndexFileDeleter(java.lang.String[] files, Directory directoryOrig, Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, boolean initialIndexExists, boolean isReaderInit) throws java.io.IOException
Initialize the deleter: find all previous commits in the Directory, incref the files they reference, call the policy to let it delete commits. This will remove any files not referenced by any of the commits.- Throws:
java.io.IOException
- if there is a low-level IO error
-
-
Method Detail
-
locked
private boolean locked()
-
inflateGens
static void inflateGens(SegmentInfos infos, java.util.Collection<java.lang.String> files, InfoStream infoStream)
Set all gens beyond what we currently see in the directory, to avoid double-write in cases where the previous IndexWriter did not gracefully close/rollback (e.g. os/machine crashed or lost power).
-
ensureOpen
void ensureOpen() throws AlreadyClosedException
- Throws:
AlreadyClosedException
-
isClosed
boolean isClosed()
-
deleteCommits
private void deleteCommits() throws java.io.IOException
Remove the CommitPoints in the commitsToDelete List by DecRef'ing all files from each SegmentInfos.- Throws:
java.io.IOException
-
refresh
void refresh() throws java.io.IOException
Writer calls this when it has hit an error and had to roll back, to tell us that there may now be unreferenced files in the filesystem. So we re-list the filesystem and delete such files. If segmentName is non-null, we will only delete files corresponding to that segment.- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
assertCommitsAreNotDeleted
boolean assertCommitsAreNotDeleted(java.util.List<IndexFileDeleter.CommitPoint> commits)
-
revisitPolicy
void revisitPolicy() throws java.io.IOException
Revisits theIndexDeletionPolicy
by calling itsIndexDeletionPolicy.onCommit(List)
again with the known commits. This is useful in cases where a deletion policy which holds onto index commits is used. The application may know that some commits are not held by the deletion policy anymore and callIndexWriter.deleteUnusedFiles()
, which will attempt to delete the unused commits again.- Throws:
java.io.IOException
-
checkpoint
public void checkpoint(SegmentInfos segmentInfos, boolean isCommit) throws java.io.IOException
For definition of "check point" see IndexWriter comments: "Clarification: Check Points (and commits)".Writer calls this when it has made a "consistent change" to the index, meaning new files are written to the index and the in-memory SegmentInfos have been modified to point to those files.
This may or may not be a commit (segments_N may or may not have been written).
We simply incref the files referenced by the new SegmentInfos and decref the files we had previously seen (if any).
If this is a commit, we also call the policy to give it a chance to remove other commits. If any commits are removed, we decref their files as well.
- Throws:
java.io.IOException
-
logInfo
private void logInfo(FileDeleter.MsgType msgType, java.lang.String msg)
-
incRef
void incRef(SegmentInfos segmentInfos, boolean isCommit) throws java.io.IOException
- Throws:
java.io.IOException
-
incRef
void incRef(java.util.Collection<java.lang.String> files)
-
decRef
void decRef(java.util.Collection<java.lang.String> files) throws java.io.IOException
Decrefs all provided files, even on exception; throws first exception hit, if any.- Throws:
java.io.IOException
-
decRef
void decRef(SegmentInfos segmentInfos) throws java.io.IOException
- Throws:
java.io.IOException
-
exists
public boolean exists(java.lang.String fileName)
-
deleteNewFiles
void deleteNewFiles(java.util.Collection<java.lang.String> files) throws java.io.IOException
Deletes the specified files, but only if they are new (have not yet been incref'd).- Throws:
java.io.IOException
-
-