- java.lang.Object
-
- org.apache.lucene.store.Directory
-
- org.apache.lucene.store.BaseDirectory
-
- org.apache.lucene.store.FSDirectory
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
- Direct Known Subclasses:
MMapDirectory
,NIOFSDirectory
,RAFDirectory
public abstract class FSDirectory extends BaseDirectory
Base class for Directory implementations that store index files in the file system. There are currently three core subclasses:NIOFSDirectory
uses java.nio's FileChannel's positional io when reading to avoid synchronization when reading from the same file. Unfortunately, due to a Windows-only Sun JRE bug this is a poor choice for Windows, but on all other platforms this is the preferred choice. Applications usingThread.interrupt()
orFuture.cancel(boolean)
should useRAFDirectory
instead. SeeNIOFSDirectory
java doc for details.MMapDirectory
uses memory-mapped IO when reading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. Due to this bug in Sun's JRE, MMapDirectory'sIndexInput.close()
is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs.
Unfortunately, because of system peculiarities, there is no single overall best implementation. Therefore, we've added the
open(java.nio.file.Path)
method, to allow Lucene to choose the best FSDirectory implementation given your environment, and the known limitations of each implementation. For users who have no reason to prefer a specific implementation, it's best to simply useopen(java.nio.file.Path)
. For all others, you should instantiate the desired implementation directly.NOTE: Accessing one of the above subclasses either directly or indirectly from a thread while it's interrupted can close the underlying channel immediately if at the same time the thread is blocked on IO. The channel will remain closed and subsequent access to the index will throw a
ClosedChannelException
. Applications usingThread.interrupt()
orFuture.cancel(boolean)
should use the slower legacyRAFDirectory
from themisc
Lucene module instead.The locking implementation is by default
NativeFSLockFactory
, but can be changed by passing in a customLockFactory
instance.- See Also:
Directory
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) class
FSDirectory.FSIndexOutput
-
Field Summary
Fields Modifier and Type Field Description protected java.nio.file.Path
directory
private java.util.concurrent.atomic.AtomicLong
nextTempFileCounter
Used to generate temp file names increateTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext)
.private java.util.concurrent.atomic.AtomicInteger
opsSinceLastDelete
private java.util.Set<java.lang.String>
pendingDeletes
Maps files that we are trying to delete (or we tried already but failed) before attempting to delete that key.-
Fields inherited from class org.apache.lucene.store.BaseDirectory
isOpen, lockFactory
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
FSDirectory(java.nio.file.Path path, LockFactory lockFactory)
Create a new FSDirectory for the named location (ctor for subclasses).
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
Closes the directory.IndexOutput
createOutput(java.lang.String name, IOContext context)
Creates a new, empty file in the directory and returns anIndexOutput
instance for appending data to this file.IndexOutput
createTempOutput(java.lang.String prefix, java.lang.String suffix, IOContext context)
Creates a new, empty, temporary file in the directory and returns anIndexOutput
instance for appending data to this file.void
deleteFile(java.lang.String name)
Removes an existing file in the directory.void
deletePendingFiles()
Try to delete any pending files that we had previously tried to delete but failed because we are on Windows and the files were still held open.protected void
ensureCanRead(java.lang.String name)
long
fileLength(java.lang.String name)
Returns the byte length of a file in the directory.protected void
fsync(java.lang.String name)
java.nio.file.Path
getDirectory()
java.util.Set<java.lang.String>
getPendingDeletions()
Returns a set of files currently pending deletion in this directory.java.lang.String[]
listAll()
Returns names of all files stored in this directory.static java.lang.String[]
listAll(java.nio.file.Path dir)
Lists all files (including subdirectories) in the directory.private static java.lang.String[]
listAll(java.nio.file.Path dir, java.util.Set<java.lang.String> skipNames)
private void
maybeDeletePendingFiles()
static FSDirectory
open(java.nio.file.Path path)
Creates an FSDirectory instance, trying to pick the best implementation given the current environment.static FSDirectory
open(java.nio.file.Path path, LockFactory lockFactory)
Just likeopen(Path)
, but allows you to also specify a customLockFactory
.private void
privateDeleteFile(java.lang.String name, boolean isPendingDelete)
void
rename(java.lang.String source, java.lang.String dest)
Renamessource
file todest
file wheredest
must not already exist in the directory.void
sync(java.util.Collection<java.lang.String> names)
Ensures that any writes to these files are moved to stable storage (made durable).void
syncMetaData()
Ensures that directory metadata, such as recent file renames, are moved to stable storage.java.lang.String
toString()
-
Methods inherited from class org.apache.lucene.store.BaseDirectory
ensureOpen, obtainLock
-
Methods inherited from class org.apache.lucene.store.Directory
copyFrom, getTempFileName, openChecksumInput, openInput
-
-
-
-
Field Detail
-
directory
protected final java.nio.file.Path directory
-
pendingDeletes
private final java.util.Set<java.lang.String> pendingDeletes
Maps files that we are trying to delete (or we tried already but failed) before attempting to delete that key.
-
opsSinceLastDelete
private final java.util.concurrent.atomic.AtomicInteger opsSinceLastDelete
-
nextTempFileCounter
private final java.util.concurrent.atomic.AtomicLong nextTempFileCounter
Used to generate temp file names increateTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext)
.
-
-
Constructor Detail
-
FSDirectory
protected FSDirectory(java.nio.file.Path path, LockFactory lockFactory) throws java.io.IOException
Create a new FSDirectory for the named location (ctor for subclasses). The directory is created at the named location if it does not yet exist.FSDirectory
resolves the given Path to a canonical / real path to ensure it can correctly lock the index directory and no other process can interfere with changing possible symlinks to the index directory inbetween. If you want to use symlinks and change them dynamically, close allIndexWriters
and create a newFSDirectory
instance.- Parameters:
path
- the path of the directorylockFactory
- the lock factory to use, or null for the default (NativeFSLockFactory
);- Throws:
java.io.IOException
- if there is a low-level I/O error
-
-
Method Detail
-
open
public static FSDirectory open(java.nio.file.Path path) throws java.io.IOException
Creates an FSDirectory instance, trying to pick the best implementation given the current environment. The directory returned uses theNativeFSLockFactory
. The directory is created at the named location if it does not yet exist.FSDirectory
resolves the given Path when calling this method to a canonical / real path to ensure it can correctly lock the index directory and no other process can interfere with changing possible symlinks to the index directory inbetween. If you want to use symlinks and change them dynamically, close allIndexWriters
and create a newFSDirectory
instance.Currently this returns
MMapDirectory
for Linux, MacOSX, Solaris, and Windows 64-bit JREs, andNIOFSDirectory
for other JREs. It is highly recommended that you consult the implementation's documentation for your platform before using this method.NOTE: this method may suddenly change which implementation is returned from release to release, in the event that higher performance defaults become possible; if the precise implementation is important to your application, please instantiate it directly, instead. For optimal performance you should consider using
MMapDirectory
on 64 bit JVMs.See above
- Throws:
java.io.IOException
-
open
public static FSDirectory open(java.nio.file.Path path, LockFactory lockFactory) throws java.io.IOException
Just likeopen(Path)
, but allows you to also specify a customLockFactory
.- Throws:
java.io.IOException
-
listAll
public static java.lang.String[] listAll(java.nio.file.Path dir) throws java.io.IOException
Lists all files (including subdirectories) in the directory.- Throws:
java.io.IOException
- if there was an I/O error during listing
-
listAll
private static java.lang.String[] listAll(java.nio.file.Path dir, java.util.Set<java.lang.String> skipNames) throws java.io.IOException
- Throws:
java.io.IOException
-
listAll
public java.lang.String[] listAll() throws java.io.IOException
Description copied from class:Directory
Returns names of all files stored in this directory. The output must be in sorted (UTF-16, java'sString.compareTo(java.lang.String)
) order.
-
fileLength
public long fileLength(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Returns the byte length of a file in the directory.This method must throw either
NoSuchFileException
orFileNotFoundException
ifname
points to a non-existing file.- Specified by:
fileLength
in classDirectory
- Parameters:
name
- the name of an existing file.- Throws:
java.io.IOException
- in case of I/O error
-
createOutput
public IndexOutput createOutput(java.lang.String name, IOContext context) throws java.io.IOException
Description copied from class:Directory
Creates a new, empty file in the directory and returns anIndexOutput
instance for appending data to this file.This method must throw
FileAlreadyExistsException
if the file already exists.- Specified by:
createOutput
in classDirectory
- Parameters:
name
- the name of the file to create.- Throws:
java.io.IOException
- in case of I/O error
-
createTempOutput
public IndexOutput createTempOutput(java.lang.String prefix, java.lang.String suffix, IOContext context) throws java.io.IOException
Description copied from class:Directory
Creates a new, empty, temporary file in the directory and returns anIndexOutput
instance for appending data to this file.The temporary file name (accessible via
IndexOutput.getName()
) will start withprefix
, end withsuffix
and have a reserved file extension.tmp
.- Specified by:
createTempOutput
in classDirectory
- Throws:
java.io.IOException
-
ensureCanRead
protected void ensureCanRead(java.lang.String name) throws java.io.IOException
- Throws:
java.io.IOException
-
sync
public void sync(java.util.Collection<java.lang.String> names) throws java.io.IOException
Description copied from class:Directory
Ensures that any writes to these files are moved to stable storage (made durable).Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index.
- Specified by:
sync
in classDirectory
- Throws:
java.io.IOException
- See Also:
Directory.syncMetaData()
-
rename
public void rename(java.lang.String source, java.lang.String dest) throws java.io.IOException
Description copied from class:Directory
Renamessource
file todest
file wheredest
must not already exist in the directory.It is permitted for this operation to not be truly atomic, for example both
source
anddest
can be visible temporarily inDirectory.listAll()
. However, the implementation of this method must ensure the content ofdest
appears as the entiresource
atomically. So oncedest
is visible for readers, the entire content of previoussource
is visible.This method is used by IndexWriter to publish commits.
-
syncMetaData
public void syncMetaData() throws java.io.IOException
Description copied from class:Directory
Ensures that directory metadata, such as recent file renames, are moved to stable storage.- Specified by:
syncMetaData
in classDirectory
- Throws:
java.io.IOException
- See Also:
Directory.sync(Collection)
-
close
public void close() throws java.io.IOException
Description copied from class:Directory
Closes the directory.
-
getDirectory
public java.nio.file.Path getDirectory()
- Returns:
- the underlying filesystem directory
-
toString
public java.lang.String toString()
- Overrides:
toString
in classBaseDirectory
-
fsync
protected void fsync(java.lang.String name) throws java.io.IOException
- Throws:
java.io.IOException
-
deleteFile
public void deleteFile(java.lang.String name) throws java.io.IOException
Description copied from class:Directory
Removes an existing file in the directory.This method must throw either
NoSuchFileException
orFileNotFoundException
ifname
points to a non-existing file.- Specified by:
deleteFile
in classDirectory
- Parameters:
name
- the name of an existing file.- Throws:
java.io.IOException
- in case of I/O error
-
deletePendingFiles
public void deletePendingFiles() throws java.io.IOException
Try to delete any pending files that we had previously tried to delete but failed because we are on Windows and the files were still held open.- Throws:
java.io.IOException
-
maybeDeletePendingFiles
private void maybeDeletePendingFiles() throws java.io.IOException
- Throws:
java.io.IOException
-
privateDeleteFile
private void privateDeleteFile(java.lang.String name, boolean isPendingDelete) throws java.io.IOException
- Throws:
java.io.IOException
-
getPendingDeletions
public java.util.Set<java.lang.String> getPendingDeletions() throws java.io.IOException
Description copied from class:Directory
Returns a set of files currently pending deletion in this directory.- Specified by:
getPendingDeletions
in classDirectory
- Throws:
java.io.IOException
-
-