- java.lang.Object
-
- org.apache.lucene.store.DataOutput
-
- org.apache.lucene.store.ByteBuffersDataOutput
-
- All Implemented Interfaces:
Accountable
public final class ByteBuffersDataOutput extends DataOutput implements Accountable
ADataOutput
storing data in a list ofByteBuffer
s.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ByteBuffersDataOutput.ByteBufferRecycler
An implementation of aByteBuffer
allocation and recycling policy.
-
Field Summary
Fields Modifier and Type Field Description static java.util.function.IntFunction<java.nio.ByteBuffer>
ALLOCATE_BB_ON_HEAP
private java.util.function.IntFunction<java.nio.ByteBuffer>
blockAllocate
ByteBuffer
supplier.private int
blockBits
Current block size:2^bits
.private java.util.function.Consumer<java.nio.ByteBuffer>
blockReuse
ByteBuffer
recycler onreset()
.private java.util.ArrayDeque<java.nio.ByteBuffer>
blocks
Blocks storing data.private java.nio.ByteBuffer
currentBlock
The current-or-next write block.static int
DEFAULT_MAX_BITS_PER_BLOCK
DefaultmaxBitsPerBlock
static int
DEFAULT_MIN_BITS_PER_BLOCK
DefaultminBitsPerBlock
private static java.nio.ByteBuffer
EMPTY
private static byte[]
EMPTY_BYTE_ARRAY
static int
LIMIT_MAX_BITS_PER_BLOCK
LargestmaxBitsPerBlock
allowedstatic int
LIMIT_MIN_BITS_PER_BLOCK
SmallestminBitsPerBlock
allowed(package private) static int
MAX_BLOCKS_BEFORE_BLOCK_EXPANSION
Maximum number of blocks at the currentblockBits
block size before we increase the block size (and thus decrease the number of blocks).private static int
MAX_CHARS_PER_WINDOW
private int
maxBitsPerBlock
Maximum block size:2^bits
.static java.util.function.Consumer<java.nio.ByteBuffer>
NO_REUSE
A singleton instance of "no-reuse" buffer strategy.private long
ramBytesUsed
Cumulative RAM usage across all blocks.-
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
-
Constructor Summary
Constructors Constructor Description ByteBuffersDataOutput()
Creates a new output with all defaults.ByteBuffersDataOutput(int minBitsPerBlock, int maxBitsPerBlock, java.util.function.IntFunction<java.nio.ByteBuffer> blockAllocate, java.util.function.Consumer<java.nio.ByteBuffer> blockReuse)
Expert: Creates a new output with custom parameters.ByteBuffersDataOutput(long expectedSize)
Create a new output, suitable for writing a file of aroundexpectedSize
bytes.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
appendBlock()
private int
blockSize()
private static int
computeBlockSizeBitsFor(long bytes)
void
copyBytes(DataInput input, long numBytes)
Copy numBytes bytes from input to ourself.void
copyTo(DataOutput output)
Copy the current content of this object into anotherDataOutput
.static ByteBuffersDataOutput
newResettableInstance()
long
ramBytesUsed()
Return the memory usage of this object in bytes.void
reset()
This method resets this object to a clean (zero-size) state and publishes any currently allocated buffers for reuse to the reuse strategy provided in the constructor.private void
rewriteToBlockSize(int targetBlockBits)
long
size()
byte[]
toArrayCopy()
Return a contiguous array with the current content written to the output.java.util.ArrayList<java.nio.ByteBuffer>
toBufferList()
Return a list of read-only view ofByteBuffer
blocks over the current content written to the output.ByteBuffersDataInput
toDataInput()
Return aByteBuffersDataInput
for the set of current buffers (toBufferList()
).java.lang.String
toString()
java.util.ArrayList<java.nio.ByteBuffer>
toWriteableBufferList()
Returns a list of writeable blocks over the (source) content buffers.void
writeByte(byte b)
Writes a single byte.void
writeBytes(byte[] b)
void
writeBytes(byte[] b, int length)
Writes an array of bytes.void
writeBytes(byte[] src, int offset, int length)
Writes an array of bytes.void
writeBytes(java.nio.ByteBuffer buffer)
void
writeInt(int v)
Writes an int as four bytes (LE byte order).void
writeLong(long v)
Writes a long as eight bytes (LE byte order).private void
writeLongString(java.lang.String s)
Writes a long string in chunksvoid
writeMapOfStrings(java.util.Map<java.lang.String,java.lang.String> map)
Writes a String map.void
writeSetOfStrings(java.util.Set<java.lang.String> set)
Writes a String set.void
writeShort(short v)
Writes a short as two bytes (LE byte order).void
writeString(java.lang.String v)
Writes a string.-
Methods inherited from class org.apache.lucene.store.DataOutput
writeGroupVInts, writeVInt, writeVLong, writeZInt, writeZLong
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
EMPTY
private static final java.nio.ByteBuffer EMPTY
-
EMPTY_BYTE_ARRAY
private static final byte[] EMPTY_BYTE_ARRAY
-
ALLOCATE_BB_ON_HEAP
public static final java.util.function.IntFunction<java.nio.ByteBuffer> ALLOCATE_BB_ON_HEAP
-
NO_REUSE
public static final java.util.function.Consumer<java.nio.ByteBuffer> NO_REUSE
A singleton instance of "no-reuse" buffer strategy.
-
DEFAULT_MIN_BITS_PER_BLOCK
public static final int DEFAULT_MIN_BITS_PER_BLOCK
DefaultminBitsPerBlock
- See Also:
- Constant Field Values
-
DEFAULT_MAX_BITS_PER_BLOCK
public static final int DEFAULT_MAX_BITS_PER_BLOCK
DefaultmaxBitsPerBlock
- See Also:
- Constant Field Values
-
LIMIT_MIN_BITS_PER_BLOCK
public static final int LIMIT_MIN_BITS_PER_BLOCK
SmallestminBitsPerBlock
allowed- See Also:
- Constant Field Values
-
LIMIT_MAX_BITS_PER_BLOCK
public static final int LIMIT_MAX_BITS_PER_BLOCK
LargestmaxBitsPerBlock
allowed- See Also:
- Constant Field Values
-
MAX_BLOCKS_BEFORE_BLOCK_EXPANSION
static final int MAX_BLOCKS_BEFORE_BLOCK_EXPANSION
Maximum number of blocks at the currentblockBits
block size before we increase the block size (and thus decrease the number of blocks).- See Also:
- Constant Field Values
-
maxBitsPerBlock
private final int maxBitsPerBlock
Maximum block size:2^bits
.
-
blockAllocate
private final java.util.function.IntFunction<java.nio.ByteBuffer> blockAllocate
ByteBuffer
supplier.
-
blockReuse
private final java.util.function.Consumer<java.nio.ByteBuffer> blockReuse
ByteBuffer
recycler onreset()
.
-
blockBits
private int blockBits
Current block size:2^bits
.
-
blocks
private final java.util.ArrayDeque<java.nio.ByteBuffer> blocks
Blocks storing data.
-
ramBytesUsed
private long ramBytesUsed
Cumulative RAM usage across all blocks.
-
currentBlock
private java.nio.ByteBuffer currentBlock
The current-or-next write block.
-
MAX_CHARS_PER_WINDOW
private static final int MAX_CHARS_PER_WINDOW
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ByteBuffersDataOutput
public ByteBuffersDataOutput(long expectedSize)
Create a new output, suitable for writing a file of aroundexpectedSize
bytes.Memory allocation will be optimized based on the
expectedSize
hint, so that there is less overhead for larger files.- Parameters:
expectedSize
- estimated size of the output file
-
ByteBuffersDataOutput
public ByteBuffersDataOutput()
Creates a new output with all defaults.
-
ByteBuffersDataOutput
public ByteBuffersDataOutput(int minBitsPerBlock, int maxBitsPerBlock, java.util.function.IntFunction<java.nio.ByteBuffer> blockAllocate, java.util.function.Consumer<java.nio.ByteBuffer> blockReuse)
Expert: Creates a new output with custom parameters.- Parameters:
minBitsPerBlock
- minimum bits per blockmaxBitsPerBlock
- maximum bits per blockblockAllocate
- block allocatorblockReuse
- block recycler
-
-
Method Detail
-
writeByte
public void writeByte(byte b)
Description copied from class:DataOutput
Writes a single byte.The most primitive data type is an eight-bit byte. Files are accessed as sequences of bytes. All other data types are defined as sequences of bytes, so file formats are byte-order independent.
- Specified by:
writeByte
in classDataOutput
- See Also:
DataInput.readByte()
-
writeBytes
public void writeBytes(byte[] src, int offset, int length)
Description copied from class:DataOutput
Writes an array of bytes.- Specified by:
writeBytes
in classDataOutput
- Parameters:
src
- the bytes to writeoffset
- the offset in the byte arraylength
- the number of bytes to write- See Also:
DataInput.readBytes(byte[],int,int)
-
writeBytes
public void writeBytes(byte[] b, int length)
Description copied from class:DataOutput
Writes an array of bytes.- Overrides:
writeBytes
in classDataOutput
- Parameters:
b
- the bytes to writelength
- the number of bytes to write- See Also:
DataInput.readBytes(byte[],int,int)
-
writeBytes
public void writeBytes(byte[] b)
-
writeBytes
public void writeBytes(java.nio.ByteBuffer buffer)
-
toBufferList
public java.util.ArrayList<java.nio.ByteBuffer> toBufferList()
Return a list of read-only view ofByteBuffer
blocks over the current content written to the output.
-
toWriteableBufferList
public java.util.ArrayList<java.nio.ByteBuffer> toWriteableBufferList()
Returns a list of writeable blocks over the (source) content buffers.This method returns the raw content of source buffers that may change over the lifetime of this object (blocks can be recycled or discarded, for example). Most applications should favor calling
toBufferList()
which returns a read-only view over the content of the source buffers.The difference between
toBufferList()
andtoWriteableBufferList()
is that read-only view of source buffers will always returnfalse
fromByteBuffer.hasArray()
(which sometimes may be required to avoid double copying).
-
toDataInput
public ByteBuffersDataInput toDataInput()
Return aByteBuffersDataInput
for the set of current buffers (toBufferList()
).
-
toArrayCopy
public byte[] toArrayCopy()
Return a contiguous array with the current content written to the output. The returned array is always a copy (can be mutated).If the
size()
of the underlying buffers exceeds maximum size of Java array, anRuntimeException
will be thrown.
-
copyBytes
public void copyBytes(DataInput input, long numBytes) throws java.io.IOException
Description copied from class:DataOutput
Copy numBytes bytes from input to ourself.- Overrides:
copyBytes
in classDataOutput
- Throws:
java.io.IOException
-
copyTo
public void copyTo(DataOutput output) throws java.io.IOException
Copy the current content of this object into anotherDataOutput
.- Throws:
java.io.IOException
-
size
public long size()
- Returns:
- The number of bytes written to this output so far.
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
writeShort
public void writeShort(short v)
Description copied from class:DataOutput
Writes a short as two bytes (LE byte order).- Overrides:
writeShort
in classDataOutput
- See Also:
DataInput.readShort()
,BitUtil.VH_LE_SHORT
-
writeInt
public void writeInt(int v)
Description copied from class:DataOutput
Writes an int as four bytes (LE byte order).- Overrides:
writeInt
in classDataOutput
- See Also:
DataInput.readInt()
,BitUtil.VH_LE_INT
-
writeLong
public void writeLong(long v)
Description copied from class:DataOutput
Writes a long as eight bytes (LE byte order).- Overrides:
writeLong
in classDataOutput
- See Also:
DataInput.readLong()
,BitUtil.VH_LE_LONG
-
writeString
public void writeString(java.lang.String v)
Description copied from class:DataOutput
Writes a string.Writes strings as UTF-8 encoded bytes. First the length, in bytes, is written as a
VInt
, followed by the bytes.- Overrides:
writeString
in classDataOutput
- See Also:
DataInput.readString()
-
writeMapOfStrings
public void writeMapOfStrings(java.util.Map<java.lang.String,java.lang.String> map)
Description copied from class:DataOutput
Writes a String map.First the size is written as an
vInt
, followed by each key-value pair written as two consecutiveString
s.- Overrides:
writeMapOfStrings
in classDataOutput
- Parameters:
map
- Input map.
-
writeSetOfStrings
public void writeSetOfStrings(java.util.Set<java.lang.String> set)
Description copied from class:DataOutput
Writes a String set.First the size is written as an
vInt
, followed by each value written as aString
.- Overrides:
writeSetOfStrings
in classDataOutput
- Parameters:
set
- Input set.
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
reset
public void reset()
This method resets this object to a clean (zero-size) state and publishes any currently allocated buffers for reuse to the reuse strategy provided in the constructor.Sharing byte buffers for reads and writes is dangerous and will very likely lead to hard-to-debug issues, use with great care.
-
newResettableInstance
public static ByteBuffersDataOutput newResettableInstance()
- Returns:
- Returns a new
ByteBuffersDataOutput
with thereset()
capability.
-
blockSize
private int blockSize()
-
appendBlock
private void appendBlock()
-
rewriteToBlockSize
private void rewriteToBlockSize(int targetBlockBits)
-
computeBlockSizeBitsFor
private static int computeBlockSizeBitsFor(long bytes)
-
writeLongString
private void writeLongString(java.lang.String s) throws java.io.IOException
Writes a long string in chunks- Throws:
java.io.IOException
-
-