org.jlab.coda.hipo

Class WriterMT

java.lang.Object

org.jlab.coda.hipo.WriterMT

All Implemented Interfaces:

java.lang.AutoCloseable

public class WriterMT
extends java.lang.Object
implements java.lang.AutoCloseable

This class is for writing Evio/HIPO files only (not buffers) while multithreading the compression of data.

At the center of how this class works is an ultra-fast ring buffer containing a store of empty records. As the user calls one of the addEvent(byte[], int, int) methods, it gradually fills one of those empty records with data. When the record is full, it's put back into the ring to wait for one of the compression thread to grab it and compress it. After compression, it's again placed back into the ring and waits for a final thread to write it to file. After being written, the record is freed up for reuse. This entire ring functionality is encapsulated in 2 classes, RecordSupply and RecordRingItem.

Since:

6.0 8/10/17

Version:

6.0

Author:

gavalian, timmer

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) class	WriterMT.RecordCompressor Class used to take data-filled record from supply, compress it, and release it back to the supply.
(package private) class	WriterMT.RecordWriter Class used to take data-filled record from supply, write it, and release it back to the supply.

Field Summary

Fields

Modifier and Type	Field and Description
(package private) java.lang.String	dictionary String containing evio-format XML dictionary to store in file header's user header.
(package private) java.nio.ByteBuffer	dictionaryFirstEventBuffer If dictionary and or firstEvent exist, this buffer contains them both as a record.
(package private) int	userHeaderLength Size in bytes of userHeader array.

Constructor Summary

Constructors

Constructor and Description
WriterMT() Default constructor is single-threaded, no compression.
WriterMT(java.nio.ByteOrder order, int maxEventCount, int maxBufferSize, CompressionType compressionType, int compressionThreads, int ringSize) Constructor with byte order.
WriterMT(HeaderType hType, java.nio.ByteOrder order, int maxEventCount, int maxBufferSize, java.lang.String dictionary, byte[] firstEvent, int firstEventLen, CompressionType compType, int compressionThreads, boolean addTrailerIndex, int ringSize) Constructor with byte order.
WriterMT(java.lang.String filename) Constructor with filename.
WriterMT(java.lang.String filename, java.nio.ByteOrder order, int maxEventCount, int maxBufferSize, CompressionType compressionType, int compressionThreads, int ringSize) Constructor with filename and byte order.

Method Summary

Modifier and Type	Method and Description
void	addEvent(byte[] buffer) Add a byte array to the current internal record.
void	addEvent(byte[] buffer, int offset, int length) Add a byte array to the current internal record.
void	addEvent(java.nio.ByteBuffer buffer) Add a ByteBuffer to the internal record.
void	addEvent(EvioBank bank) Add an EvioBank to the internal record.
void	addEvent(EvioNode node) Add an EvioNode to the internal record.
boolean	addTrailer() Does this writer add a trailer to the end of the file?
void	addTrailer(boolean addingTrailer) Set whether this writer adds a trailer to the end of the file.
boolean	addTrailerWithIndex() Does this writer add a trailer with a record index to the end of the file?
void	addTrailerWithIndex(boolean addTrailingIndex) Set whether this writer adds a trailer with a record index to the end of the file.
void	close() Close opened file.
(package private) java.nio.ByteBuffer	createDictionaryRecord() Create a buffer representation of a record containing the dictionary and/or the first event.
java.nio.ByteBuffer	createHeader(byte[] userHdr) Create and return a buffer containing a general file header followed by the user header given in the argument.
(package private) java.nio.ByteBuffer	createHeader(java.nio.ByteBuffer userHdr) Return a buffer with a general file header followed by the given user header (userHdr).
java.nio.ByteOrder	getByteOrder() Get the file's byte order.
(package private) CompressionType	getCompressionType() Get the compression type for the file being written.
FileHeader	getFileHeader() Get the file header.
void	open(java.lang.String filename) Open a new file and write file header with no user header.
void	open(java.lang.String filename, byte[] userHdr) Open a file and write file header with given user's header.
void	reset() Get this object ready for re-use.
WriterMT	setCompressionType(CompressionType compression) Convenience method that sets compression type for the file.
void	writeRecord(RecordOutputStream rec) Appends the record to the file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

userHeaderLength

int userHeaderLength

Size in bytes of userHeader array.

dictionary

java.lang.String dictionary

String containing evio-format XML dictionary to store in file header's user header.

dictionaryFirstEventBuffer

java.nio.ByteBuffer dictionaryFirstEventBuffer

If dictionary and or firstEvent exist, this buffer contains them both as a record.

Constructor Detail

WriterMT

public WriterMT()

Default constructor is single-threaded, no compression. Little endian. No file is opened. Any file will have little endian byte order. 1M max event count and 8M max buffer size.

WriterMT

public WriterMT(java.nio.ByteOrder order,
                int maxEventCount,
                int maxBufferSize,
                CompressionType compressionType,
                int compressionThreads,
                int ringSize)
         throws java.lang.IllegalArgumentException

Constructor with byte order. No file is opened. Any dictionary will be placed in the user header which will create a conflict if user tries to call open(String, byte[]) with another user header array.

Parameters:

order - byte order of written file

maxEventCount - max number of events a record can hold. Value <= O means use default (1M).

maxBufferSize - max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.

compressionType - type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip)

compressionThreads - number of threads doing compression simultaneously

ringSize - number of records in supply ring, must be multiple of 2 and >= compressionThreads.

Throws:

java.lang.IllegalArgumentException - if invalid compression type.

WriterMT

public WriterMT(HeaderType hType,
                java.nio.ByteOrder order,
                int maxEventCount,
                int maxBufferSize,
                java.lang.String dictionary,
                byte[] firstEvent,
                int firstEventLen,
                CompressionType compType,
                int compressionThreads,
                boolean addTrailerIndex,
                int ringSize)
         throws java.lang.IllegalArgumentException

Constructor with byte order. No file is opened. Any dictionary will be placed in the user header which will create a conflict if user tries to call open(String, byte[]) with another user header array.

Parameters:

hType - the type of the file. If set to HeaderType.HIPO_FILE, the header will be written with the first 4 bytes set to HIPO.

order - byte order of written file

maxEventCount - max number of events a record can hold. Value <= O means use default (1M).

maxBufferSize - max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.

dictionary - string holding an evio format dictionary to be placed in userHeader.

firstEvent - byte array containing an evio event to be included in userHeader. It must be in the same byte order as the order argument.

firstEventLen - number of valid bytes in firstEvent.

compType - type of data compression to do (one, lz4 fast, lz4 best, gzip)

compressionThreads - number of threads doing compression simultaneously

addTrailerIndex - if true, we add a record index to the trailer.

ringSize - number of records in supply ring, must be multiple of 2 and >= compressionThreads.

Throws:

java.lang.IllegalArgumentException - if invalid compression type.

WriterMT

public WriterMT(java.lang.String filename)
         throws HipoException

Constructor with filename. The output file will be created with no user header. File byte order is little endian.

Parameters:

filename - output file name

Throws:

HipoException - if filename arg is null or bad.

WriterMT

public WriterMT(java.lang.String filename,
                java.nio.ByteOrder order,
                int maxEventCount,
                int maxBufferSize,
                CompressionType compressionType,
                int compressionThreads,
                int ringSize)
         throws HipoException

Constructor with filename and byte order. The output file will be created with no user header.

Parameters:

filename - output file name

order - byte order of written file or null for default (little endian)

maxEventCount - max number of events a record can hold. Value of O means use default (1M).

maxBufferSize - max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.

compressionType - type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip)

compressionThreads - number of threads doing compression simultaneously

ringSize - number of records in supply ring, must be multiple of 2 and >= compressionThreads.

Throws:

HipoException - if filename arg is null or bad.

Method Detail

createDictionaryRecord

java.nio.ByteBuffer createDictionaryRecord()

Create a buffer representation of a record containing the dictionary and/or the first event. No compression.

Returns:

buffer representation of record containing dictionary and/or first event, of zero size if first event and dictionary don't exist.

getByteOrder

public java.nio.ByteOrder getByteOrder()

Get the file's byte order.

Returns:

file's byte order.

getFileHeader

public FileHeader getFileHeader()

Get the file header.

Returns:

file header.

getCompressionType

CompressionType getCompressionType()

Get the compression type for the file being written.

Returns:

compression type for the file being written.

addTrailer

public boolean addTrailer()

Does this writer add a trailer to the end of the file?

Returns:

true if this writer adds a trailer to the end of the file, else false.

addTrailer

public void addTrailer(boolean addingTrailer)

Set whether this writer adds a trailer to the end of the file.

Parameters:

addingTrailer - if true, at the end of file, add an ending header (trailer) with no index of records and no following data. Update the file header to contain a file offset to the trailer.

addTrailerWithIndex

public boolean addTrailerWithIndex()

Does this writer add a trailer with a record index to the end of the file?

Returns:

true if this writer adds a trailer with a record index to the end of the file, else false.

addTrailerWithIndex

public void addTrailerWithIndex(boolean addTrailingIndex)

Set whether this writer adds a trailer with a record index to the end of the file.

Parameters:

addTrailingIndex - if true, at the end of file, add an ending header (trailer) with an index of all records but with no following data. Update the file header to contain a file offset to the trailer.

open

public final void open(java.lang.String filename)
                throws HipoException

Open a new file and write file header with no user header.

Parameters:

filename - output file name

Throws:

HipoException - if filename arg is null or bad, if this method already called without being followed by reset.

open

public final void open(java.lang.String filename,
                       byte[] userHdr)
                throws HipoException

Open a file and write file header with given user's header. User header is automatically padded when written.

Parameters:

filename - disk file name.

userHdr - byte array representing the optional user's header.

Throws:

HipoException - if filename arg is null or bad, if this method already called without being followed by reset.

setCompressionType

public final WriterMT setCompressionType(CompressionType compression)

Convenience method that sets compression type for the file. The compression type is set for internal record. The record data will be compressed according to the given type.

Parameters:

compression - compression type

Returns:

this object

createHeader

public java.nio.ByteBuffer createHeader(byte[] userHdr)

Create and return a buffer containing a general file header followed by the user header given in the argument. If user header is not padded to 4-byte boundary, it's done here.

Parameters:

userHdr - byte array containing a user-defined header, may be null.

Returns:

ByteBuffer containing a file header followed by the user-defined header

createHeader

java.nio.ByteBuffer createHeader(java.nio.ByteBuffer userHdr)

Return a buffer with a general file header followed by the given user header (userHdr). The buffer is cleared and set to desired byte order prior to writing. If user header is not padded to 4-byte boundary, it's done here.

Parameters:

userHdr - buffer containing a user-defined header which must be READY-TO-READ!

Returns:

buffer containing a file header followed by the user-defined header.

writeRecord

public void writeRecord(RecordOutputStream rec)
                 throws java.lang.IllegalArgumentException,
                        HipoException

Appends the record to the file. Using this method in conjunction with addEvent(byte[], int, int) is not thread-safe.

Parameters:

rec - record object

Throws:

java.lang.IllegalArgumentException - if arg's byte order is opposite to output endian.

HipoException - if we cannot replace internal buffer of the current record if it needs to be expanded since the buffer was provided by the user.

addEvent

public void addEvent(byte[] buffer,
                     int offset,
                     int length)
              throws HipoException

Add a byte array to the current internal record. If the length of the buffer exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). And another record will be obtained from the supply to receive the buffer. Using this method in conjunction with writeRecord(RecordOutputStream) is not thread-safe.

Parameters:

buffer - array to add to the file.

offset - offset into array from which to start writing data.

length - number of bytes to write from array.

Throws:

HipoException - if buffer arg is null.

addEvent

public void addEvent(byte[] buffer)
              throws HipoException

Add a byte array to the current internal record. If the length of the buffer exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). And another record will be obtained from the supply to receive the buffer. Using this method in conjunction with writeRecord(RecordOutputStream) is not thread-safe.

Parameters:

buffer - array to add to the file.

Throws:

HipoException - if buffer arg is null.

addEvent

public void addEvent(java.nio.ByteBuffer buffer)
              throws HipoException

Add a ByteBuffer to the internal record. If the length of the buffer exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). And another record will be obtained from the supply to receive the buffer. Using this method in conjunction with writeRecord() is not thread-safe. The byte order of event's data must match the byte order given in constructor!

Parameters:

buffer - array to add to the file.

Throws:

HipoException - if buffer arg is null or byte order is wrong.

addEvent

public void addEvent(EvioBank bank)
              throws HipoException

Add an EvioBank to the internal record. If the length of the bank exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). Internal record will be reset to receive new buffers. Using this method in conjunction with writeRecord() is not thread-safe.

Parameters:

bank - event to add to the file.

Throws:

HipoException - if arg is null.

addEvent

public void addEvent(EvioNode node)
              throws HipoException

Add an EvioNode to the internal record. If the length of the data exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). Internal record will be reset to receive new buffers. Using this method in conjunction with writeRecord() is not thread-safe. The byte order of node's data must match the byte order given in constructor!

Parameters:

node - node to add to the file.

Throws:

HipoException - if node is null or does not correspond to a bank.

reset

public void reset()

Get this object ready for re-use. Follow calling this with call to open(String).

close

public void close()

Close opened file. If the output record contains events, they will be flushed to file. Trailer and its optional index written if requested.

The addEvent or addRecord methods must no longer be called.

Specified by:

close in interface java.lang.AutoCloseable