This class is for writing Evio/HIPO files only (not buffers). More...

#include <WriterMT.h>

Public Member Functions
	WriterMT ()
	Default constructor. More...

	WriterMT (const ByteOrder &order, uint32_t maxEventCount, uint32_t maxBufferSize, Compressor::CompressionType compType, uint32_t compressionThreads)
	Constructor with byte order. More...

	WriterMT (const HeaderType &hType, const ByteOrder &order=ByteOrder::ENDIAN_LITTLE, uint32_t maxEventCount=0, uint32_t maxBufferSize=0, const std::string &dictionary="", uint8_t *firstEvent=nullptr, uint32_t firstEventLen=0, Compressor::CompressionType compressionType=Compressor::UNCOMPRESSED, uint32_t compressionThreads=1, bool addTrailerIndex=false, uint32_t ringSize=16)
	Constructor with byte order. More...

	WriterMT (const std::string &filename)
	Constructor with filename. More...

	WriterMT (const std::string &filename, const ByteOrder &order, uint32_t maxEventCount, uint32_t maxBufferSize, Compressor::CompressionType compressionType, uint32_t compressionThreads)
	Constructor with filename & byte order. More...

	~WriterMT ()=default

const ByteOrder &	getByteOrder () const
	Get the file's byte order. More...

FileHeader &	getFileHeader ()
	Get the file header. More...

Compressor::CompressionType	getCompressionType ()
	Get the internal record's header. More...

bool	addTrailer () const
	Does this writer add a trailer to the end of the file/buffer? More...

void	addTrailer (bool add)
	Set whether this writer adds a trailer to the end of the file/buffer. More...

bool	addTrailerWithIndex ()
	Does this writer add a trailer with a record index to the end of the file? Or, if writing to a buffer, is a trailer added with no index? More...

void	addTrailerWithIndex (bool addTrailingIndex)
	Set whether this writer adds a trailer with a record index to the end of the file. More...

void	open (const std::string &filename)
	Open a new file and write file header with no user header. More...

void	open (const std::string &filename, uint8_t *userHdr, uint32_t userLen)
	Open a file and write file header with given user header. More...

std::shared_ptr< ByteBuffer >	createHeader (uint8_t *userHdr, uint32_t userLen)
	Create and return a buffer containing a general file header followed by the user header given in the argument. More...

std::shared_ptr< ByteBuffer >	createHeader (ByteBuffer &userHdr)
	Return a buffer with a general file header followed by the given user header (userHdr). More...

void	writeRecord (RecordOutput &record)
	Appends the record to the file. More...

void	addEvent (uint8_t *buffer, uint32_t offset, uint32_t length)
	Add a byte array to the current internal record. More...

void	addEvent (ByteBuffer &buffer)
	Add a ByteBuffer to the internal record. More...

void	addEvent (EvioNode &node)
	Add an EvioNode to the internal record. More...

void	reset ()
	Get this object ready for re-use. More...

void	close ()
	Close opened file. More...

Detailed Description

This class is for writing Evio/HIPO files only (not buffers).

It's able to multithread the compression of data.

At the center of how this class works is an ultra-fast ring buffer containing a store of empty records. As the user calls one of the addEvent methods, it gradually fills one of those empty records with data. When the record is full, it's put back into the ring to wait for one of the compression thread to grab it and compress it. After compression, it's again placed back into the ring and waits for a final thread to write it to file. After being written, the record is freed up for reuse. This entire ring functionality is encapsulated in 2 classes, RecordSupply and RecordRingItem.

Version: 6.0

Since: 6.0 5/13/19

Author: timmer

Constructor & Destructor Documentation

evio::WriterMT::WriterMT ( )

Default constructor.

Compression is single-threaded, no compression. Little endian. No file is opened. Any file will have little endian byte order. 1M max event count and 8M max buffer size.

evio::WriterMT::WriterMT	(	const ByteOrder &	order,
		uint32_t	maxEventCount,
		uint32_t	maxBufferSize,
		Compressor::CompressionType	compType,
		uint32_t	compressionThreads
	)

Constructor with byte order.

No file is opened. File header type is evio file (HeaderType#EVIO_FILE). Any dictionary will be placed in the user header which will create a conflict if user tries to call open(const std::string &, uint8_t *, uint32_t) with another user header array.

Parameters

order	byte order of written file.
maxEventCount	max number of events a record can hold. Value of O means use default (1M).
maxBufferSize	max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.
compType	type of data compression to do (one, lz4 fast, lz4 best, gzip).
compressionThreads	number of threads doing compression simultaneously.

evio::WriterMT::WriterMT	(	const HeaderType &	hType,
		const ByteOrder &	order = `ByteOrder::ENDIAN_LITTLE`,
		uint32_t	maxEventCount = `0`,
		uint32_t	maxBufferSize = `0`,
		const std::string &	dictionary = `""`,
		uint8_t *	firstEvent = `nullptr`,
		uint32_t	firstEventLen = `0`,
		Compressor::CompressionType	compType = `Compressor::UNCOMPRESSED`,
		uint32_t	compressionThreads = `1`,
		bool	addTrailerIndex = `false`,
		uint32_t	ringSize = `16`
	)

explicit

Constructor with byte order.

The given file is opened so any subsequent call to open will fail. This method places the dictionary and first event into the file header's user header.

Parameters

hType	the type of the file. If set to HeaderType#HIPO_FILE, the header will be written with the first 4 bytes set to HIPO.
order	byte order of written file.
maxEventCount	max number of events a record can hold. Value of O means use default (1M).
maxBufferSize	max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.
dictionary	string holding an evio format dictionary to be placed in userHeader.
firstEvent	byte array containing an evio event to be included in userHeader. It must be in the same byte order as the order argument.
firstEventLen	number of valid bytes in firstEvent.
compType	type of data compression to do (one, lz4 fast, lz4 best, gzip)
compressionThreads	number of threads doing compression simultaneously
addTrailerIndex	if true, we add a record index to the trailer.
ringSize	number of records in supply ring, must be multiple of 2 and >= compressionThreads.

References evio::RecordHeader::HEADER_SIZE_BYTES, and evio::HeaderType::HIPO_FILE.

evio::WriterMT::WriterMT ( const std::string & filename )

explicit

Constructor with filename.

The output file will be created with no user header. File byte order is little endian.

Parameters

filename output file name

References open().

evio::WriterMT::WriterMT	(	const std::string &	filename,
		const ByteOrder &	order,
		uint32_t	maxEventCount,
		uint32_t	maxBufferSize,
		Compressor::CompressionType	compType,
		uint32_t	compressionThreads
	)

Constructor with filename & byte order.

The output file will be created with no user header. Ring size is 16.

Parameters

filename	output file name
order	byte order of written file or null for default (little endian)
maxEventCount	max number of events a record can hold. Value of O means use default (1M).
maxBufferSize	max number of uncompressed data bytes a record can hold. Value of < 8MB results in default of 8MB.
compType	type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip)
compressionThreads	number of threads doing compression simultaneously

References open().

evio::WriterMT::~WriterMT ( )

default

Member Function Documentation

void evio::WriterMT::addEvent	(	uint8_t *	buffer,
		uint32_t	offset,
		uint32_t	length
	)

Add a byte array to the current internal record.

If the length of the buffer exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). And another record will be obtained from the supply to receive the buffer. Using this method in conjunction with writeRecord(RecordOutput &) is not thread-safe.

Parameters

buffer	array to add to the file.
offset	offset into array from which to start writing data.
length	number of bytes to write from array.

void evio::WriterMT::addEvent ( ByteBuffer & buffer )

Add a ByteBuffer to the internal record.

If the length of the buffer exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). Internal record will be reset to receive new buffers. Using this method in conjunction with writeRecord() is not thread-safe. The byte order of event's data must match the byte order given in constructor!

Parameters

buffer array to add to the file.

Exceptions

EvioException if cannot write to file or buffer arg's byte order is wrong.

References evio::ByteBuffer::order().

void evio::WriterMT::addEvent ( EvioNode & node )

Add an EvioNode to the internal record.

If the length of the data exceeds the maximum size of the record, the record will be written to the file (compressed if the flag is set). Internal record will be reset to receive new buffers. Using this method in conjunction with writeRecord() is not thread-safe. The byte order of node's data must match the byte order given in constructor!

Parameters

node	node to add to the file.

Exceptions

EvioException if cannot write to file or node arg's byte order is wrong.

References evio::EvioNode::getBuffer().

bool evio::WriterMT::addTrailer ( ) const

Does this writer add a trailer to the end of the file/buffer?

Returns: true if this writer adds a trailer to the end of the file/buffer, else false.

void evio::WriterMT::addTrailer ( bool add )

Set whether this writer adds a trailer to the end of the file/buffer.

Parameters

add	if true, at the end of file/buffer, add an ending header (trailer) with no index of records and no following data. Update the file header to contain a file offset to the trailer.

bool evio::WriterMT::addTrailerWithIndex ( )

Does this writer add a trailer with a record index to the end of the file? Or, if writing to a buffer, is a trailer added with no index?

Returns: if writing to a file: true if this writer adds a trailer with a record index to the end of the file, else false. If writing to a buffer, true if this writer adds a traile to the end of the buffer, else false.

void evio::WriterMT::addTrailerWithIndex ( bool addTrailingIndex )

Set whether this writer adds a trailer with a record index to the end of the file.

Or, if writing to a buffer, set whether a trailer is added with no index.

Parameters

addTrailingIndex if true, at the end of file, add an ending header (trailer) with an index of all records but with no following data. Update the file header to contain a file offset to the trailer. If true, and writing to a buffer, add a trailer with no index to the end of the buffer.

void evio::WriterMT::close ( )

Close opened file.

If the output record contains events, they will be flushed to file. Trailer and its optional index written if requested.

The addEvent or addRecord methods must no longer be called.

References evio::ByteOrder::ENDIAN_LOCAL, evio::FileHeader::RECORD_COUNT_OFFSET, and SWAP_32.

std::shared_ptr< ByteBuffer > evio::WriterMT::createHeader	(	uint8_t *	userHdr,
		uint32_t	userLen
	)

Create and return a buffer containing a general file header followed by the user header given in the argument.

If user header is not padded to 4-byte boundary, it's done here.

Parameters

userHdr	byte array containing a user-defined header, may be null.
userLen	array length in bytes.

Returns: ByteBuffer containing a file header followed by the user-defined header

References evio::FileHeader::getLength(), evio::FileHeader::HEADER_SIZE_BYTES, evio::FileHeader::reset(), evio::FileHeader::setBitInfo(), evio::FileHeader::setUserHeaderLength(), and evio::FileHeader::writeHeader().

Referenced by open().

std::shared_ptr< ByteBuffer > evio::WriterMT::createHeader ( ByteBuffer & userHdr )

Return a buffer with a general file header followed by the given user header (userHdr).

The buffer is cleared and set to desired byte order prior to writing. If user header is not padded to 4-byte boundary, it's done here.

Parameters

userHdr buffer containing a user-defined header which must be READY-TO-READ!

Returns: buffer containing a file header followed by the user-defined header.

Exceptions

EvioException if writing to buffer, not file.

References evio::ByteBuffer::array(), evio::ByteBuffer::arrayOffset(), evio::FileHeader::getLength(), evio::FileHeader::HEADER_SIZE_BYTES, evio::ByteBuffer::position(), evio::ByteBuffer::remaining(), evio::FileHeader::reset(), evio::FileHeader::setBitInfo(), evio::FileHeader::setUserHeaderLength(), and evio::FileHeader::writeHeader().

const ByteOrder & evio::WriterMT::getByteOrder ( ) const

Get the file's byte order.

Returns: file's byte order.

Compressor::CompressionType evio::WriterMT::getCompressionType ( )

Get the internal record's header.

Returns: internal record's header. Get the internal record used to add events to file.; internal record used to add events to file. Convenience method that gets compression type for the file being written.; compression type for the file being written.

FileHeader & evio::WriterMT::getFileHeader ( )

Get the file header.

Returns: file header.

void evio::WriterMT::open ( const std::string & filename )

Open a new file and write file header with no user header.

Parameters

filename output file name

Exceptions

EvioException	if open already called without being followed by calling close.
IOException	if file cannot be found or IO error writing to file

Referenced by WriterMT().

void evio::WriterMT::open	(	const std::string &	filename,
		uint8_t *	userHdr,
		uint32_t	userLen
	)

Open a file and write file header with given user header.

User header is automatically padded when written.

Parameters

filename	name of file to write to.
userHdr	byte array representing the optional user's header. If this is null AND dictionary and/or first event are given, the dictionary and/or first event will be placed in its own record and written as the user header.
userLen	length of userHdr in bytes.

Exceptions

EvioException	if filename arg is bad, or if open() was already called without being followed by reset().
IOException	if file cannot be found or IO error writing to file

References createHeader(), and evio::FileHeader::getLength().

void evio::WriterMT::reset ( )

Get this object ready for re-use.

Follow calling this with call to open(const std::string &).

References evio::FileHeader::reset().

void evio::WriterMT::writeRecord ( RecordOutput & rec )

Appends the record to the file.

Using this method in conjunction with addEvent() is not thread-safe.

Parameters

rec	record object

Exceptions

EvioException if record's byte order is opposite to output endian.

References evio::RecordOutput::getByteOrder().

The documentation for this class was generated from the following files:

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation