org.jlab.coda.hipo

Class RecordOutputStream

java.lang.Object

org.jlab.coda.hipo.RecordOutputStream

public class RecordOutputStream
extends java.lang.Object

Class which handles the creation and use of Evio and HIPO Records.

 RECORD STRUCTURE:

               Uncompressed                                      Compressed

    +----------------------------------+            +----------------------------------+
    |       General Record Header      |            |       General Record Header      |
    +----------------------------------+            +----------------------------------+

    +----------------------------------+ ---------> +----------------------------------+
    |           Index Array            |            |        Compressed Data           |
    +----------------------------------+            |             Record               |
                                                    |                                  |
    +----------------------------------+            |                                  |
    |           User Header            |            |                  ----------------|
    |           (Optional)             |            |                  |    Pad 3      |
    |                  ----------------|            +----------------------------------+
    |                  |    Pad 1      |           ^
    +----------------------------------+          /
                                                 /
    +----------------------------------+       /
    |           Data Record            |     /
    |                                  |    /
    |                  ----------------|   /
    |                  |    Pad 2      | /
    +----------------------------------+




 GENERAL RECORD HEADER STRUCTURE ( see RecordHeader.java )

    +----------------------------------+
  1 |         Record Length            | // 32bit words, inclusive
    +----------------------------------+
  2 +         Record Number            |
    +----------------------------------+
  3 +         Header Length            | // 14 (words)
    +----------------------------------+
  4 +       Event (Index) Count        |
    +----------------------------------+
  5 +      Index Array Length          | // bytes
    +-----------------------+---------+
  6 +       Bit Info        | Version  | // version (8 bits)
    +-----------------------+----------+
  7 +      User Header Length          | // bytes
    +----------------------------------+
  8 +          Magic Number            | // 0xc0da0100
    +----------------------------------+
  9 +     Uncompressed Data Length     | // bytes
    +------+---------------------------+
 10 +  CT  |  Data Length Compressed   | // CT = compression type (4 bits)
    +----------------------------------+
 11 +        General Register 1        | // UID 1st (64 bits)
    +--                              --+
 12 +                                  |
    +----------------------------------+
 13 +        General Register 2        | // UID 2nd (64 bits)
    +--                              --+
 14 +                                  |
    +----------------------------------+
 

Since:

6.0 9/6/17

Version:

6.0

Author:

gavalian, timmer

Constructor Summary

Constructors

Constructor and Description
RecordOutputStream() Default, no-arg constructor.
RecordOutputStream(java.nio.ByteBuffer buffer, int maxEventCount, CompressionType compressionType, HeaderType hType) Constructor with arguments.
RecordOutputStream(java.nio.ByteOrder order, int maxEventCount, int maxBufferSize, CompressionType compressionType) Constructor with arguments.
RecordOutputStream(java.nio.ByteOrder order, int maxEventCount, int maxBufferSize, CompressionType compressionType, HeaderType hType) Constructor with arguments.

Method Summary

Modifier and Type	Method and Description
boolean	addEvent(byte[] event) Adds an event's ByteBuffer into the record.
boolean	addEvent(byte[] event, int position, int eventLen) Adds an event's data into the record.
boolean	addEvent(byte[] event, int position, int eventLen, int extraDataLen) Adds an event's data into the record.
boolean	addEvent(java.nio.ByteBuffer event) Adds an event's ByteBuffer into the record.
boolean	addEvent(java.nio.ByteBuffer event, int extraDataLen) Adds an event's ByteBuffer into the record.
boolean	addEvent(EvioBank event) Adds an event's ByteBuffer into the record.
boolean	addEvent(EvioBank event, int extraDataLen) Adds an event's ByteBuffer into the record.
boolean	addEvent(EvioNode node) Adds an event's ByteBuffer into the record.
boolean	addEvent(EvioNode node, int extraDataLen) Adds an event's ByteBuffer into the record.
void	build() Builds the record.
void	build(java.nio.ByteBuffer userHeader) Builds the record.
java.nio.ByteBuffer	getBinaryBuffer() Get the internal ByteBuffer used to construct binary representation of this record.
java.nio.ByteOrder	getByteOrder() Get the byte order of the record to be built.
int	getEventCount() Get the number of events written so far into the buffer
RecordHeader	getHeader() Get the general header of this record.
int	getInternalBufferCapacity() Get the capacity of the internal buffer in bytes.
int	getMaxEventCount() Get the max number of events that can be accepted into this record.
int	getUncompressedSize() Get the current uncompressed size of the record in bytes.
int	getUserBufferSize() Get the number of initially available bytes to be written into in the user-given buffer, that goes from position to limit.
boolean	hasUserProvidedBuffer() Was the internal buffer provided by the user?
boolean	oneTooMany() Does adding one more event exceed the event count limit?
void	reset() Reset internal buffers.
boolean	roomForEvent(int length) Is there room in this record's memory for an additional event of the given length in bytes (length NOT including accompanying index).
void	setBuffer(java.nio.ByteBuffer buf) Reset internal buffers and set the buffer in which to build this record.
void	setStartingBufferPosition(int pos) Set the starting position of the user-given buffer being written into.
void	transferDataForReading(RecordOutputStream rec) Copy the contents of the arg into this object and get data buffer ready for reading.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RecordOutputStream

public RecordOutputStream()

Default, no-arg constructor. Little endian. LZ4 compression.

RecordOutputStream

public RecordOutputStream(java.nio.ByteOrder order,
                          int maxEventCount,
                          int maxBufferSize,
                          CompressionType compressionType)

Constructor with arguments. Record header type defaults to HIPO record.

Parameters:

order - byte order of built record byte arrays.

maxEventCount - max number of events this record can hold. Value <= O means use default (1M).

maxBufferSize - max number of uncompressed data bytes this record can hold. Value of < 8MB results in default of 8MB.

compressionType - type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip).

RecordOutputStream

public RecordOutputStream(java.nio.ByteOrder order,
                          int maxEventCount,
                          int maxBufferSize,
                          CompressionType compressionType,
                          HeaderType hType)

Constructor with arguments.

Parameters:

order - byte order of built record byte arrays.

maxEventCount - max number of events this record can hold. Value <= O means use default (1M).

maxBufferSize - max number of uncompressed data bytes this record can hold. Value of < 8MB results in default of 8MB.

compressionType - type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip).

hType - type of record header to use.

RecordOutputStream

public RecordOutputStream(java.nio.ByteBuffer buffer,
                          int maxEventCount,
                          CompressionType compressionType,
                          HeaderType hType)

Constructor with arguments.

Parameters:

buffer - buffer in which to put constructed (and compressed) binary record. Must have position and limit set to accept new data.

maxEventCount - max number of events this record can hold. Value <= O means use default (1M).

compressionType - type of data compression to do (0=none, 1=lz4 fast, 2=lz4 best, 3=gzip).

hType - type of record header to use.

Method Detail

setBuffer

public void setBuffer(java.nio.ByteBuffer buf)

Reset internal buffers and set the buffer in which to build this record. The given buffer should be made ready to receive new data by setting its position and limit properly. Its byte order is set to the same as this writer's. The argument ByteBuffer can be retrieved by calling getBinaryBuffer().

Parameters:

buf - buffer in which to build this record.

transferDataForReading

public void transferDataForReading(RecordOutputStream rec)
                            throws HipoException

Copy the contents of the arg into this object and get data buffer ready for reading. Copy all data up to the buffer limit (not capacity). If the arg has more data than will fit, increase buffer sizes. If the arg has more events than our allowed max, increase the max.

Parameters:

rec - object to copy, must be read to read.

Throws:

HipoException - if we cannot replace internal buffer if it needs to be expanded since it was provided by the user.

getMaxEventCount

public int getMaxEventCount()

Get the max number of events that can be accepted into this record.

Returns:

max number of events that can be accepted into this record.

getUserBufferSize

public int getUserBufferSize()

Get the number of initially available bytes to be written into in the user-given buffer, that goes from position to limit. The user-given buffer is referenced by recordBinary.

Returns:

umber of initially available bytes to be written into in the user-given buffer.

getUncompressedSize

public int getUncompressedSize()

Get the current uncompressed size of the record in bytes. This does NOT count any user header.

Returns:

current uncompressed size of the record in bytes.

getInternalBufferCapacity

public int getInternalBufferCapacity()

Get the capacity of the internal buffer in bytes. This is the upper limit of the memory needed to store this uncompressed record.

Returns:

capacity of the internal buffer in bytes.

getHeader

public RecordHeader getHeader()

Get the general header of this record.

Returns:

general header of this record.

getEventCount

public int getEventCount()

Get the number of events written so far into the buffer

Returns:

number of events written so far into the buffer

getBinaryBuffer

public java.nio.ByteBuffer getBinaryBuffer()

Get the internal ByteBuffer used to construct binary representation of this record.

Returns:

internal ByteBuffer used to construct binary representation of this record.

hasUserProvidedBuffer

public boolean hasUserProvidedBuffer()

Was the internal buffer provided by the user?

Returns:

true if internal buffer provided by user.

getByteOrder

public java.nio.ByteOrder getByteOrder()

Get the byte order of the record to be built.

Returns:

byte order of the record to be built.

roomForEvent

public boolean roomForEvent(int length)

Is there room in this record's memory for an additional event of the given length in bytes (length NOT including accompanying index).

Parameters:

length - length of data to add in bytes

Returns:

true if room in record, else false.

oneTooMany

public boolean oneTooMany()

Does adding one more event exceed the event count limit?

Returns:

true if one more event exceeds count limit, else false.

addEvent

public boolean addEvent(byte[] event)

Adds an event's ByteBuffer into the record. If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's byte array.

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

addEvent

public boolean addEvent(byte[] event,
                        int position,
                        int eventLen)

Adds an event's data into the record. If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's byte array.

position - offset into event byte array from which to begin reading.

eventLen - number of bytes from byte array to add.

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

addEvent

public boolean addEvent(byte[] event,
                        int position,
                        int eventLen,
                        int extraDataLen)

Adds an event's data into the record. Can specify the length of additional data to follow the event (such as an evio trailer record) to see if by adding this event everything will fit in the available memory.

If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's byte array.

position - offset into event byte array from which to begin reading.

eventLen - number of bytes from byte array to add.

extraDataLen - additional data bytes to follow event (e.g. trailer length).

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

addEvent

public boolean addEvent(java.nio.ByteBuffer event)

Adds an event's ByteBuffer into the record. If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's ByteBuffer object.

Returns:

true if event was added; false if the buffer is full, the event count limit is exceeded, or this single event cannot fit into the user-provided buffer.

addEvent

public boolean addEvent(java.nio.ByteBuffer event,
                        int extraDataLen)

Adds an event's ByteBuffer into the record. Can specify the length of additional data to follow the event (such as an evio trailer record) to see if by adding this event everything will fit in the available memory.

If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's ByteBuffer object.

extraDataLen - additional data bytes to follow event (e.g. trailer length).

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

addEvent

public boolean addEvent(EvioNode node)
                 throws HipoException

Adds an event's ByteBuffer into the record. If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor! This method is not thread-safe with respect to the node as it's backing ByteBuffer's limit and position may be concurrently changed.

Parameters:

node - event's EvioNode object

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

Throws:

HipoException - if node does not correspond to a bank.

addEvent

public boolean addEvent(EvioNode node,
                        int extraDataLen)
                 throws HipoException

Adds an event's ByteBuffer into the record. Can specify the length of additional data to follow the event (such as an evio trailer record) to see if by adding this event everything will fit in the available memory.

If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor! This method is not thread-safe with respect to the node as it's backing ByteBuffer's limit and position may be concurrently changed.

Parameters:

node - event's EvioNode object

extraDataLen - additional data bytes to follow event (e.g. trailer length).

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

Throws:

HipoException - if node does not correspond to a bank.

addEvent

public boolean addEvent(EvioBank event)

Adds an event's ByteBuffer into the record. If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's EvioBank object.

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

addEvent

public boolean addEvent(EvioBank event,
                        int extraDataLen)

Adds an event's ByteBuffer into the record. Can specify the length of additional data to follow the event (such as an evio trailer record) to see if by adding this event everything will fit in the available memory.

If a single event is too large for the internal buffers, more memory is allocated. On the other hand, if the buffer was provided by the user, then obviously the buffer cannot be expanded and false is returned.

The byte order of event must match the byte order given in constructor!

Parameters:

event - event's EvioBank object.

extraDataLen - additional data bytes to follow event (e.g. trailer length).

Returns:

true if event was added; false if the event was not added because the count limit would be exceeded or the buffer is full and cannot be expanded since it's user-provided.

reset

public void reset()

Reset internal buffers. The buffer is ready to receive new data. Also resets the header including removing any compression. If data buffer externally provided, the starting position is set to 0.

setStartingBufferPosition

public void setStartingBufferPosition(int pos)

Set the starting position of the user-given buffer being written into. Calling this may be necessary from EventWriter(Unsync) when a common record (dictionary + first event) is written after the constructor for this object has been called, but before any events have been written. This method should not be called in general as it will MESS UP THE WRITING OF DATA!

Parameters:

pos - position in buffer to start writing.

build

public void build()

Builds the record. Compresses data, header is constructed, then header and data written into internal buffer. This method may be called multiple times in succession without any problem. The buffer obtained with getBinaryBuffer() has its position and limit set for reading.

build

public void build(java.nio.ByteBuffer userHeader)

Builds the record. Compresses data, header is constructed, then header and data written into internal buffer. If user header is not padded to 4-byte boundary, it's done here. This method may be called multiple times in succession without any problem.

Parameters:

userHeader - user's ByteBuffer which must be READY-TO-READ!