org.jlab.coda.hipo

Class Reader

java.lang.Object

org.jlab.coda.hipo.Reader

public class Reader
extends java.lang.Object

Reader class that reads files stored in the HIPO format.

 File has this structure:

    +----------------------------------+
    |      General File Header         |
    +----------------------------------+
    +----------------------------------+
    |         Index (optional)         |
    +----------------------------------+
    +----------------------------------+
    |     User Header (optional)       |
    +----------------------------------+
    +----------------------------------+
    |                                  |
    |            Record 1              |
    |                                  |
    |                                  |
    |                                  |
    +----------------------------------+
                   ...
    +----------------------------------+
    |                                  |
    |            Record N              |
    |                                  |
    |                                  |
    |                                  |
    +----------------------------------+
    +----------------------------------+
    |       Trailer (optional)         |
    +----------------------------------+
    +----------------------------------+
    |    Trailer's Index (optional)    |
    +----------------------------------+



 Buffer or streamed data has this structure:

    +----------------------------------+
    |                                  |
    |            Record 1              |
    |                                  |
    |                                  |
    |                                  |
    +----------------------------------+
                   ...
    +----------------------------------+
    |                                  |
    |            Record N              |
    |                                  |
    |                                  |              RecordPosition
    |                                  |
    +----------------------------------+
    +----------------------------------+
    |       Trailer (optional)         |
    +----------------------------------+

 The important thing with a buffer or streaming is for the last header or
 trailer to set the "last record" bit.

 

Something to keep in mind is one can intersperse sequential calls (getNextEvent, getPrevEvent, or getNextEventNode) with random access calls (getEvent or getEventNode), and the sequence remains unchanged after the random access.

Since:

6.0 08/10/2017

Version:

6.0

Author:

gavalian, timmer

See Also:

FileHeader, RecordInputStream

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Reader.RecordPosition Internal class to keep track of the records in the file/buffer.

Constructor Summary

Constructors

Constructor and Description
Reader() Default constructor.
Reader(java.nio.ByteBuffer buffer) Constructor for reading buffer with evio data.
Reader(java.nio.ByteBuffer buffer, EvioNodeSource pool) Constructor for reading buffer with evio data.
Reader(java.nio.ByteBuffer buffer, EvioNodeSource pool, boolean checkRecordNumSeq) Constructor for reading buffer with evio data.
Reader(java.io.RandomAccessFile file) Constructor with file already opened.
Reader(java.lang.String filename) Constructor with filename.
Reader(java.lang.String filename, boolean forceScan) Constructor with filename.

Method Summary

Modifier and Type	Method and Description
java.nio.ByteBuffer	addStructure(int eventNumber, java.nio.ByteBuffer addBuffer) This method adds an evio container (bank, segment, or tag segment) as the last structure contained in an event.
void	close() This closes the file.
static void	findRecordInfo(java.nio.ByteBuffer buf, int offset, int[] info) Reads data from a record header in order to determine things like the bitInfo word, various lengths, etc.
void	forceScanFile() Scan file to find all records and store their position, length, and event count.
java.nio.ByteBuffer	getBuffer() Get the buffer being read, if any.
int	getBufferOffset() Get the beginning position of the buffer being read.
java.nio.ByteOrder	getByteOrder() Get the byte order of the file/buffer being read.
boolean	getCheckRecordNumberSequence() Get whether or not record numbers are enforced to be sequential.
int	getCurrentRecord() Get the index of the current record.
RecordInputStream	getCurrentRecordStream() Get the current record stream.
java.lang.String	getDictionary() Get the XML format dictionary if there is one.
java.nio.ByteBuffer	getEvent(java.nio.ByteBuffer buf, int index) Get a byte array representing the specified event from the file/buffer and place it in the given buf.
byte[]	getEvent(int index) Get a byte array representing the specified event from the file/buffer.
int	getEventCount() Get the number of events in file/buffer.
EvioNode	getEventNode(int index) Get an EvioNode representing the specified event from the buffer.
java.util.ArrayList<EvioNode>	getEventNodes() Get the list of EvioNode objects contained in the buffer being read.
FileHeader	getFileHeader() Get the file header from reading a file.
java.lang.String	getFileName() Get the name of the file being read.
long	getFileSize() Get the size of the file being read, in bytes.
byte[]	getFirstEvent() Get a byte array representing the first event.
RecordHeader	getFirstRecordHeader() Get the first record header from reading a file/buffer.
byte[]	getNextEvent() Get a byte array representing the next event from the file/buffer while sequentially reading.
EvioNode	getNextEventNode() Get an EvioNode representing the next event from the buffer while sequentially reading.
int	getNumEventsRemaining() Get the number of events remaining in the file/buffer.
byte[]	getPrevEvent() Get a byte array representing the previous event from the sequential queue.
int	getRecordCount() Get the number of records read from the file/buffer.
int	getRecordEventCount() Get the number of events in current record.
java.util.List<Reader.RecordPosition>	getRecordPositions() Returns the list of record positions in the file.
int	getVersion() Get the Evio format version number of the file/buffer being read.
boolean	hasDictionary() Does this evio file/buffer have an associated XML dictionary?
boolean	hasFirstEvent() Does this evio file/buffer have an associated first event?
boolean	hasNext() Checks if the file has an event to read next.
boolean	hasPrev() Checks if the stream has previous event to be accessed through, getPrevEvent()
boolean	isClosed() Has close() been called (without reopening by calling setBuffer(ByteBuffer))?
boolean	isCompressed() Is the data in the file/buffer compressed?
boolean	isEvioFormat() Does this file/buffer contain non-evio format events?
boolean	isFile() Is a file being read?
static void	main(java.lang.String[] args)
void	open(java.lang.String filename) Opens an input stream in binary mode.
void	open(java.lang.String filename, boolean scan) Opens an input stream in binary mode.
boolean	readRecord(int index) Reads record from the file/buffer at the given record index.
java.nio.ByteBuffer	readUserHeader() Reads user header of the file header/first record header of buffer.
java.nio.ByteBuffer	removeStructure(EvioNode removeNode) This method removes the data, represented by the given node, from the buffer.
java.nio.ByteBuffer	scanBuffer() This method scans a buffer to find all records and store their position, length, and event count.
void	scanFile(boolean force) Scans the file to index all the record positions.
void	scanUncompressedBuffer() Scan buffer to find all records and store their position, length, and event count.
void	setBuffer(java.nio.ByteBuffer buf) This method can be used to avoid creating additional Reader objects by reusing this one with another buffer.
void	setBuffer(java.nio.ByteBuffer buf, EvioNodeSource pool) This method can be used to avoid creating additional Reader objects by reusing this one with another buffer.
void	show()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Reader

public Reader()

Default constructor. Does nothing. The open(String) method has to be called to open the input stream. Also forceScanFile() needs to be called to find records.

Reader

public Reader(java.io.RandomAccessFile file)

Constructor with file already opened. The method forceScanFile() needs to be called to find records.

Parameters:

file - open file

Reader

public Reader(java.lang.String filename)
       throws java.io.IOException,
              HipoException

Constructor with filename. Creates instance and opens the input stream with given name. Uses existing indexes in file before scanning.

Parameters:

filename - input file name.

Throws:

java.io.IOException - if error reading file

HipoException - if file is not in the proper format or earlier than version 6

Reader

public Reader(java.lang.String filename,
              boolean forceScan)
       throws java.io.IOException,
              HipoException

Constructor with filename. Creates instance and opens the input stream with given name.

Parameters:

filename - input file name.

forceScan - if true, force a scan of file, else use existing indexes first.

Throws:

java.io.IOException - if error reading file

HipoException - if file is not in the proper format or earlier than version 6

Reader

public Reader(java.nio.ByteBuffer buffer)
       throws HipoException

Constructor for reading buffer with evio data. Buffer must be ready to read with position and limit set properly. If the given buffer contains compressed data, it is uncompressed into another buffer. The buffer containing the newly uncompressed data then becomes the internal buffer of this object. It can be obtained by calling getBuffer().

Parameters:

buffer - buffer with evio data.

Throws:

HipoException - if buffer too small, not in the proper format, or earlier than version 6

Reader

public Reader(java.nio.ByteBuffer buffer,
              EvioNodeSource pool)
       throws HipoException

Constructor for reading buffer with evio data. Buffer must be ready to read with position and limit set properly. If the given buffer contains compressed data, it is uncompressed into another buffer. The buffer containing the newly uncompressed data then becomes the internal buffer of this object. It can be obtained by calling getBuffer().

Parameters:

buffer - buffer with evio data.

pool - pool of EvioNode objects for garbage-free operation.

Throws:

HipoException - if buffer too small, not in the proper format, or earlier than version 6

Reader

public Reader(java.nio.ByteBuffer buffer,
              EvioNodeSource pool,
              boolean checkRecordNumSeq)
       throws HipoException

Constructor for reading buffer with evio data. Buffer must be ready to read with position and limit set properly. If the given buffer contains compressed data, it is uncompressed into another buffer. The buffer containing the newly uncompressed data then becomes the internal buffer of this object. It can be obtained by calling getBuffer().

Parameters:

buffer - buffer with evio data.

pool - pool of EvioNode objects for garbage-free operation.

checkRecordNumSeq - if true, check to see if all record numbers are in order, if not throw exception.

Throws:

HipoException - if buffer too small, not in the proper format, or earlier than version 6; if checkRecordNumSeq is true and records are out of sequence.

Method Detail

open

public final void open(java.lang.String filename)
                throws java.io.IOException,
                       HipoException

Opens an input stream in binary mode. Scans for records in the file and stores record information in internal array. Each record can be read from the file.

Parameters:

filename - input file name

Throws:

java.io.IOException - if file not found or error opening file

HipoException - if file is not in the proper format or earlier than version 6

open

public final void open(java.lang.String filename,
                       boolean scan)
                throws java.io.IOException,
                       HipoException

Opens an input stream in binary mode. Scans for records in the file and stores record information in internal array. Each record can be read from the file.

Parameters:

filename - input file name

scan - if true, call scanFile(false).

Throws:

java.io.IOException - if file not found or error opening file

HipoException - if file is not in the proper format or earlier than version 6

close

public void close()
           throws java.io.IOException

This closes the file.

Throws:

java.io.IOException - if error accessing file

isClosed

public boolean isClosed()

Has close() been called (without reopening by calling setBuffer(ByteBuffer))?

Returns:

true if this object closed, else false.

isFile

public boolean isFile()

Is a file being read?

Returns:

true if a file is being read, false if it's a buffer.

setBuffer

public void setBuffer(java.nio.ByteBuffer buf)
               throws HipoException

This method can be used to avoid creating additional Reader objects by reusing this one with another buffer. If the given buffer contains compressed data, it is uncompressed into another buffer. The buffer containing the newly uncompressed data then becomes the internal buffer of this object. It can be obtained by calling getBuffer().

Parameters:

buf - ByteBuffer to be read

Throws:

HipoException - if buf arg is null, buffer too small, not in the proper format, or earlier than version 6

setBuffer

public void setBuffer(java.nio.ByteBuffer buf,
                      EvioNodeSource pool)
               throws HipoException

This method can be used to avoid creating additional Reader objects by reusing this one with another buffer. The pool is not reset in this method. Caller may do that prior to calling this method. If the given buffer contains compressed data, it is uncompressed into another buffer. The buffer containing the newly uncompressed data then becomes the internal buffer of this object. It can be obtained by calling getBuffer().

Parameters:

buf - ByteBuffer to be read

pool - pool of EvioNode objects to use when parsing buf.

Throws:

HipoException - if buf arg is null, buffer too small, not in the proper format, or earlier than version 6

getFileName

public java.lang.String getFileName()

Get the name of the file being read.

Returns:

name of the file being read or null if none.

getFileSize

public long getFileSize()

Get the size of the file being read, in bytes.

Returns:

size of the file being read, in bytes, or 0 if none.

getBuffer

public java.nio.ByteBuffer getBuffer()

Get the buffer being read, if any.

Returns:

buffer being read, if any.

getBufferOffset

public int getBufferOffset()

Get the beginning position of the buffer being read.

Returns:

the beginning position of the buffer being read.

getFileHeader

public FileHeader getFileHeader()

Get the file header from reading a file. Will return null if reading a buffer.

Returns:

file header from reading a file.

getFirstRecordHeader

public RecordHeader getFirstRecordHeader()

Get the first record header from reading a file/buffer.

Returns:

first record header from reading a file/buffer.

getByteOrder

public java.nio.ByteOrder getByteOrder()

Get the byte order of the file/buffer being read.

Returns:

byte order of the file/buffer being read.

getVersion

public int getVersion()

Get the Evio format version number of the file/buffer being read.

Returns:

Evio format version number of the file/buffer being read.

isCompressed

public boolean isCompressed()

Is the data in the file/buffer compressed?

Returns:

true if data is compressed.

isEvioFormat

public boolean isEvioFormat()

Does this file/buffer contain non-evio format events?

Returns:

true if all events are in evio format, else false.

getDictionary

public java.lang.String getDictionary()

Get the XML format dictionary if there is one.

Returns:

XML format dictionary, else null.

hasDictionary

public boolean hasDictionary()

Does this evio file/buffer have an associated XML dictionary?

Returns:

true if this evio file/buffer has an associated XML dictionary, else false.

getFirstEvent

public byte[] getFirstEvent()

Get a byte array representing the first event.

Returns:

byte array representing the first event. Null if none.

hasFirstEvent

public boolean hasFirstEvent()

Does this evio file/buffer have an associated first event?

Returns:

true if this evio file/buffer has an associated first event, else false.

getEventCount

public int getEventCount()

Get the number of events in file/buffer.

Returns:

number of events in file/buffer.

getRecordCount

public int getRecordCount()

Get the number of records read from the file/buffer.

Returns:

number of records read from the file/buffer.

getRecordPositions

public java.util.List<Reader.RecordPosition> getRecordPositions()

Returns the list of record positions in the file.

Returns:

the list of record positions in the file.

getEventNodes

public java.util.ArrayList<EvioNode> getEventNodes()

Get the list of EvioNode objects contained in the buffer being read. To be used internally to evio.

Returns:

list of EvioNode objects contained in the buffer being read.

getCheckRecordNumberSequence

public boolean getCheckRecordNumberSequence()

Get whether or not record numbers are enforced to be sequential.

Returns:

true if record numbers are enforced to be sequential.

getNumEventsRemaining

public int getNumEventsRemaining()

Get the number of events remaining in the file/buffer. Useful only if doing a sequential read.

Returns:

number of events remaining in the file/buffer

getNextEvent

public byte[] getNextEvent()
                    throws HipoException

Get a byte array representing the next event from the file/buffer while sequentially reading. If the previous call was to getPrevEvent(), this will get the event past what that returned. Once the last event is returned, this will return null.

Returns:

byte array representing the next event or null if there is none.

Throws:

HipoException - if file/buffer not in hipo format

getPrevEvent

public byte[] getPrevEvent()
                    throws HipoException

Get a byte array representing the previous event from the sequential queue. If the previous call was to getNextEvent(), this will get the event previous to what that returned. If this is called before getNextEvent, it will always return null. Once the first event is returned, this will return null.

Returns:

byte array representing the previous event or null if there is none.

Throws:

HipoException - if the file/buffer is not in HIPO format

getNextEventNode

public EvioNode getNextEventNode()

Get an EvioNode representing the next event from the buffer while sequentially reading. Calling this and calling getNextEvent() have the same effect in terms of advancing the same internal counter. If the previous call was to getPrevEvent(), this will get the event past what that returned. Once the last event is returned, this will return null.

Returns:

EvioNode representing the next event or null if no more events, reading a file or data is compressed.

readUserHeader

public java.nio.ByteBuffer readUserHeader()
                                   throws java.io.IOException

Reads user header of the file header/first record header of buffer. The returned ByteBuffer also contains endianness of the file/buffer.

Returns:

ByteBuffer containing the user header of the file/buffer.

Throws:

java.io.IOException - if error reading file

getEvent

public byte[] getEvent(int index)
                throws HipoException

Get a byte array representing the specified event from the file/buffer. If index is out of bounds, null is returned.

Parameters:

index - index of specified event within the entire file/buffer, contiguous starting at 0.

Returns:

byte array representing the specified event or null if index is out of bounds.

Throws:

HipoException - if file/buffer not in hipo format

getEvent

public java.nio.ByteBuffer getEvent(java.nio.ByteBuffer buf,
                                    int index)
                             throws HipoException

Get a byte array representing the specified event from the file/buffer and place it in the given buf. If no buf is given (arg is null), create a buffer internally and return it. If index is out of bounds, null is returned.

Parameters:

buf - buffer in which to place event data.

index - index of specified event within the entire file/buffer, contiguous starting at 0.

Returns:

buf or null if buf is null or index out of bounds.

Throws:

HipoException - if file/buffer not in hipo format, or if buf has insufficient space to contain event (buf.capacity() < event size).

getEventNode

public EvioNode getEventNode(int index)

Get an EvioNode representing the specified event from the buffer. If index is out of bounds, null is returned.

Parameters:

index - index of specified event within the entire buffer, starting at 0.

Returns:

EvioNode representing the specified event or null if index is out of bounds, reading a file or data is compressed.

hasNext

public boolean hasNext()

Checks if the file has an event to read next.

Returns:

true if the next event is available, false otherwise

hasPrev

public boolean hasPrev()

Checks if the stream has previous event to be accessed through, getPrevEvent()

Returns:

true if previous event is accessible, false otherwise

getRecordEventCount

public int getRecordEventCount()

Get the number of events in current record.

Returns:

number of events in current record.

getCurrentRecord

public int getCurrentRecord()

Get the index of the current record.

Returns:

index of the current record.

getCurrentRecordStream

public RecordInputStream getCurrentRecordStream()

Get the current record stream.

Returns:

current record stream.

readRecord

public boolean readRecord(int index)
                   throws HipoException

Reads record from the file/buffer at the given record index.

Parameters:

index - record index (starting at 0).

Returns:

true if valid index and successful reading record, else false.

Throws:

HipoException - if file/buffer not in hipo format

findRecordInfo

public static void findRecordInfo(java.nio.ByteBuffer buf,
                                  int offset,
                                  int[] info)
                           throws HipoException,
                                  java.nio.BufferUnderflowException

Reads data from a record header in order to determine things like the bitInfo word, various lengths, etc. Does not change the position or limit of buffer.

Parameters:

buf - buffer containing evio header.

offset - byte offset into buffer.

info - array in which to store header info. Elements are:

0. bit info word
1. record length in bytes (inclusive)
2. compression type
3. header length in bytes
4. index array length in bytes (not padded)
5. user header length in bytes
6. uncompressed data length in bytes (w/o record header)

Throws:

java.nio.BufferUnderflowException - if not enough data in buffer.

HipoException - null arg(s), negative offset, or info.length < 7.

scanBuffer

public java.nio.ByteBuffer scanBuffer()
                               throws HipoException,
                                      java.nio.BufferUnderflowException

This method scans a buffer to find all records and store their position, length, and event count. It also finds all events and then creates and stores their associated EvioNode objects.

The difficulty with doing this is that the buffer may contain compressed data. It must then be uncompressed into a different buffer. There are 2 possibilities. First, if the buffer being parsed is too small to hold its uncompressed form, then a new, larger buffer is created, filled with the uncompressed data and then given as the return value of this method. Second, if the buffer being parsed is large enough to hold its uncompressed form, the data is uncompressed into a temporary holding buffer. When all decompresssion and parsing is finished, the contents of the temporary buffer are copied back into the original buffer which then becomes the return value.

Returns:

buffer containing uncompressed data. This buffer may be different than the one originally scanned if the data was compressed and the uncompressed length is greater than the original buffer could hold.

Throws:

HipoException - if buffer not in the proper format or earlier than version 6; if checkRecordNumberSequence is true and records are out of sequence.

java.nio.BufferUnderflowException - if not enough data in buffer.

scanUncompressedBuffer

public void scanUncompressedBuffer()
                            throws HipoException

Scan buffer to find all records and store their position, length, and event count. Also finds all events and then creates and stores their associated EvioNode objects.

Throws:

HipoException - if buffer too small, not in the proper format, or earlier than version 6; if checkRecordNumberSequence is true and records are out of sequence.

forceScanFile

public void forceScanFile()
                   throws java.io.IOException,
                          HipoException

Scan file to find all records and store their position, length, and event count. Safe to call this method successively.

Throws:

java.io.IOException - if error reading file

HipoException - if file is not in the proper format or earlier than version 6; if checkRecordNumberSequence is true and records are out of sequence.

scanFile

public void scanFile(boolean force)
              throws java.io.IOException,
                     HipoException

Scans the file to index all the record positions. It takes advantage of any existing indexes in file.

Parameters:

force - if true, force a file scan even if header or trailer have index info.

Throws:

java.io.IOException - if error reading file

HipoException - if file is not in the proper format or earlier than version 6

removeStructure

public java.nio.ByteBuffer removeStructure(EvioNode removeNode)
                                    throws HipoException

This method removes the data, represented by the given node, from the buffer. It also marks all nodes taken from that buffer as obsolete. They must not be used anymore.

Parameters:

removeNode - evio structure to remove from buffer

Returns:

ByteBuffer updated to reflect the node removal

Throws:

HipoException - if object closed; if node was not found in any event; if internal programming error; if buffer has compressed data;

addStructure

public java.nio.ByteBuffer addStructure(int eventNumber,
                                        java.nio.ByteBuffer addBuffer)
                                 throws HipoException

This method adds an evio container (bank, segment, or tag segment) as the last structure contained in an event. It is the responsibility of the caller to make sure that the buffer argument contains valid evio data (only data representing the structure to be added - not in file format with record header and the like) which is compatible with the type of data stored in the given event.

To produce such evio data use BaseStructure.write(ByteBuffer), BaseStructure.write(ByteBuffer) or BaseStructure.write(ByteBuffer) depending on whether a bank, seg, or tagseg is being added.

The given buffer argument must be ready to read with its position and limit defining the limits of the data to copy.

Parameters:

eventNumber - number of event to which addBuffer is to be added

addBuffer - buffer containing evio data to add (not evio file format, i.e. no record headers)

Returns:

a new ByteBuffer object which is created and filled with all the data including what was just added.

Throws:

HipoException - if eventNumber out of bounds; if addBuffer is null; if addBuffer arg is empty or has non-evio format; if addBuffer is opposite endian to current event buffer; if added data is not the proper length (i.e. multiple of 4 bytes); if the event number does not correspond to an existing event; if there is an internal programming error; if object closed

show

public void show()

main

public static void main(java.lang.String[] args)