What is CODA?
CODA is a software and hardware tool kit from which data acquisition systems may be constructed. In order for this description of the tool kit to make any sense the model of data acquisition implemented by CODA has to be understood. The typical CODA data acquisition system is constructed from modular components (usually programs) which are spread over a network of processors. These processors may take the form of embedded CAMAC, VME or FASTBUS modules, or PC / Workstation systems. The primary function of each component is to take data from some input source(s), manipulate it and pass it to an output. The components are distributed and interconnected in such a way that the overall result is that data from a detector is collected and archived on long-term storage media. The components are acted upon by a group of utilities that perform such functions as configuration, data flow control and monitoring the "health" of the system.
The CODA system is under continuous development as the needs of experiments at Jefferson lab evolve. The first stable production version of CODA was version 1.5. CODA 1.4 was designed on ULTRIX and was supported on Hewlett-Packard HP-UX, Sun Microsystems Solaris and DEC ULTRIX machines.
Author's note
Although ULTRIX versions of CODA 1.5 binaries still exist ULTRIX was taken out of service at Jefferson lab in fall 1998. We now have no operational ULTRIX machines to perform bug fixes etc at Jefferson lab. CODA versions above 2.x were never tested on ULTRIX. Version 2.0 of CODA was developed between 1994 and 1998 to meet the needs of the CLAS detector. Version 2.1 was released late in 1998 for general use at Jefferson lab where it has almost completely replaced CODA 1.5. CODA is currently (January 1999) supported on the Solaris and Linux operating systems.
Data acquisition system evolution.
The pull architecture.
In the 1970's early 1980's most data acquisition systems designed followed the pattern of a central host machine connected to a tree like structure, for example a CAMAC branch highway or FASTBUS segment interconnect, the leaves on which were digitizing electronics. There was very little processing power outside the central host. The introduction of the microprocessor lead to increasing embedded processing power outside the central host. The data acquisition architecture was still relatively unmodified the central host pulled data from the front-end systems. In the mid 1980's a new architecture emerged in which much more of the processing power was in the embedded processors which pushed data downstream. The push architecture lead to the development of the shared memory event
builder. In this system the embedded processors have direct interfaces to a memory area which is shared between all the embedded processors and the host machine. The embedded controllers push data into the memory and the host pulls it out in the correct sequence to generate complete events.
Historical note
The first design of a data acquisition system for the CLAS detector took the form of embedded FASTBUS processors interfaced to a shared memory event builder. The problem with this architecture was that much of the hardware and all of the software was usually custom designed. This was time consuming and costly.
The network based event builder.
Sometime in the late 1980's early 1990's several groups (including ours) started work on the same idea. Commercial network hardware was approaching the data bandwidth of the custom build data-links used for data acquisition at a much lower cost. Also high-speed network switches were being designed for the telecommunications industry which looked amazingly similar in function to the shared memory event builder. The idea was to link the embedded controllers to a commercial network switch that could then be linked to one, or more, host machines. Essentially the network switch would take the place of the custom-built event builder. This model was chosen for CODA since it has several clear advantages over other models including, low cost, ease of maintenance, scalability, ease of upgrade.
The CODA model
CODA implements a model in which a data acquisition system is composed of distributed heterogeneous processors connected by commercial network components. ( That is not to say that custom data-links will not work!) The first CODA systems that were deployed transferred data over 10 Mbit/S Ethernet. Later Hall C installed FDDI and achieved 30Mbit/S. There was a brief period of time where ATM was investigated for hall B (CLAS) but this medium was quickly overtaken in commercial popularity by 100Mbit/S Ethernet and was too costly to implement on a large scale.
As of fall 1998 all major DAQ systems at Jefferson lab were using 100Mbit/S Ethernet. The CLAS detector uses four 100Mbit/S links in parallel for a sustained transfer of over 11Mbyte/S. (the limiting factor here is the RAID disk drive used to buffer the data on the host machine).
In coda the embedded processors can be in CAMAC, FASTBUS or VME. The CODA software is designed to be extremely portable however as of January 1999 the only real-time operating system supported on the embedded controllers is VxWorks. CODA has been ported to Linux but as of January 1999 we have not yet tested real time Linux for embedded processors. We are also investigating LynxOS.
Data flow
In CODA a hierarchy of processes manipulate the flow of data from the detector to a storage
device.
In the diagram embedded controllers called "Readout Controllers" or "ROCs" read out different components of a detector. The data is buffered in the controller and when enough data has accumulated it is transmitted over the network. The buffering in the controllers is critical to reducing the overhead required to run a commercial network protocol such as TCP/IP. When running at low data rate CODA switches into a less efficient method of data transmission where events are passed over the network one at a time.
Since the data for a particular event is spread over data streams from several embedded controllers it is necessary to merge the data streams. This function is performed by the "Event Builder" (EB) which caches incoming buffers of events from the different controllers then merges the data streams in such a way that data which was taken concurrently in time (i.e. belongs to the same "event" in the detector) appears together. The event builder also formats the outgoing data stream and provides some measure of flow control to incoming and outgoing data.
The event builder in CODA is a software event builder, it runs as a UNIX process and inserts formatted events into a buffer manager called the "Event Transport system" (ET). The Et system uses UNIX shared memory protected by inter process semaphores to allow many processes access to the same events for analysis, filtering, monitoring, display etc.
Author's note
The original DD (Data Distribution) system used by CODA was replaced by a new buffer manager, the "Event transfer" or ET system in 1999. The ET system has been very successful and has replaced the DD system at Brookhaven where it was originally written. The ET system may be configured to direct the data stream to an "event recorder" process that stores the events in a format of the user's choice. In contrast to other data acquisition software suites CODA does not have the capability to write tapes directly. The philosophy is that data should be staged on high speed RAID disk arrays and archived to tape by commercially available software such as OSM. For example hall B data is written to a RAID-0 disk array backed up by the computer center to a StorageTek tape silo using
OSM.
The event builder and event recorder are designed so that they are relatively independent of the event format. There is a default CODA data format but the user can supply shared libraries for the event builder and event recorder which generate and write events in any format. For example halls A and C use CODA format which hall B (CLAS) use BOS and FPACK. The idea is that as far as CODA is concerned all event formats provide adequate information for event building and storage. It is up to the user to provide routines to decode and encode the required format.
Control and monitoring
Clearly a CODA system can become complex and therefore difficult to control and monitor due to its distributed nature. Several tools are available to make these tasks easier:
RunControl
RunControl is the main GUI for CODA it provides the user with the ability to chose a configuration for a particular data taking run and control all aspects of data taking. RunControl is written in C++ and contains an internal representation of all of the components of the data acquisition system. The main departure between the CODA 2.0 and 1.5 RunControl programs is the ability to have multiple
sessions where more than one user can use different components from a common pool. For example different detector development groups working on different aspects of the detector at the same time.
Configuration database
RunControl and all of the component parts of CODA require configuration information to operate. This information is stored in a relational database. In the case of the 2.x CODA version this is the msql database.
CMLOG
cmlog provides a central message recording and reporting facility for CODA. Each CODA component may post a message using the subroutine dalogmsg(class,fmt,...), giving a message classification, a printf like format string and a variable list of arguments. The message is time-stamped and tagged with the process id and other useful information by the message source. The cmlog system has the ability to display the formatted messages, according to selection criteria, on one or more X-window displays. cmlog is a third partypackage in that it was written by Jie Chen who used to work for the data acquisition group but actually wrote it for the Jefferson lab accelerator control group.
http://www.jlab.org/cdev_and CODA CDEV service
CDEV is a whole world by itself, see the web page, in short it is a common C/C++ API to control systems. The idea is that the user should not have to know in depth details of the control system but should be able to send simple plain text commands to named objects. The RunControl server for CODA communicates with the RunControl GUI via a C++ communication library called ACE, CODA includes a CDEV service which uses ACE to communicate with the RunControl server.
Scripting
All of the components of CODA (ROC, EB, ER etc) contain an interpreter for the Tcl scripting language. In fact many parts of CODA are written in Tcl. A general purpose Tcl shell is provided as part of the CODA package which implements Tcl commands which allow direct interaction with CODA. In conjunction with the Tk user interface package this allows interfaces to CODA to be written entirely in Tcl. Some examples are listed below.
dbedit
The dbedit tool is written in Tcl which gives the CODA user direct access to the msql database.
xevent
CODA contains a large number of component packages which are "glued" into applications using the TCL scripting language. One advantage of this approach is that the "informed user" has direct access to the internals of many CODA components. The xevent utility is written in TCL and uses the CODA inter-process communication system to extract the current event count from any CODA component and display it in an X-window display.
xpart
The xpart program is also written in Tcl and provides a mechanism for displaying the state of the internal buffer manager on any ROC.
Further reading
CODA is a complex system but its basic features are relatively simple to understand. Data is read from the detector by a readout list in a Readout Controller and may be either stored directly at this point or passed over a network to an Event Builder that builds and formats the event. The events may be stored at this point or passed into a buffer manager. One possible consumer on the buffer manager is an Event Recorder that formats or stores data to disk. Data is archived from disk by software outside of CODA. The entire system is controlled by Run Control using information from a relational database. Tools exist for monitoring the system and to assist the input of configuration information into the database. All of these components and tools are described in detail in these pages under the user guide section. CODA related papers are also stored on this web site.
