VxWorks to Linux HOWTO

This page provides some insight, instructions, and examples on how to migrate your module libraries and programs from the vxWorks operating system to the Linux operating system. We will assume that the use of the Intel-Based VME controller will be for taking data with CODA, or parallel with CODA.


Basic differences with respect to VME access

Access to VME address spaces

In vxWorks, the VME windows (the map between various VME address spaces and PCI space) is created at boot. In Linux, this is done at program execution with the routine:

int  vmeOpenDefaultWindows();

This routine does the following

  • Opens the device file to the VME kernel driver
  • Opens default VME windows to PCI space.
  • Maps those PCI space windows to Virtual Memory (available to current process)
  • Maps VME bridge registers to Virtual Memory
  • Creates a Shared Mutex for locking access to the VME Bus (if not already created).

Because virtual memory is allocated, the programs must make a call to

int  vmeCloseDefaultWindows();

to release this memory before it exits. The VME windows to PCI space will always remain open after the first call to vmeOpenDefaultWindows(). What will change, is the map of this PCI memory to Virtual Memory for each executed process.

Example: Acquiring Virtual Memory address of a VME module

After vmeOpenDefaultWindows() has been successfully called, one can gain access to a VME module's registers with

int  vmeBusToLocalAdrs(int vmeAdrsSpace, char *vmeBusAdrs, char **pLocalAdrs);

I have a VME TIR device that responds to A16 (Address Modifier = 0x29) at VME address 0x0ed0. To get its Virtual Memory address:

unsigned int vmeAdr = 0x0ed0;
unsigned int lAdr;
int stat;

stat = vmeBusToLocalAdrs(0x29,(char *)vmeAddr,(char **)&lAdr);

if (stat != 0)

  printf("tirInit: ERROR: Error in vmeBusToLocalAdrs res=%d \n",stat);
  printf("TIR VME (USER) address = 0x%.8x (0x%.8x)\n",vmeAdr,lAdr);

Here... USER address refers to the Virtual Memory address of the TIR that is available to the User Process. This is very similar to what is done to get a local address from the VME address, in vxWorks.

Routine changes

We've attempted to ease the transition with respect to the change from a vxWorks controller to Linux controller by maintaining the basic names of the routines and keeping their argument structure in tact where possible. The table below shows the differences.


vxWorks Linux Notes
STATUS sysBusToLocalAdrs (int adrsSpace, char *busAdrs, char **pLocalAdrs); int vmeBusToLocalAdrs(int vmeAdrsSpace, char *vmeBusAdrs, char **pLocalAdrs);
int vxMemProbe(char *addr, int mode, int size, char *rval); int vmeMemProbe(char *addr, int size, char *rval); Linux, mode is read only
STATUS intConnect (VOIDFUNCPTR *vector, VOIDFUNCPTR routine, int parameter); int vmeIntConnect(unsigned int vector, unsigned int level, VOIDFUNCPTR routine, unsigned int arg);
int intDisconnect(int vector) int vmeIntDisconnect(unsigned int level); vector != level
int sysIntEnable(int); NONE
int sysIntDisable(int); NONE
int intLock(); NONE
void intUnlock(int key); NONE
STATUS usrVmeDmaConfig(unsigned int addrType, unsigned int dataType, unsigned int sstMode); int vmeDmaConfig(unsigned int addrType, unsigned int dataType, unsigned int sstMode);
STATUS sysVmeDmaSend (UINT32 locAdrs, UINT32 vmeAdrs, int size, BOOL toVme); int vmeDmaSend(unsigned int locAdrs, unsigned int vmeAdrs, int size); Linux, always from VME
int sysVmeDmaDone(int pcnt, int pflag); int vmeDmaDone(); vxWorks: returns size - (#bytes transferred)
Linux: returns (#bytes transferred)

Little-endian vs Big-endian

A very good discussion of endianness is described on the Wikipedia page. To summarize, vxWorks presents data to/from the VME bus as Little-Endian. Where Linux presents data to/from the VME bus as Big-Endian.

Because most of our applications and libraries were written assuming vxWorks, we have decided to keep this ordering and make the port to Linux using Little-Endian. To mitigate this, routines have been written to support byte-swapping when it is deemed necessary, at compilation.

  • For VME Writes:
void vmeWrite16(volatile unsigned short *addr, unsigned short val);
void vmeWrite32(volatile unsigned int *addr, unsigned int val);
  • For VME Reads:
unsigned short vmeRead16(volatile unsigned short *addr);
unsigned int vmeRead32(volatile unsigned int *addr) 

Example: Reading and Writing with "correct" byte swapping

In the tir library, we have defined a structure that will be mapped to a VME module's registers:

struct vme_tir
    volatile unsigned short tir_csr;
    volatile unsigned short tir_vec;
    volatile unsigned short tir_dat;
    volatile unsigned short tir_oport;
    volatile unsigned short tir_iport;

Once this structure has been mapped to the TIR, our original vxWorks library read from the csr register like this:

unsigned short read_csr;
read_csr = tirPtr->tir_csr;

To perform the same operation, and maintain its byte ordering regardless of operating system:

unsigned short read_csr;
read_csr = vmeRead16(&tirPtr->tir_csr);

Likewise for writing to VME registers, the original code:

tirPtr->tir_csr = 0x80; /* Reset the board */

must be changed to:

vmeWrite16(&tirPtr->tir_csr,0x80); /* Reset the board */


DMA (Block Transfers)

VME DMA (Block Transfers) transfers data from VME to the Linux Controller's physical memory (other types of DMA are possible, but we limit our description to this example). Before performing a DMA, allocate physical memory and map it into User space (virtual memory) with:

#define BUFFER_SIZE 2048
#define NBUFFERS 10
DMA_MEM_ID vmeIN  = dmaPCreate("vmeIN", BUFFER_SIZE, NBUFFERS, 0);
DMA_MEM_ID vmeOUT = dmaPCreate("vmeOUT", 0, 0, 0);

where BUFFER_SIZE is the size of an individual memory buffer (in bytes), and NBUFFERS is the number of buffers. Here, we are creating 10 2048 Byte buffers.vmeIN is our queue of buffers to transfer VME data to physical memory. When the transfer is completed, it is moved to the vmeOUT queue for use for further data processing or moving it along to a process that ships it off over the network. When vmeOUT is done with this buffer, it makes itself available back to thevmeIN queue. It is important to allocate memory in the configuration portion of data acquisition, as it is time consuming process. In the current implementation, the maximum size of BUFFER_SIZE is about 4MBytes.

Initialize the queues with


A view of these initialized buffers/queues is observed by executing the routine dmaPStatsAll()

Address    total  free   busy   size  incr  (KBytes)  Name
---------- -----  ----   ----  -----  ----  --------  ----
0xacf08000    10    10     0    2104     0  (21)      vmeIN
0xacf01000     0     0     0      56     0  (0)       vmeOUT

Where the size of each buffer is adjusted to account for the internal memory structures of the queue.

During a readout trigger, one of these buffers is obtained from the queue with:

extern DMANODE *the_event;
extern unsigned int *dma_dabufp;
unsigned int event_number = 1234; /* change this to whatever you want */

The GETEVENT macro pulls a buffer from the vmeIN queue and makes it available for DMA. event_number is placed within it's memory structure to provide a means for further identification (its up to the user on how to fill this variable). Access to the data buffer is available through the dma_dabufp pointer. It is similar to the rol->dabufp used with previous CODA Readout-Lists. Learn by example...

Add a word to the buffer:

*dma_dabufp++ = 0x12345678;

Or a byteswapped word:

*dma_dabufp++ = LSWAP(0x12345678); 

Performing the transfer

Once a buffer is available, use it to initiate a DMA:

int retVal = vmeDmaSend(dma_dabufp, VME_ADDR, MAX_BYTES);

If MAX_BYTES is greater than the amount of bytes left in the buffer, it will be truncated to the amount of bytes left in the buffer. retVal will be OK (0) if successful, otherwise ERROR (-1). The DMA is an asynchronous transfer that starts upon return of vmeDmaSend(...). You can wait for its completion with:

int retVal = vmeDmaDone();

retVal will be the number of bytes in the transfer, or ERROR (-1). If you want to add more data to the buffer, you should increment the dma_dabufp pointer by the amount of 32bit words in the transfer. I.e.

dma_dabufp += (retVal)>>2;

Make this buffer available to the vmeOUT queue with the macro:


Do something with the vmeOUT queue

/* Grab a buffer from the vmeOUT queue */
DMANODE *outEvent = dmaPGetItem(vmeOUT);
/* Get it's length, type, and event number */
int length       = outEvent->length;
int type         = outEvent->type;
int event_number = outEvent->nevent; 
/* Do something with the data... I'm just printing it to stdout */
int i;
for(i=0; i<length; i++)
   printf(" %d:  0x%08x\n", i, outEvent->data[i]);
/* put the buffer back into the vmeIN queue */

Queue cleanup

When done with the allocated memory.. it's best to clean up so it's not lost forever (or until reboot):


Different types of VME DMA

Anytime before performing a vmeDmaSend(...), you can configure the type of VME DMA with:

/* Setup Address and data modes for DMA transfers
*  vmeDmaConfig(addrType, dataType, sstMode);
*  addrType = 0 (A16)    1 (A24)    2 (A32)
*  dataType = 0 (D16)    1 (D32)    2 (BLK32) 3 (MBLK) 4 (2eVME) 5 (2eSST)
*  sstMode  = 0 (SST160) 1 (SST267) 2 (SST320)
vmeDmaConfig(2,5,1); /* Optimal DMA configuration for the FADC250 */

Library/Application Modifications

Header Files

Changes to Makefile

Running the CODA Linux ROC

The CODA Linux ROC executable is here:


And has the following command-line arguments:

-name:        Name of object
-type:        Type of object
-objects:     Name and type of this object
-msqld_host:  Name of host to connect to for msql access
-msqld_port:  TCP port to connect to for msql access
-et_filename: Filename of the ET system
-session:     Name of current Session
                Default value: "$SESSION"
-i:           Interactive mode
-w:           Startup Watchdog task
-q:           quiet mode
-r:           Error Recover mode
-f:           startup_file
-f0:          startup_file
-debug:       debug mode "data:api:"
Generic options for all commands:
-help:        Print summary of command-line options and abort

These options are identical to those used for the vxWorks ROC.

Example: Linux ROC execution

I have a Linux ROC to be run on dafarm42.jlab.org. It's ROC Name is "roc42" and I will run it in "interactive" mode. I execute:

$CODA_BIN/coda_roc_v3 -i -t ROC -name roc42

Any shared libraries (.so) that are used within the CODA readout list used for any CODA configuration, must exist within the current LD_LIBRARY_PATHenvironment variable.