This page provides some insight, instructions, and examples on how to migrate your module libraries and programs from the vxWorks operating system to the Linux operating system. We will assume that the use of the Intel-Based VME controller will be for taking data with CODA, or parallel with CODA.
- 1 Basic differences with respect to VME access
- 2 DMA (Block Transfers)
- 3 Library/Application Modifications
- 4 Changes to Makefile
- 5 Running the CODA Linux ROC
In vxWorks, the VME windows (the map between various VME address spaces and PCI space) is created at boot. In Linux, this is done at program execution with the routine:
This routine does the following
- Opens the device file to the VME kernel driver
- Opens default VME windows to PCI space.
- Maps those PCI space windows to Virtual Memory (available to current process)
- Maps VME bridge registers to Virtual Memory
- Creates a Shared Mutex for locking access to the VME Bus (if not already created).
Because virtual memory is allocated, the programs must make a call to
to release this memory before it exits. The VME windows to PCI space will always remain open after the first call to vmeOpenDefaultWindows(). What will change, is the map of this PCI memory to Virtual Memory for each executed process.
After vmeOpenDefaultWindows() has been successfully called, one can gain access to a VME module's registers with
int vmeBusToLocalAdrs(int vmeAdrsSpace, char *vmeBusAdrs, char **pLocalAdrs);
I have a VME TIR device that responds to A16 (Address Modifier = 0x29) at VME address 0x0ed0. To get its Virtual Memory address:
unsigned int vmeAdr = 0x0ed0;
unsigned int lAdr;
stat = vmeBusToLocalAdrs(0x29,(char *)vmeAddr,(char **)&lAdr);
if (stat != 0)
printf("tirInit: ERROR: Error in vmeBusToLocalAdrs res=%d \n",stat);
printf("TIR VME (USER) address = 0x%.8x (0x%.8x)\n",vmeAdr,lAdr);
Here... USER address refers to the Virtual Memory address of the TIR that is available to the User Process. This is very similar to what is done to get a local address from the VME address, in vxWorks.
We've attempted to ease the transition with respect to the change from a vxWorks controller to Linux controller by maintaining the basic names of the routines and keeping their argument structure in tact where possible. The table below shows the differences.
|STATUS sysBusToLocalAdrs (int adrsSpace, char *busAdrs, char **pLocalAdrs);||int vmeBusToLocalAdrs(int vmeAdrsSpace, char *vmeBusAdrs, char **pLocalAdrs);|
|int vxMemProbe(char *addr, int mode, int size, char *rval);||int vmeMemProbe(char *addr, int size, char *rval);||Linux, mode is read only|
|STATUS intConnect (VOIDFUNCPTR *vector, VOIDFUNCPTR routine, int parameter);||int vmeIntConnect(unsigned int vector, unsigned int level, VOIDFUNCPTR routine, unsigned int arg);|
|int intDisconnect(int vector)||int vmeIntDisconnect(unsigned int level);||vector != level|
|void intUnlock(int key);||NONE|
|STATUS usrVmeDmaConfig(unsigned int addrType, unsigned int dataType, unsigned int sstMode);||int vmeDmaConfig(unsigned int addrType, unsigned int dataType, unsigned int sstMode);|
|STATUS sysVmeDmaSend (UINT32 locAdrs, UINT32 vmeAdrs, int size, BOOL toVme);||int vmeDmaSend(unsigned int locAdrs, unsigned int vmeAdrs, int size);||Linux, always from VME|
|int sysVmeDmaDone(int pcnt, int pflag);||int vmeDmaDone();||vxWorks: returns size - (#bytes transferred)
Linux: returns (#bytes transferred)
A very good discussion of endianness is described on the Wikipedia page. To summarize, vxWorks presents data to/from the VME bus as Little-Endian. Where Linux presents data to/from the VME bus as Big-Endian.
Because most of our applications and libraries were written assuming vxWorks, we have decided to keep this ordering and make the port to Linux using Little-Endian. To mitigate this, routines have been written to support byte-swapping when it is deemed necessary, at compilation.
- For VME Writes:
void vmeWrite16(volatile unsigned short *addr, unsigned short val);
void vmeWrite32(volatile unsigned int *addr, unsigned int val);
- For VME Reads:
unsigned short vmeRead16(volatile unsigned short *addr);
unsigned int vmeRead32(volatile unsigned int *addr)
In the tir library, we have defined a structure that will be mapped to a VME module's registers:
volatile unsigned short tir_csr;
volatile unsigned short tir_vec;
volatile unsigned short tir_dat;
volatile unsigned short tir_oport;
volatile unsigned short tir_iport;
Once this structure has been mapped to the TIR, our original vxWorks library read from the csr register like this:
unsigned short read_csr;
read_csr = tirPtr->tir_csr;
To perform the same operation, and maintain its byte ordering regardless of operating system:
unsigned short read_csr;
read_csr = vmeRead16(&tirPtr->tir_csr);
Likewise for writing to VME registers, the original code:
tirPtr->tir_csr = 0x80; /* Reset the board */
must be changed to:
vmeWrite16(&tirPtr->tir_csr,0x80); /* Reset the board */
VME DMA (Block Transfers) transfers data from VME to the Linux Controller's physical memory (other types of DMA are possible, but we limit our description to this example). Before performing a DMA, allocate physical memory and map it into User space (virtual memory) with:
#define BUFFER_SIZE 2048
#define NBUFFERS 10
DMA_MEM_ID vmeIN = dmaPCreate("vmeIN", BUFFER_SIZE, NBUFFERS, 0);
DMA_MEM_ID vmeOUT = dmaPCreate("vmeOUT", 0, 0, 0);
where BUFFER_SIZE is the size of an individual memory buffer (in bytes), and NBUFFERS is the number of buffers. Here, we are creating 10 2048 Byte buffers.vmeIN is our queue of buffers to transfer VME data to physical memory. When the transfer is completed, it is moved to the vmeOUT queue for use for further data processing or moving it along to a process that ships it off over the network. When vmeOUT is done with this buffer, it makes itself available back to thevmeIN queue. It is important to allocate memory in the configuration portion of data acquisition, as it is time consuming process. In the current implementation, the maximum size of BUFFER_SIZE is about 4MBytes.
Initialize the queues with
A view of these initialized buffers/queues is observed by executing the routine dmaPStatsAll()
Address total free busy size incr (KBytes) Name
---------- ----- ---- ---- ----- ---- -------- ----
0xacf08000 10 10 0 2104 0 (21) vmeIN
0xacf01000 0 0 0 56 0 (0) vmeOUT
Where the size of each buffer is adjusted to account for the internal memory structures of the queue.
During a readout trigger, one of these buffers is obtained from the queue with:
extern DMANODE *the_event;
extern unsigned int *dma_dabufp;
unsigned int event_number = 1234; /* change this to whatever you want */
The GETEVENT macro pulls a buffer from the vmeIN queue and makes it available for DMA. event_number is placed within it's memory structure to provide a means for further identification (its up to the user on how to fill this variable). Access to the data buffer is available through the dma_dabufp pointer. It is similar to the rol->dabufp used with previous CODA Readout-Lists. Learn by example...
Add a word to the buffer:
*dma_dabufp++ = 0x12345678;
Or a byteswapped word:
*dma_dabufp++ = LSWAP(0x12345678);
Once a buffer is available, use it to initiate a DMA:
int retVal = vmeDmaSend(dma_dabufp, VME_ADDR, MAX_BYTES);
If MAX_BYTES is greater than the amount of bytes left in the buffer, it will be truncated to the amount of bytes left in the buffer. retVal will be OK (0) if successful, otherwise ERROR (-1). The DMA is an asynchronous transfer that starts upon return of vmeDmaSend(...). You can wait for its completion with:
int retVal = vmeDmaDone();
retVal will be the number of bytes in the transfer, or ERROR (-1). If you want to add more data to the buffer, you should increment the dma_dabufp pointer by the amount of 32bit words in the transfer. I.e.
dma_dabufp += (retVal)>>2;
Make this buffer available to the vmeOUT queue with the macro:
DMANODE *outEvent = dmaPGetItem(vmeOUT);
int length = outEvent->length;
int type = outEvent->type;
int event_number = outEvent->nevent;
for(i=0; i<length; i++)
printf(" %d: 0x%08x\n", i, outEvent->data[i]);
When done with the allocated memory.. it's best to clean up so it's not lost forever (or until reboot):
Anytime before performing a vmeDmaSend(...), you can configure the type of VME DMA with:
/* Setup Address and data modes for DMA transfers
* vmeDmaConfig(addrType, dataType, sstMode);
* addrType = 0 (A16) 1 (A24) 2 (A32)
* dataType = 0 (D16) 1 (D32) 2 (BLK32) 3 (MBLK) 4 (2eVME) 5 (2eSST)
* sstMode = 0 (SST160) 1 (SST267) 2 (SST320)
vmeDmaConfig(2,5,1); /* Optimal DMA configuration for the FADC250 */
The CODA Linux ROC executable is here:
And has the following command-line arguments:
-name: Name of object
-type: Type of object
-objects: Name and type of this object
-msqld_host: Name of host to connect to for msql access
-msqld_port: TCP port to connect to for msql access
-et_filename: Filename of the ET system
-session: Name of current Session
Default value: "$SESSION"
-i: Interactive mode
-w: Startup Watchdog task
-q: quiet mode
-r: Error Recover mode
-debug: debug mode "data:api:"
Generic options for all commands:
-help: Print summary of command-line options and abort
These options are identical to those used for the vxWorks ROC.
I have a Linux ROC to be run on dafarm42.jlab.org. It's ROC Name is "roc42" and I will run it in "interactive" mode. I execute:
$CODA_BIN/coda_roc_v3 -i -t ROC -name roc42
Any shared libraries (.so) that are used within the CODA readout list used for any CODA configuration, must exist within the current LD_LIBRARY_PATHenvironment variable.