VxWorks to Linux HOWTO

This page provides some insight, instructions, and examples on how to migrate your module libraries and programs from the vxWorks operating system to the Linux operating system. We will assume that the use of the Intel-Based VME controller will be for taking data with CODA, or parallel with CODA.

Contents

• 1 Basic differences with respect to VME access
  • 1.1 Access to VME address spaces
    • 1.1.1 Example: Acquiring Virtual Memory address of a VME module
  • 1.2 Routine changes
  • 1.3 Little-endian vs Big-endian
    • 1.3.1 Example: Reading and Writing with "correct" byte swapping
• 2 DMA (Block Transfers)
  • 2.1 Performing the transfer
  • 2.2 Do something with the vmeOUT queue
  • 2.3 Queue cleanup
  • 2.4 Different types of VME DMA
• 3 Library/Application Modifications
  • 3.1 Header Files
• 4 Changes to Makefile
• 5 Running the CODA Linux ROC
  • 5.1 Example: Linux ROC execution

Basic differences with respect to VME access

Access to VME address spaces

In vxWorks, the VME windows (the map between various VME address spaces and PCI space) is created at boot. In Linux, this is done at program execution with the routine:

This routine does the following

• Opens the device file to the VME kernel driver

• Opens default VME windows to PCI space.
• Maps those PCI space windows to Virtual Memory (available to current process)
• Maps VME bridge registers to Virtual Memory
• Creates a Shared Mutex for locking access to the VME Bus (if not already created).

Because virtual memory is allocated, the programs must make a call to

to release this memory before it exits. The VME windows to PCI space will always remain open after the first call to vmeOpenDefaultWindows(). What will change, is the map of this PCI memory to Virtual Memory for each executed process.

Example: Acquiring Virtual Memory address of a VME module

After vmeOpenDefaultWindows() has been successfully called, one can gain access to a VME module's registers with

I have a VME TIR device that responds to A16 (Address Modifier = 0x29) at VME address 0x0ed0. To get its Virtual Memory address:

unsigned int vmeAdr = 0x0ed0;
unsigned int lAdr;
int stat;

stat = vmeBusToLocalAdrs(0x29,(char *)vmeAddr,(char **)&lAdr);

if (stat != 0)

{
  printf("tirInit: ERROR: Error in vmeBusToLocalAdrs res=%d \n",stat);
}
else
{
  printf("TIR VME (USER) address = 0x%.8x (0x%.8x)\n",vmeAdr,lAdr);
}

Here... USER address refers to the Virtual Memory address of the TIR that is available to the User Process. This is very similar to what is done to get a local address from the VME address, in vxWorks.

Routine changes

We've attempted to ease the transition with respect to the change from a vxWorks controller to Linux controller by maintaining the basic names of the routines and keeping their argument structure in tact where possible. The table below shows the differences.

Little-endian vs Big-endian

A very good discussion of endianness is described on the Wikipedia page. To summarize, vxWorks presents data to/from the VME bus as Little-Endian. Where Linux presents data to/from the VME bus as Big-Endian.

Because most of our applications and libraries were written assuming vxWorks, we have decided to keep this ordering and make the port to Linux using Little-Endian. To mitigate this, routines have been written to support byte-swapping when it is deemed necessary, at compilation.

• For VME Writes:

• For VME Reads:

Example: Reading and Writing with "correct" byte swapping

In the tir library, we have defined a structure that will be mapped to a VME module's registers:

Once this structure has been mapped to the TIR, our original vxWorks library read from the csr register like this:

To perform the same operation, and maintain its byte ordering regardless of operating system:

Likewise for writing to VME registers, the original code:

tirPtr->tir_csr = 0x80; /* Reset the board */

must be changed to:

vmeWrite16(&tirPtr->tir_csr,0x80); /* Reset the board */

DMA (Block Transfers)

VME DMA (Block Transfers) transfers data from VME to the Linux Controller's physical memory (other types of DMA are possible, but we limit our description to this example). Before performing a DMA, allocate physical memory and map it into User space (virtual memory) with:

where BUFFER_SIZE is the size of an individual memory buffer (in bytes), and NBUFFERS is the number of buffers. Here, we are creating 10 2048 Byte buffers.vmeIN is our queue of buffers to transfer VME data to physical memory. When the transfer is completed, it is moved to the vmeOUT queue for use for further data processing or moving it along to a process that ships it off over the network. When vmeOUT is done with this buffer, it makes itself available back to thevmeIN queue. It is important to allocate memory in the configuration portion of data acquisition, as it is time consuming process. In the current implementation, the maximum size of BUFFER_SIZE is about 4MBytes.

Initialize the queues with

A view of these initialized buffers/queues is observed by executing the routine dmaPStatsAll()

Where the size of each buffer is adjusted to account for the internal memory structures of the queue.

During a readout trigger, one of these buffers is obtained from the queue with:

The GETEVENT macro pulls a buffer from the vmeIN queue and makes it available for DMA. event_number is placed within it's memory structure to provide a means for further identification (its up to the user on how to fill this variable). Access to the data buffer is available through the dma_dabufp pointer. It is similar to the rol->dabufp used with previous CODA Readout-Lists. Learn by example...

Add a word to the buffer:

Or a byteswapped word:

Performing the transfer

Once a buffer is available, use it to initiate a DMA:

If MAX_BYTES is greater than the amount of bytes left in the buffer, it will be truncated to the amount of bytes left in the buffer. retVal will be OK (0) if successful, otherwise ERROR (-1). The DMA is an asynchronous transfer that starts upon return of vmeDmaSend(...). You can wait for its completion with:

retVal will be the number of bytes in the transfer, or ERROR (-1). If you want to add more data to the buffer, you should increment the dma_dabufp pointer by the amount of 32bit words in the transfer. I.e.

Make this buffer available to the vmeOUT queue with the macro:

Do something with the vmeOUT queue

Queue cleanup

When done with the allocated memory.. it's best to clean up so it's not lost forever (or until reboot):

Different types of VME DMA

Anytime before performing a vmeDmaSend(...), you can configure the type of VME DMA with:

Library/Application Modifications

Header Files

Changes to Makefile

Running the CODA Linux ROC

The CODA Linux ROC executable is here:

And has the following command-line arguments:

These options are identical to those used for the vxWorks ROC.

Example: Linux ROC execution

I have a Linux ROC to be run on dafarm42.jlab.org. It's ROC Name is "roc42" and I will run it in "interactive" mode. I execute:

Any shared libraries (.so) that are used within the CODA readout list used for any CODA configuration, must exist within the current LD_LIBRARY_PATHenvironment variable.