Radboud University Nijmegen

Real-time Linux (Xenomai)

Exercise #1: The Basics of Real-Time Linux

Real-Time Linux Introduction

Linux is a free Unix-like operating system that runs on a variety of platforms, including PCs. Numerous Linux distributions such as Red Hat, Debian and Mandrake bundle the Linux OS with tools, productivity software, games, etc.

  • The Linux scheduler, like that of other OSes such as Windows or MacOS, is designed for best average response, so it feels fast and interactive even when running many programs.
    • However, it doesn't guarantee that any particular task will always run by a given deadline. A task may be suspended for an arbitrarily long time, for example while a Linux device driver services a disk interrupt.
    • Scheduling guarantees are offered by real-time operating systems (RTOSes), such as QNX, LynxOS or VxWorks. RTOSes are typically used for control or communications applications, not for general purpose computing.
  • Linux has been adapted for real-time support. These adaptations are termed "Real-Time Linux" (RT Linux).
  • Numerous versions of RT Linux are available, free or commercial. Three commonly available free RT Linux versions are
    • RTLinux, developed by New Mexico Tech and now maintained by Wind River Systems.
    • RTAI, The Real-Time Application Interface (RTAI), developed by the Milan Polytechnical University and available at https://www.rtai.org/.
    • Xenomai, a spin off of  RTAI Linux. While RTAI is focused on lowest technically feasible latencies, Xenomai also considers clean extensibility (RTOS skins), portability, and maintainability as very important goals. It is available at www.xenomai.org.
  • These RT Linux systems are patches to the basic Linux kernel source code. Instructions for building an RT Linux system from a the Linux source code are provided with these RT Linux systems. Briefly, this involves setting up the basic Linux system, getting the latest Linux kernel source code from www.kernel.org, patching the kernel source code, and compiling the patched kernel.

Objectives

The following are the primary objectives of this exercise:

  • To learn how Xenomai real-time Linux works and to build a small Xenomai real-time task.

 

How does RT Linux work?

The general idea of RT Linux is that a small real-time kernel runs beneath Linux, meaning that the real-time kernel has a higher priority than the Linux kernel. Real-time tasks are executed by the real-time kernel, and normal Linux programs are allowed to run when no real-time tasks have to be executed. Linux can be considered as the idle task of the real-time scheduler. When this idle task runs, it executes its own scheduler and schedules the normal Linux processes.  Since the real-time kernel has a higher priority, a normal Linux process is preempted when a real-time task becomes ready to run and the real-time task is executed immediately.

How is the real-time kernel given higher priority than Linux kernel?

Basically, an operating system is driven by interrupts, which can be considered as the heartbeats of a computer:

  • All programs running in an OS are scheduled by a scheduler which is driven by timer interrupts of a clock to reschedule at certain times.
  • An executing program can block or voluntary give up the CPU in which case the scheduler is informed by means of a software interrupt (system call).
  • Hardware can generate interrupts to interrupt the normal scheduled work of the OS for fast handling of hardware.

RT Linux uses the flow of interrupts to give the real-time kernel a higher priority than the Linux kernel:

  • When an interrupt arrives, it is first given to the real-time kernel, and not to the Linux kernel. But interrupts are stored to give them later to Linux when the real-time kernel is done.
  • As first in row, the real-time kernel can run its real-time tasks driven by these interrupts.
  • Only when the real-time kernel is not running anything, the interrupts which were stored are passed on to the Linux kernel.
  • As second in row, Linux can schedule its own processes driven by these interrupt.

Hence, when a normal Linux program runs and a new interrupt arrives:

  • it is first handled by an interrupt handler set by the real-time kernel;
  • the code in the interrupt handler awakes a real-time task;
  • immediately after the interrupt handler, the real-time scheduler is called ;
  • the real-time scheduler observes that another real-time task is ready to run, so it puts the Linux kernel to sleep, and awakes the real-time task.
  • Hence, to the real-time kernel and Linux kernel coexist on a single machine a special way of passing of the interrupts between real-time kernel and the Linux kernel is needed. Each flavor of RT Linux does this is in its own way. Xenomai uses an interrupt pipeline from the Adeos project. For more information, see also Life with Adeos.

RT Linux Tasks are not Linux Programs

  • In real-time Linux we can make a real-time program by programming real-time threads. None-real-time programs in real-time Linux just use the conventional Linux threads.
  • A real-time task can be executed in kernel space using a kernel module, but it can also be executed in user space using a normal C program.
  • Running in user space instead of in kernel space gives a little extra overhead, but with the following advantages :
    • A bug in a kernel module can crash the whole system, however a bug in a user space program can only crash the program.
    • In a kernel model one can only use the real-time API and the limited kernel API, but in a real-time user space program the real-time API and the whole Linux API can be used! However when using the Linux API in user space, the program cannot be scheduled by the real-time scheduler (HARD real-time) but must be scheduled by the Linux scheduler (SOFT real-time). So calling a Linux API call from a real-time user space task will degrade its performance from HARD to SOFT real-time. After the call the task will return to the real-time scheduler.

Xenomai

  • The Xenomai project was launched in August 2001.
  • Xenomai is based on an abstract RTOS core, usable for building any kind of real-time interfaces, over a nucleus which exports a set of generic RTOS services. Any number of RTOS personalities called “skins” can then be built over the nucleus, providing their own specific interface to the applications, by using the services of a single generic core to implement it.
  • The following skins on the generic core are implemented :
  • Xenomai allows to run real-time threads either strictly in kernel space, or within the address space of a Linux process. A real-time task in user space still has the benefit of memory protection, but is scheduled by Xenomai directly, and no longer by the Linux kernel. The worst case scheduling latency of such kind of task is always near to the hardware limits and predictable, since Xenomai is not bound to synchronizing with the Linux kernel activity in such a context, and can preempt any regular Linux activity with no delay. Hence, he preferred execution environment for Xenomai applications is user space context.
  • But there might be a few cases where running some of the real-time code embodied into kernel modules is required, especially with legacy systems or very low-end platforms with under-performing MMU hardware. For this reason, Xenomai's native API provides the same set of real-time services in a seamless manner to applications, regardless of their execution space. Additionally, some applications may need real-time activities in both spaces to cooperate, therefore special care has been taken to allow the latter to work on the exact same set of API objects.
  • In our terminology, the terms "thread" and "task" have the same meaning. When talking about a Xenomai task we refer to real-time task in user space, i.e., within the address space of a Linux process, not to be confused with regular Linux task/thread.

Xenomai API

  • In our exercise we will use the native Xenomai skin and run real-time tasks in user space because working with them is easier and still HARD real-time.  
  • The Native Xenomai API is flexible and well documented.
  • The most important characteristics of the Xenomai API are :
    • Context independence

Xenomai's native API provides the same set of real-time services in a seamless manner to application regardless of their execution mode is user or kernel mode. 

    • Seamless topology

Xenomai objects (eg. a semaphore, task, etc..) are always reachable even when the location of the object is another execution space than were the task using it is running. e.g. user space task running in user space can use a semaphore which is stored in kernel space.
To this end, each category of services in Xenomai's native API defines a uniform descriptor type to represent the objects it manages. For instance, a task will always be represented by a RT_TASK descriptor, a message queue by a RT_QUEUE descriptor, and a semaphore by a RT_SEM descriptor, regardless of the execution space from which they are used.
Xenomai uses a unified registry, to index each object descriptor under a symbolic name given by the user. By doing this, Xenomai enables you to retrieve any descriptor associated with a registered object in any location of your application.

C is the Preferred Language

  • Linux is written in the C programming language (with some assembly language), as is Xenomai, so C is the preferred language for writing RT Linux programs.
  • Other languages (e.g., C++) can be used. For instance, see the Xeno-- project which is providing a thin object-oriented software interface to Xenomai’s C-based skins. 
  • The examples in this tutorial are all written in C.
  • C programs are normally compiled into full executable programs
  • In C, a program's "entry point" where execution begins is a function called 'main()'.
  • Each real-time task is associated with a C function that is called when the task is scheduled to run. 

Creating a task

  • When you create a real-time task in Xenomai the RT_TASK structure is used as the descriptor to refer to this task.
  • An RT_TASK data structure is used to hold all the information about a task:
    • the task function to be executed by the real-time task,
    • any initial argument passed to it,
    • the size of the stack allocated for its variables,
    • its priority,
    • whether or not it uses floating-point math,
    • and a "signal handler" that will be called when the task becomes active.
  • The task is created by calling
int rt_task_create (RT_TASK *task, const char *name, int stack_size, int priority, int mode)
    • 'task' is a pointer to an RT_TASK type structure which must have been declared before and whose structure is filled.
    • 'name'  is an  ASCII string standing for the symbolic name of the task. This symbolic name you can use to retrieve the task structure anywhere else in your code by calling the rt_task_bind() function.
    • 'stack_size' is the size of the stack to be used by the new task.
    • 'priority' is the priority to be given the task. The highest priority is 99, while the lowest is 1.
    • 'mode' is a set of flags which affect the task :
      • T_FPU allows the task to use the FPU (Floating Point Unit) whenever available on the platform. This flag is forced for user-space tasks.
      • T_SUSP causes the task to start in suspended mode. In such a case, the thread will have to be explicitly resumed using the rt_task_resume() service for its execution to actually begin.
      • T_CPU(cpuid) specifies that the new task runs on the CPU with number cpuid. CPU identifiers range from 0 to RTHAL_NR_CPUS - 1 (inclusive).
      • T_JOINABLE (user-space only) allows another task to wait on the termination of the new task. This implies that rt_task_join() is actually called for this task to clean up any user-space located resources after its termination.

Starting a task

  • A task can be started by calling :
int rt_task_start (RT_TASK *task, void(*task_func)(void *arg), void *arg)
    • 'task' is a pointer to an RT_TASK type structure which must be already initialized by a call to rt_task_create().
    • 'task_function' is the the task function to be executed by this real-time task.
    • 'arg' is the void pointer argument given to the task function.

If you did specify T_SUSP as a mode flag to rt_task_create(), the task will be started in a suspended mode. In that case you  have to explicitly resume the task with  rt_task_resume(). Otherwise, the real time task is ready to execute immediately.

Hello World

The following "ex01.c" C code illustrates the creating and start of a "demo" task which writes a message.

#include <stdio.h>

#include <signal.h>

#include <unistd.h>

#include <sys/mman.h>

 

#include <native/task.h>

#include <native/timer.h>

 

#include  <rtdk.h>

RT_TASK demo_task;

 

void demo(void *arg)

{

  RT_TASK *curtask;

  RT_TASK_INFO curtaskinfo;

 

  // hello world

  rt_printf("Hello World!\n");

 

  // inquire current task

  curtask=rt_task_self();

  rt_task_inquire(curtask,&curtaskinfo);

 

  // print task name

  rt_printf("Task name : %s \n", curtaskinfo.name);

}

 

int main(int argc, char* argv[])

{

  char  str[10] ;

 

  // Perform auto-init of rt_print buffers if the task doesn't do so

  rt_print_auto_init(1);

 

  // Lock memory : avoid memory swapping for this program

  mlockall(MCL_CURRENT|MCL_FUTURE);

 

  rt_printf("start task\n");

 

  /*

   * Arguments: &task,

   *            name,

   *            stack size (0=default),

   *            priority,

   *            mode (FPU, start suspended, ...)

   */

  sprintf(str,"hello");

  rt_task_create(&demo_task, str, 0, 50, 0);

 

  /*

   * Arguments: &task,

   *            task function,

   *            function argument

   */

  rt_task_start(&demo_task, &demo, 0);

}

 

In the program above, memory is locked by calling 

    mlockall(MCL_CURRENT|MCL_FUTURE);

so that it cannot be swapped to disk. Preventing swapping is a must when real-time performance must be reached. Hence, Xenomai generates a warning when this is forgotten when its API is called.

The rdtk real-time printing library

Note that in the above code we used the command 'rt_printf()' instead of the familiar 'printf()' command. We did not use 'printf()' because it uses Linux syscalls and therefore terribly slows down the real-time thread. Instead we use the 'rt_printf' command  from the rdtk real-time printing  library. This is a stand-alone real-time library for printf services. It is embedded into the Xenomai user space part but actually doesn't depend on any Xenomai service, just using the plain POSIX API. The librtprint API looks much like the printf(3) man page:

  • rt_vfprintf
  • rt_vprintf
  • rt_fprintf
  • rt_printf

The basic idea of librdtk is to keep the print side as cheap as possible: no locks, no syscalls at all. Each thread that uses rt_printf & friends has its own local ring buffer. A central (per process) non-RT output thread takes care of forwarding the content of all thread ring buffers to the output streams. The message order is preserved loosely by a global sequence counter (arch-dependent CPU-wide or system-wide atomic_inc_return might be used in the future). The output thread uses periodic polling (default: 10 Hz) to avoid the need for special signalling mechanisms.
To start the non-RT output thread one always has to call the following at the beginning of a Xenomai programma :

rt_print_auto_init(1);

This performs auto-init of the rt_print buffers and start the non-RT output thread.

Compiling a Xenomai program

To compile the above program, first obtain the CFLAGS and LDFLAGS by running  

> xeno-config --xeno-cflags

and 

> xeno-config --xeno-ldflags

To set these flags, use 

> export CFLAGS=`xeno-config --xeno-cflags`

and 

> export LDFLAGS=`xeno-config --xeno-ldflags`

Finally, compile the ex01.c program against the 'native' library implementing the Xenomai Native API and the 'rtdk' print library 

> gcc $CFLAGS $LDFLAGS  -lnative -lrtdk  ex01.c -o ex01

By default the executable 'ex01' is linked against the dynamic libraries in /usr/xenomai/lib. Because these are special and not known to linux, we first have to tell Linux where to find them by setting a library path environment variable :

> export LD_LIBRARY_PATH=/usr/xenomai/lib

Than run the program :

> ./ex01

Exercises

Exercise 1a.

Compile and run program ex01.c to get some basic experience in working with Xenomai tasks. First, try a manual build by compiling with the commands above. Next, try an automated the build with this Makefile (using commands "make" and "./ex01").

Exercise 1b.

Introduce an error by commenting out the assignment to the current task as follows:

//curtask=rt_task_self();

Save the resulting program as ex01b.c Build and run this program, observe the result.

Next introduce a return value for the 'rt_task_inquire' call and print the error code as described in the part about "Error handling" of the Tips page on the Xenomai pages of Robot Lab site.

Build and run the new version of the program.

Last Updated: 22 August 2011 (Jozef Hooman)
Created by: Harco Kuppens
h.kuppens@cs.ru.nl