When Linux is mentioned, usually General Purpose Operating System (GPOS) is related to it, but over time, the need to have the same benefits as Real-Time Operating System (RTOS) for Linux has become more stringent. The challenge for any real-time system is to meet the given timing constrains in spite of the number and type of random asynchronous events. This is no simple task and an extensive number of papers and researches were done on theory of the real-time systems. Another challenge for a real-time system would be to have an upper limit on latency, called a scheduling deadline. Depending on how systems meet this challenge, they can be split into hard, firm, and soft:
- Hard real-time system: This represents system for which a deadline miss will result in a complete system failure.
- Firm real-time system: This represents systems for which a deadline miss is acceptable but the system quality can be degraded. Also, after the deadline is missed, the result that is offered is not useful anymore.
- Soft real-time system: This represents systems for which missing of deadlines degrades the usefulness of the received result and consequently, of the quality of the system. In these kind of systems, the meeting of the deadline is seen as a goal than as a strict requirement.
There are multiple reasons for Linux not being suitable as a RTOS:
- Paging: The page swap process through virtual memory is without limits. There is no method in place to know the time that will pass until you can get a page from a disk, and this implies that there is no upper limit to the delay caused by the fault in a page.
- Coarsed-grained synchronization: Here, the definition of the Linux kernel is not preemptible. This means that once a process is inside the kernel context, it cannot be preempted until it exits the context. At an event occurrence, the new event needs to wait for scheduling until the already available one exits the kernel context.
- Batching: An operation can be batched for a more efficient use of resources. The simplest example of this is the page freeing process. Instead of freeing each separate page, Linux is able to pass multiple pages and clean as many as possible.
- Request reordering: The I/O requests can be reordered for processes, making the process of using hardware more efficient.
- Fairness in scheduling: This is a UNIX heritage and refers to the fact that a scheduler tries to be fair with all running processes. This property offers the possibility of lower priority processes that have been waiting for a long time to be scheduled before higher priority ones.
The first thing anyone can do to improve the latency of the standard Linux operating system would be to try and make a change to the scheduling policies. The default Linux time sharing scheduling policies are called SCHED_OTHER, and they use a fairness algorithm, giving all processes zero priority, the lowest one available. Other such scheduling policies are SCHED_BATCH for batch scheduling of the processes and the SCHED_IDLE, which is suitable for the scheduling of extremely low priority jobs. The alternatives to this scheduling policy are SCHED_FIFO and SCHED_RR. Both of them are intended as real-time policies and are time-critical applications that require precise control processes and their latencies.
The other available implementation is interrupt abstraction. This approach is based on the fact that not all systems require a hard real-time determinism and most of them only require a section of their task to be executed in a real-time context. The idea behind this approach is to run Linux with the priority of an idle task under a real-time kernel and non-real-time tasks to continue to execute them as they normally do. This implementation fakes the disabling of an interrupt for the real-time kernel, but in fact, it is passed to the real-time kernel. For this type of implementation, there are three available solutions:
- RTLinux: It represents the original implementation of the interrupt abstraction approach and was developed at the Institute of Mining and Technology, New Mexico. Although it still has an open source implementation, most of the development is now done through FSMLabs engineers, later required by the Wind River System on the commercial version of it. The commercial support for RTLinux ended in August 2011.
- RTAI: It is an enhancement made to the RTLinux solution developed in the department of Aerospace Engineering from the Politecnico di Milano. This project is a very active with a high number of developers and has current releases available.
- Xenomai: It represents the third implementation. It's history is a bit twisted: it appeared in August 2001, only to be merged with RTAI in 2013 to generate a real-time operating system that was fit for production. However, the fusion was dispersed in 2005 and it became an independent project again.
The following diagram presents a basic RTLinux architecture.
A similar architecture, as shown in the preceding diagram, applies to the two other solutions since both of them were born from the RTLinux implementation. The difference between them is at the implementation level and each offers various benefits.
The PREEMPT_RT patches are the first option for every developer when a real-time solution is required. For some developers, the PREEMPT_RT patches transform Linux into a real-time solution suitable for their needs. This solution could not replace a real-time operation system, but is, in fact, suitable for a large number of systems.
The PREEMPT_RT patch transforms Linux from a general purpose operating system into a preemptible one using the following tricks:
- Protecting critical sections with the preemptible
rwlock_t preemptibleandspinlock_t. The use of the old solutions is still available usingraw_spinlock_t, which shares the same API asspinlock_t. - The kernel locking mechanisms is preempted by using
rtmutexes. - A priority inversion and priority inheritance mechanism is implemented for
mutexes,spinlocksandrw_semaphores. - Converting the available Linux timer API into one with a high resolution timer that offers the possibility of having timeouts.
- Implementing the usage of kernel threads for interrupt handlers. The real-time preemption patch treats soft interrupt handlers into the kernel thread context using a
task_structlike structure for every user space process. There is also the possibility of registering an IRQ into the kernel context.
Before moving to the actual configuration part, you should download a suitable version for the kernel. The best inspiration source is https://www.kernel.org/, which should be the starting point because it does not contain any extra patches. After the source code is received, the corresponding rt patches version can be downloaded from https://www.kernel.org/pub/linux/kernel/projects/rt/. The kernel version chosen for this demonstration is the 3.12 kernel version, but if any other kernel version is required, the same steps can be taken with a similar end result. The development of the real-time preemption patches is very active, so any missing version support is covered very fast. Also, for other sublevel versions, the patches can be found in the incr or older subdirectories of that particular kernel version. The following is the example for sublevel versions:
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.12.38.tar.xz wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.12/patch-3.12.38-rt52.patch.gz
After the source code is received, the sources need to be unpacked and the patches applied:
tar xf linux-3.12.38.tar.xz cd linux-3.12.38/ gzip -cd ../patch-3.12.38-rt52.patch.gz | patch -p1
The next step involves the configuration of the kernel sources. The configuration differs from one architecture to another, but the general idea remains. The following configurations are required for a QEMU ARM machine supported inside Poky. To enable the PREEMPT_RT support for a machine, there are multiple options available. You can implement a low-latency support version, which is most suitable for a desktop computer using a kernel configuration fragment similar to this:
CONFIG_GENERIC_LOCKBREAK=y CONFIG_TREE_PREEMPT_RCU=y CONFIG_PREEMPT_RCU=y CONFIG_UNINLINE_SPIN_UNLOCK=y CONFIG_PREEMPT=y CONFIG_PREEMPT__LL=y CONFIG_PREEMPT_COUNT=y CONFIG_DEBUG_PREEMPT=y CONFIG_RCU_CPU_STALL_VERBOSE=y
This option is one of the most often used and it also constitutes the primary source of usage of the PREEMPT_RT patches. The alternative of this would be to enable the fully preemptive support for the PREEMPT_RT patches using a configuration similar to this:
CONFIG_PREEMPT_RT_FULL=y CONFIG_HZ_1000=y CONFIG_HZ=1000
In the following image, the CONFIG_PREEMPT_RCU and CONFIG_PREEMPT_RT_FULL configurations are available. More information related to RCU is available at https://lwn.net/Articles/262464/.
The third image contains the CONFIG_PREEMPT__LL configuration. Another interesting configuration is CONFIG_PREEMPT_VOLUNTARY, which also reduces the latency along with the CONFIG_PREEMPT__LL configuration, for a desktop computer.
One interesting argument against the low-latency desktop option is available at https://sevencapitalsins.wordpress.com/2007/08/10/low-latency-kernel-wtf/.
For more information regarding the PREEMPT_RT patch, refer to http://varun-anand.com/preempt.html and http://www.versalogic.com/mediacenter/whitepapers/wp_linux_rt.asp.
After the kernel image is obtained with the newly applied and configured real-time preemptible kernel patch, it needs to be booted to make sure the activity is done appropriately so that the end result can be usable. Using the uname –a command, the patch rt* revision number is visible and should be applied to the kernel version. Of course, there are other methods that can used to identify this information. An alternative for the uname –a command is the dmesg command on its output the string real-time preemption support should be visible, but only one method should be enough. The following image offers a representation of how the uname –a command output should look:
Taking a look at the list of processes, it can be seen, as mentioned earlier, that the IRQ handler is treated using kernel threads. This information is visible in the next ps command output due to the fact that it is put between square brackets. Single IRQ handlers are represented by the task_struct structures that are similar to the user space ones, making them easily controllable from the user space:
The next bit of information that needs to be gathered involves the formatting of the interrupt process entries, which are a bit different than the ones used for a vanilla kernel. This output is visible by inspecting the /proc/interrupts file:
I (IRQ_INPROGRESS): This refers to the IRQ handler that is activeD (IRQ_DISABLED): This represents the IRQ as being disabledP (IRQ_PENDING): The IRQ here is presented as being in a pending stateR (IRQ_REPLAY): In this state, the IRQ has been replied to, but no ACK is received yetA (IRQ_AUTODETECT): This represents the IRQ as being in an autodetect stateW (IRQ_WAITING): This refers to the IRQ being in an autodetect state, but not seen yetL (IRQ_LEVEL): The IRQ is in a level-triggered stateM (IRQ_MASKED): This represents the state in which the IRQ is not visible as being masked anymoreN (IRQ_NODELAY): This is the state in which the IRQ must be executed immediately
Inside Yocto, kernel recipes with PREEMPT_RT patches are applied. For the moment, there are only two recipes that incorporate the PREEMPT_RT patch; both are available inside the meta layer. The recipes that refer to kernel versions 3.10 and 3.14 and their naming are linux-yocto-rt_3.10.bb and linux-yocto-rt_3.14.bb. The –rt ending in the naming indicates that these recipes fetch the PREEMPT_RT branches of the Linux kernel versions maintained by the Yocto community.
The format for the 3.14 kernel recipe is presented here:
As shown, one of the recipes seemed to have a duplicated line and a patch is necessary to remove it:
Linux, a general purpose operating system that is optimized for throughput, is the exact opposite of what a real-time operating system is all about. Of course it offers a high throughput by using a large, multilayered cache, which is a nightmare for a hard real-time operating process.
In order to have a real-time Linux, there are two available options:
- The first one involves the use of the PREEMPT_RT patches, which offer preemption by minimizing the latency and executing all activities in a thread context.
- The second solution involves the use of real-time extensions that act as layers between Linux and the hardware used for the management of real-time tasks. This second solution includes the previously mentioned RTLinux, RTAI, and XENOMAI solutions, as well as other commercial solutions and variations that involve moving the layer and also separating it in multiple components.
Another disadvantage of the preempt-rt patches is that they need to be ported from one kernel version to another and adapted from one architecture or software vendor to another. This only implies that knowledge of the Linux kernel should be available in-house for a particular vendor and it should adapt the solution for each of its available kernels. This fact alone has made it less likeable for BSP or Linux operating system providers.
actual configuration part, you should download a suitable version for the kernel. The best inspiration source is https://www.kernel.org/, which should be the starting point because it does not contain any extra patches. After the source code is received, the corresponding rt patches version can be downloaded from https://www.kernel.org/pub/linux/kernel/projects/rt/. The kernel version chosen for this demonstration is the 3.12 kernel version, but if any other kernel version is required, the same steps can be taken with a similar end result. The development of the real-time preemption patches is very active, so any missing version support is covered very fast. Also, for other sublevel versions, the patches can be found in the incr or older subdirectories of that particular kernel version. The following is the example for sublevel versions:
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.12.38.tar.xz wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.12/patch-3.12.38-rt52.patch.gz
After the source code is received, the sources need to be unpacked and the patches applied:
tar xf linux-3.12.38.tar.xz cd linux-3.12.38/ gzip -cd ../patch-3.12.38-rt52.patch.gz | patch -p1
The next step involves the configuration of the kernel sources. The configuration differs from one architecture to another, but the general idea remains. The following configurations are required for a QEMU ARM machine supported inside Poky. To enable the PREEMPT_RT support for a machine, there are multiple options available. You can implement a low-latency support version, which is most suitable for a desktop computer using a kernel configuration fragment similar to this:
CONFIG_GENERIC_LOCKBREAK=y CONFIG_TREE_PREEMPT_RCU=y CONFIG_PREEMPT_RCU=y CONFIG_UNINLINE_SPIN_UNLOCK=y CONFIG_PREEMPT=y CONFIG_PREEMPT__LL=y CONFIG_PREEMPT_COUNT=y CONFIG_DEBUG_PREEMPT=y CONFIG_RCU_CPU_STALL_VERBOSE=y
This option is one of the most often used and it also constitutes the primary source of usage of the PREEMPT_RT patches. The alternative of this would be to enable the fully preemptive support for the PREEMPT_RT patches using a configuration similar to this:
CONFIG_PREEMPT_RT_FULL=y CONFIG_HZ_1000=y CONFIG_HZ=1000
In the following image, the CONFIG_PREEMPT_RCU and CONFIG_PREEMPT_RT_FULL configurations are available. More information related to RCU is available at https://lwn.net/Articles/262464/.
The third image contains the CONFIG_PREEMPT__LL configuration. Another interesting configuration is CONFIG_PREEMPT_VOLUNTARY, which also reduces the latency along with the CONFIG_PREEMPT__LL configuration, for a desktop computer.
One interesting argument against the low-latency desktop option is available at https://sevencapitalsins.wordpress.com/2007/08/10/low-latency-kernel-wtf/.
For more information regarding the PREEMPT_RT patch, refer to http://varun-anand.com/preempt.html and http://www.versalogic.com/mediacenter/whitepapers/wp_linux_rt.asp.
After the kernel image is obtained with the newly applied and configured real-time preemptible kernel patch, it needs to be booted to make sure the activity is done appropriately so that the end result can be usable. Using the uname –a command, the patch rt* revision number is visible and should be applied to the kernel version. Of course, there are other methods that can used to identify this information. An alternative for the uname –a command is the dmesg command on its output the string real-time preemption support should be visible, but only one method should be enough. The following image offers a representation of how the uname –a command output should look:
Taking a look at the list of processes, it can be seen, as mentioned earlier, that the IRQ handler is treated using kernel threads. This information is visible in the next ps command output due to the fact that it is put between square brackets. Single IRQ handlers are represented by the task_struct structures that are similar to the user space ones, making them easily controllable from the user space:
The next bit of information that needs to be gathered involves the formatting of the interrupt process entries, which are a bit different than the ones used for a vanilla kernel. This output is visible by inspecting the /proc/interrupts file:
I (IRQ_INPROGRESS): This refers to the IRQ handler that is activeD (IRQ_DISABLED): This represents the IRQ as being disabledP (IRQ_PENDING): The IRQ here is presented as being in a pending stateR (IRQ_REPLAY): In this state, the IRQ has been replied to, but no ACK is received yetA (IRQ_AUTODETECT): This represents the IRQ as being in an autodetect stateW (IRQ_WAITING): This refers to the IRQ being in an autodetect state, but not seen yetL (IRQ_LEVEL): The IRQ is in a level-triggered stateM (IRQ_MASKED): This represents the state in which the IRQ is not visible as being masked anymoreN (IRQ_NODELAY): This is the state in which the IRQ must be executed immediately
Inside Yocto, kernel recipes with PREEMPT_RT patches are applied. For the moment, there are only two recipes that incorporate the PREEMPT_RT patch; both are available inside the meta layer. The recipes that refer to kernel versions 3.10 and 3.14 and their naming are linux-yocto-rt_3.10.bb and linux-yocto-rt_3.14.bb. The –rt ending in the naming indicates that these recipes fetch the PREEMPT_RT branches of the Linux kernel versions maintained by the Yocto community.
The format for the 3.14 kernel recipe is presented here:
As shown, one of the recipes seemed to have a duplicated line and a patch is necessary to remove it:
Linux, a general purpose operating system that is optimized for throughput, is the exact opposite of what a real-time operating system is all about. Of course it offers a high throughput by using a large, multilayered cache, which is a nightmare for a hard real-time operating process.
In order to have a real-time Linux, there are two available options:
- The first one involves the use of the PREEMPT_RT patches, which offer preemption by minimizing the latency and executing all activities in a thread context.
- The second solution involves the use of real-time extensions that act as layers between Linux and the hardware used for the management of real-time tasks. This second solution includes the previously mentioned RTLinux, RTAI, and XENOMAI solutions, as well as other commercial solutions and variations that involve moving the layer and also separating it in multiple components.
Another disadvantage of the preempt-rt patches is that they need to be ported from one kernel version to another and adapted from one architecture or software vendor to another. This only implies that knowledge of the Linux kernel should be available in-house for a particular vendor and it should adapt the solution for each of its available kernels. This fact alone has made it less likeable for BSP or Linux operating system providers.
recipes with PREEMPT_RT patches are applied. For the moment, there are only two recipes that incorporate the PREEMPT_RT patch; both are available inside the meta layer. The recipes that refer to kernel versions 3.10 and 3.14 and their naming are linux-yocto-rt_3.10.bb and linux-yocto-rt_3.14.bb. The –rt ending in the naming indicates that these recipes fetch the PREEMPT_RT branches of the Linux kernel versions maintained by the Yocto community.
The format for the 3.14 kernel recipe is presented here:
As shown, one of the recipes seemed to have a duplicated line and a patch is necessary to remove it:
Linux, a general purpose operating system that is optimized for throughput, is the exact opposite of what a real-time operating system is all about. Of course it offers a high throughput by using a large, multilayered cache, which is a nightmare for a hard real-time operating process.
In order to have a real-time Linux, there are two available options:
- The first one involves the use of the PREEMPT_RT patches, which offer preemption by minimizing the latency and executing all activities in a thread context.
- The second solution involves the use of real-time extensions that act as layers between Linux and the hardware used for the management of real-time tasks. This second solution includes the previously mentioned RTLinux, RTAI, and XENOMAI solutions, as well as other commercial solutions and variations that involve moving the layer and also separating it in multiple components.
Another disadvantage of the preempt-rt patches is that they need to be ported from one kernel version to another and adapted from one architecture or software vendor to another. This only implies that knowledge of the Linux kernel should be available in-house for a particular vendor and it should adapt the solution for each of its available kernels. This fact alone has made it less likeable for BSP or Linux operating system providers.
In order to have a real-time Linux, there are two available options:
- The first one involves the use of the PREEMPT_RT patches, which offer preemption by minimizing the latency and executing all activities in a thread context.
- The second solution involves the use of real-time extensions that act as layers between Linux and the hardware used for the management of real-time tasks. This second solution includes the previously mentioned RTLinux, RTAI, and XENOMAI solutions, as well as other commercial solutions and variations that involve moving the layer and also separating it in multiple components.
Another disadvantage of the preempt-rt patches is that they need to be ported from one kernel version to another and adapted from one architecture or software vendor to another. This only implies that knowledge of the Linux kernel should be available in-house for a particular vendor and it should adapt the solution for each of its available kernels. This fact alone has made it less likeable for BSP or Linux operating system providers.
Having a real-time operating system may not always be enough for everyone. Some people would also require real-time optimized applications running over the operating system. To make sure an rt-application can be designed and interacted with, the required determinism is necessary on the operating system and hardware. With regard to the hardware configuration, the requirements involve a low-latency interrupt handling. The mechanisms causing the ISR latencies should register values around tens of microseconds.
To write an application, there are some things that need to be taken care of, such as making sure that the use of swap is disabled to diminish latencies caused by page faults. The use of global variables or arrays should be kept to a minimum. The 99 priority number is not configured to run an application, and other spin locks are not implemented instead, it uses priority inheritance futexes. Also avoid input/output operations and data sharing between applications.
The Linux operating system was for a very long time seen as a GPOS, but in the last couple of years, some projects tried to change this by modifying the Linux kernel into a RTOS. One such project is the PREEMPT_RT patch, which was mentioned previously.
In this section of the chapter, I will discuss a series of tests that could be executed for both versions of the Linux OS with or without applying the PREEMPT_RT patches. I should mention that for those of you who are interested in some actual results, there are a number of papers available that try to investigate the latency effect of the PREEMPT_RT or its advantages or disadvantages. One such example is available at http://www.versalogic.com/downloads/whitepapers/real-time_linux_benchmark.pdf.
Before continuing further, I believe it is my duty to define a number of technical terms that are necessary to properly understand some information:
- Interrupt latency: This indicates the time that has elapsed since an interrupt was generated and until the execution has been started in the interrupt handler.
- Scheduling latency: This represents the time between the wake up signal of an event and a scheduler that has the opportunity to schedule a thread for it. It is also called a dispatch latency.
- Worst-case latency: This indicates the time that has passed since a demand was issued and until the response to that demand was received.
- Context-switch: This represents the switching of the CPU from one process or thread to another. It only occurs in the kernel mode.
The LPPTest is included in the PREEMPT_RT patch and it contains a Linux driver that only changes a bit value on a parallel port to identify the response time. Another driver responds to the change in a bit value and a user space application that measures the results. The files to look for are drivers/char/lpptest.c and scripts/testlpp.c. To perform this test, two machines are required: one to send the signal and the other one to receive and send the response. This requirement is stringent since the use of a loopback cable can mess with the measurements.
RealFeel is a test for interrupt processing. The program uses /dev/rtc to fire a periodic interrupt, measures the duration between one interrupt to another, and compares it with the expected value. At the end, it prints the variation from the expected value indefinitely so that the variations can be exported in a log file to process later.
Linux Real-Time Benchmarking Framework (LRTB) represents a set of scripts and drivers that are used to evaluate various performance counters for the Linux kernel with a real-time addition. It measures the load imposed by real-time patches and their ability to obtain a more deterministic response to interrupts.
The PREEMPT_RT patch improves the preemptiveness of the Linux kernel, but this does not mean it is the best solution to use. The usefulness of PREEMPT_RT patch can differ if various aspects of the application domain changes. With regard to the PREEMPT_RT patch, it is ready to be used in a hard real-time system. One conclusion cannot be made, but I must admit that it can be considered hard real-time material if it is used in life sustaining or mission-critical systems. This is a decision for everybody to make, and for this testing is required. One opinion that supports this is from Steven Rostedt, a Linux kernel developer who is the maintainer of the stable version of the real-time Linux kernel patch for Red Hat. It is available at http://www.linux.com/news/featured-blogs/200-libby-clark/710319-intro-to-real-time-linux-for-embedded-developers.
The meta-realtime layer is an initiative maintained by Bruce Ashfield from WindRiver, which planned to create a place where real-time activities related to the Linux kernel or system development. It was created as the placeholder for PREEMPT_RT, SCHED_DEADLINE, POSIX real-time, and alternative paring of general purpose operating systems and real-time operating systems, whether this involved a user space RTOS, a hypervisor, or an AMP solution. Also, this is where system partitioning, CPU isolation, and other related applications s reside. Of course, none of this would be considered complete without some performance profiling and benchmarking applications available for the whole Linux operating system.
The next and last available application is rt-app, which is used as a test application for the simulation of real-time loads on a system. It does this by starting multiple threads at given periods of time. It offers support for SCHED_FIFO, SCHED_OTHER, SCHED_RR, SCHED_DEADLINE, as well as the Adaptive Quality of Service Architecture (AQuoSA) framework, which is an open source project that tries to offer adaptive Quality of Service (QoS) for the Linux kernel.
Besides the included packages, the layer also contains an image that incorporates them, but this is not nearly enough to make this layer one that contains substantial content. Although it does not contain a vast amount of information inside it, this layer has been presented in this chapter because it contains the starting point and offers a development point of view of all the information presented until now. Of course, a number of applications that should reside in this layer are already spread across multiple other layers, such as the idlestat package that is available in meta-linaro. However, this does not constitute the central point of this explanation. I only wanted to point out the most suitable place that can contain any real-time relate activities, and in my opinion, meta-realtime is this place.