Let’s get an in-depth understanding of real-time systems, analysing the real-time characteristics of two prominent operating systems: Linux with PREEMPT_RT patch and QNX. By examining their real-time characteristics, determinism, performance, and features, developers can get an idea of how the real-time behaviours of Linux with PREEMPT_RT patch compare with QNX.
Real-time operating systems (RTOS) play a critical role in the development of hard real-time systems and meeting the stringent timing requirements of real-time applications. QNX (Quantum Neutrino eXecutive) stands out as a preferred choice for hard real-time systems, such as medical systems, due to its real-time capabilities and features.
Linux, in its standard form, is not considered a hard real-time operating system. However, efforts have been made to make Linux more suitable for real-time applications. Linux with PREEMPT_RT patch attempts to enhance the real-time capabilities of the standard Linux kernel by adding specific real-time scheduling policies, reducing interrupt latency, and providing mechanisms for prioritising time-critical tasks. These improvements have made Linux with PREEMPT_RT patch more predictable and responsive.
A real-time operating system is specialised, designed to meet the stringent timing requirements of hard real-time systems. These systems often have critical applications in industries such as aerospace, automotive, industrial automation, and medical devices. Nowadays, medical industries are migrating from (proprietary) RTOS such as QNX to Linux (open source) drawn by licence, community support, security, and cost considerations. Linux with strong real-time capabilities is required for this scenario, and for understanding how PREEMPT_RT patch improves real-time capabilities.
Figure 1 lists some business challenges associated with using RTOS.
RTOS like QNX are used and deployed in real-time systems in critical industries, such as medical devices. Industries considering migrating from RTOS to Linux need a detailed understanding of Linux with the PREEMPT_RT patch — its real-time capabilities, behaviour, and features. Additionally, a comparison between Linux with PREEMPT_RT patch and QNX is necessary. The problem is that this information and comparison results are not available. This article will address this issue comprehensively. We also do not know how to use PREEMPT_RT patch and the real-time behaviours, features and latency reduction offered by it. This issue will be addressed here with PREEMPT_RT patch guidelines and case studies.
Real-time systems are computer systems designed to respond to input and produce output within a specified timeframe, known as a deadline. The primary characteristic that distinguishes real-time systems from other computer systems is their ability to guarantee timely and predictable responses to events or tasks. These systems are widely used in various domains, including industrial control, automotive systems, aerospace, medical devices, multimedia, and telecommunications.
There are two main types of real-time systems.
Hard real-time systems: Meeting deadlines is crucial in hard real-time systems. Failure to complete a task within its specified deadline can lead to dangerous consequences, system failure, or compromise safety. Examples include flight control systems in aeroplanes, medical devices for life support, and airbag deployment in vehicles.
Soft real-time systems: In soft real-time systems, meeting deadlines is important, but occasional misses may be tolerable without dangerous consequences. While timely execution is desirable, the system can still provide useful results even if some deadlines are missed. Examples of soft real-time systems include streaming multimedia applications, online video conferencing, and online gaming. The user experience may degrade if deadlines are missed, but it doesn’t lead to system failure or severe safety issues.
Apart from these main types, there is a middle ground known as ‘firm real-time systems’, a hybrid between hard and soft real-time systems. Firm real-time systems prioritise meeting most deadlines, but occasional misses are allowed if they don’t significantly impact the overall system behaviour. These systems are often used in multimedia streaming and telecommunications applications.
QNX is a real-time operating system known for its strong real-time characteristics, reliability, and low and deterministic latencies. Originally developed by QNX Software Systems, it is now a part of BlackBerry Limited and is widely used across various industries and applications that require real-time performance, determinism, and high reliability. Its characteristics are:
Deterministic behaviour: QNX is designed to be highly deterministic, guaranteeing that critical tasks and processes meet their deadlines consistently. This attribute is particularly crucial in safety-critical applications, such as automotive systems, industrial automation, and medical devices.
Real-time scheduling: QNX employs a priority based preemptive scheduler, allowing tasks with higher priorities to preempt lower-priority tasks. This scheduling mechanism ensures that time-critical tasks receive immediate attention without interference from lower-priority tasks.
Fault tolerance and reliability: QNX is designed with a strong focus on fault tolerance and reliability. Its microkernel architecture isolates critical components, reducing the impact of failures and enhancing system resilience.
Resource management: QNX provides efficient resource management, allowing developers to allocate CPU time, memory, and other system resources to tasks in a controlled and predictable manner. This capability is essential for meeting the timing requirements of real-time applications.
Preemption: QNX supports preemption, enabling a higher-priority task to interrupt the execution of a lower-priority task promptly.
Fast interrupt handling: QNX provides efficient interrupt handling mechanisms. When an interrupt occurs, the kernel quickly determines the appropriate handler for the interrupt and dispatches it. This minimises the time spent in the interrupt handling routine.
Latencies: QNX is known for its low and deterministic latencies, making it suitable for real-time applications with strict timing requirements. The latencies in a real-time operating system like QNX refer to the time it takes for the system to respond to an event or an interrupt and start executing a specific task. There are several factors that can influence latencies in QNX:
- Context switching
- Task scheduling
- Kernel design
- Interrupt handling
It’s important to note that while QNX offers low and deterministic latencies, the actual latencies experienced in a specific system can depend on various factors, such as the hardware platform, system workload, and application design. The above techniques are used to optimise systems for low latencies and predictable behaviour.
Linux also possesses some real-time capabilities. Examples include EDF and PREEMPT_RT patch.
EDF: EDF (earliest deadline first) is a scheduling algorithm commonly used in real-time operating systems and some real-time extensions for general-purpose operating systems, including Linux. It prioritises tasks based on their deadlines, giving preference to tasks with the earliest deadlines for execution.
In standard Linux, the default scheduling algorithm is the completely fair scheduler (CFS), which is designed for fairness and responsiveness in non-real-time workloads. However, there are ways to implement EDF in Linux for real-time applications. One way to enable EDF scheduling in Linux is by using the PREEMPT_RT (real-time) patch.
PREEMPT_RT patch: The PREEMPT_RT patch is a set of patches for the Linux kernel that aim to turn Linux into a soft real-time operating system. The goal of the PREEMPT_RT patch is to improve the responsiveness and predictability of the Linux kernel for real-time applications while retaining the general-purpose capabilities of Linux.
The main features introduced by the PREEMPT_RT patch are listed in Table 1.
|Soft and hard real-time support
|Kernel locking and synchronisation
When the PREEMPT_RT patch is applied to the Linux kernel and booted, the real-time features become available. Real-time tasks can be created, and use the real-time scheduling classes to achieve deterministic and timely execution.
To use real-time capabilities, you need to configure the applications to run with elevated privileges (often as root) since real-time tasks have higher scheduling priorities and require special permissions.
PREEMPT_RT patch significantly improves real-time performance; it may not be suitable for certain ultra-high-performance real-time applications where absolute timing guarantees are critical. This patch is an external addition to the Linux kernel and may not always be in sync with the latest official kernel releases. Therefore, it is crucial to carefully select the kernel version and the corresponding PREEMPT_RT patch version that are compatible and well-maintained for your application needs. Additionally, the kernel configuration options and tuning for real-time performance should be done with care and understanding of specific requirements.
We take a look at a test conducted in the PREEMPT_RT patch, using the Cyclictest command. Cyclictest is a command-line utility in Linux, often used for measuring real-time performance and latency of the system. It is particularly helpful for testing and analysing the Linux kernel’s ability to handle periodic tasks with strict timing requirements.
To use Cyclictest, you need to install it on your Linux system. It is often available in most distribution repositories. If it is not installed already, use the package manager specific to your distribution. For example, on Ubuntu and Debian-based systems, the following command can be used to install Cyclictest:
sudo apt-get install rt-tests
Once installed, run Cyclictest with distinct options. The basic syntax is as follows:
Here are some common options used with Cyclictest:
-t: Specifies the number of threads to use (default is 1)
-n: Specifies the number of iterations (default is 10000)
-i: Specifies the interval (in microseconds) between iterations (default is 1000µs)
-l: Specifies the maximum thread lock latency (in microseconds) before considering a result as an outlier (default is 10000µs)
-m: Specifies the CPU mask, which restricts the test to specific CPUs (default is all CPUs)
-p: Prints the raw latency results
-H: Prints the latency histogram
-S: Prints the latency statistics (mean, min, max, etc)
-q: Quiet mode; suppresses the summary and histogram outputs.
Here is a Cyclictest sample command:
cyclictest -p 90 -t80 -m -n 10000 -l 100000 –priospread
This command uses the values given below with options.
- The option -p is used for thread priority level and -p as 90.
- The option –t is used for number of threads and -t as 80.
- The option -m is used to enable lockall, which locks all memory used by the ‘cyclictest’ process into RAM.
- Number of iterations or number of test cycles -n is 10000.
- The option -l sets the length of each test cycle in microseconds and -l as 100000.
The option -priospread is used to enable ‘priority spreading’. Priority spreading is a feature in the Linux kernel that helps mitigate priority inversion issues.
The kernel configurations given below enable the PREEMPT_RT patch. The latencies are calculated with all the kernel configurations as well as with the enabled PREEMPT_RT patch. The testing is done with ‘cyclictest’ Linux command-line utility with some required options. Hackbench load is also given with the Cyclictest command.
The kernel config options (including PREEMPT_RT Patch) are:
Kernel version: 6.1.35 Hardware: Raspberry Pi 4 model B (64-bit)
Kernel configuration options for the server should be ‘No Forced Preemption’ as shown in Figure 2.
These changes are reflected in the .config file as shown below:
CONFIG_PREEMPT_NONE_BUILD=y CONFIG_PREEMPT_NONE=y CONFIG_PREEMPT_VOLUNTARY is not set. CONFIG_PREEMPT is not set. CONFIG_PREEMPT_DYNAMIC is not set. CONFIG_SCHED_CORE is not set.
Kernel configuration options for the desktop should be ‘Voluntary Kernel Preemption’ as shown in Figure 3.
The changes are reflected in the .config file, as shown below:
CONFIG_PREEMPT_VOLUNTARY_BUILD=y CONFIG_PREEMPT_NONE is not set. CONFIG_PREEMPT_VOLUNTARY=y CONFIG_PREEMPT is not set. CONFIG_PREEMPT_DYNAMIC is not set. CONFIG_SCHED_CORE is not set.
Figure 4 shows kernel configurations for ‘Preemptible Kernel’ (low latency desktop).
These changes are reflected in .config as shown below:
CONFIG_PREEMPT_BUILD=y CONFIG_PREEMPT_NONE is not set. CONFIG_PREEMPT_VOLUNTARY is not set. CONFIG_PREEMPT=y CONFIG_PREEMPT_COUNT=y CONFIG_PREEMPTION=y CONFIG_PREEMPT_DYNAMIC is not set. CONFIG_SCHED_CORE is not set.
Figure 5 shows kernel configurations for ‘Fully Preemptible Kernel’ (real-time).
These changes are reflected in .config as shown below:
CONFIG_HAVE_PREEMPT_LAZY=y CONFIG_PREEMPT_LAZY=y CONFIG_PREEMPT_NONE is not set. CONFIG_PREEMPT_VOLUNTARY is not set. CONFIG_PREEMPT is not set. CONFIG_PREEMPT_RT=y CONFIG_PREEMPT_COUNT=y CONFIG_PREEMPTION=y CONFIG_SCHED_CORE is not set.
Table 2 lists all kernel config and cyclic test observations.
|No Forced Pre-emption (server)
|Voluntary Kernel Pre-emption (desktop)
|Preemptible Kernel (low latency desktop)
|Fully Preemptible Kernel (real-time)
|Compiled kernel without applied RT-patch
|Compiled kernel without applied RT-patch
|Compiled kernel without applied RT-patch
|Compiled kernel with applied RT-patch
|hackbench -l 100000000
|hackbench -l 100000000
|hackbench -l 100000000
Figure 6 shows a histogram of No Forced Preemption (server), Figure 7 of Voluntary Kernel Preemption (desktop), Figure 8 of Preemptible Kernel (low latency desktop), and Figure 9 of Fully Preemptible Kernel (real-time).
Figure 10 shows some PREEMPT_RT patch benefits.
Guidelines for using PREEMPT_RT patch in Linux
- Select a compatible kernel version: Choose a kernel version that is compatible with the PREEMPT_RT patch we intend to apply.
- Hardware and driver support: Ensure that hardware and drivers are supported by the selected kernel version.
- Patch the kernel: Apply the PREEMPT_RT patch to the chosen kernel version. Follow the patch’s documentation and instructions carefully.
- Configuration options: Enable necessary real-time options in the kernel configuration.
- Compile the source code: Compile the source code after patching as per the instructions.
- Real-time scheduling policies: Configure the kernel to support real-time scheduling policies like SCHED_FIFO and SCHED_RR. These are essential for managing real-time tasks.
- Minimise interrupt handling latency: Optimise interrupt handling to minimise latency. Make the interrupt service routines as efficient as possible.
- Community and support: Engage with the PREEMPT_RT community and mailing lists for assistance, advice, and troubleshooting.
- Real-time thread priority: Real-time threads should be assigned the highest priority (SCHED_FIFO or SCHED_RR) to ensure they are not preempted by lower-priority tasks.
- Use POSIX real-time APIs: Leverage the POSIX real-time APIs, such as pthreads with SCHED_FIFO, for managing real-time threads. These APIs provide mechanisms for setting thread priorities, managing synchronisation, and avoiding priority inversion.
- Minimise blocking and I/O: Real-time threads should minimise blocking operations, especially those that may involve waiting on external events. If blocking is necessary, use non-blocking I/O or asynchronous I/O mechanisms.
- Avoid busy-waiting: Avoid busy-waiting (e.g., spinlocks) as it can lead to high CPU usage and increased system latency. Instead, use blocking mechanisms or conditional waiting (e.g., pthread_cond_wait) to efficiently manage synchronisation.
- Manage memory carefully: Minimise dynamic memory allocation during real-time operation as it can introduce non-deterministic behaviour. Allocate memory statically or use memory pools to ensure deterministic access.
- Resource management: Properly manage resources, such as file descriptors, network connections, and hardware peripherals, to avoid contention and resource exhaustion. Release resources when they are no longer needed.
- Error handling: Implement robust error handling and fault tolerance mechanisms in your real-time application to handle unexpected situations and recover gracefully.
Comparison of QNX and Real-Time Linux
QNX and Real-Time Linux are both popular choices for building real-time and embedded systems, but they have different characteristics, features, and performance considerations. Table 3 gives a comparison of QNX and Real-Time Linux in terms of real-time performance.
|Better determinism and real-time performance out of the box due to its microkernel architecture
|Real-Time Linux can achieve good determinism with proper configuration and patches
|Has lower latencies and faster context switching, making it suitable for hard real-time applications
|Achieves low latencies with specialised patches but might have slightly higher latencies compared to QNX
|Ease of use
|QNX is designed specifically for real-time applications and provides a comprehensive environment for embedded systems development
|Real-Time Linux may require more configuration and tuning to achieve optimal real-time performance
|Ecosystem and compatibility
|QNX has a more focused ecosystem tailored to embedded and real-time applications.
|Real-Time Linux benefits from the vast Linux ecosystem, including a wide range of device drivers and software libraries
In making your decision, consider the following factors.
Timing requirements: PREEMPT-RT patch converts standard Linux to Real-Time Linux. The latter has good determinism to achieve real-time capabilities and reduces scheduling latency. However, QNX may be slightly better compared to Real-Time Linux if your application requires hard real-time capabilities and precise determinism, though the latter can be used based on the requirements.
Reliability and safety: QNX is often used in safety-critical systems including medical devices. It is designed with safety in mind and provides features for safety certification. Real-Time Linux can be used in safety-critical applications, but achieving safety certification may require additional effort and adherence to industry-specific safety standards.
Ecosystem and tools: Consider the availability of development tools, libraries, and drivers that match your project’s requirements.
Familiarity: If your team is already experienced with Linux development, Real-Time Linux with PREEMPT_RT patch, kernel config options, IRQ handling and thread priorities may be a more seamless choice. SCHED_FIFO, SCHED_RR are often used for Real-Time threads. These factors will improve real-time capabilities in the Linux environment.
Project scope: The size and complexity of the project may influence which platform is more suitable.
In conclusion, both QNX and Real-Time Linux are good options for developing real-time and embedded systems; each comes with its own strengths and considerations. The choice depends on the specific application requirements, the level of real-time performance needed, familiarity with the platforms, the available tools, ecosystem, and the level of determinism needed.