Real-Time Linux Applicability for Hard Real-Time Systems

Let’s get an in-depth understanding of real-time systems, analysing the real-time characteristics of two prominent operating systems: Linux with PREEMPT_RT patch and QNX. By examining their real-time characteristics, determinism, performance, and features, developers can get an idea of how the real-time behaviours of Linux with PREEMPT_RT patch compare with QNX.

Real-time operating systems (RTOS) play a critical role in the development of hard real-time systems and meeting the stringent timing requirements of real-time applications. QNX (Quantum Neutrino eXecutive) stands out as a preferred choice for hard real-time systems, such as medical systems, due to its real-time capabilities and features.

Linux, in its standard form, is not considered a hard real-time operating system. However, efforts have been made to make Linux more suitable for real-time applications. Linux with PREEMPT_RT patch attempts to enhance the real-time capabilities of the standard Linux kernel by adding specific real-time scheduling policies, reducing interrupt latency, and providing mechanisms for prioritising time-critical tasks. These improvements have made Linux with PREEMPT_RT patch more predictable and responsive.

Business challenges

A real-time operating system is specialised, designed to meet the stringent timing requirements of hard real-time systems. These systems often have critical applications in industries such as aerospace, automotive, industrial automation, and medical devices. Nowadays, medical industries are migrating from (proprietary) RTOS such as QNX to Linux (open source) drawn by licence, community support, security, and cost considerations. Linux with strong real-time capabilities is required for this scenario, and for understanding how PREEMPT_RT patch improves real-time capabilities.

Figure 1 lists some business challenges associated with using RTOS.

Problem statement

RTOS like QNX are used and deployed in real-time systems in critical industries, such as medical devices. Industries considering migrating from RTOS to Linux need a detailed understanding of Linux with the PREEMPT_RT patch — its real-time capabilities, behaviour, and features. Additionally, a comparison between Linux with PREEMPT_RT patch and QNX is necessary. The problem is that this information and comparison results are not available. This article will address this issue comprehensively. We also do not know how to use PREEMPT_RT patch and the real-time behaviours, features and latency reduction offered by it. This issue will be addressed here with PREEMPT_RT patch guidelines and case studies.

Real-time systems

Real-time systems are computer systems designed to respond to input and produce output within a specified timeframe, known as a deadline. The primary characteristic that distinguishes real-time systems from other computer systems is their ability to guarantee timely and predictable responses to events or tasks. These systems are widely used in various domains, including industrial control, automotive systems, aerospace, medical devices, multimedia, and telecommunications.

There are two main types of real-time systems.

Hard real-time systems: Meeting deadlines is crucial in hard real-time systems. Failure to complete a task within its specified deadline can lead to dangerous consequences, system failure, or compromise safety. Examples include flight control systems in aeroplanes, medical devices for life support, and airbag deployment in vehicles.

Soft real-time systems: In soft real-time systems, meeting deadlines is important, but occasional misses may be tolerable without dangerous consequences. While timely execution is desirable, the system can still provide useful results even if some deadlines are missed. Examples of soft real-time systems include streaming multimedia applications, online video conferencing, and online gaming. The user experience may degrade if deadlines are missed, but it doesn’t lead to system failure or severe safety issues.

Apart from these main types, there is a middle ground known as ‘firm real-time systems’, a hybrid between hard and soft real-time systems. Firm real-time systems prioritise meeting most deadlines, but occasional misses are allowed if they don’t significantly impact the overall system behaviour. These systems are often used in multimedia streaming and telecommunications applications.

QNX

QNX is a real-time operating system known for its strong real-time characteristics, reliability, and low and deterministic latencies. Originally developed by QNX Software Systems, it is now a part of BlackBerry Limited and is widely used across various industries and applications that require real-time performance, determinism, and high reliability. Its characteristics are:

Deterministic behaviour: QNX is designed to be highly deterministic, guaranteeing that critical tasks and processes meet their deadlines consistently. This attribute is particularly crucial in safety-critical applications, such as automotive systems, industrial automation, and medical devices.

Real-time scheduling: QNX employs a priority based preemptive scheduler, allowing tasks with higher priorities to preempt lower-priority tasks. This scheduling mechanism ensures that time-critical tasks receive immediate attention without interference from lower-priority tasks.

Fault tolerance and reliability: QNX is designed with a strong focus on fault tolerance and reliability. Its microkernel architecture isolates critical components, reducing the impact of failures and enhancing system resilience.

Resource management: QNX provides efficient resource management, allowing developers to allocate CPU time, memory, and other system resources to tasks in a controlled and predictable manner. This capability is essential for meeting the timing requirements of real-time applications.

Preemption: QNX supports preemption, enabling a higher-priority task to interrupt the execution of a lower-priority task promptly.

Fast interrupt handling: QNX provides efficient interrupt handling mechanisms. When an interrupt occurs, the kernel quickly determines the appropriate handler for the interrupt and dispatches it. This minimises the time spent in the interrupt handling routine.

Latencies: QNX is known for its low and deterministic latencies, making it suitable for real-time applications with strict timing requirements. The latencies in a real-time operating system like QNX refer to the time it takes for the system to respond to an event or an interrupt and start executing a specific task. There are several factors that can influence latencies in QNX:

• Context switching
• Task scheduling
• Kernel design
• Interrupt handling

It’s important to note that while QNX offers low and deterministic latencies, the actual latencies experienced in a specific system can depend on various factors, such as the hardware platform, system workload, and application design. The above techniques are used to optimise systems for low latencies and predictable behaviour.

Linux

Linux also possesses some real-time capabilities. Examples include EDF and PREEMPT_RT patch.

EDF: EDF (earliest deadline first) is a scheduling algorithm commonly used in real-time operating systems and some real-time extensions for general-purpose operating systems, including Linux. It prioritises tasks based on their deadlines, giving preference to tasks with the earliest deadlines for execution.

In standard Linux, the default scheduling algorithm is the completely fair scheduler (CFS), which is designed for fairness and responsiveness in non-real-time workloads. However, there are ways to implement EDF in Linux for real-time applications. One way to enable EDF scheduling in Linux is by using the PREEMPT_RT (real-time) patch.

PREEMPT_RT patch: The PREEMPT_RT patch is a set of patches for the Linux kernel that aim to turn Linux into a soft real-time operating system. The goal of the PREEMPT_RT patch is to improve the responsiveness and predictability of the Linux kernel for real-time applications while retaining the general-purpose capabilities of Linux.

The main features introduced by the PREEMPT_RT patch are listed in Table 1.

Feature	Description
Preemption	Enhances the kernel’s ability to be preempted at more points during its execution This allows higher-priority tasks to interrupt lower-priority ones more quickly, reducing latencies and improving responsiveness
Kernel preemption	Introduces kernel preemption, which allows high-priority kernel threads to preempt lower-priority ones Improves the responsiveness of the kernel to real-time events
Priority inheritance	Implements priority inheritance protocols to prevent priority inversion, a situation where a higher-priority task is blocked by a lower-priority task holding a shared resource Priority inheritance ensures that the resource-holding task temporarily inherits the priority of the blocked task, preventing priority inversions
High-resolution timers	Provides support for high-resolution timers, which allow applications to set timers with finer granularity, making it easier to meet tight timing constraints
Real-time scheduling	Enhances the Linux scheduler to support real-time scheduling processes, such as SCHED_FIFO and SCHED_RR in addition to the standard CFS scheduler These policies allow developers to create real-time tasks with strict timing requirements
Throttle mechanism	Introduces throttle mechanism to limit the execution time of low-priority tasks, preventing them from excessively delaying high-priority real-time tasks
Interrupt handling	The patch optimises interrupt handling to reduce the latency associated with servicing interrupts It reduces the time spent in interrupt context, making the system more responsive to real-time events
Soft and hard real-time support	PREEMPT_RT patch offers both soft and hard real-time capabilities Soft real-time systems aim to meet timing constraints most of the time but allow occasional deadline misses. Hard real-time systems must meet timing constraints strictly, without any misses
Kernel locking and synchronisation	To ensure data consistency and prevent race conditions, the patch addresses kernel locking and synchronisation mechanisms to handle concurrent access to shared data

 

When the PREEMPT_RT patch is applied to the Linux kernel and booted, the real-time features become available. Real-time tasks can be created, and use the real-time scheduling classes to achieve deterministic and timely execution.

To use real-time capabilities, you need to configure the applications to run with elevated privileges (often as root) since real-time tasks have higher scheduling priorities and require special permissions.

PREEMPT_RT patch significantly improves real-time performance; it may not be suitable for certain ultra-high-performance real-time applications where absolute timing guarantees are critical. This patch is an external addition to the Linux kernel and may not always be in sync with the latest official kernel releases. Therefore, it is crucial to carefully select the kernel version and the corresponding PREEMPT_RT patch version that are compatible and well-maintained for your application needs. Additionally, the kernel configuration options and tuning for real-time performance should be done with care and understanding of specific requirements.

Cyclictest

We take a look at a test conducted in the PREEMPT_RT patch, using the Cyclictest command. Cyclictest is a command-line utility in Linux, often used for measuring real-time performance and latency of the system. It is particularly helpful for testing and analysing the Linux kernel’s ability to handle periodic tasks with strict timing requirements.

To use Cyclictest, you need to install it on your Linux system. It is often available in most distribution repositories. If it is not installed already, use the package manager specific to your distribution. For example, on Ubuntu and Debian-based systems, the following command can be used to install Cyclictest:

sudo apt-get install rt-tests

Once installed, run Cyclictest with distinct options. The basic syntax is as follows:

cyclictest <options>

Here are some common options used with Cyclictest:

-t: Specifies the number of threads to use (default is 1)

-n: Specifies the number of iterations (default is 10000)

-i: Specifies the interval (in microseconds) between iterations (default is 1000µs)

-l: Specifies the maximum thread lock latency (in microseconds) before considering a result as an outlier (default is 10000µs)

-m: Specifies the CPU mask, which restricts the test to specific CPUs (default is all CPUs)

-p: Prints the raw latency results

-H: Prints the latency histogram

-S: Prints the latency statistics (mean, min, max, etc)

-q: Quiet mode; suppresses the summary and histogram outputs.

Here is a Cyclictest sample command:

cyclictest -p 90 -t80 -m -n 10000 -l 100000 –priospread

This command uses the values given below with options.

• The option -p is used for thread priority level and -p as 90.
• The option –t is used for number of threads and -t as 80.
• The option -m is used to enable lockall, which locks all memory used by the ‘cyclictest’ process into RAM.
• Number of iterations or number of test cycles -n is 10000.
• The option -l sets the length of each test cycle in microseconds and -l as 100000.

The option -priospread is used to enable ‘priority spreading’. Priority spreading is a feature in the Linux kernel that helps mitigate priority inversion issues.

Kernel configurations

The kernel configurations given below enable the PREEMPT_RT patch. The latencies are calculated with all the kernel configurations as well as with the enabled PREEMPT_RT patch. The testing is done with ‘cyclictest’ Linux command-line utility with some required options. Hackbench load is also given with the Cyclictest command.

The kernel config options (including PREEMPT_RT Patch) are:

Kernel version: 6.1.35
Hardware: Raspberry Pi 4 model B (64-bit)

Kernel configuration options for the server should be ‘No Forced Preemption’ as shown in Figure 2.

These changes are reflected in the .config file as shown below:

CONFIG_PREEMPT_NONE_BUILD=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_VOLUNTARY is not set.
CONFIG_PREEMPT is not set.
CONFIG_PREEMPT_DYNAMIC is not set.
CONFIG_SCHED_CORE is not set.

Kernel configuration options for the desktop should be ‘Voluntary Kernel Preemption’ as shown in Figure 3.

The changes are reflected in the .config file, as shown below:

CONFIG_PREEMPT_VOLUNTARY_BUILD=y
CONFIG_PREEMPT_NONE is not set.
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_PREEMPT is not set.
CONFIG_PREEMPT_DYNAMIC is not set.
CONFIG_SCHED_CORE is not set.

Figure 4 shows kernel configurations for ‘Preemptible Kernel’ (low latency desktop).

These changes are reflected in .config as shown below:

CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT_NONE is not set.
CONFIG_PREEMPT_VOLUNTARY is not set.
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC is not set.
CONFIG_SCHED_CORE is not set.

Figure 5 shows kernel configurations for ‘Fully Preemptible Kernel’ (real-time).

These changes are reflected in .config as shown below:

CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
CONFIG_PREEMPT_NONE is not set.
CONFIG_PREEMPT_VOLUNTARY is not set.
CONFIG_PREEMPT is not set.
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_SCHED_CORE is not set.

Table 2 lists all kernel config and cyclic test observations.

Name	No Forced Pre-emption (server)	Voluntary Kernel Pre-emption (desktop)	Preemptible Kernel (low latency desktop)	Fully Preemptible Kernel (real-time)
Kernel compilation	Compiled kernel without applied RT-patch	Compiled kernel without applied RT-patch	Compiled kernel without applied RT-patch	Compiled kernel with applied RT-patch
Hackbench load	hackbench -l 100000000	hackbench -l 100000000	hackbench -l 100000000	hackbench -l 100000000
Cyclictest duration	10 min	10 min	10 min	10 min
Latency	13548us	541us	699us	156us

 

Figure 6 shows a histogram of No Forced Preemption (server), Figure 7 of Voluntary Kernel Preemption (desktop), Figure 8 of Preemptible Kernel (low latency desktop), and Figure 9 of Fully Preemptible Kernel (real-time).

Figure 10 shows some PREEMPT_RT patch benefits.

Guidelines for using PREEMPT_RT patch in Linux

Kernel guidelines

• Select a compatible kernel version: Choose a kernel version that is compatible with the PREEMPT_RT patch we intend to apply.
• Hardware and driver support: Ensure that hardware and drivers are supported by the selected kernel version.
• Patch the kernel: Apply the PREEMPT_RT patch to the chosen kernel version. Follow the patch’s documentation and instructions carefully.
• Configuration options: Enable necessary real-time options in the kernel configuration.
• Compile the source code: Compile the source code after patching as per the instructions.
• Real-time scheduling policies: Configure the kernel to support real-time scheduling policies like SCHED_FIFO and SCHED_RR. These are essential for managing real-time tasks.
• Minimise interrupt handling latency: Optimise interrupt handling to minimise latency. Make the interrupt service routines as efficient as possible.
• Community and support: Engage with the PREEMPT_RT community and mailing lists for assistance, advice, and troubleshooting.

Application guidelines

• Real-time thread priority: Real-time threads should be assigned the highest priority (SCHED_FIFO or SCHED_RR) to ensure they are not preempted by lower-priority tasks.
• Use POSIX real-time APIs: Leverage the POSIX real-time APIs, such as pthreads with SCHED_FIFO, for managing real-time threads. These APIs provide mechanisms for setting thread priorities, managing synchronisation, and avoiding priority inversion.
• Minimise blocking and I/O: Real-time threads should minimise blocking operations, especially those that may involve waiting on external events. If blocking is necessary, use non-blocking I/O or asynchronous I/O mechanisms.
• Avoid busy-waiting: Avoid busy-waiting (e.g., spinlocks) as it can lead to high CPU usage and increased system latency. Instead, use blocking mechanisms or conditional waiting (e.g., pthread_cond_wait) to efficiently manage synchronisation.
• Manage memory carefully: Minimise dynamic memory allocation during real-time operation as it can introduce non-deterministic behaviour. Allocate memory statically or use memory pools to ensure deterministic access.
• Resource management: Properly manage resources, such as file descriptors, network connections, and hardware peripherals, to avoid contention and resource exhaustion. Release resources when they are no longer needed.
• Error handling: Implement robust error handling and fault tolerance mechanisms in your real-time application to handle unexpected situations and recover gracefully.

Comparison of QNX and Real-Time Linux

QNX and Real-Time Linux are both popular choices for building real-time and embedded systems, but they have different characteristics, features, and performance considerations. Table 3 gives a comparison of QNX and Real-Time Linux in terms of real-time performance.

Characteristics	QNX	Real-Time Linux
Determinism	Better determinism and real-time performance out of the box due to its microkernel architecture	Real-Time Linux can achieve good determinism with proper configuration and patches
Latency	Has lower latencies and faster context switching, making it suitable for hard real-time applications	Achieves low latencies with specialised patches but might have slightly higher latencies compared to QNX
Ease of use	QNX is designed specifically for real-time applications and provides a comprehensive environment for embedded systems development	Real-Time Linux may require more configuration and tuning to achieve optimal real-time performance
Ecosystem and compatibility	QNX has a more focused ecosystem tailored to embedded and real-time applications.	Real-Time Linux benefits from the vast Linux ecosystem, including a wide range of device drivers and software libraries

In making your decision, consider the following factors.

Timing requirements: PREEMPT-RT patch converts standard Linux to Real-Time Linux. The latter has good determinism to achieve real-time capabilities and reduces scheduling latency. However, QNX may be slightly better compared to Real-Time Linux if your application requires hard real-time capabilities and precise determinism, though the latter can be used based on the requirements.

Reliability and safety: QNX is often used in safety-critical systems including medical devices. It is designed with safety in mind and provides features for safety certification. Real-Time Linux can be used in safety-critical applications, but achieving safety certification may require additional effort and adherence to industry-specific safety standards.

Ecosystem and tools: Consider the availability of development tools, libraries, and drivers that match your project’s requirements.

Familiarity: If your team is already experienced with Linux development, Real-Time Linux with PREEMPT_RT patch, kernel config options, IRQ handling and thread priorities may be a more seamless choice. SCHED_FIFO, SCHED_RR are often used for Real-Time threads. These factors will improve real-time capabilities in the Linux environment.

Project scope: The size and complexity of the project may influence which platform is more suitable.

In conclusion, both QNX and Real-Time Linux are good options for developing real-time and embedded systems; each comes with its own strengths and considerations. The choice depends on the specific application requirements, the level of real-time performance needed, familiarity with the platforms, the available tools, ecosystem, and the level of determinism needed.