Compile and Deploy a Real-Time Kernel to the ComfilePi

Last updated on Jan 24, 2024

The generic kernel employed the the default ComfilePi operating system is not a real-time kernel. That is fine for many, if not most, use cases. For those use cases that may require real-time operation, we provide the following procedure.

In addition to compiling and deploying a realtime kernel, other OS configurations may be necessary to improve the realtime performance.

Please note that COMFILE Technology does not maintain the Raspberry Pi kernel or the Linux real-time patches, so those wishing to employ this capability should assume responsibility for it.

Kernel Versioning

The Raspberry Pi Foundation maintains the kernel for Raspberry Pi-based products at https://github.com/raspberrypi/linux. The real-time kernel patches are maintained by a different group at https://cdn.kernel.org/pub/linux/kernel/projects/rt/. Compile errors and/or patch errors can arise if the 3 digit version number of the Raspberry Pi kernel and the real time patch do not match. The Raspberry Pi Foundation and the real-time patch maintainers do no coordinate their releases, so one must search the kernel's commit history for a “Linux x.y.z” commit that matches the real-time kernel patch version, then use the “Merge remote-tracking…” commit that follows it. At the time of writing this, the latest such commit for the 5.10.y branch was here:

Build Script

Given the versioning limitations explained above, the following shell script cross-compiles kernel version 6.1.26. It will download all necessary software and source code, cross-compile the real-time kernel, and package the resulting assests for deployment to a ComfilePi panel PC. It was tested on a Debian instance of Windows Subsystem for Linux.

#!/bin/bash
 
# You may need to install the following packages
# apt update -y
# apt install -y wget git bc bison flex libssl-dev make libc6-dev libncurses5-dev crossbuild-essential-$ARCH
 
KERNEL_VERSION=6.1
RT_PATCH_VERSION=$KERNEL_VERSION.26-rt8
COMMIT_HASH=dbcb82357ef21be47841ba39d117b626d715af31
DEFCONFIG=bcm2711_defconfig
ARCH=arm64
TARGET=aarch64-linux-gnu
KERNEL_NAME=kernel8
 
KERNEL_BRANCH=rpi-$KERNEL_VERSION.y
COMPILER=$TARGET-
 
# All work will be done in $PWD/rtkernel.  The final result will be packaged as $PWD/rtkernel/result
WORK_DIR=`pwd`/rtkernel
rm -rf $WORK_DIR
mkdir $WORK_DIR
cd $WORK_DIR
 
# download the kernel
git clone --depth=1 --branch $KERNEL_BRANCH https://github.com/raspberrypi/linux
 
# download the RT patch
wget http://cdn.kernel.org/pub/linux/kernel/projects/rt/$KERNEL_VERSION/older/patch-$RT_PATCH_VERSION.patch.gz
gunzip patch-$RT_PATCH_VERSION.patch.gz
 
# Need this to get precisely the correct kernel version
cd linux
git fetch origin $COMMIT_HASH
git reset --hard $COMMIT_HASH
 
# Patch the kernel
patch -p1 < ../patch-$RT_PATCH_VERSION.patch
 
 # Prepare configuration
make ARCH=$ARCH CROSS_COMPILE=$COMPILER $DEFCONFIG
 
 # Configure to use the "Fully Preemptible Kernel (Real-Time)" kernel
./scripts/config --set-val CONFIG_PREEMPT_RT y
./scripts/config --set-val CONFIG_RCU_BOOST y
./scripts/config --set-val CONFIG_RCU_BOOST_DELAY 500
./scripts/config --set-val CONFIG_RCU_NOCB_CPU y
./scripts/config --set-val CONFIG_NO_HZ_FULL y
./scripts/config --set-val CONTEXT_TRACKING_USER_FORCE n
./scripts/config --set-val RCU_NOCB_CPU_DEFAULT_ALL n
./scripts/config --set-val RCU_NOCB_CPU_CB_BOOST y
 
# Compile
make -j32 ARCH=$ARCH CROSS_COMPILE=$COMPILER Image.gz modules dtbs
 
# copy assets to the root file system
make ARCH=$ARCH CROSS_COMPILE=$COMPILER INSTALL_MOD_PATH=modules_to_install modules_install
 
 # Out of linux, out of work_dir
cd ../../
 
# copy assets to the $PROJECT_DIR/result directory
RESULT_DIR=$WORK_DIR/result
mkdir -p $RESULT_DIR/boot/overlays
cp ${WORK_DIR}/linux/arch/$ARCH/boot/Image.gz ${RESULT_DIR}/boot/$KERNEL_NAME.img
cp -r ${WORK_DIR}/linux/modules_to_install/lib/* ${RESULT_DIR}/lib/
cp ${WORK_DIR}/linux/arch/$ARCH/boot/dts/broadcom/*.dtb ${RESULT_DIR}/boot/
cp ${WORK_DIR}/linux/arch/$ARCH/boot/dts/overlays/*.dtb* ${RESULT_DIR}/boot/overlays/
cp ${WORK_DIR}/linux/arch/$ARCH/boot/dts/overlays/README ${RESULT_DIR}/boot/overlays/

Deploy to the ComfilePi

After running the build script above, the rtkernel/result directory will contain 2 directories: boot and lib. The boot directory contains assets that must be deployed to the ComfilePi's FAT32 partition (i.e. the /boot directory) and the lib directory contains assets that must be deployed to the ComfilePi's operating system root partition (i.e. the /lib directory).

On the build PC:

# Package up the result into a single file
tar czf rtkernel.tgz -C rtkernel/result/ .

# transfer the file to the ComfilePi
scp rtkernel.tgz pi@192.168.0.5:

Download rtkernel_6.1.26-rt8.tgz (MD5: cadb563fc3cc61e9a2a5fc571a08004a)

On the ComfilePi:

# Install the kernel assets
sudo tar xzf rtkernel.tgz --directory / --keep-directory-symlink --no-same-owner

After rebooting, verify that the new kernel is loaded and running

$uname -a
Linux raspberrypi 6.1.21-rt8-v8+ #1 SMP PREEMPT_RT Fri May 12 10:19:19 KST 2023 aarch64 GNU/Linux

References

Creating a real-time, low latency configuration for a Raspberry Pi using make menuconfig involves setting various kernel options. Below is a table that outlines some of the important options that you should configure for a real-time, low jitter, and low latency system. This table assumes you are using a Linux kernel that has the PREEMPT_RT patch applied, which is essential for full real-time capabilities.

Menu Entry	Option	Location in `make menuconfig`	Notes
Preemption Model	Fully Preemptible Kernel (RT)	General Setup → Preemption Model	Select "Fully Preemptible Kernel (RT)" for lowest latency.
Timer Frequency	1000 HZ	Processor type and features → Timer frequency (Hz)	Higher frequency timers can improve responsiveness but may increase CPU utilization.
High Resolution Timer Support	Enabled	Processor type and features → High Resolution Timer Support	Needed for accurate timing.
Control Group support	Enabled	General Setup → Control Group Support	Useful for managing resources.
CPU Frequency scaling	Disabled	CPU Power Management → CPU Frequency scaling	Disabling can reduce variability in performance, improving latency.
Real-time group scheduling	Enabled	General Setup → Group scheduling for SCHED_RR/FIFO	Allows for better control of real-time tasks scheduling.
Lock debugging	Disabled	Kernel hacking → Lock Debugging (spinlocks, mutexes, etc.)	Disabling can reduce overhead but makes debugging harder.
Real-Time Threading Support	Enabled	General Setup → POSIX Message Queues	Required for real-time applications.
Interrupt pipeline	Enabled	Kernel Features → Interrupt pipeline	Necessary if the kernel supports the interrupt pipeline feature for better RT behavior.
Thread IRQs	Enabled	Kernel Features → Thread IRQs	Threads IRQs for better real-time performance.

Please note that the availability and exact location of these options may vary depending on the specific kernel version and configuration. Additionally, be aware that enabling real-time features may affect overall system performance and power consumption.

It is recommended to consult the documentation for the specific kernel version you are working with and to test your configuration thoroughly to ensure it meets your real-time requirements. Some of these settings may require a deeper understanding of the system's workload and the trade-offs involved in configuring a real-time system.