1. Overview
In this article, we’ll discuss the various frameworks and tools we use for testing the Linux kernel. First, we’ll start up by covering the comprehensive Linux Test Project. Then, we’ll go through the other testing tools and frameworks.
Finally, we’ll discuss the process of testing new Linux drivers that we cannot test with the regular Kernel testing tools.
2. The Linux Test Project (LTP)
The Linux Test Project is an open-source collection of automation tools that we can use to create a build process, carry out automated tests, and debug the Linux kernel. The LTP project is developed by IBM, Cisco, RHEL, and other organizations.
2.1. The Build System
The build system of LTP gives structure to a Linux kernel project. Not only that, it uses the new Makefile system, which provides an enhanced build process, thereby making the maintenance of a Linux project more manageable.
2.2. C Test Cases
We can write LTP test cases in the C language or a portable POSIX compliant shell language. The test cases consume the LTP Test API, which is solely written in the C language. Moreover, we can write the test cases for glibc and the musl C libraries.
3. Autotest
Autotest is a testing framework that is designed specifically for testing out the Linux kernel. It provides modules for setting up completely automated test grids.
The Autotest modules include:
- A client module to carry out the actual execution of the tests inside the project directory
- A server module that makes it easy for us to manage the test clients on multiple remote machines
- An Autotest database module that stores the results of the test grids
- A scheduler that executes cron jobs related to the test grids
- A command-line interface for using Autotest
- A web front-end for visualizing test results and scheduling jobs
For rigorous testing, we can also combine LTP test cases with the Autotest scheduler because Autotest doesn’t provide the C Test API for testing Linux kernels.
4. Kmemleak
Kmemleak is a development feature of the mainline Linux kernel. The memory allocated by the kernel is, sometimes, not freed. Therefore, it can become both a performance issue and a security risk.
With Kmemleak, we can check for memory leaks produced by the kernel. However, it doesn’t fix the issue but instead traces the possible leaks and reports them to /sys/kernel/debug/memleak. In a way, Kmemleak is very similar to a garbage collector in high-level languages such as Java and C#.
Furthermore, if we’re working on a Linux kernel, we can use this feature by enabling “CONFIG_DEBUG_KMEMLEAK” in the kernel config. On most popular distributions, this feature will be disabled by default:
$ zgrep "CONFIG_DEBUG_KMEMLEAK" /proc/config.gz
# CONFIG_DEBUG_KMEMLEAK is not set
It makes sense because we wouldn’t need it in production.
5. Kmemcheck
Like Kmemleak, Kmemcheck is also a development feature of the Linux kernel. However, they both perform different tasks.
Kmemcheck is responsible for checking whether the kernel code accesses an uninitialized structure in the memory. For instance, the kernel allocates memory blocks but doesn’t fill them and, then, tries to access that location. This can lead to serious bugs and performance issues.
We should note that this feature is only available for x86 systems.
6. Linaro LAVA
LAVA (Linaro Automation and Validation Architecture) is a continuous integration for deploying operating systems onto physical and virtual hardware for running tests. These tests include boot, bootloader, and system-level testing. It is specifically designed for ARM systems.
We can use LAVA to check for the validity of the changes that we make to the kernel code. Not only that, but we can also check whether the kernel is optimized for both speed and size. LAVA will then write these metrics to a file that we can further analyze.
The LAVA framework is mostly used for testing out mobile operating systems due to the fact that the majority of mobile phones use ARM or some form of ARM architecture.
7. Step Debuggers
A step debugger is a tool that allows us to execute a program’s code line-by-line. It provides us with an easy way to monitor the program flow and figure out obscure errors. However, we should know that step debuggers do not perform unit testing.
7.1. GDB
On Linux, we have GDB (GNU Debugger) and KGDB (Kernel GDB). GDB is a generic debugger that allows us to debug the code of many programming languages, including C. Although we cannot debug the Kernel code with GDB directly, we can set up QEMU with GDB to easily go through the kernel code in execution.
7.2. KGDB
On the other hand, KGDB is a debugger for BSD operating systems as well as Linux. In order to use KGDB, we’d need two machines that are connected through a serial connection or a firewire link. One machine will run the kernel while the other will run the debugger. These two machines communicate through the GDB remote protocol.
8. Device Driver Testing
Testing device drivers for Linux can become very tedious because there is no silver bullet for testing device drivers for Linux. Therefore, all the testing will have to be done manually on physical hardware. Although we can test the drivers on a virtual machine, it might not work properly due to the abstraction layer added by the virtual machine.
For testing the drivers, we’d need to load the driver manually after booting up Linux to check for any initialization errors. Afterward, we can use the step debuggers and, in some cases, execute automated tests.
9. Conclusion
In this article, we briefly went through the different tools and frameworks that we commonly use for different purposes to test the Linux kernel. We went through LTP, Autotest, Linux in-built testing capabilities, the LAVA framework, and step debuggers.
Finally, we briefly touched on the subject of testing device drivers for Linux.