Step 3 – Configuring the Linux kernel

Configuring the kernel is perhaps the most critical step in the kernel build process. One of the many reasons Linux is a critically acclaimed OS is its versatility. It’s a common misconception to think that there is a separate Linux kernel code base for an enterprise-class server, a data center, a workstation, and a tiny, embedded Linux device – no, they all use the very same unified Linux kernel source! Thus, carefully configuring the kernel for a particular use case (server, desktop, embedded, or hybrid/custom) is a powerful feature and a requirement. This is precisely what we are delving into here.

You’ll probably find that this topic – arriving at a working kernel config – tends to be a long and winding discussion, but it’s ultimately worth it; do take the time and trouble to read through it.

Next, we cover a bit of required background on the kernel build system.

.

Minimally understanding the Kconfig/Kbuild build system

The infrastructure that the Linux kernel uses to configure the kernel is known as the Kconfig system, and to build it, there is the Kbuild infrastructure. Without delving into the gory details, the Kconfig + Kbuild system ties together the complex kernel configuration and build process by separating out the work into logical streams:

• Kconfig – the infrastructure to configure the kernel; it consists of two logical parts:
  • The Kconfig language: it’s used to specify the syntax within the various Kconfig[.*] files, which in effect specify the “menus” where kernel config options are selected.
  • The Kconfig parsers: tools that intelligently parse the Kconfig[.*] files, figure out dependencies and auto-selections, and generate the menu system. Among them is the commonly used make menuconfig, which internally invokes the mconf tool (the code’s under scripts/kconfig).
• Kbuild – the support infrastructure to build the source code into kernel binary components. It mainly uses a recursive make style build, originating at the kernel top-level Makefile, which, in turn recursively parses the content of hundreds of Makefiles embedded into sub-directories within the source (as required).

To help you gain a better understanding, let’s look at a few of the key components that go into the Kconfig/Kbuild system:

• The CONFIG_FOO symbols
• The menu specification file(s), named Kconfig[.*]
• The Makefile(s)
• The overall kernel config file – .config – itself.

The purposes of these components are summarized as follows:

Table 2.3: Major components of the Kconfig+Kbuild build system

How the Kconfig+Kbuild system works – a minimal take

Now that we know a few details, here’s a simplified take on how it’s tied together and works:

• First, the user (you) configures the kernel using some kind of menu system provided by Kconfig.
• The kernel config directives selected via this menu system UI are written into a few auto-generated headers and a final .config file, using a CONFIG_FOO={y|m} syntax, or, CONFIG_FOO is simply commented out (implying “don’t build FOO at all”).
• Next, the Kbuild per-component Makefiles (invoked via the kernel top-level Makefile) typically specify a directive FOO like this:

  obj-$(CONFIG_FOO) += FOO.o
  

• A FOO component could be anything – a core kernel feature, a device driver, a filesystem, a debug directive, and so on. Recall, the value of CONFIG_FOO may be y, or m, or not exist; this accordingly has the build either build the component FOO into the kernel (when its value is y), or as a module (when its value is m)! If commented out, it isn’t built at all, simple. In effect, the above-mentioned Makefile directive, at build time, expands into one of these three for a given kernel component FOO:

  obj-y += FOO.o      # build the feature FOO into the kernel image
  obj-m += FOO.o     # build the feature FOO as a discrete kernel module (a                     #  foo.ko file)
  <if CONFIG_FOO is null>        # do NOT build feature FOO
  

To see an instance of this in action, check out the Kbuild file (more details on Kconfig files are in the Understanding the Kconfig* files section) in the root of the kernel source tree:

$ cat Kconfig
…
# Kbuild for top-level directory of the kernel
…
# Ordinary directory descending
# ---------------------------------------------------------------------------
obj-y                   += init/
obj-y                   += usr/
obj-y                   += arch/$(SRCARCH)/
obj-y                   += $(ARCH_CORE)
obj-y                   += kernel/
[ … ]
obj-$(CONFIG_BLOCK)     += block/
obj-$(CONFIG_IO_URING)  += io_uring/
obj-$(CONFIG_RUST)      += rust/
obj-y                   += $(ARCH_LIB)
[ … ]
obj-y                   += virt/
obj-y                   += $(ARCH_DRIVERS)

Interesting! We can literally see how the top-level Makefile will descend into other directories, with the majority being set to obj-y; in effect, build it in (in a few cases it’s parametrized, becoming obj-y or obj-m depending on how the user selected the option).

Great. Let’s move along now; the key thing to do is to get ourselves a working .config file. How can we do so? We do this iteratively. We begin with a “default” configuration – the topic of the following section – and carefully work our way up to a custom config.

Arriving at a default configuration

So, how do you decide on the initial kernel configuration to begin with? Several techniques exist; a few common ones are as follows:

• Don’t specify anything; Kconfig will pull in a default kernel configuration (as all kernel configs have a default value)
• Use the existing distribution’s kernel configuration
• Build a custom configuration based on the kernel modules currently loaded in memory

The first approach has the benefit of simplicity. The kernel will handle the details, giving you a default configuration. The downside is that the default config can be very large (this is the case when building Linux for an x86_64-based desktop or server-type system); a huge number of options are turned on by default, just in case you need it, which can make the build time very long and the kernel image size very large. Typically, of course, you are then expected to manually configure the kernel to the desired settings.

This brings up the question, where is the default kernel config stored? The Kconfig system uses a priority list fallback scheme to retrieve a default configuration if none is specified. The priority list and its order (the first being the highest priority) is as follows:

• .config
• /lib/modules/$(uname -r)/.config
• /etc/kernel-config
• /boot/config-$(uname -r)
• ARCH_DEFCONFIG (if defined)
• arch/${ARCH}/defconfig

From the list, you can see that the Kconfig system first checks for the presence of a .config file in the root of the kernel source tree; if found, it picks up all the config values from there. If it doesn’t exist, it next looks at the path /lib/modules/$(uname -r)/.config. If found, the values found in that file will be used as the defaults. If not found, it checks the next one in the preceding priority list, and so on… You can see this shown in Figure 2.8.

A diagram (inspired by Cao Jin’s articles) that attempts to communicate the kernel’s Kconfig/Kbuild system is shown here. The diagram conveys more information than has been covered by now; worry not, we’ll get to it.

Figure 2.8: The kernel’s Kconfig/Kbuild system in a simplified form

Right, let’s now get to figuring out how exactly to get a working kernel config!

Obtaining a good starting point for kernel configuration

This brings us to a really important point: while playing around with the kernel configuration is okay to do as a learning exercise, for a production system it’s critical that you base your custom config on a proven – known, tested, and working – kernel configuration.

Here, to help you understand the nuances of selecting a valid starting point for kernel configuration, we will see three approaches to obtaining a starting point for a typical kernel configuration:

• First, an easy (but sub-optimal) approach where you simply emulate the existing distribution’s kernel configuration.
• Next, a more optimized approach where you base the kernel configuration on the existing system’s in-memory kernel modules. This is the localmodconfig approach.
• Finally, a word on the approach to follow for a typical embedded Linux project.

Let’s examine each of these approaches in a bit more detail. In terms of configuring the kernel you’ve downloaded and extracted in the previous two steps, don’t do anything right now; read the sections that follow, and then, in the Getting going with the localmodconfig approach section, we’ll have you actually get started.

Kernel config using distribution config as a starting point

The typical target system for using this approach is a x86_64 desktop or server Linux system. Let’s configure the kernel to all defaults:

$ make mrproper
  CLEAN   scripts/basic
  CLEAN   scripts/kconfig
  CLEAN   include/config include/generated .config

To ensure we begin with a clean slate, we run make mrproper first; be careful, it cleans pretty much everything, including the .config if it exists.

Next, we perform the make defconfig step, which, as the make help command output shows (try it out! See Figure 2.10), gives us a new config:

$ make defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
 [ … ]
 HOSTLD  scripts/kconfig/conf
*** Default configuration is based on 'x86_64_defconfig'
#
# configuration written to .config
# 

The building of the mconf utility itself is performed first (under scripts/kconfig), and then the config is generated. Here it is:

$ ls -l .config 
-rw-rw-r-- 1 c2kp c2kp 136416 Apr 29 08:12 .config

There, done: we now have an “all-defaults” kernel config saved in .config!

What if there’s no defconfig file for your arch under arch/${ARCH}/configs? Then, at least on x86_64, you can simply copy in the existing distro default kernel config:

cp /boot/config-$(uname -r) ${LKP_KSRC}/.config

Here, we simply copy the existing Linux distribution’s (here, it’s our Ubuntu 22.04 LTS guest VM) config file into the .config file in the root of the kernel source tree, thereby making the distribution config the starting point, which can then be further edited. As already mentioned, the downside of this quick approach is that the config tends to be large, thus resulting in a large-footprint kernel image.

Also, FYI, once the kernel config is generated in any manner (like via make defconfig), every kernel config FOO is shown as an empty file within include/config.

Tuned kernel config via the localmodconfig approach

The typical target system for using this approach is a (typically x86_64) desktop or server Linux system.

This second approach is more optimized than the previous one – a good one to use when the goal is to begin with a kernel config that is based on your existing running system and is thus (usually) relatively compact compared to the typical default config on a desktop or server Linux system.

Here, we provide the Kconfig system with a snapshot of the kernel modules currently running on the system by simply redirecting the output of lsmod into a temporary file and then providing that file to the build. This can be achieved as follows:

lsmod > /tmp/lsmod.now
cd ${LKP_KSRC}
make LSMOD=/tmp/lsmod.now localmodconfig

The lsmod utility simply lists all the kernel modules currently residing in system kernel memory. We will see more on this in Chapter 4, Writing Your First Kernel Module – Part 1.

We save its output in a temporary file, which we then pass via the LSMOD environment variable to the Makefile's localmodconfig target. The job of this target is to configure the kernel in a manner as to only include the base functionality plus the functionality provided by these kernel modules and leave out the rest, in effect giving us a reasonable facsimile of the current kernel (or of whichever kernel the lsmod output represents). We use precisely this technique to configure our 6.1 kernel in the upcoming Getting going with the localmodconfig approach section. We also show only this approach as step 1 (1 in the circle) in Figure 2.8.

Kernel config for typical embedded Linux systems

The typical target system for using this approach is usually a small embedded Linux system. The goal here is to begin with a proven – a known, tested, and working – kernel configuration for our embedded Linux project. Well, how exactly can we achieve this?

Before going further, let me mention this: the initial discussion here will be shown to be the older approach to configuring (the AArch32 or ARM-32 arch) embedded Linux; we shall then see the “correct” and modern approach for modern platforms.

Interestingly, for the AArch32 at least, the kernel code base itself contains known, tested, and working kernel configuration files for various well-known hardware platforms.

Assuming our target is ARM-32 based, we merely must select the one that matches (or is the nearest match to) our embedded target board. These kernel config files are present within the kernel source tree in the arch/<arch>/configs/ directory. The config files are in the format <platform-name>_defconfig.

A quick peek is in order; see the following screenshot showing, for the ARM-32, the existing board-specific kernel code under arch/arm/mach-<foo> and platform config files under arch/arm/configs on the v6.1.25 Linux kernel code base:

Figure 2.9: The contents of arch/arm and arch/arm/configs on the 6.1.25 Linux kernel

Whoah, quite a bit! The directories arch/arm/mach-<foo> represent hardware platforms (boards or machines) that Linux has been ported to (typically by the silicon vendor); the board-specific code is within these directories.

Similarly, working default kernel config files for these platforms are also contributed by them and are under the arch/arm/configs folder of the form <foo>_defconfig; as can clearly be seen in the lower portion of Figure 2.9.

Thus, for example, if you find yourself configuring the Linux kernel for a hardware platform having, say, an i.MX 7 SoC from NXP on it, please don’t start with an x86_64 kernel config file as the default. It won’t work. Even if you manage it, the kernel will not build/work cleanly. Pick the appropriate kernel config file: for our example here, perhaps the imx_v6_v7_defconfig file would be a good starting point. You can copy this file into .config in the root of your kernel source tree and then proceed to fine-tune it to your project-specific needs.

As another example, the Raspberry Pi (https://www.raspberrypi.org/) is a very popular hobbyist and production platform. The kernel config file – within its kernel source tree – used as a base for it is this one: arch/arm/configs/bcm2835_defconfig. The filename reflects the fact that Raspberry Pi boards use a Broadcom 2835-based SoC. You can find details regarding kernel compilation for the Raspberry Pi here: https://www.raspberrypi.org/documentation/linux/kernel/building.md. Hang on, though, we will be covering at least some of this in Chapter 3, Building the 6.x Linux Kernel from Source – Part 2, in the Kernel build for the Raspberry Pi section.

The modern approach – using the Device Tree

Okay, a word of caution! For AArch32, as we saw, you’ll find the platform-specific config files under arch/arm/configs as well as the board-specific kernel code under arch/arm/mach-<foo>, where foo is the platform name. Well, the reality is that this approach – keeping board-specific config and kernel source files within the Linux OS code base – is considered to be exactly the wrong one for an OS! Linus has made it amply clear – the “mistakes” made in older versions of Linux, which happened when the ARM-32 was the popular arch, must never be repeated for other arches. Then how does one approach this? The answer, for the modern (32 and 64-bit) ARM and PPC architectures, is to use the modern Device Tree (DT) approach.

Very basically, the DT holds all platform hardware topology details. It is effectively the board or platform layout; it’s not code, it’s a description of the hardware platform analogous to VHDL. BSP-specific code and drivers still need to be written, but now have a neat way to be “discovered” or enumerated by the kernel at boot when it parses the DTB (Device Tree Blob) that’s passed along by the bootloader. The DTB is generated as part of the build process, by invoking the DTC (Device Tree Compiler) on the DT source files for the platform.

So, nowadays, you’ll find that the majority of embedded projects (for ARM and PPC at least) will use the DT to good effect. It also helps OEMs and ODMs/vendors by allowing usage of essentially the same kernel with platform/model-specific tweaks built into the DT. Think of the dozens of Android phone models from popular OEMs with mostly the same stuff but just a few hardware differences; one kernel will typically suffice! This dramatically eases the maintenance burden. For the curious – the DT sources – the .dts files can be found here: arch/<arch>/boot/dts.

The lesson on keeping board-specific stuff outside the kernel code base as far as is possible seems to have been learned well for the AArch64 (ARM-64). Compare its clean, well-organized, and uncluttered config and DTS folders (arch/arm64/configs/: it has only one file, a defconfig) to AArch32. Even the DTS files (look under arch/arm64/boot/dts/) are well organized compared to AArch32:

6.1.25 $ ls arch/arm64/configs/
defconfig
6.1.25 $ ls arch/arm64/boot/dts/
actions/      amazon/     apm/          bitmain/     exynos/       intel/       marvell/      nuvoton/    realtek/      socionext/   tesla/        xilinx/        allwinner/    amd/        apple/        broadcom/    freescale/    lg/          mediatek/     nvidia/     renesas/      sprd/        ti/           altera/   amlogic/      arm/        cavium/       hisilicon/   Makefile      microchip/     qcom/         rockchip/   synaptics/    toshiba/

So, with modern embedded projects and the DT, how is one to go about kernel/BSP tasks? Well, there are several approaches; the BSP work is:

• Carried out by an in-house BSP or platform team.
• Provided as a BSP “package” by a vendor (often the silicon vendor that you’ve partnered with) along with reference hardware.
• Outsourced to an external company or consultant that you’ve partnered with. Several companies exist in this space – among them are Siemens (ex Mentor Graphics), Timesys, and WindRiver.
• Often nowadays, with the project being built and integrated via sophisticated builder software like Yocto or Buildroot, the vendors contribute BSP layers, which are then integrated into the product by the build team.

  Design: A bit off-topic, but I think it’s important: when working on projects (especially embedded), teams have repeatedly shown the undesirable tendency to directly employ vendor SDK APIs to perform device-specific work in their apps and drivers. Now, at first glance, this might seem fine; it can become a huge burden when the realization dawns that, hey, requirements change, devices themselves change, and thus your tightly coupled software simply breaks! You had the apps (and drivers) tie into the device hardware with hardly any separation.

  The solution, of course, is to use a loosely coupled architecture, with what’s essentially a HAL (Hardware Abstraction Layer) to allow apps to interface with devices seamlessly. This also allows for the device-specific code to be changed without affecting the higher layers (apps). Designing this way might seem obvious in the abstract, essentially leveraging the information-hiding idea, but can be difficult to ensure in practice; do always keep this in mind. The fact is, Linux’s device model encourages this loose coupling. The Further reading section has some good links on these design approaches, within the Generic online and book resources... section (here: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E/blob/main/Further_Reading.md#generic-online-and-book-resources--miscellaneous-very-useful).

Right, that concludes the three approaches to setting up a starting point for kernel configuration.

Seeing all available config options

As a matter of fact, with regard to kernel config, we have just scratched the surface. Many more techniques to explicitly generate the kernel configuration in a given manner are encoded into the Kconfig system itself! How? Via configuration targets to make. See them by running make help in the root of your kernel source; they’re under the Configuration targets heading:

Figure 2.10: Output from make help on an x86_64 (6.1.25 kernel) with interesting lines highlighted

Let’s experiment with a couple of other approaches as well – the oldconfig one to begin with:

$ make mrproper
[ … ]
$ make oldconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  [ … ]
 HOSTLD  scripts/kconfig/conf
 #
 # using defaults found in /boot/config-5.19.0-41-generic
 #
 [ … ]
 * Restart config...
 [ … ]
 *
 Control Group support (CGROUPS) [Y/?] y
  Favor dynamic modification latency reduction by default   (CGROUP_FAVOR_DYNMODS) [N/y/?] (NEW)                  << waits for input here >>
 [ … ]

It works of course, but you’ll have to press the Enter key (a number of times, perhaps) to accept defaults for any and every newly detected kernel config… (or you can specify a value explicitly; we shall see more on these new kernel configs in the section that follows). This is quite normal, but a bit of an annoyance. There’s an easier way : using the olddefconfig target; as its help line says – “Same as oldconfig but sets new symbols to their default value without prompting”:

$ make olddefconfig
#
# using defaults found in /boot/config-5.19.0-41-generic
#
.config:10301:warning: symbol value 'm' invalid for ANDROID_BINDER_IPC
.config:10302:warning: symbol value 'm' invalid for ANDROID_BINDERFS
#
# configuration written to .config
#
Done.

Viewing and setting new kernel configs

Another quick experiment: we clean up and then copy in the distro kernel config. Keep a backup if you want your existing .config.

$ make mrproper
$ cp /boot/config-5.19.0-41-generic .config
cp: overwrite '.config'? y
$ 

We have the distro defaults in .config. Now, think on it: we’re currently running the 5.19.0-41-generic distro kernel but are intending to build a new kernel, 6.1.25. So, the new kernel’s bound to have at least a few new kernel configs. In cases like this, when you attempt to configure the kernel, the Kconfig system will question you: it will display every single new config option and the available values you can set it to, with the default one in square brackets, in the console window. You’re expected to select the values for the new config options it encounters. You will see this as a series of questions and a prompt to answer them on the command line.

The kernel provides two interesting mechanisms to see all new kernel configs:

• listnewconfig – list new options
• helpnewconfig – list new options and help text

Running the first merely lists every new kernel config variable:

$ make listnewconfig
[ … ]
CONFIG_CGROUP_FAVOR_DYNMODS=n
CONFIG_XEN_PV_MSR_SAFE=y
CONFIG_PM_USERSPACE_AUTOSLEEP=n
[ … ]
CONFIG_TEST_DYNAMIC_DEBUG=n

There are lots of them – I got 108 new configs – so I’ve truncated the output here.

We can see all the new configs, though the output’s not very helpful in understanding what exactly they mean. Running the helpnewconfig target solves this – you can now see the “Help” (from the config option’s Kconfig file) for every new kernel config:

$ make helpnewconfig
[ … ] 
CONFIG_CGROUP_FAVOR_DYNMODS:
This option enables the favordynmods mount option by default, which reduces the latencies of dynamic cgroup modifications such as task migrations and controller on/offs at the cost of making hot path operations such as forks and exits more expensive.
Say N if unsure.
Symbol: CGROUP_FAVOR_DYNMODS [=n]
Type  : bool
Defined at init/Kconfig:959
  Prompt: Favor dynamic modification latency reduction by default
[ … ]
CONFIG_TEST_DYNAMIC_DEBUG:
This module registers a tracer callback to count enabled pr_debugs in a do_debugging function, then alters their enablements, calls the function, and compares counts.
If unsure, say N.
Symbol: TEST_DYNAMIC_DEBUG [=n]
[ … ]
$ 

Don’t worry about understanding the Kconfig syntax for now; we shall cover it in the Customizing the kernel menu, Kconfig, and adding our own menu item section.

The LMC_KEEP environment variable

Also, did you notice in Figure 2.10 that the localmodconfig and localyesconfig targets can optionally include an environment variable named LMC_KEEP (LMC is LocalModConfig)?

Its meaning is straightforward: setting LMC_KEEP to some colon-delimited values has the Kconfig system preserve the original configs for the specified paths. An example might look like this: "drivers/usb:drivers/gpu:fs". In effect, it says, “Keep these modules enabled.”

This is a feature introduced in the 5.8 kernel (the commit’s here: https://github.com/torvalds/linux/commit/c027b02d89fd42ecee911c39e9098b9609a5ca0b). So, to make use of it, you could run the config command like this:

make LSMOD=/tmp/mylsmod \
  LMC_KEEP="drivers/usb:drivers/gpu:fs" \
              localmodconfig

Tuned config via the streamline_config.pl script

Interestingly, the kernel provides many helper scripts that can perform useful housekeeping, debugging, and other tasks within the scripts/ directory. A good example with respect to what we’re discussing here is the scripts/kconfig/streamline_config.pl Perl script. It’s ideally suited to situations where your distro kernel has too many modules or built-in kernel features enabled, and you just want the ones that you’re using right now – the ones that the currently loaded modules provide, like localmodconfig. Run this script with all the modules you want loaded up, saving its output to, ultimately, .config. Then run make oldconfig and the config’s ready!

As a sidebar, here’s how the original author of this script – Steven Rostedt – describes what it does (https://github.com/torvalds/linux/blob/master/scripts/kconfig/streamline_config.pl):

[…]
# Here's what I did with my Debian distribution.
#
#    cd /usr/src/linux-2.6.10
#    cp /boot/config-2.6.10-1-686-smp .config
#    ~/bin/streamline_config > config_strip
#    mv .config config_sav
#    mv config_strip .config
#    make oldconfig
[…]

You can try it if you wish.

Getting going with the localmodconfig approach

Now (finally!) let’s get hands-on and create a reasonably sized base kernel configuration for our 6.1.25 LTS kernel by using the localmodconfig technique. As mentioned, this existing-kernel-modules-only approach is a good one when the goal is to obtain a starting point for kernel config on an x86-based system by keeping it tuned to the current host.

Before going any further, it’s a good idea to clean up the source tree, especially if you ran the experiments we worked on previously. Be careful: this command will wipe everything, including the .config:

make mrproper

As described previously, let’s first obtain a snapshot of the currently loaded kernel modules, and then have the build system operate upon it by specifying the localmodconfig target, like so:

lsmod > /tmp/lsmod.now cd ${LKP_KSRC}
make LSMOD=/tmp/lsmod.now localmodconfig

Now, when you run the make [...] localmodconfig command just shown, it’s entirely possible, indeed probable, that there will be a difference in the configuration options between the kernel you are currently configuring (version 6.1.25) and the kernel you are currently running on the build machine (for myself, the host kernel is $(uname -r) = 5.19.0-41-generic). In such cases, as explained in the Viewing and setting new kernel configs section, the Kconfig system will question you for each new config; pressing Enter accepts the default.

Now let’s make use of the localmodconfig command.

Enter the following commands (if not already done):

$ uname -r
5.19.0-41-generic
$ lsmod > /tmp/lsmod.now
$ make LSMOD=/tmp/lsmod.now localmodconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  [ ... ]
using config: '/boot/config-5.19.0-41-generic'
System keyring enabled but keys "debian/canonical-certs.pem" not found. Resetting keys to default value.
*
* Restart config...
*
* Control Group support
*
Control Group support (CGROUPS) [Y/?] y
  Favor dynamic modification latency reduction by default (CGROUP_FAVOR_DYNMODS) [N/y/?] (NEW)   ↵
  Memory controller (MEMCG) [Y/n/?] y
[ … ]
Userspace opportunistic sleep (PM_USERSPACE_AUTOSLEEP) [N/y/?] (NEW)   ↵
[ … ]
Multi-Gen LRU (LRU_GEN) [N/y/?] (NEW)   ↵
[ … ]
 Rados block device (RBD) (BLK_DEV_RBD) [N/m/y/?] n
 Userspace block driver (Experimental) (BLK_DEV_UBLK) [N/m/y/?] (NEW)   ↵
[ … ]
  Pine64 PinePhone Keyboard (KEYBOARD_PINEPHONE) [N/m/y/?] (NEW)   ↵
[ … ]
  Intel Meteor Lake pinctrl and GPIO driver (PINCTRL_METEORLAKE) [N/m/y/?] (NEW)   ↵
[ … ]
Test DYNAMIC_DEBUG (TEST_DYNAMIC_DEBUG) [N/m/y/?] (NEW)   ↵
[ … ]
#
# configuration written to .config
#
$ ls -l .config
-rw-rw-r-- 1 c2kp c2kp 170136 Apr 29 09:56 .config
   

After pressing the Enter key (↵) many times – we’ve highlighted this in the output block just above, showing just a few of the many new options encountered – the interrogation mercifully finishes and the Kconfig system writes the newly generated configuration to a file named .config in the current working directory. Note that we truncated the previous output as it’s simply too voluminous, and unnecessary, to reproduce fully.

The preceding steps take care of generating the .config file via the localmodconfig approach. Before we conclude this section, here are a few additional points to note:

• To ensure a completely clean slate, run make mrproper or make distclean in the root of the kernel source tree, useful when you want to restart the kernel build procedure from scratch; rest assured, it will happen one day! Note that doing this deletes the kernel configuration file(s) too. Keep a backup before you begin, if required.
• Here, in this chapter, all the kernel configuration steps and the screenshots pertaining to it have been performed on an x86_64 Ubuntu 22.04 LTS guest VM, which we use as the host to ultimately build a brand-spanking-new 6.1 LTS Linux kernel. The precise names, presence, and content of the menu items seen, as well as the look and feel of the menu system (the UI), can and do vary based on (a) the architecture (CPU) and (b) the kernel version.
• As mentioned earlier, on a production system or project, the platform or BSP team, or indeed the embedded Linux BSP vendor company if you have partnered with one, will provide a good known, working, and tested kernel config file. Use this as a starting point by copying it into the .config file in the root of the kernel source tree. Alternatively, builder software like Yocto or Buildroot might be employed.

As you gain experience with building the kernel, you will realize that the effort in setting up the kernel configuration correctly the first time is higher; and, of course, the time required for the very first build is a lengthy one. Once done correctly, though, the process typically becomes much simpler – a recipe to run repeatedly.

Now, let’s learn how to use a useful and intuitive UI to fine-tune our kernel configuration.

Tuning our kernel configuration via the make menuconfig UI

Okay, great, we now have an initial kernel config file (.config) generated for us via the localmodconfig Makefile target, as shown in detail in the previous section, which is a good starting point. Typically, we now further fine-tune our kernel configuration. One way to do this – in fact, the recommended way – is via the menuconfig Makefile target. This target has the Kbuild system generate a pretty sophisticated C-based program executable (scripts/kconfig/mconf), which presents to the end user a neat menu-based UI. This is step 2 in Figure 2.8. In the following output block, when (within the root of our kernel source tree), we invoke the command for the first time, the Kbuild system builds the mconf executable and invokes it:

$ make menuconfig
 UPD scripts/kconfig/.mconf-cfg
 HOSTCC scripts/kconfig/mconf.o
 HOSTCC scripts/kconfig/lxdialog/checklist.o
 […]
 HOSTLD  scripts/kconfig/mconf

Of course, a picture is no doubt worth a thousand words, so here’s what the menuconfig UI looks like on my VM.

Figure 2.11: The main menu for kernel configuration via make menuconfig (on x86-64)

As experienced developers, or indeed anyone who has sufficiently used a computer, well know, things can and do go wrong. Take, for example, the following scenario – running make menuconfig for the first time on a freshly installed Ubuntu system:

$ make menuconfig
 UPD     scripts/kconfig/.mconf-cfg
 HOSTCC  scripts/kconfig/mconf.o
 YACC    scripts/kconfig/zconf.tab.c
/bin/sh: 1: bison: not found
scripts/Makefile.lib:196: recipe for target 'scripts/kconfig/zconf.tab.c' failed
make[1]: *** [scripts/kconfig/zconf.tab.c] Error 127
Makefile:539: recipe for target 'menuconfig' failed
make: *** [menuconfig] Error 2
$

Hang on, don’t panic. Read the failure messages carefully. The line after YACC [...] provides the clue: /bin/sh: 1: bison: not found. Ah! So, install bison with the following command:

sudo apt install bison

Now, all should be well. Well, almost; again, on a freshly baked Ubuntu guest, make menuconfig then complains that flex wasn’t installed. So, we installed it (you guessed it: via sudo apt install flex). Also, specifically on Ubuntu, you need the libncurses5-dev package installed. On Fedora, do sudo dnf install ncurses-devel.

Moving along, the Kconfig+Kbuild open-source framework provides clues to the user via its UI. Look at Figure 2.11; you’ll often see symbols prefixed to the menus (like [*], <>, -*-, (), and so on); these symbols and their meaning are as follows:

• [.]: In-kernel feature, Boolean option. It’s either On or Off; the ‘.’ shown will be replaced by * or a space:
  • [*]: On, feature compiled and built in to the kernel image (y)
  • [ ]: Off, not built at all (n)
• <.>: A feature that could be in one of three states. This is known as tristate; the . shown will be replaced by *, M, or a space):
  • <*>: On, feature compiled and built in the kernel image (y)
  • <M>: Module, feature compiled and built as a kernel module (an LKM) (m)
  • < >: Off, not built at all (n)
• {.}: A dependency exists for this config option; hence, it’s required to be built or compiled as either a module (m) or to the kernel image (y).
• -*-: A dependency requires this item to be compiled in (y).
• (...): Prompt: an alphanumeric input is required. Press the Enter key while on this option and a prompt box appears.
• <Menu name> --->: A sub-menu follows. Press Enter on this item to navigate to the sub-menu.

Again, the empirical approach is key. Let’s perform a few experiments with the make menuconfig UI to see how it works. This is precisely what we’ll learn in the next section.

Sample usage of the make menuconfig UI

To get a feel for using the Kbuild menu system via the convenient menuconfig target, let’s turn on a quite interesting kernel config. It’s named Kernel .config support and allows one to see the content of the kernel config while running that kernel! Useful, especially during development and testing. For security reasons, it’s typically turned off in production, though.

A couple of nagging questions remain:

• Q. Where is it?

  A. It’s located as an item under the General Setup main menu (we’ll see it soon enough).

• Q. What is it set to by default?

  A. To the value <M>, meaning it will be built as a kernel module by default.

As a learning experiment, we’ll set it to the value [*] (or y), building it into the very fabric of the kernel. In effect, it will be always on. Okay, let’s get it done!

1. Fire up the kernel config UI:

   make menuconfig
   

   You should see a terminal UI as in Figure 2.11. The very first item is usually a submenu labeled General Setup ---> ; press the Enter key while on it; this will lead you into the General Setup submenu, within which many items are displayed; navigate (by pressing the down arrow) to the item named Kernel .config support:

   

   Figure 2.12: A screenshot of the General Setup menu items, with the relevant one highlighted (on x86-64)

2. We can see in the preceding screenshot that we’re configuring the 6.1.25 kernel on an x86, the highlighted menu item is Kernel .config support, and, from its <M> prefix, that it’s a tristate menu item that’s set to the choice <M> for “module,” to begin with (by default).
3. Keeping this item (Kernel .config support) highlighted, use the right arrow key to navigate to the < Help > button on the bottom toolbar and press the Enter key while on the < Help > button. Or, simply press ? while on an option! The screen should now look something like this:

   

   Figure 2.13: Kernel configuration via make menuconfig; an example Help screen (with the name of the kernel config macro highlighted)

   The help screen is quite informative. Indeed, several of the kernel config help screens are very well populated and helpful. Unfortunately, some just aren’t.

4. Okay, next, press Enter while on the < Exit > button so that we go back to the previous screen.
5. Change the value by pressing the spacebar; doing this has the current menu item’s value toggle between <*> (always on), < > (off), and <M> (module). Keep it on <*>, meaning “always on.”
6. Next, though it’s now turned on, the ability to actually view the kernel config is provided via a pseudofile under procfs; the very next item below this one is the relevant one:

   [ ]   Enable access to .config through /proc/config.gz
   

7. You can see it’s turned off by default ([ ]); turn it on by navigating to it and pressing the spacebar. It now shows as [*]:

Figure 2.14: A truncated screenshot showing how we’ve turned on the ability to view the kernel config

8. Right, we’re done for now; press the right arrow or Tab key, navigate to the < Exit > button, and press Enter while on it; you’re back at the main menu screen. Repeat this, pressing < Exit > again; the UI asks you if you’d like to save this configuration. Select < Yes > (by pressing Enter while on the Yes button):

Figure 2.15: Save the modified kernel config prompt

9. The new kernel config is now saved within the .config file. Let’s quickly verify this. I hope you noticed that the exact name of the kernel configs we modified – which is the macro as seen by the kernel source – is:
   1. CONFIG_IKCONFIG for the Kernel .config support option.
   2. CONFIG_IKCONFIG_PROC for the Enable access to .config through /proc/config.gz option.

   How do we know? It’s in the top-left corner of the Help screen! Look again at Figure 2.13.

Done. Of course, the actual effect won’t be seen until we build and boot from this kernel. Now, what exactly does turning this feature on achieve? When turned on, the currently running kernel’s configuration settings can be looked up at any time in two ways:

• By running the scripts/extract-ikconfig script.
• By directly reading the content of the /proc/config.gz pseudofile. Of course, it’s gzip compressed; first uncompress it, and then read it. zcat /proc/config.gz does the trick!

As a further learning exercise, why not further modify the default kernel config (of our 6.1.25 Linux kernel for the x86-64 architecture) of a couple more items? For now, don’t stress out regarding the precise meaning of each of these kernel config options; it’s just to get some practice with the Kconfig system. So, run make menuconfig, and within it make changes by following the format seen just below.

Format:

• Kernel config we’re working with:
  • What it means
  • Where to navigate
  • Name of menu item (and kernel config macro CONFIG_FOO within parentheses)
  • Its default value
  • Value to change it to

Right, here’s what to try out; let’s begin with:

Local version:

• Meaning: the string to append to the kernel version. Take uname –r as an example; in effect, it’s the “z” or EXTRAVERSION component in the w.x.y.z kernel version nomenclature).
• Navigate to: General Setup.
• Menu item: Local version – append to kernel release (CONFIG_LOCALVERSION); press Enter once here and you’ll get a prompt box.
• Default value: NULL.
• Change to: anything you like; prefixing a hyphen to the localversion is considered good practice; for example, -lkp-kernel.

Next:

• Timer frequency. You’ll learn the details regarding this tunable in Chapter 10, The CPU Scheduler – Part 1:
  • Meaning: the frequency at which the timer (hardware) interrupt is triggered.
  • Navigate to: Processor type and features | Timer frequency (250 HZ) ---> . Keep scrolling until you find the second menu item.
  • Menu item: Timer frequency (CONFIG_HZ).
  • Default value: 250 HZ.
  • Change to: 300 HZ.

Look up the Help screens for each of the kernel configs as you work with them. Great; once done, save and exit the UI.

Verifying the kernel config within the config file

But where’s the new kernel configuration saved? This is repeated as it’s important: the kernel configuration is written into a simple ASCII text file in the root of the kernel source tree, named .config. That is, it’s saved in ${LKP_KSRC}/.config.

As mentioned earlier, every single kernel config option is associated with a config variable of the form CONFIG_<FOO>, where <FOO>, of course, is replaced with an appropriate name. Internally, these become macros that the build system and indeed the kernel source code uses.

Thus, to verify whether the kernel configs we just modified will take effect, let’s appropriately grep the kernel config file:

$ grep -E "CONFIG_IKCONFIG|CONFIG_LOCALVERSION|CONFIG_HZ_300" .config
CONFIG_LOCALVERSION="-lkp-kernel"
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_HZ_300=y
$

Aha! The configuration file now reflects the fact that we have indeed modified the relevant kernel configs; the values too show up.

So, by now, I expect you’ve modified the kernel config to suit the values just seen.

During our quick adventure with the Kconfig/Kbuild system so far, quite a lot has occurred under the hood. The next section examines some remaining points: a little bit more regarding Kconfig/Kbuild, searching within the menu system, cleanly visualizing the differences between the original and modified kernel configuration files, using a script to edit the config, security concerns and tips on addressing them; plenty still to learn!

Kernel config – exploring a bit more

The creation of, or edits to, the .config file within the root of the kernel source tree via the make menuconfig UI or other methods is not the final step in how the Kconfig system works with the configuration. No, it now proceeds to internally invoke a hidden target called syncconfig, which was earlier misnamed silentoldconfig. This target has Kconfig generate a few header files that are further used in the setup to build the kernel.

These files include some meta-headers under include/config, as well as the include/generated/autoconf.h header file, which stores the kernel config as C macros, thus enabling both the kernel Makefiles and kernel code to make decisions based on whether a kernel feature is available.

Now that we’ve covered sufficient ground, take another look at Figure 2.8, the high-level diagram (inspired by Cao Jin’s articles) that attempts to communicate the kernel’s Kconfig/Kbuild system. This diagram, in the Kconfig portion, only shows the common make menuconfig UI; note that several other UI approaches exist, which are make config and make {x|g|n}config. Those are not shown here.

Searching within the menuconfig UI

Moving along, what if – when running make menuconfig – you are looking for a particular kernel configuration option but are having difficulty spotting it? No problem: the menuconfig UI system has a Search Configuration Parameter feature. Just as with the famous vi editor (yes, [g]vi[m] is still our favorite text editor!), press the / (forward slash) key to have a search dialog pop up, then enter your search term with or without CONFIG_ preceding it, and select the < Ok > button to have it go on its way.

The following couple of screenshots show the search dialog and the result dialog. As an example, we searched for the term vbox:

Figure 2.16: Kernel configuration via make menuconfig: searching for a config parameter

The result dialog in Figure 2.17 for the preceding search is interesting. It reveals several pieces of information regarding the configuration options:

• The config directive. Just prefix CONFIG_ onto whatever it shows in Symbol:.
• The Type of config (Boolean, tristate, alphanumeric, and so on).
• The Prompt string.
• Importantly, so you can find it, its Location in the menu system.
• Its internal dependencies (Depends on:) if any.
• The Kconfig file and line number n within it (Defined at <path/to/foo.Kconfig*:n>) where this particular kernel config is defined. We’ll cover more on this in coming sections.
• Any config option it auto-selects (Selects:) if it itself is selected.

The following is a partial screenshot of the result dialog:

Figure 2.17: Kernel configuration via make menuconfig: truncated screenshot of the result dialog from the preceding search

All the information driving the menu display and selections is present in an ASCII text file used by the Kbuild system – this file is typically named Kconfig. There are actually several of them. Their precise names and locations are shown in the Defined at ... line.

Looking up the differences in configuration

The moment the .config kernel configuration file is to be written to, the Kconfig system checks whether it already exists, and if so, it backs it up with the name .config.old. Knowing this, we can always differentiate the two to see the changes we have just wrought. However, using your typical diff utility to do so makes the differences quite hard to interpret. The kernel helpfully provides a better way, a console-based script that specializes in doing precisely this. The scripts/diffconfig script within the kernel source tree is useful for this. Pass it the --help parameter to see a usage screen.

Let’s try it out:

$ scripts/diffconfig .config.old .config
HZ 250 -> 300
HZ_250 y -> n
HZ_300 n -> y
LOCALVERSION "" -> "-lkp-kernel"
$ 

If you modified the kernel configuration changes as shown in the preceding section, you should see an output like that shown in the preceding code block via the kernel’s diffconfig script. It clearly shows us exactly which kernel config options we changed and how. In fact, you don’t even need to pass the .config* parameters; it uses these by default.

Using the kernel’s config script to view/edit the kernel config

On occasion, there’s a need to edit or query the kernel configuration directly, checking for or modifying a given kernel config. We’ve learned to do so via the super make menuconfig UI. Here we learn that there’s perhaps an easier, and more importantly, non-interactive and thus scriptable, way to achieve the same – via a Bash script within the kernel source: scripts/config.

Running it without any parameters will result in a useful help screen being displayed; do check it out. An example will help regarding its usage.

The ability to look up the current kernel config’s very useful, so let’s ensure these kernel configs are turned on. Just for this example, let’s first explicitly disable the relevant kernel configs and then enable them:

$ scripts/config --disable IKCONFIG --disable IKCONFIG_PROC
$ grep IKCONFIG .config
# CONFIG_IKCONFIG is not set
# CONFIG_IKCONFIG_PROC is not set
$ scripts/config --enable IKCONFIG --enable IKCONFIG_PROC
$ grep IKCONFIG .config
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

Voila, done.

Configuring the kernel for security

Before we finish, a quick note on something critical: kernel security. While user-space-security-hardening technologies have vastly grown, kernel-space-security-hardening technologies are playing catch-up. Careful configuration of the kernel’s config options does indeed play a key role in determining the security posture of a given Linux kernel; the trouble is, there are so many options and opinions that it’s often hard to check what’s a good idea security-wise and what isn’t.

Alexander Popov has written a very useful Python script named kconfig-hardened-check. It can be run to check and compare a given kernel configuration, via the usual config file, to a set of predetermined hardening preferences sourced from various Linux kernel security projects:

• The Kernel Self Protection Project (KSPP; link: https://kernsec.org/wiki/index.php/Kernel_Self_Protection_Project)
• The last public grsecurity patch
• The CLIP OS
• The security lockdown LSM
• Direct feedback from Linux kernel maintainers

You can clone the kconfig-hardened-check project from its GitHub repository at https://github.com/a13xp0p0v/kconfig-hardened-check and try it out! FYI, my Linux Kernel Debugging book does cover using this script in more detail. Below, a screenshot of its help screen will help you get started with it:

Figure 2.18: A screenshot showing the super kconfig-hardened-check script’s help screen

A quick, useful tip: using the kconfig-hardened-check script, one can easily generate a security-conscious kernel config file like this (here, as an example, for the AArch64):

kconfig-hardened-check -g ARM64 > my_kconfig_hardened

(The output file is called the config fragment.) Now, practically speaking, what if you have an already existing kernel config file for your product? Can we merge both? Indeed we can! The kernel provides a script to do just this: scripts/kconfig/merge_config.sh. Run it, passing as parameters the pathname to the original (perhaps non-secure) kernel config file and then the path to the just-generated secure kernel config fragment; the result is the merger of both (additional parameters to merge_config.sh allow you to control it further; do check it out.

An example can be found here: https://github.com/a13xp0p0v/kconfig-hardened-check#generating-a-kconfig-fragment-with-the-security-hardening-options).

Also, you’re sure to come across the fact that new-ish GCC plugins exist (CONFIG_GCC_PLUGINS) providing some cool arch-specific security features. For example, auto-initialization of local/heap variables, entropy generation at boot, and so on. However, they often don’t even show up in the menu. Typically they’re here: General architecture-dependent options | GCC plugins, as the support isn’t installed by default. On x86 at least, try installing the gcc-<ver#>-plugin-dev package, where ver# is the GCC version number, and then retry configuring.

Miscellaneous tips – kernel config

A few remaining tips follow with regard to kernel configuration:

• When building the x86 kernel for a VM using VirtualBox (as we are here), when configuring the kernel, you might find it useful to set CONFIG_ISO9660_FS=y; it subsequently allows VirtualBox to have the guest mount the Guest Additions virtual CD and install the (pretty useful!) guest additions. Typically stuff that improves performance in the VM and allows better graphics, USB capabilities, clipboard and file sharing, and so on.
• When building a custom kernel, we at times want to write/build eBPF programs (an advanced topic not covered here) or stuff similar to it. In order to do so, some in-kernel headers are required. You can explicitly ensure this by setting the kernel config CONFIG_IKHEADERS=y (or to m; from 5.2 onward). This results in a /sys/kernel/kheaders.tar.xz file being made available, which can be extracted elsewhere to provide the headers.
  • Further, while talking about eBPF, modern kernels have the ability to generate some debug information, called BPF Type Format (BTF) metadata. This can be enabled by selecting the kernel config CONFIG_DEBUG_INFO_BTF=y. This also requires the pahole tool to be installed. More on the BTF metadata can be found within the official kernel documentation here: https://www.kernel.org/doc/html/next/bpf/btf.html.
  • Now, when this option is turned on, another kernel config – CONFIG_MODULE_ALLOW_BTF_MISMATCH – becomes relevant when building kernel modules. This is a topic we cover in depth in the following two chapters. If CONFIG_DEBUG_INFO_BTF is enabled, it’s a good idea to set this latter config to Yes, as otherwise, your modules may not be allowed to load up if the BTF metadata doesn’t match at load time.
• Next, the kernel build should, in theory at least, generate no errors or even warnings. To ensure this, to treat warnings as errors, set CONFIG_WERROR=y. Within the now familiar make menuconfig UI, it’s under General Setup | Compile the kernel with warnings as errors, and is typically off by default.
• There’s an interesting script here: scripts/get_feat.pl; its help screen shows how you can leverage it to list the kernel feature support matrix for the machine or for a given architecture. For example, to see the kernel feature support matrix for the AArch64, do this:

  scripts/get_feat.pl --arch arm64 ls
  

• Next, an unofficial “database” of sorts, of all available kernel configs – and the kernel versions they’re supported upon – is available at the Linux Kernel Driver database (LKDDb) project site here: https://cateee.net/lkddb/web-lkddb/.
• Kernel boot config: At boot, you can always override some kernel features via the powerful kernel command-line parameters. They’re thoroughly documented here: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html. While this is very helpful, sometimes we need to pass along more parameters as key-value pairs, essentially of the form key=value, extending the kernel command line. This can be done by populating a small kernel config file called the boot config. This boot config feature depends on the kernel config’s BOOT_CONFIG being y. It’s under the General Setup menu and is typically on by default.
  • It can be used in two ways: by attaching a boot config to the initrd or initramfs image (we cover initrd in the following chapter) or by embedding a boot config into the kernel itself. For the latter, you’ll need to create the boot config file, pass the directive CONFIG_BOOT_CONFIG_EMBED_FILE="x/y/z" in the kernel config, and rebuild the kernel. Note that kernel command-line parameters will take precedence over the boot config parameters. On boot, if enabled and used, the boot config parameters are visible via /proc/bootconfig. Details regarding the boot config are in the official kernel documentation here: https://elixir.bootlin.com/linux/v6.1.25/source/Documentation/admin-guide/bootconfig.rst.

You’re sure to come across many other useful kernel config settings and scripts, including those for hardening the kernel; keep a keen eye out.

Alright! You have now completed the first three steps of the Linux kernel build – quite a thing. Of course, we will complete the remaining four steps of the kernel build process in the following chapter. We will end this chapter with a final section on learning another useful skill – how to customize the kernel UI menu.