Book Image

Linux Kernel Programming Part 2 - Char Device Drivers and Kernel Synchronization

By : Kaiwan N. Billimoria
Book Image

Linux Kernel Programming Part 2 - Char Device Drivers and Kernel Synchronization

By: Kaiwan N. Billimoria

Overview of this book

Linux Kernel Programming Part 2 - Char Device Drivers and Kernel Synchronization is an ideal companion guide to the Linux Kernel Programming book. This book provides a comprehensive introduction for those new to Linux device driver development and will have you up and running with writing misc class character device driver code (on the 5.4 LTS Linux kernel) in next to no time. You'll begin by learning how to write a simple and complete misc class character driver before interfacing your driver with user-mode processes via procfs, sysfs, debugfs, netlink sockets, and ioctl. You'll then find out how to work with hardware I/O memory. The book covers working with hardware interrupts in depth and helps you understand interrupt request (IRQ) allocation, threaded IRQ handlers, tasklets, and softirqs. You'll also explore the practical usage of useful kernel mechanisms, setting up delays, timers, kernel threads, and workqueues. Finally, you'll discover how to deal with the complexity of kernel synchronization with locking technologies (mutexes, spinlocks, and atomic/refcount operators), including more advanced topics such as cache effects, a primer on lock-free techniques, deadlock avoidance (with lockdep), and kernel lock debugging techniques. By the end of this Linux kernel book, you'll have learned the fundamentals of writing Linux character device driver code for real-world projects and products.

Related Content you might be interested in

Current Title:

Small book image
Linux Kernel Programming Part 2 - Char Device Drivers and Kernel Synchronization
Kaiwan N. Billimoria
Mar, 2021 | 452 pages
Small book image
Linux Kernel Programming
Kaiwan N Billimoria
Feb, 2024 | 826 pages
Small book image
Linux Kernel Debugging
Kaiwan N Billimoria
Aug, 2022 | 638 pages
Small book image
Mastering Linux Kernel Development
CH Raghav Maruthi
Oct, 2017 | 354 pages
Table of Contents (11 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
1
Section 1: Character Device Driver Basics
Section 1: Character Device Driver Basics
Free Chapter
2
Writing a Simple misc Character Device Driver
Writing a Simple misc Character Device Driver
Technical requirements
Getting started with writing a simple misc character device driver
Understanding the device basics
A quick note on the Linux Device Model
Writing the misc driver code – part 1
Understanding the connection between the process, the driver, and the kernel
Handling unsupported methods
Writing the misc driver code – part 2
Writing the misc driver code – part 3
Testing our simple misc driver
Copying data from kernel to user space and vice versa
Leveraging kernel APIs to perform the data transfer
A misc driver with a secret
Writing the 'secret' misc device driver's code
Our secret driver – the init code
Our secret driver – the read method
Our secret driver – the write method
Our secret driver – cleanup
Our secret driver – the user space test app
Issues and security concerns
Hacking the secret driver
Bad driver – buggy read()
Bad driver – buggy write() – a privesc!
User space test app modifications
Device driver modifications
Let's get root now
Summary
Questions
Further reading
3
User-Kernel Communication Pathways
User-Kernel Communication Pathways
Technical requirements
Approaches to communicating/interfacing a kernel driver with a user space C app
Interfacing via the proc filesystem (procfs)
Understanding the proc filesystem
Directories under /proc
The purpose behind the proc filesystem
procfs is off-bounds to driver authors
Using procfs to interface with the user space
Basic procfs APIs
The four procfs files we will create
Trying out the dynamic debug_level procfs control
Dynamically controlling debug_level via procfs
A few misc procfs APIs
Interfacing via the sys filesystem (sysfs)
Creating a sysfs (pseudo) file in code
Creating a simple platform device
Platform devices
Tying it all together – setting up the device attributes and creating the sysfs file
The code for implementing our sysfs file and its callbacks
The "one value per sysfs file" rule
Interfacing via the debug filesystem (debugfs)
Checking for the presence of debugfs
Looking up the debugfs API documentation
An interfacing example with debugfs
Creating and using the first debugfs file
Creating and using the second debugfs file
Helper debugfs APIs for working on numeric globals
Removing the debugfs pseudo file(s)
Seeing a kernel bug – an Oops!
Debugfs – actual users
Interfacing via netlink sockets
Advantages using sockets
Understanding what a netlink socket is
Writing the user space netlink socket application
Writing the kernel-space netlink socket code as a kernel module
Trying out our netlink interfacing project
Interfacing via the ioctl system call
Using ioctl in the user and kernel space
User space – using the ioctl system call
Kernel space – using the ioctl system call
ioctl as a debug interface
Comparing the interfacing methods – a table
Summary
Questions
Further reading
4
Working with Hardware I/O Memory
Working with Hardware I/O Memory
Technical requirements
Accessing hardware I/O memory from the kernel
Understanding the issue with direct access
The solution – mapping via I/O memory or I/O port
Asking the kernel's permission
Understanding and using memory-mapped I/O
Using the ioremap*() APIs
The newer breed – the devm_* managed APIs
Obtaining the device resources
All in one with the devm_ioremap_resource() API
Looking up the new mapping via /proc/iomem
MMIO – performing the actual I/O
Performing 1- to 8-byte reads and writes on MMIO memory regions
Performing repeating I/O on MMIO memory regions
Setting and copying on MMIO memory regions
Understanding and using port-mapped I/O
PMIO – performing the actual I/O
A PIO example – the i8042
Looking up the port(s) via /proc/ioports
Port I/O – a few remaining points to note
Summary
Questions
Further reading
5
Handling Hardware Interrupts
Handling Hardware Interrupts
Technical requirements
Hardware interrupts and how the kernel handles them
Allocating the hardware IRQ
Allocating your interrupt handler with request_irq()
Freeing the IRQ line
Setting interrupt flags
Understanding level- and edge-triggered interrupts – a brief note
Code view 1 – the IXGB network driver
Implementing the interrupt handler routine
Interrupt context guidelines – what to do and what not to do
Don't block – spotting possibly blocking code paths
Interrupt masking – the defaults and controlling it
Keep it fast
Writing the interrupt handler routine itself
Code view 2 – the i8042 driver's interrupt handler
Code view 3 – the IXGB network driver's interrupt handler
IRQ allocation – the modern way – the managed interrupt facility
Working with the threaded interrupts model
Employing the threaded interrupt model – the API
Employing the managed threaded interrupt model – the recommended way
Code view 4 – the STM32 F7 microcontroller's threaded interrupt handler
Internally implementing the threaded interrupt
Why use threaded interrupts?
Threaded interrupts – to really make it real time
Constraints when using a threaded handler
Working with either hardirq or threaded handlers
Enabling and disabling IRQs
The NMI
Viewing all allocated interrupt (IRQ) lines
Understanding and using top and bottom halves
Specifying and using a tasklet
Initializing the tasklet
Running the tasklet
Understanding the kernel softirq mechanism
Available softirqs and what they are for
Understanding how the kernel runs softirqs
Running tasklets
Employing the ksoftirqd kernel threads
Softirqs and concurrency
Hardirqs, tasklets, and threaded handlers – what to use when
Fully figuring out the context
Viewing the context – examples
How Linux prioritizes activities
A few remaining FAQs answered
Load balancing interrupts and IRQ affinity
Does the kernel maintain separate IRQ stacks?
Measuring metrics and latency
Measuring interrupts with [e]BPF
Measuring time servicing individual hardirqs
Measuring time servicing individual softirqs
Using Ftrace to get a handle on system latencies
Finding the interrupts disabled worst-case time latency with Ftrace
Other tools
Summary
Questions
Further reading
6
Working with Kernel Timers, Threads, and Workqueues
Working with Kernel Timers, Threads, and Workqueues
Technical requirements
Delaying for a given time in the kernel
Understanding how to use the *delay() atomic APIs
Understanding how to use the *sleep() blocking APIs
Taking timestamps within kernel code
Let's try it – how long do delays and sleeps really take?
The "sed" drivers – to demo kernel timers, kthreads, and workqueues
Setting up and using kernel timers
Using kernel timers
Our simple kernel timer module – code view 1
Our simple kernel timer module – code view 2
Our simple kernel timer module – running it
sed1 – implementing timeouts with our demo sed1 driver
Deliberately missing the bus
Creating and working with kernel threads
A simple demo – creating a kernel thread
Running the kthread_simple kernel thread demo
The sed2 driver – design and implementation
sed2 – the design
sed2 driver – code implementation
sed2 – trying it out
Querying and setting the scheduling policy/priority of a kernel thread
Using kernel workqueues
The bare minimum workqueue internals
Using the kernel-global workqueue
Initializing the kernel-global workqueue for your task – INIT_WORK()
Having your work task execute – schedule_work()
Variations of scheduling your work task
Cleaning up – canceling or flushing your work task
A quick summary of the workflow
Our simple work queue kernel module – code view
Our simple work queue kernel module – running it
The sed3 mini project – a very brief look
Summary
Questions
Further reading
7
Section 2: Delving Deeper
Section 2: Delving Deeper
8
Kernel Synchronization - Part 1
Kernel Synchronization - Part 1
Critical sections, exclusive execution, and atomicity
What is a critical section?
A classic case – the global i ++
Concepts – the lock
A summary of key points
Concurrency concerns within the Linux kernel
Multicore SMP systems and data races
Preemptible kernels, blocking I/O, and data races
Hardware interrupts and data races
Locking guidelines and deadlocks
Mutex or spinlock? Which to use when
Determining which lock to use – in theory
Determining which lock to use – in practice
Using the mutex lock
Initializing the mutex lock
Correctly using the mutex lock
Mutex lock and unlock APIs and their usage
Mutex lock – via [un]interruptible sleep?
Mutex locking – an example driver
The mutex lock – a few remaining points
Mutex lock API variants
The mutex trylock variant
The mutex interruptible and killable variants
The mutex io variant
The semaphore and the mutex
Priority inversion and the RT-mutex
Internal design
Using the spinlock
Spinlock – simple usage
Spinlock – an example driver
Test – sleep in an atomic context
Testing on a 5.4 debug kernel
Testing on a 5.4 non-debug distro kernel
Locking and interrupts
Using spinlocks – a quick summary
Summary
Questions
Further reading
9
Kernel Synchronization - Part 2
Kernel Synchronization - Part 2
Using the atomic_t and refcount_t interfaces
The newer refcount_t versus older atomic_t interfaces
The simpler atomic_t and refcount_t interfaces
Examples of using refcount_t within the kernel code base
64-bit atomic integer operators
Using the RMW atomic operators
RMW atomic operations – operating on device registers
Using the RMW bitwise operators
Using bitwise atomic operators – an example
Efficiently searching a bitmask
Using the reader-writer spinlock
Reader-writer spinlock interfaces
A word of caution
The reader-writer semaphore
Cache effects and false sharing
Lock-free programming with per-CPU variables
Per-CPU variables
Working with per-CPU
Allocating, initialization, and freeing per-CPU variables
Performing I/O (reads and writes) on per-CPU variables
Per-CPU – an example kernel module
Per-CPU usage within the kernel
Lock debugging within the kernel
Configuring a debug kernel for lock debugging
The lock validator lockdep – catching locking issues early
Examples – catching deadlock bugs with lockdep
Example 1 – catching a self deadlock bug with lockdep
Fixing it
Example 2 – catching an AB-BA deadlock with lockdep
lockdep – annotations and issues
lockdep annotations
lockdep issues
Lock statistics
Viewing lock stats
Memory barriers – an introduction
An example of using memory barriers in a device driver
Summary
Questions
Further reading
10
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

To get the most out of this book

To get the most out of this book, we expect you to have knowledge and experience of the following:

  • Know your way around a Linux system, on the command line (the shell).
  • The C programming language.
  • Know how to write simple kernel modules via the Loadable Kernel Module (LKM) framework
  • Understand (at least the basics) of key Linux kernel internals concepts: kernel architecture, memory management (plus common dynamic memory alloc/de-alloc APIs), and CPU scheduling.
  • It's not mandatory, but experience with Linux kernel programming concepts and technologies will help greatly.

Ideally, we highly recommend reading this book's companion, Linux Kernel Programming, first.

The details on hardware and software requirements for this book, as well as their installation, are shown here:

Chapter number

Software required (with version)

Free /

proprietary

Download links to the software

Hardware specifications

OS required

All chapters

A recent Linux distribution; we use Ubuntu 18.04 LTS (as well as Fedora 31 / Ubuntu 20.04 LTS); any of these will be suitable. Recommend you install the Linux OS as a virtual machine (VM), using Oracle VirtualBox 6.x (or later) as the hypervisor

Free (open source)

Ubuntu (desktop): https://ubuntu.com/download/desktop

Oracle VirtualBox: https://www.virtualbox.org/wiki/Downloads

Required: a modern relatively powerful PC or laptop equipped with 4 GB RAM (minimally; the more the better), 25 GB free disk space, and a good internet connection.

Optional: we also use the Raspberry Pi 3B+ as a test bed.

Linux VM on a Windows host
-OR- 
Linux as a stand-alone OS

Detailed installation steps (software-wise):

  1. Install Linux as a VM on a host Windows system; follow one of these tutorials:
  2. Install the required software packages on the Linux VM:
    1. Log in to your Linux guest VM and first run the following commands within a Terminal window (on a shell):
sudo apt update
sudo apt install gcc make perl
    1. Install the Oracle VirtualBox Guest Additions now. Reference: How to Install VirtualBox Guest Additions in Ubuntu: https://www.tecmint.com/install-virtualbox-guest-additions-in-ubuntu/
      (This step only applies if you are running Ubuntu as a VM using Oracle VirtualBox as the hypervisor app.)
    2. To install the packages, take the following steps:
      1. Within the Ubuntu VM, first run the sudo apt update command
      2. Now, run the sudo apt install git fakeroot build-essential tar ncurses-dev tar xz-utils libssl-dev bc stress python3-distutils libelf-dev linux-headers-$(uname -r) bison flex libncurses5-dev util-linux net-tools linux-tools-$(uname -r) exuberant-ctags cscope sysfsutils curl perf-tools-unstable gnuplot rt-tests indent tree pstree smem hwloc bpfcc-tools sparse flawfinder cppcheck tuna hexdump trace-cmd virt-what command in a single line.
  2. Useful resources:
  3. Detailed instructions, as well as additional useful projects, installing a cross-toolchain for ARM, and more, are described in Chapter 1, Kernel Workspace Setup, of this book's companion guide, Linux Kernel Programming, Kaiwan N Billimoria, Packt Publishing.

We have tested all the code in this book (it has its own GitHub repository as well) on these platforms:

  • x86_64 Ubuntu 18.04 LTS guest OS (running on Oracle VirtualBox 6.1)
  • x86_64 Ubuntu 20.04.1 LTS guest OS (running on Oracle VirtualBox 6.1)
  • x86_64 Ubuntu 20.04.1 LTS native OS
  • ARM Raspberry Pi 3B+ (running both its distro kernel as well as our custom 5.4 kernel); lightly tested.

If you are using the digital version of this book, we advise you to type the code yourself or, better, access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

For this book, we'll log in as the user named llkd. I strongly recommend that you follow the empirical approach: not taking anyone's word on anything at all, but trying it out and experiencing it for yourself. Hence, this book gives you many hands-on experiments and kernel driver code examples that you can and must try out yourself; this will greatly aid you in making real progress and deeply learning and understanding various aspects of Linux driver/kernel development.

Previous Section
Next Section