Increasing the serviceability of an embedded (Android) system (Advanced)
At the maintenance stage of a software's lifecycle, techniques to restore the failure generation scene are critical. These include system logging, system debugging, and tracing.
Getting ready
There are some tools available to fix reported failures as quickly as possible.
Android/Linux comes with logging utilities, such as RAM console and Logger/Logcat. They capture data that will allow the conditions associated with the failure to be determined.
The Ftrace-based systrace
tool is also added for function-level tracing.
For debugging, gdbserver
is available; besides this, the BusyBox top
, iostat
, and devmem
options may give some help. In our example, we will only learn about some of them.
How to do it...
Let's talk about debug filesystems, such as debugfs
, sysfs
, and procfs
and tools, such as Ftrace
, top
, iostat
, and devmem
:
Kernel debugging interfaces, such as
debugfs
,sysfs
, andprocfs
:To debug the Linux kernel, we should enable its debugging features with
CONFIG_DEBUG_KERNEL=y
, and further,CONFIG_DEBUG_FS=y
creates a virtual debug filesystem (debugfs
). We can mount it at runtime.shell@android:/ # mount -t debugfs none /sys/kernel/debug/ shell@android:/ # ls /sys/kernel/debug/ bdi hid mmc0 suspend_stats wakeup_sources binder memblock sched_features tracing
In an Android system, there is a
/d
directory linked to/sys/kernel/debug/
under the root directory. As we've mentioned before, the Android ramdisk has already mountedsysfs
andprocfs
to the system. They are useful kernel interfaces, and we can use them to communicate with the kernel, monitor the runtime kernel status, and diagnose issues.Ftrace – tracing kernel functions and events:
Ftrace is an internal tracing framework in the Linux kernel; it can be used to trace kernel functions. The interface can be found at
/sys/kernel/debug/tracing
. To use it, the following configuration options should be enabled:CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y
Provided we've configured this and started an emulator with this kernel, we can mount
debugfs
on our Android system, and then enter the tracing directory.shell@android:/ # cd /sys/kernel/debug/tracing
For example, if want to trace kernel functions, we can enable the function tracer.
shell@android:/ # echo function > current_tracer shell@android:/ # echo 1 > tracing_on shell@android:/ # ls shell@android:/ # echo 0 > tracing_on shell@android:/ # cat trace | head -20 tracer: function entries-in-buffer/entries-written: 60151/335518 #P:1 _-----=> irqs-off / _----=> need-resched | / _---=> hardirq /softirq || / _--=> preempt-depth ||| / delay TASK-PID CPU# |||| TIMESTAMP FUNCTION | | | |||| | | ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid ActivityManager-328 [000] .... 173.080000: pid_nr_ns <-next_ tgid ActivityManager-328 [000] .... 173.080000: pid_task <-next_tgid ActivityManager-328 [000] .... 173.080000: find_ge_pid <-next_tgid ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid ActivityManager-328 [000] .... 173.080000: pid_nr_ns <-next_tgid ActivityManager-328 [000] .... 173.080000: pid_task <-next_tgid ActivityManager-328 [000] .... 173.080000: find_ge_pid <-next_tgid ActivityManager-328 [000] .... 173.080000: find_pid_ns <-find_ge_pid
In reality, Ftrace also provides the other specific tracers, such as the irqsoff and preemptoff tracers. Both of them can be used to find the bottlenecks that may increase system latency. Tracers that are precompiled are listed in the
available_tracers
interface.shell@android:/ # cat /sys/kernel/debug/tracing/available_tracers blk function_graph mmiotrace wakeup_rt wakeup function nop
Besides, it also provides tracepoint- and kprobe-based event tracers that trace some specific group of kernel functions. These groups include a subsystem or a particular kernel module, such as ext4, IRQ, GPIO, power, syscalls, or workqueue. Look at the available events by going to
/sys/kernel/debug/tracing/available_events
. To learn more about Ftrace, read the documents under theDocumentation/trace
directory of theLinux kernel source code.Top – monitors process statuses:
The
top
applet of BusyBox can be used to monitor processes' status, including CPU and memory utilization. It can display Linux processes, including CPU occupancy, memory usage, and the status of each process. For example:shell@android:/ # busybox top Mem: 253776K used, 511960K free, 0K shrd, 7920K buff, 131144K cached CPU: 75.7% usr 23.0% sys 1.1% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq Load average: 3.92 0.92 0.30 14/245 592 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 318 57 1000 S 224m 29.9 0 25.1 system_server 56 1 1000 S 60308 7.8 0 24.1 /system/bin/surfaceflinger 377 57 10019 R 162m 21.6 0 14.0 {ndroid.systemui} com.android.syst 460 57 10022 S 169m 22.7 0 11.0 {ndroid.launcher} com.android.laun 452 57 1001 S 162m 21.6 0 9.1 {m.android.phone} com.android.phon 527 57 10009 S 159m 21.2 0 4.1 {d.process.acore} android.process. 91 1 1003 S < 23036 3.0 0 3.3 /system/bin/bootanimation 59 1 1013 S 21000 2.7 0 2.1 /system/bin/mediaserver 382 51 0 R 21072 2.7 0 1.3 /system/bin/fsck_msdos -p -f /dev/ 559 65 2000 R 1596 0.2 0 0.4 busybox top 44 2 0 SW 0 0.0 0 0.4 [mmcqd/0] 57 1 0 S 140m 18.7 0 0.2 zygote /bin/app_process -Xzygote / 50 1 1000 S 888 0.1 0 0.2 /system/bin/servicemanager 1 0 0 S 348 0.0 0 0.2 /init 567 57 10009 D 153m 20.4 0 0.0 {ndroid.contacts} com.android.cont 505 57 10001 S 151m 20.2 0 0.0 {d.process.media} android.process. 439 57 10003 S 151m 20.1 0 0.0 {putmethod.latin} com.android.inpu 482 57 10031 S 150m 20.0 0 0.0 {m.android.music} com.android.musi 53 1 0 S 9716 1.2 0 0.0 /system/bin/netd 58 1 1019 S 6612 0.8 0 0.0 /system/bin/drmserver
We can see the verbose outputs.
PID
shows the process ID,PPID
shows the parent process ID of the current process,USER
specifies the user ID of each process,STAT
represents the status of process, whereR
is for run andS
is for sleep,VSZ
shows the virtual memory size used by the process, and%CPU
shows the percentage of CPU occupancy of the process.According to the outputs, we can find which process consumes more system power and system memory and which process is dead, so we can take proper decisions to improve code efficiency and system performance.
iostat – monitors the I/O status
The
iostat
applet of BusyBox is a tool to report CPU and I/O statistics. It will show the average occupation for CPU in idle state; I/O wait response; and the user, system, and steal modes.The device status will also be displayed in read/write speed. We can show only CPU stats with the
-c
command and device stats with the-d
option:shell@android:/d/tracing # busybox iostat -cd Linux 3.4.0-g18554c7 (localhost) 07/02/13 _armv7l_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.30 0.03 1.20 0.00 0.00 97.47 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn mmcblk0 0.22 1.80 0.00 25114 1
devmem – monitors the hardware status:
The
devmem
applet of BusyBox is a useful tool to read from or write values to a physical address. This is a very efficient trick to read the hardware register or write an expected value to that register. For example:Read the
0x1000000
physical address.shell@android:/ # devmem 0x10000000 0x00000000
Write
0xF000000F
to the0x10000000
physical address:shell@android:/ # devmem 0x10000000 32 0xF000000F shell@android:/ # devmem 0x10000000 0xF000000F
In a real Android device, the SOC (System On Chip) has its own CoreSight TM (the ARM debug and trace technology is the most complete on-chip debug and real-time trace solution); it can access from 0 to 4 GB address space, but there are some limits to using these spaces. Some part of this space is used as an SFR (Special Function Register) area, and some are used for the physical memory area. So, use
devmem
to see the value of an SFR. We can read its status and change its value if it is writable; this is often used while the system stays in an unstable state.
How it works...
Faults should be filtered out before the maintenance stage, but not all of them can be discovered. If failures occur, logging methods must be provided to record and report the failure scene. With the failure logs, the tracing tools can be used to track the system status and find the problematic areas; lastly, if necessary, use a debugger to debug interactively to locate the problems more precisely.
There's more...
Introduce the Ftrace-based systrace.py
tool provided by Android.
# systrace.py --disk --time=10 -o mynewtrace.html
The output mynewtrace.html
shows the kernel activities graphically and allows us to track the system status easily. The output is in HTML5 format and can be parsed by most modern browsers. For more information about it, visit http://developer.android.com/tools/help/systrace.html.
Other similar tools are listed at http://www.elinux.org/Toolbox.
We have demonstrated some useful tools provided by BusyBox. With these tools, we can optimize the embedded system for size, stability, power consumption, boot speed, and serviceability. But this is just an initiation for us. We hope you will dig into it further and make more progress in your own embedded system development.