Even after extensive quality assurance testing, embedded systems in-field also fail and need to be debugged. Moreover, often the failure is not something that can be easily reproduced in a laboratory environment, so we are left with production, often hardened, systems to debug.
Assuming we have designed our system with the aforementioned scenario in mind, our first debugging choice is usually to extract as much information about the failing system, for example, by obtaining and analyzing a core dump of the misbehaving processes.
In the process of debugging embedded Linux systems, we can use the same toolbox as standard Linux systems. One of these tools enables applications to generate a memory core dump upon crashing. This assumes that we have enough disk space to store the application's entire memory map, and that writing to disk is quick enough that it will not drag the system to a halt.
Once the memory core dump is generated, we use the host's GDB to analyze...