Book Image

Learning Linux Binary Analysis

By : Ryan "elfmaster" O'Neill
5 (1)
Book Image

Learning Linux Binary Analysis

5 (1)
By: Ryan "elfmaster" O'Neill

Overview of this book

Learning Linux Binary Analysis is packed with knowledge and code that will teach you the inner workings of the ELF format, and the methods used by hackers and security analysts for virus analysis, binary patching, software protection and more. This book will start by taking you through UNIX/Linux object utilities, and will move on to teaching you all about the ELF specimen. You will learn about process tracing, and will explore the different types of Linux and UNIX viruses, and how you can make use of ELF Virus Technology to deal with them. The latter half of the book discusses the usage of Kprobe instrumentation for kernel hacking, code patching, and debugging. You will discover how to detect and disinfect kernel-mode rootkits, and move on to analyze static code. Finally, you will be walked through complex userspace memory infection analysis. This book will lead you into territory that is uncharted even by some experts; right into the world of the computer hacker.
Table of Contents (17 chapters)
Learning Linux Binary Analysis
Credits
About the Author
Acknowledgments
About the Reviewers
www.PacktPub.com
Preface
Index

A ptrace anti-debugging trick


The ptrace command can be used as an anti-debugging technique. Often when a hacker doesn't want their program to be easily debugged, they include certain anti-debugging techniques. One popular way in Linux is to use ptrace with the PTRACE_TRACEME request so that it traces the process of itself.

Remember that a process can only have one tracer at a time, so if a process is already being traced and a debugger tries to attach using ptrace, it says Operation not permitted. PTRACE_TRACEME can also be used to check whether your program is already being debugged. You can use the code in the following section to check this.

Is your program being traced?

Let's take a look at a code snippet that will use ptrace to find out whether your program is already being traced:

if (ptrace(PTRACE_TRACEME, 0) < 0) {
printf("This process is being debugged!!!\n");
exit(1);
}

The preceding code works because it should only fail if the program is already being traced. So, if ptrace returns...