Book Image

Mastering Malware Analysis

By : Alexey Kleymenov, Amr Thabet
Book Image

Mastering Malware Analysis

By: Alexey Kleymenov, Amr Thabet

Overview of this book

With the ever-growing proliferation of technology, the risk of encountering malicious code or malware has also increased. Malware analysis has become one of the most trending topics in businesses in recent years due to multiple prominent ransomware attacks. Mastering Malware Analysis explains the universal patterns behind different malicious software types and how to analyze them using a variety of approaches. You will learn how to examine malware code and determine the damage it can possibly cause to your systems to ensure that it won't propagate any further. Moving forward, you will cover all aspects of malware analysis for the Windows platform in detail. Next, you will get to grips with obfuscation and anti-disassembly, anti-debugging, as well as anti-virtual machine techniques. This book will help you deal with modern cross-platform malware. Throughout the course of this book, you will explore real-world examples of static and dynamic malware analysis, unpacking and decrypting, and rootkit detection. Finally, this book will help you strengthen your defenses and prevent malware breaches for IoT devices and mobile platforms. By the end of this book, you will have learned to effectively analyze, investigate, and build innovative solutions to handle any malware incidents.

Related Content you might be interested in

Current Title:

Small book image
Mastering Malware Analysis
Alexey Kleymenov | Amr Thabet
Jun, 2019 | 562 pages
Small book image
ChatGPT Prompts Book - Precision Prompts, Priming, Training & AI Writing Techniques for Mortals
Oliver Theobald
Feb, 2024 | 104 pages
Small book image
GPT-3
Sandra Kublik | Shubham Saboo
Feb, 2023 | 150 pages
Small book image
Generative AI for Cloud Solutions
Anurag Karuparti | Paul Singh
Apr, 2024 | 300 pages
Table of Contents (18 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Free Chapter
1
Section 1: Fundamental Theory
Section 1: Fundamental Theory
2
A Crash Course in CISC/RISC and Programming Basics
A Crash Course in CISC/RISC and Programming Basics
Basic concepts
Registers
Memory
Virtual memory
Stack
Branches, loops, and conditions
Exceptions, interrupts, and communicating with other devices
Assembly languages
CISC versus RISC
Types of instructions
Becoming familiar with x86 (IA-32 and x64)
Registers
Special registers
The instruction structure
opcode
dest
src
The instruction set
Data manipulation instructions
Data transfer instructions
Flow control instructions
Arguments, local variables, and calling conventions (in x86 and x64)
stdcall
Arguments
Local variables
cdecl
fastcall
thiscall
The x64 calling convention
Exploring ARM assembly
Basics
Instruction sets
Basics of MIPS
Basics
The instruction set
Diving deep into PowerPC
Basics
The instruction set
Covering the SuperH assembly
Basics
The instruction set
Working with SPARC
Basics
The instruction set
Moving from assembly to high-level programming languages
Arithmetic statements
If conditions
While loop conditions
Summary
3
Section 2: Diving Deep into Windows Malware
Section 2: Diving Deep into Windows Malware
4
Basic Static and Dynamic Analysis for x86/x64
Basic Static and Dynamic Analysis for x86/x64
Working with the PE header structure
Why PE?
Exploring PE structure
MZ header
PE header
File header
Optional header
Data directory
Section table
PE+ (x64 PE)
PE header analysis tools
Static and dynamic linking
Static linking
Dynamic linking
Dynamic link libraries
Application programming interface
Dynamic API loading
Using PE header information for static analysis
How to use PE header for incident handling
How to use a PE header for threat intelligence
PE loading and process creation
Basic terminology
What's process?
Virtual memory to physical memory mapping
Threads
Important data structures: TIB, TEB, and PEB
Process loading step by step
PE file loading step by step
WOW64 processes
Dynamic analysis with OllyDbg/Immunity Debugger
Debugging tools
How to analyze a sample with OllyDbg
Types of breakpoints
Step into/step over breakpoint
INT3 breakpoint
Memory breakpoints
Hardware breakpoints
Modifying the program execution
Patching—modifying the program's assembly instructions
Change EFlags
Modifying the instruction pointer value
Changing the program data
Debugging malicious services
What is service?
Attaching to the service
Summary
5
Unpacking, Decryption, and Deobfuscation
Unpacking, Decryption, and Deobfuscation
Exploring packers
Exploring packing and encrypting tools
Identifying a packed sample
Technique 1 – checking PE tool static signatures
Technique 2 – evaluating PE section names
Technique 3 – using stub execution signs
Technique 4 – detecting a small import table
Automatically unpacking packed samples
Technique 1 – the official unpacking process
Technique 2 – using OllyScript with OllyDbg
Technique 3 – using generic unpackers
Technique 4 – emulation
Technique 5 – memory dumps
Manual unpacking using OllyDbg
Technique 6 – memory breakpoint on execution
Step 1 – setting the breakpoints
Step 2 – turning on Data Execution Prevention
Step 3 – preventing any further attempts to change memory permissions
Step 4 – executing and getting the OEP
Technique 7 – call stack backtracing
Step 1 – setting the breakpoints
Step 2 – following the call stack
Step 3 – reaching the OEP
Technique 8 – monitoring memory allocated spaces for unpacked code
Technique 9 – in-place unpacking
Technique 10 – stack restoration-based
Dumping the unpacked sample and fixing the import table
Dumping the process
Fixing the import table
Identifying different encryption algorithms and functions
Types of encryption algorithms
Basic encryption algorithms
How to identify encryption functions
String search detection techniques for simple algorithms
The basics of X-RAYING
Simple static encryption
Other encryption algorithms
X-RAYING tools for malware analysis and detection
Identifying the RC4 encryption algorithm
The RC4 encryption algorithm
Key-scheduling algorithm
Pseudo-random generation algorithm
Identifying RC4 algorithms in a malware sample
Standard symmetric and asymmetric encryption algorithms
Extracting information from Windows cryptography APIs
Step 1 – initializing and connecting to the cryptographic service provider (CSP)
Step 2 – preparing the key
Step 3 – encrypting or decrypting the data
Step 4 – freeing the memory
Cryptography API next generation (CNG)
Applications of encryption in modern malware – Vawtrak banking Trojan
String and API name encryption
Network communication encryption
Using IDA for decryption and unpacking
IDA tips and tricks
Static analysis
Dynamic analysis
Classic and new syntax of IDA scripts
Dynamic string decryption
Dynamic WinAPIs resolution
Summary
6
Inspecting Process Injection and API Hooking
Inspecting Process Injection and API Hooking
Understanding process injection
What's process injection?
Why process injection?
DLL injection
Windows-supported DLL injection
A simple DLL injection technique
Working with process injection
Getting the list of running processes
Code injection
Advanced code injection-reflective DLL injection
Stuxnet secret technique-process hollowing
Dynamic analysis of code injection
Technique 1—debug it where it is
Technique 2—attach to the targeted process
Technique 3—dealing with process hollowing
Memory forensics techniques for process injection
Technique 1—detecting code injection and reflective DLL injection
Technique 2—detecting process hollowing
Technique 3—detecting process hollowing using the HollowFind plugin
Understanding API hooking
Why API hooking?
Working with API hooking
Inline API hooking
Inline API hooking with trampoline
Inline API hooking with a length disassembler
Detecting API hooking using memory forensics
Exploring IAT hooking
Summary
7
Bypassing Anti-Reverse Engineering Techniques
Bypassing Anti-Reverse Engineering Techniques
Exploring debugger detection
Direct check for debugger presence
Detecting a debugger through an environment change
Detecting a debugger using parent processes
Handling debugger breakpoints evasion
Detecting software breakpoints (INT3)
Detecting single-stepping breakpoints (trap flag)
Detecting a trap flag using the SS register
Detecting single-stepping using timing techniques
Evading hardware breakpoints
What is structured exception handling?
Detecting and removing hardware breakpoints
Memory breakpoints
Escaping the debugger
Process injection
TLS callbacks
Windows events callbacks
Obfuscation and anti-disassemblers
Encryption
Junk code insertion
Code transportation
Dynamic API calling with checksum
Proxy functions and proxy argument stacking
Detecting and evading behavioral analysis tools
Finding the tool process
Searching for the tool window
Detecting sandboxes and virtual machines
Different output between virtual machines and real machines
Detecting virtualization processes and services
Detecting virtualization through registry keys
Detecting virtual machines using PowerShell
Detecting sandboxes by using default settings
Other techniques
Summary
8
Understanding Kernel-Mode Rootkits
Understanding Kernel-Mode Rootkits
Kernel mode versus user mode
Protection rings
Windows internals
The infrastructure of Windows
The execution path from user mode to kernel mode
Rootkits and device drivers
What is a rootkit?
Types of rootkits
What is a device driver?
Hooking mechanisms
SSDT hooking
Hooking the SYSENTER entry function
Modifying SSDT in an x86 environment
Modifying SSDT in an x64 environment
Hooking SSDT functions
IRP hooking
Devices and major functions
Attaching to a device
Modifying the IRP response and setting a completion routine
DKOM
The kernel objects—EPROCESS and ETHREAD
How do rootkits perform an object manipulation attack?
Process injection in kernel mode
Executing the inject code using APC queuing
KPP in x64 systems (PatchGuard)
Bypassing driver signature enforcement
Bypassing PatchGuard—the Turla example
Bypassing PatchGuard—GhostHook
Disabling PatchGuard using the Command Prompt
Static and dynamic analysis in kernel mode
Static analysis
Tools
Tips and tricks
Dynamic and behavioral analysis
Tools
Monitors
Rootkit detectors
Setting up a testing environment
Setting up the debugger
Stopping at the driver's entry point
Loading the driver
Restoring the debugging state
Summary
9
Section 3: Examining Cross-Platform Malware
Section 3: Examining Cross-Platform Malware
10
Handling Exploits and Shellcode
Handling Exploits and Shellcode
Getting familiar with vulnerabilities and exploits
Types of vulnerabilities
Stack overflow vulnerability
Heap overflow vulnerabilities
The use-after-free vulnerability
Logical vulnerabilities
Types of exploits
Cracking the shellcode
What's shellcode?
Linux shellcode in x86-64
Getting the absolute address
Null-free shellcode
Local shell shellcode
Reverse shell shellcode
Linux shellcode for ARM
Null-free shellcode
Windows shellcode
Getting the Kernel32.dll's ImageBase
Getting the required APIs from Kernel32.dll
The download and execute shellcode
Static and dynamic analysis of exploits
Analysis workflow
Shellcode analysis
Exploring bypasses for exploit mitigation technologies
Data execution prevention (DEP/NX)
Return-oriented programming
Address space layout randomization
DEP and partial ASLR
DEP and full ASLR – partial ROP and chaining multiple vulnerabilities
DEP and full ASLR – heap spray technique
Other mitigation technologies
Analyzing Microsoft Office exploits
File structures
Compound file binary format
Rich text format
Office open XML format
Static and dynamic analysis of MS Office exploits
Static analysis
Dynamic analysis
Studying malicious PDFs
File structure
Static and dynamic analysis of PDF files
Static analysis
Dynamic analysis
Summary
11
Reversing Bytecode Languages: .NET, Java, and More
Reversing Bytecode Languages: .NET, Java, and More
The basic theory of bytecode languages
Object-oriented programming
Inheritance
Polymorphism
.NET explained
.NET file structure
.NET COR20 header
Metadata streams
How to identify a .NET application from PE characteristics
The CIL language instruction set
Pushing into stack instructions
Pulling out a value from the stack
Mathematical and logical operations
Branching instructions
CIL language to higher-level languages
Local variable assignments
Local variable assignment with a method return value
Basic branching statements
Loops statements
.NET malware analysis
.NET analysis tools
Static and dynamic analysis (with Dnspy)
.NET static analysis
.NET dynamic analysis
Patching a .NET sample
Dealing with obfuscation
Obfuscated names for classes, methods, and others
Encrypted strings inside the binary
The sample is obfuscated using an obfuscator
The essentials of Visual Basic
File structure
P-code versus native code
Common p-code instructions
Dissecting Visual Basic samples
Static analysis
P-code
Native code
Dynamic analysis
P-code
Native code
The internals of Java samples
File structure
JVM instructions
Static analysis
Dynamic analysis
Dealing with anti-reverse engineering solutions
Python—script language internals
File structure
Bytecode instructions
Analyzing compiled Python
Static analysis
Dynamic analysis
Summary
12
Scripts and Macros: Reversing, Deobfuscation, and Debugging
Scripts and Macros: Reversing, Deobfuscation, and Debugging
Classic shell script languages
Windows batch scripting
Bash
VBScript explained
Basic syntax
Static and dynamic analysis
Deobfuscation
Those evil macros inside documents
Basic syntax
Static and dynamic analysis
Besides macros
The power of PowerShell
Basic syntax
Static and dynamic analysis
Handling JavaScript
Basic syntax
Static and dynamic analysis
Anti-reverse engineering tricks
Behind C and C—even malware has its own backend
Things to focus on
Static and dynamic analysis
Other script languages
Where to start from
Questions to answer
Summary
13
Section 4: Looking into IoT and Other Platforms
Section 4: Looking into IoT and Other Platforms
14
Dissecting Linux and IoT Malware
Dissecting Linux and IoT Malware
Explaining ELF files
ELF structure
System calls
Filesystem
Network
Process management
Other
Syscalls in assembly
Common anti-reverse engineering tricks
Exploring common behavioral patterns
Initial delivery and lateral movement
Persistence
Privilege escalation
Interaction with the command and control server
Attacking stage
Static and dynamic analysis of x86 (32- and 64-bit) samples
Static analysis
File type detectors
Data carving
Disassemblers
Actual tools
Engines
How to choose
Dynamic analysis
Tracers
Network monitors
Debuggers
Binary emulators
Radare2 cheat sheet
Anti-reverse engineering techniques
Learning Mirai, its clones, and more
High-level functionality
Propagation
Weaponry
Self-defense
Later derivatives
Other widespread families
Static and dynamic analysis of RISC samples
ARM
MIPS
PowerPC
SuperH
SPARC
Handling other architectures
What to start from
Summary
15
Introduction to macOS and iOS Threats
Introduction to macOS and iOS Threats
Understanding the role of the security model
macOS
Security policies
Filesystem hierarchy and encryption
Directory structure
Encryption
Apps protection
Gatekeeper
App sandbox
Other technologies
iOS
System security
Data encryption and password management
Apps' security
File formats and APIs
Mach-O
Thin
Fat
Application bundles (.app)
Info.plist
macOS
iOS
Installer packages (.pkg)
Apple disk images (.dmg)
iOS app store packages (.ipa)
APIs
Static and dynamic analyses of macOS and iOS samples
Static analysis
Retrieving samples
Disassemblers and decompilers
Auxiliary tools and libraries
Dynamic and behavioral analysis
macOS
Debuggers
Monitoring and dynamic instrumentation
Network analysis
iOS
Installers and loaders
Debuggers
Dumping and decryption
Monitors and in-memory patching
Network analysis
Attack stages
Jailbreaks on demand
Penetration
Deployment and persistence
macOS
iOS
Action phase
macOS
iOS
Other attack techniques
macOS
iOS
Advanced techniques
Anti-reverse-engineering (RE) tricks
Misusing dynamic data exchange (DDE)
User hiding
Use of AppleScript
API hijacking
Rootkits for Mac—do they exist?
Analysis workflow
Summary
16
Analyzing Android Malware Samples
Analyzing Android Malware Samples
(Ab)using Android internals
File hierarchy
Android security model
Process management
Filesystem
App permissions
Security services
Console
To root or not to root?
Understanding Dalvik and ART
Dalvik VM (DVM)
Android runtime (ART)
APIs
File formats
DEX
ODEX
OAT
VDEX
ART
ELF
APK
Bytecode set
Malware behavior patterns
Attack stages
Penetration
Deployment
Action phase
Advanced techniques—investment pays off
Patching system libraries
Keylogging
Self-defense
Rootkits—get it covered
Static and dynamic analysis of threats
Static analysis
Disassembling and data extraction
Decompiling
Dynamic analysis
Android debug bridge
Emulators
Behavioral analysis and tracing
Debuggers
Analysis workflow
Summary
17
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Hooking the SYSENTER entry function

When a user-mode application executes sysenter (int 0x2e in Windows 2000 and earlier versions), the processor switches the execution to kernel mode and, in particular, to a specific address stored in the Model Specific Register (MSR). MSRs are the control registers that are used for debugging, monitoring, toggling, or disabling various CPU features.

There are three important registers for the user-mode-to-kernel-mode switching process using sysenter:

  • MSR 0x174 (IA32_SYSENTER_CS): This stores the CS segment register value, which is available after using sysenter; here, the SS segment register will be a CS value + 8.
  • MSR 0x175 (IA32_SYSENTER_ESP): This stores the value of the kernel-mode stack pointer once sysenter is executed; it is where the arguments generally will be copied to.
  • MSR 0x176 (IA32_SYSENTER_EIP): This is the new EIP value after executing sysenter. It points to the KiSystemService function on x86 or the KiSystemCall64 function on x86-64...
Previous Section
End of Section 13
Next Section