Book Image

Mastering Malware Analysis

By : Alexey Kleymenov, Amr Thabet

Book Image

Mastering Malware Analysis

By: Alexey Kleymenov, Amr Thabet

Overview of this book

With the ever-growing proliferation of technology, the risk of encountering malicious code or malware has also increased. Malware analysis has become one of the most trending topics in businesses in recent years due to multiple prominent ransomware attacks. Mastering Malware Analysis explains the universal patterns behind different malicious software types and how to analyze them using a variety of approaches. You will learn how to examine malware code and determine the damage it can possibly cause to your systems to ensure that it won't propagate any further. Moving forward, you will cover all aspects of malware analysis for the Windows platform in detail. Next, you will get to grips with obfuscation and anti-disassembly, anti-debugging, as well as anti-virtual machine techniques. This book will help you deal with modern cross-platform malware. Throughout the course of this book, you will explore real-world examples of static and dynamic malware analysis, unpacking and decrypting, and rootkit detection. Finally, this book will help you strengthen your defenses and prevent malware breaches for IoT devices and mobile platforms. By the end of this book, you will have learned to effectively analyze, investigate, and build innovative solutions to handle any malware incidents.

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Free Chapter

Section 1: Fundamental Theory

Section 1: Fundamental Theory

A Crash Course in CISC/RISC and Programming Basics

A Crash Course in CISC/RISC and Programming Basics

Branches, loops, and conditions

Exceptions, interrupts, and communicating with other devices

Assembly languages

CISC versus RISC

Types of instructions

Becoming familiar with x86 (IA-32 and x64)

Special registers

The instruction structure

The instruction set

Data manipulation instructions

Data transfer instructions

Flow control instructions

Arguments, local variables, and calling conventions (in x86 and x64)

Local variables

The x64 calling convention

Exploring ARM assembly

Instruction sets

The instruction set

Diving deep into PowerPC

The instruction set

Covering the SuperH assembly

The instruction set

Working with SPARC

The instruction set

Moving from assembly to high-level programming languages

Arithmetic statements

While loop conditions

Section 2: Diving Deep into Windows Malware

Section 2: Diving Deep into Windows Malware

Basic Static and Dynamic Analysis for x86/x64

Basic Static and Dynamic Analysis for x86/x64

Working with the PE header structure

Exploring PE structure

Optional header

PE header analysis tools

Static and dynamic linking

Dynamic linking

Dynamic link libraries

Application programming interface

Dynamic API loading

Using PE header information for static analysis

How to use PE header for incident handling

How to use a PE header for threat intelligence

PE loading and process creation

Basic terminology

What's process?

Virtual memory to physical memory mapping

Important data structures: TIB, TEB, and PEB

Process loading step by step

PE file loading step by step

WOW64 processes

Dynamic analysis with OllyDbg/Immunity Debugger

Debugging tools

How to analyze a sample with OllyDbg

Types of breakpoints

Step into/step over breakpoint

INT3 breakpoint

Memory breakpoints

Hardware breakpoints

Modifying the program execution

Patching—modifying the program's assembly instructions

Modifying the instruction pointer value

Changing the program data

Debugging malicious services

What is service?

Attaching to the service

Unpacking, Decryption, and Deobfuscation

Unpacking, Decryption, and Deobfuscation

Exploring packers

Exploring packing and encrypting tools

Identifying a packed sample

Technique 1 – checking PE tool static signatures

Technique 2 – evaluating PE section names

Technique 3 – using stub execution signs

Technique 4 – detecting a small import table

Automatically unpacking packed samples

Technique 1 – the official unpacking process

Technique 2 – using OllyScript with OllyDbg

Technique 3 – using generic unpackers

Technique 4 – emulation

Technique 5 – memory dumps

Manual unpacking using OllyDbg

Technique 6 – memory breakpoint on execution

Step 1 – setting the breakpoints

Step 2 – turning on Data Execution Prevention

Step 3 – preventing any further attempts to change memory permissions

Step 4 – executing and getting the OEP

Technique 7 – call stack backtracing

Step 1 – setting the breakpoints

Step 2 – following the call stack

Step 3 – reaching the OEP

Technique 8 – monitoring memory allocated spaces for unpacked code

Technique 9 – in-place unpacking

Technique 10 – stack restoration-based

Dumping the unpacked sample and fixing the import table

Dumping the process

Fixing the import table

Identifying different encryption algorithms and functions

Types of encryption algorithms

Basic encryption algorithms

How to identify encryption functions

String search detection techniques for simple algorithms

The basics of X-RAYING

Simple static encryption

Other encryption algorithms

X-RAYING tools for malware analysis and detection

Identifying the RC4 encryption algorithm

The RC4 encryption algorithm

Key-scheduling algorithm

Pseudo-random generation algorithm

Identifying RC4 algorithms in a malware sample

Standard symmetric and asymmetric encryption algorithms

Extracting information from Windows cryptography APIs

Step 1 – initializing and connecting to the cryptographic service provider (CSP)

Step 2 – preparing the key

Step 3 – encrypting or decrypting the data

Step 4 – freeing the memory

Cryptography API next generation (CNG)

Applications of encryption in modern malware – Vawtrak banking Trojan

String and API name encryption

Network communication encryption

Using IDA for decryption and unpacking

IDA tips and tricks

Static analysis

Dynamic analysis

Classic and new syntax of IDA scripts

Dynamic string decryption

Dynamic WinAPIs resolution

Inspecting Process Injection and API Hooking

Inspecting Process Injection and API Hooking

Understanding process injection

What's process injection?

Why process injection?

Windows-supported DLL injection

A simple DLL injection technique

Working with process injection

Getting the list of running processes

Advanced code injection-reflective DLL injection

Stuxnet secret technique-process hollowing

Dynamic analysis of code injection

Technique 1—debug it where it is

Technique 2—attach to the targeted process

Technique 3—dealing with process hollowing

Memory forensics techniques for process injection

Technique 1—detecting code injection and reflective DLL injection

Technique 2—detecting process hollowing

Technique 3—detecting process hollowing using the HollowFind plugin

Understanding API hooking

Why API hooking?

Working with API hooking

Inline API hooking

Inline API hooking with trampoline

Inline API hooking with a length disassembler

Detecting API hooking using memory forensics

Exploring IAT hooking

Bypassing Anti-Reverse Engineering Techniques

Bypassing Anti-Reverse Engineering Techniques

Exploring debugger detection

Direct check for debugger presence

Detecting a debugger through an environment change

Detecting a debugger using parent processes

Handling debugger breakpoints evasion

Detecting software breakpoints (INT3)

Detecting single-stepping breakpoints (trap flag)

Detecting a trap flag using the SS register

Detecting single-stepping using timing techniques

Evading hardware breakpoints

What is structured exception handling?

Detecting and removing hardware breakpoints

Memory breakpoints

Escaping the debugger

Process injection

Windows events callbacks

Obfuscation and anti-disassemblers

Junk code insertion

Code transportation

Dynamic API calling with checksum

Proxy functions and proxy argument stacking

Detecting and evading behavioral analysis tools

Finding the tool process

Searching for the tool window

Detecting sandboxes and virtual machines

Different output between virtual machines and real machines

Detecting virtualization processes and services

Detecting virtualization through registry keys

Detecting virtual machines using PowerShell

Detecting sandboxes by using default settings

Other techniques

Understanding Kernel-Mode Rootkits

Understanding Kernel-Mode Rootkits

Kernel mode versus user mode

Protection rings

Windows internals

The infrastructure of Windows

The execution path from user mode to kernel mode

Rootkits and device drivers

What is a rootkit?

Types of rootkits

What is a device driver?

Hooking mechanisms

Hooking the SYSENTER entry function

Modifying SSDT in an x86 environment

Modifying SSDT in an x64 environment

Hooking SSDT functions

Devices and major functions

Attaching to a device

Modifying the IRP response and setting a completion routine

The kernel objects—EPROCESS and ETHREAD

How do rootkits perform an object manipulation attack?

Process injection in kernel mode

Executing the inject code using APC queuing

KPP in x64 systems (PatchGuard)

Bypassing driver signature enforcement

Bypassing PatchGuard—the Turla example

Bypassing PatchGuard—GhostHook

Disabling PatchGuard using the Command Prompt

Static and dynamic analysis in kernel mode

Static analysis

Tips and tricks

Dynamic and behavioral analysis

Rootkit detectors

Setting up a testing environment

Setting up the debugger

Stopping at the driver's entry point

Loading the driver

Restoring the debugging state

Section 3: Examining Cross-Platform Malware

Section 3: Examining Cross-Platform Malware

Handling Exploits and Shellcode

Handling Exploits and Shellcode

Getting familiar with vulnerabilities and exploits

Types of vulnerabilities

Stack overflow vulnerability

Heap overflow vulnerabilities

The use-after-free vulnerability

Logical vulnerabilities

Types of exploits

Cracking the shellcode

What's shellcode?

Linux shellcode in x86-64

Getting the absolute address

Null-free shellcode

Local shell shellcode

Reverse shell shellcode

Linux shellcode for ARM

Null-free shellcode

Windows shellcode

Getting the Kernel32.dll's ImageBase

Getting the required APIs from Kernel32.dll

The download and execute shellcode

Static and dynamic analysis of exploits

Analysis workflow

Shellcode analysis

Exploring bypasses for exploit mitigation technologies

Data execution prevention (DEP/NX)

Return-oriented programming

Address space layout randomization

DEP and partial ASLR

DEP and full ASLR – partial ROP and chaining multiple vulnerabilities

DEP and full ASLR – heap spray technique

Other mitigation technologies

Analyzing Microsoft Office exploits

File structures

Compound file binary format

Rich text format

Office open XML format

Static and dynamic analysis of MS Office exploits

Static analysis

Dynamic analysis

Studying malicious PDFs

Static and dynamic analysis of PDF files

Static analysis

Dynamic analysis

Reversing Bytecode Languages: .NET, Java, and More

Reversing Bytecode Languages: .NET, Java, and More

The basic theory of bytecode languages

Object-oriented programming

.NET file structure

.NET COR20 header

Metadata streams

How to identify a .NET application from PE characteristics

The CIL language instruction set

Pushing into stack instructions

Pulling out a value from the stack

Mathematical and logical operations

Branching instructions

CIL language to higher-level languages

Local variable assignments

Local variable assignment with a method return value

Basic branching statements

Loops statements

.NET malware analysis

.NET analysis tools

Static and dynamic analysis (with Dnspy)

.NET static analysis

.NET dynamic analysis

Patching a .NET sample

Dealing with obfuscation

Obfuscated names for classes, methods, and others

Encrypted strings inside the binary

The sample is obfuscated using an obfuscator

The essentials of Visual Basic

P-code versus native code

Common p-code instructions

Dissecting Visual Basic samples

Static analysis

Dynamic analysis

The internals of Java samples

JVM instructions

Static analysis

Dynamic analysis

Dealing with anti-reverse engineering solutions

Python—script language internals

Bytecode instructions

Analyzing compiled Python

Static analysis

Dynamic analysis

Scripts and Macros: Reversing, Deobfuscation, and Debugging

Scripts and Macros: Reversing, Deobfuscation, and Debugging

Classic shell script languages

Windows batch scripting

VBScript explained

Static and dynamic analysis

Those evil macros inside documents

Static and dynamic analysis

The power of PowerShell

Static and dynamic analysis

Handling JavaScript

Static and dynamic analysis

Anti-reverse engineering tricks

Behind C and C—even malware has its own backend

Things to focus on

Static and dynamic analysis

Other script languages

Where to start from

Questions to answer

Section 4: Looking into IoT and Other Platforms

Section 4: Looking into IoT and Other Platforms

Dissecting Linux and IoT Malware

Dissecting Linux and IoT Malware

Explaining ELF files

Process management

Syscalls in assembly

Common anti-reverse engineering tricks

Exploring common behavioral patterns

Initial delivery and lateral movement

Privilege escalation

Interaction with the command and control server

Attacking stage

Static and dynamic analysis of x86 (32- and 64-bit) samples

Static analysis

File type detectors

Dynamic analysis

Network monitors

Binary emulators

Radare2 cheat sheet

Anti-reverse engineering techniques

Learning Mirai, its clones, and more

High-level functionality

Later derivatives

Other widespread families

Static and dynamic analysis of RISC samples

Handling other architectures

What to start from

Introduction to macOS and iOS Threats

Introduction to macOS and iOS Threats

Understanding the role of the security model

Security policies

Filesystem hierarchy and encryption

Directory structure

Apps protection

Other technologies

System security

Data encryption and password management

File formats and APIs

Application bundles (.app)

Installer packages (.pkg)

Apple disk images (.dmg)

iOS app store packages (.ipa)

Static and dynamic analyses of macOS and iOS samples

Static analysis

Retrieving samples

Disassemblers and decompilers

Auxiliary tools and libraries

Dynamic and behavioral analysis

Monitoring and dynamic instrumentation

Network analysis

Installers and loaders

Dumping and decryption

Monitors and in-memory patching

Network analysis

Jailbreaks on demand

Deployment and persistence

Other attack techniques

Advanced techniques

Anti-reverse-engineering (RE) tricks

Misusing dynamic data exchange (DDE)

Use of AppleScript

Rootkits for Mac—do they exist?

Analysis workflow

Analyzing Android Malware Samples

Analyzing Android Malware Samples

(Ab)using Android internals

Android security model

Process management

App permissions

Security services

To root or not to root?

Understanding Dalvik and ART

Dalvik VM (DVM)

Android runtime (ART)

Malware behavior patterns

Advanced techniques—investment pays off

Patching system libraries

Rootkits—get it covered

Static and dynamic analysis of threats

Static analysis

Disassembling and data extraction

Dynamic analysis

Android debug bridge

Behavioral analysis and tracing

Analysis workflow

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Auxiliary tools and libraries

The following are the auxiliary tools and libraries for static analysis:

plutil: This tool is very useful when we need to convert the binary version of .plist to readable formats, such as XML. For non-macOS platforms, it is installed together with iTunes.
otool/MachOView: A Mac console tools that allows us to view different parts of Mach-O files.
class-dump/class-dump-z: These tools can be used to generate Objective-C headers from Mach-O files.
LIEF: A cross-platform library can be used to both parse and modify Mach-O executables.
Capstone: A cross-platform disassembly framework that powers multiple reverse-engineering tools.

Apart from this, many basic universal tools, such as file, strings, or nm can be used to extract information from executables.