Book Image

Mastering Reverse Engineering

By : Reginald Wong
Book Image

Mastering Reverse Engineering

By: Reginald Wong

Overview of this book

If you want to analyze software in order to exploit its weaknesses and strengthen its defenses, then you should explore reverse engineering. Reverse Engineering is a hackerfriendly tool used to expose security flaws and questionable privacy practices.In this book, you will learn how to analyse software even without having access to its source code or design documents. You will start off by learning the low-level language used to communicate with the computer and then move on to covering reverse engineering techniques. Next, you will explore analysis techniques using real-world tools such as IDA Pro and x86dbg. As you progress through the chapters, you will walk through use cases encountered in reverse engineering, such as encryption and compression, used to obfuscate code, and how to to identify and overcome anti-debugging and anti-analysis tricks. Lastly, you will learn how to analyse other types of files that contain code. By the end of this book, you will have the confidence to perform reverse engineering.
Table of Contents (20 chapters)
Title Page
Copyright and Credits
Packt Upsell
Contributors
Preface
Index

Index

A

  • addition operation / Addition and subtraction
  • analysis summary / Analysis summary
  • anti-debugging tricks
    • about / Anti-debugging tricks
    • IsDebuggerPresent / IsDebuggerPresent
    • flags, debugging in PEB / Debug flags in the PEB
    • information, debugging from NtQueryInformationProcess / Debugger information from NtQueryInformationProcess
    • timing tricks / Timing tricks
    • code execution, passing via SEH / Passing code execution via SEH
  • anti-dumping tricks / Anti-dumping tricks
  • anti-emulation tricks / Anti-emulation tricks
  • anti-VM tricks
    • about / Anti-VM tricks
    • VM running process names / VM running process names
    • existence of VM files/directories / Existence of VM files and directories
    • default MAC address / Default MAC address
    • registry entries made by VMs / Registry entries made by VMs
    • VM devices / VM devices
    • CPUID results / CPUID results
  • API Monitor / Monitoring tools
  • APIs
    • calling / Calling APIs
    • Windows API libraries / Common Windows API libraries
    • functions / Short list of common API functions
    • learning / Learning about the APIs
    • keylogger / Keylogger
    • regenum / regenum
    • server / The server
  • Application Program Interface (API) / Disassemblers
  • arithmetic operations
    • about / Arithmetic operations
    • addition / Addition and subtraction
    • subtraction / Addition and subtraction
    • decrement instruction / Increment and decrement instructions
    • increment instruction / Increment and decrement instructions
    • division instruction / Multiplication and division instructions
    • multiplication instruction / Multiplication and division instructions
    • signed operations / Other signed operations
  • ARM-compiled executables
    • analyzing / Analysis in unfamiliar environments
  • assemblers
    • about / Popular assemblers
    • MASM / MASM
    • NASM / NASM
    • FASM / FASM
  • assessment
    • ideas / Assessment and static analysis
    • file types / File types and header analysis
    • header analysis / File types and header analysis
  • attack tools
    • about / Attack tools
    • Metasploit / Attack tools
    • ExploitPack / Attack tools
  • automated analysis
    • tools / Dynamic analysis
  • automated dynamic analysis
    • about / Automated dynamic analysis
    • Cuckoo / Automated dynamic analysis
    • ThreatAnalyzer / Automated dynamic analysis
    • Joe Sandbox / Automated dynamic analysis
    • Buster Sandbox Analyzer (BSA) / Automated dynamic analysis
    • Regshot / Automated dynamic analysis
  • automation tools
    • about / Automation tools
    • Python / Automation tools
    • Yara / Automation tools
    • Visual Studio / Automation tools
  • autoruns / Autoruns

B

  • bases
    • about / Bases
    • converting between / Converting between bases
  • basic analysis lab setup / Basic analysis lab setup, Our setup
  • basic instructions
    • about / Basic instructions
    • opcode bytes / Opcode bytes
    • data, copying / Copying data
    • arithmetic operations / Arithmetic operations
    • bitwise algebra / Bitwise algebra
    • control flow / Control flow
    • stack manipulation / Stack manipulation
  • BEYE
    • about / Disassemblers
    • reference / Disassemblers
  • binary analysis tools / Binary analysis tools
  • binary arithmetic / Binary arithmetic
  • binary numbers
    • about / Binary numbers
    • bases / Bases
    • signed numbers / Signed numbers
  • bintext
    • reference / Try it yourself
  • BinText / Strings
  • bitwise algebra
    • about / Bitwise algebra
    • NOT / Bitwise algebra
    • AND / Bitwise algebra
    • OR / Bitwise algebra
    • XOR / Bitwise algebra
    • SHL/SAL / Bitwise algebra
    • SHR/SAR / Bitwise algebra
    • ROL / Bitwise algebra
    • ROR / Bitwise algebra
  • Bless / Editing tools
  • Bochs
    • about / Emulators
    • MBR debugging / MBR debugging with Bochs
  • Buster Sandbox Analyzer (BSA) / Automated dynamic analysis

C

  • Capstone
    • about / Disassemblers
    • reference / Disassemblers
  • CaptureBAT / Monitoring tools
  • CFF Explorer
    • about / File type information
    • download link / Static analysis
  • code assembly / Code assembly
  • Complex Instruction Set Computing (CISC) / Emulation
  • compressors / Packers or compressors
  • control flow / Control flow
  • CPU architectures / Emulation
  • CPUID
    • reference / CPUID results
  • crypters / Crypters
  • Cryptographic Service Provider (CSP) / Encrypting and decrypting a file
  • Cuckoo / Dynamic analysis, Automated dynamic analysis
  • Cyberchef / Other file-types

D

  • data assembly
    • on stack / Data assembly on the stack
    • in memory regions / Assembly of data in other memory regions
  • deadlisting / Deadlisting
  • debuggers
    • about / Debuggers, Debuggers
    • x86dbg / Debuggers
    • IDA Pro / Debuggers
    • OllyDebug / Debuggers
    • Immunity Debugger / Debuggers
    • Windbg / Debuggers
    • GDB / Debuggers
    • Radare / Debuggers
  • debugging
    • about / Debugging, Debugging
    • unknown image / The unknown image
    • analysis summary / Analysis summary
  • decompilers
    • about / Decompilers, Decompilers, Decompilers
    • ILSpy / ILSpy – C# Decompiler
    • Snowman / Decompilers
    • Hex-Rays / Decompilers
    • dotPeek / Decompilers
    • iLSpy / Decompilers
  • decrement instruction / Increment and decrement instructions
  • default command-line tools
    • strings / Default command-line tools
    • md5sum / Default command-line tools
    • file / Default command-line tools
  • Detect-it-Easy (DiE) / File type information
  • disassemblers
    • about / Disassemblers, Disassemblers
    • IDA Pro / Disassemblers
    • Radare / Disassemblers
    • Capstone / Disassemblers
    • Hopper / Disassemblers
    • BEYE / Disassemblers
    • HIEW / Disassemblers
  • disk filesystems / The filesystem
  • division instruction / Multiplication and division instructions
  • dlroW olleH
    • about / dlroW olleH
    • information / What have we gathered so far?
    • dynamic analysis / Dynamic analysis
    • debugging / Going further with debugging
  • dotPeek
    • reference / Decompilers
    • about / Decompilers
  • dynamic analysis
    • about / Dynamic analysis
    • memory regions / Memory regions and the mapping of a process
    • mapping process / Memory regions and the mapping of a process
    • memory process / Memory regions and the mapping of a process
    • process monitoring / Process and thread monitoring
    • thread monitoring / Process and thread monitoring
    • network traffic / Network traffic
    • system changes, monitoring / Monitoring system changes
    • post-execution differences / Post-execution differences
    • debugging / Debugging

E

  • editing tools
    • about / Editing tools
    • HxD Hex Editor / Editing tools
    • Bless / Editing tools
    • Notepad++ / Editing tools
    • BEYE / Editing tools
    • HIEW / Editing tools
  • emulation
    • about / Emulation
    • of Windows, under x86 host / Emulation of Windows and Linux under an x86 host
    • of Linux, under x86 host / Emulation of Windows and Linux under an x86 host
  • emulators
    • about / Emulators
    • QEMU / Emulators
    • Bochs / Emulators
  • encrypted data identification
    • about / Encrypted data identification
    • loop codes / Loop codes
    • simple arithmetic / Simple arithmetic
    • simple XOR decryption / Simple XOR decryption
  • environment setup, for tools
    • virtual machines / Virtual machines
    • Windows / Windows
    • Linux / Linux
  • errors
    • dealing with / Dealing with common errors when building
  • exceptions / Causing exceptions
  • executable
    • in unpacked state / How about an executable in its unpacked state?
  • executable files / Packers, crypters, obfuscators, protectors and SFX
  • ExifTool / File type information
  • ExploitPack
    • about / Attack tools
    • reference / Attack tools

F

  • Falcon Sandbox / Online service sites
  • FASM
    • about / FASM
    • download link / FASM
    • installing / Installation of FASM
    • working / It works!
  • file
    • information, extracting from / Extracting useful information from file
    • about / file
    • encrypting / Encrypting and decrypting a file
    • decrypting / Encrypting and decrypting a file
  • File-types / Other file-types
  • filesystem / The filesystem
  • file type information tools
    • PEiD / File type information
    • TrID / File type information
    • CFF Explorer / File type information
    • PE Explorer / File type information
    • Detect-it-Easy (DiE) / File type information
    • ExifTool / File type information
  • Flare
    • about / Flare
    • reference / Flare
  • FLASM
    • about / FLASM
    • reference / FLASM

G

  • GDB / Debuggers

H

  • hash information / Other information
  • HashTab / Hash identifying
  • Hello World
    • in Radare2 / Hello World in Radare2
    • about / Hello World
  • hello world program, Linux
    • about / Linux executable – hello world
    • dlroW olleH / dlroW olleH
    • password / What is the password?
  • Hex-Rays / Decompilers
  • HIEW / Disassemblers
  • Hopper / Disassemblers
  • HTML scripts
    • analyzing / Analysis of HTML scripts
  • HxD
    • reference / Extracting useful information from file
  • HxD Hex Editor / Editing tools

I

  • IDA (Interactive Disassembler) / IDA (Interactive Disassembler)
  • IDA Pro
    • reference / Try it yourself, Disassemblers
    • about / Disassemblers, Debuggers
  • iLSpy
    • reference / Decompilers
    • about / Decompilers
  • ILSpy / ILSpy – C# Decompiler
  • Immunity Debugger
    • reference / Debuggers
    • about / Debuggers
  • increment instruction / Increment and decrement instructions
  • inetsim / Network tools
  • information
    • extracting, from file / Extracting useful information from file
  • information gathering tools
    • about / Information gathering tools
    • file type information / File type information
    • hash identifying / Hash identifying
    • strings / Strings
    • monitoring tools / Monitoring tools
    • default command-line tools / Default command-line tools
  • initial file information
    • obtaining / Initial file information
  • initial static analysis
    • about / Initial static analysis
    • initial file information / Initial file information
    • deadlisting / Deadlisting
  • IsDebuggerPresent / IsDebuggerPresent
  • ISO, for Ubuntu installer
    • reference / Setup

J

  • Joe Sandbox / Automated dynamic analysis
  • JPEXS SWF decompiler / JPEXS SWF decompiler 
    • reference / JPEXS SWF decompiler 
    • about / JPEXS SWF decompiler 

K

  • keylogger / Keylogger

L

  • LEA / MOV and LEA
  • Linux / Linux
  • Linux ARM guest
    • in QEMU / Linux ARM guest in QEMU
  • Linux ELF file
    • reference / A quick review on how native executables are loaded by the OS
  • Lubuntu / Linux

M

  • malware
    • handling / Malware handling
    • about / Typical malware behavior
  • malware delivery
    • about / Malware delivery
    • email / Email
    • instant messenger / Instant messenger
    • computer network / The computer network
    • media storage / Media storage
    • exploits / Exploits and compromised websites
    • compromised websites / Exploits and compromised websites
    • software piracy / Software piracy, Malware file properties
  • malware persistence
    • about / Persistence
    • run keys / Run keys
    • load values / Load and Run values
    • run values / Load and Run values
    • BootExecute value / Load and Run values
    • Winlogon key / Load and Run values
    • policy scripts keys / Load and Run values
    • AppInit_DLLs values / Load and Run values
    • services keys / Load and Run values
    • file associations / Load and Run values
    • startup values / Startup values
    • Image file execution options key / The Image File Execution Options key
  • Malwr
    • reference / Dynamic analysis
    • about / Online service sites
  • MASM
    • about / MASM
    • download link / MASM
  • Master Boot Record (MBR) / Virtual machines
  • MASTIFF
    • about / MASTIFF
    • example / MASTIFF
    • download link / MASTIFF
  • MBR debugging
    • with Bochs / MBR debugging with Bochs
  • memory
    • about / Memory
    • processes, dumping from / Dumping processes from memory
  • memory addressing
    • about / Memory addressing
    • endianness / Endianness
  • Memory Boot Record (MBR) / Emulators
  • memory dumping
    • with VirtualBox / Memory dumping with VirtualBox
  • Metasploit
    • about / Attack tools
    • reference / Attack tools
  • MinGW
    • reference / NASM
  • mitmproxy / Network tools
  • monitoring tools
    • about / Monitoring tools, Monitoring tools
    • SysInternals Suite's Procmon or Process Monitor / Monitoring tools
    • API Monitor / Monitoring tools
    • CaptureBAT / Monitoring tools
  • MOV / MOV and LEA
  • MS Office macro analysis
    • about / MS Office macro analysis
    • performing / MS Office macro analysis
  • multiplication instruction / Multiplication and division instructions

N

  • NASM
    • about / NASM
    • reference / NASM
  • native executables
    • loading, by OS / A quick review on how native executables are loaded by the OS
  • network tools
    • about / Network tools
    • tcpdump / Network tools
    • Wireshark / Network tools
    • mitmproxy / Network tools
    • inetsim / Network tools
  • network traffic analysis / Network traffic analysis
  • Notepad++
    • about / Editing tools
    • reference / Analysis of HTML scripts
  • NtQueryInformationProcess
    • reference / Debugger information from NtQueryInformationProcess

O

  • obfuscation techniques
    • about / Other obfuscation techniques
    • control flow flattening obfuscation / Control flow flattening obfuscation
    • garbage code insertion / Garbage code insertion
    • with metamorphic engine / Code obfuscation with a metamorphic engine
    • dynamic library loading / Dynamic library loading
    • PEB information usage / Use of PEB information
  • obfuscators / Obfuscators
  • OllyDebug
    • about / Ollydebug, Debuggers
    • download link / Ollydebug
    • reference / Debuggers
  • online service sites
    • VirusTotal / Online service sites
    • Malwr / Online service sites
    • Falcon Sandbox / Online service sites
    • whois.domaintools.com / Online service sites
    • robtex.com / Online service sites
    • debuggex.com / Online service sites
  • opcode bytes / Opcode bytes
  • operating system environment
    • about / The operating system environment
    • filesystem / The filesystem
    • memory / Memory
    • registry system / The registry system

P

  • packed executable
    • unpacking / Unpacking
  • packer
    • about / Packers or compressors
    • used, for debugging / Debugging though the packer
  • password
    • about / What is the password?
    • static analysis / Static analysis
    • quick run / A quick run
    • deadlisting / Deadlisting
    • dynamic analysis with debugging / Dynamic analysis with debugging
    • decompilers / Decompilers
  • payload / Payload – the evil within
  • PDF file analysis
    • about / PDF file analysis
    • performing / PDF file analysis
  • PE executables / PE executables
  • PE Explorer / File type information
  • PEiD
    • about / PEid and TrID, File type information
    • reference / PEid and TrID
  • practical reverse engineering, of Windows Executable
    • preparing for / Things to prepare
    • initial static analysis / Initial static analysis
    • debugging / Debugging
  • Process Environment Block (PEB) / Use of PEB information, Anti-debugging tricks
  • processes
    • dumping, from memory / Dumping processes from memory
  • Process explorer tool / The Process explorer
  • processlist / processlist
  • program
    • about / Dissecting the program 
    • dissecting / Dissecting the program 
  • protectors / Protectors
  • Python
    • reference / Automation tools
    • about / Automation tools
  • python-magic
    • about / python-magic
    • download link / python-magic

Q

  • QEMU / Emulators
  • Quickhash
    • about / Hash identifying
    • reference / Static analysis

R

  • rabin2 / Hello World in Radare2
  • Radare
    • about / Disassemblers, Debuggers, Setup
    • reference / Disassemblers
  • Radare2
    • Hello World / Hello World in Radare2
  • Read Time-Stamp Counter (RDTSC) / Timing tricks
  • Reduced Instruction Set Computing (RISC) / Emulation
  • regenum / regenum
  • registers / Registers
  • registry system / The registry system
  • RegShot / Dynamic analysis, Automated dynamic analysis
  • resource forks / The filesystem
  • reverse engineering
    • about / Reverse engineering
  • reverse engineering, as process
    • about / Reverse engineering as a process
    • approval, seeking / Seeking approval
    • static analysis / Static analysis
    • dynamic analysis / Dynamic analysis
    • low-level analysis / Low-level analysis
    • reporting / Reporting
  • reverse engineering, Linux
    • setup / Setup

S

  • Sandboxie
    • about / Dynamic analysis
    • reference / Dynamic analysis
  • SEH
    • setting up / A typical SEH setup
  • self-extracting archives (SFX) / SFX  Self-extracting archives
  • signed numbers / Signed numbers
  • signed operations
    • NEA / Other signed operations
    • MOVSX / Other signed operations
    • CBW / Other signed operations
    • CWDE / Other signed operations
    • CWD / Other signed operations
    • IMUL/IDIV / Other signed operations
  • Snowman
    • reference / Decompilers
    • about / Decompilers
  • software forensic tools
    • about / Software forensic tools
    • references / Software forensic tools
  • stack
    • about / Data assembly on the stack
    • data assembly / Data assembly on the stack
  • stack manipulation / Stack manipulation
  • static analysis
    • about / Assessment and static analysis, Static analysis, Analysis in unfamiliar environments
    • trying / Try it yourself
  • strings / Strings
  • Strings
    • reference / Deadlisting
  • Structured Error Handlers (SEH) / Use of PEB information
  • Structured Exception Handler (SEH) / Anti-debugging tricks
  • subtraction operation / Addition and subtraction
  • SWF file analysis
    • about / SWF file analysis
    • performing / SWF file analysis
  • SWFTools
    • about / SWFTools
    • reference / SWFTools
  • SysInternals suite
    • reference / Technical requirements
  • SysInternals Suite's Procmon or Process Monitor / Monitoring tools
  • SysInternals Suite's string / Strings

T

  • tcpdump / Network tools
  • Thread Environment Block (TEB) / Debug flags in the PEB
  • Thread Information Block (TIB) / Use of PEB information, Debug flags in the PEB
  • ThreatAnalyzer / Dynamic analysis, Automated dynamic analysis
  • tools
    • about / Tools, Tools
    • binary analysis tools / Binary analysis tools
    • disassemblers / Disassemblers
    • debuggers / Debuggers
    • monitoring tools / Monitoring tools
    • decompilers / Decompilers
    • autoruns / Autoruns
    • Process explorer / The Process explorer
    • environment setup / Analysis environments
    • information gathering tools / Information gathering tools
  • Transmission Control Protocol (TCP) / Network traffic analysis
  • TrID
    • about / PEid and TrID, File type information
    • reference / PEid and TrID

U

  • Ubuntu forums
    • reference / Setup
  • UPX
    • reference / Try it yourself
  • UPX tool
    • about / The UPX tool
    • download link / The UPX tool
  • User Datagram Protocol (UDP) / Network traffic analysis

V

  • Vectored Exception Handler
    • reference / A typical SEH setup
  • VirtualBox
    • reference / Technical requirements, Our setup
    • downloading / Our setup
    • memory dumping / Memory dumping with VirtualBox
  • virtualization software
    • VMWare Workstation / Virtual machines
    • VirtualBox / Virtual machines
    • Qemu (Quick Emulator) / Virtual machines
    • Bochs / Virtual machines
    • Microsoft Hyper-V / Virtual machines
  • virtual machines, Microsoft
    • download link / To get the most out of this book, Technical requirements
  • VirtualProtect / Debugging though the packer
  • VirusTotal / Online service sites
  • Visual Studio / Automation tools
  • Visual Studio Community edition
    • reference / Hello World
  • Volatility
    • about / Software forensic tools
    • reference / Software forensic tools
    • download link / Extracting the process to a file using Volatility
    • used, for extracting process to file / Extracting the process to a file using Volatility

W

  • whois.domaintools.com / Online service sites
  • WinDbg
    • about / WinDbg, Debuggers
    • download link / WinDbg
  • Windows / Windows
  • Windows 7 32-bit
    • download link / Our setup
  • Windows API libraries
    • KERNEL32 / Common Windows API libraries
    • USER32 / Common Windows API libraries
    • ADVAPI32 / Common Windows API libraries
    • MSVCRT / Common Windows API libraries
    • WININET / Common Windows API libraries
    • WS2_32 / Common Windows API libraries
    • URLMON / Common Windows API libraries
    • NETAPI32 / Common Windows API libraries
  • Windows PE file
    • reference / A quick review on how native executables are loaded by the OS
  • Wireshark
    • about / Network tools
    • reference / Network tools, Network traffic analysis

X

  • x64dbg
    • about / x64dbg
    • download link / x64dbg, Debugging
  • x86
    • about / x86, Emulation
    • registers / Registers
    • memory addressing / Memory addressing
  • x86 Debuggers
    • about / x86 Debuggers
    • WinDbg / WinDbg
    • OllyDebug / Ollydebug
    • x64dbg / x64dbg
  • x86dbg
    • about / Debuggers
    • reference / Debuggers
    • file, decrypting with / Decrypting with x86dbg
  • XXXSWF
    • about / XXXSWF
    • reference / XXXSWF

Y

  • Yara / Automation tools

Z

  • 7-zip / Hash identifying