Doing reverse code engineering starts off with understanding the meaning of every bit and byte. Simply viewing the bytes contained requires developing tools that aid in the reading of files and objects. Parsing and adding meaning to every byte would require another tool. Reverse engineering has evolved with tools that are continuously updated when encountering new software technology. Here, we have categorized these tools into binary analysis tools, disassemblers, decompilers, debuggers, and monitoring tools.
Binary analysis tools are used to parse binary files and extract information about the file. An analyst would be able to identify which applications are able to read or execute the binary. File types are generally identified from their magic header bytes. These Magic Header bytes are usually located at the beginning of a file. For example, a Microsoft executable file, an EXE file, begin with the MZ header (MZ is believed to be the initials of Mark Zbikowski, a developer from Microsoft during the DOS days). Microsoft Office Word documents, on the other hand, have these first four bytes as their Magic Header:
The hexadecimal bytes in the preceding screenshot read as DOCFILE Other information such as text string also give hints. The following screenshot shows information indicating that the program was most likely built using Window Forms:
Disassemblers are used to view the low-level code of a program. Reading low-level code requires knowledge of assembly language. Analysis done with a disassembler gives information about the execution conditions and system interactions that a program will carry out when executed. However, the highlights when reading low-level code are when the program uses Application Program Interface (API) functions. The following screenshot shows a code snippet of a program module that uses the GetJob() API. This API is used to get information about the printer job, as shown here:
Disassemblers can show the code tree, but the analyst can verify which branch the code flows to by using a debugger. A debugger does actual execution per line of code. The analyst can trace through codes such as loops, conditional statements, and API execution. Since debuggers are categorized under dynamic analysis and perform a step-wise execution of code, debugging is done in an enclosed environment. Various file types have different disassemblers. In a .NET compiled executable, it is best to instead disassemble the p-code and work out what each operator means.
Monitoring tools are used to monitor system behaviors regarding file, registry, memory, and network. These tools usually tap or hook on APIs or system calls, then log information such as newly created processes, updated files, new registry entries, and incoming SMB packets are generated by reporting tools.
Decompilers are similar to disassemblers. They are tools that attempt to restore the high-level source code of program unlike disassemblers that attempt to restore the low-level (assembly language) source code of a program.
These tools work hand in hand with each other. The logs generated from monitoring tools can be used to trace the actual code from the disassembled program. The same applies when debugging, where the analyst can see the overview of the low-level code from the disassembly, while being able to predict where to place breakpoints based on the monitoring tools' logs.