Most encryption algorithms that are used by malware consist of basic mathematical and logical instructions—that is, xor, add, sub, rol, and ror. These instructions are reversible, and you don't lose data while encrypting with them compared to shl, shr, where it is possible to lose some bits from the left and right. This also happens with and, or, which can lead to the loss of data when using or with 1 or and with 0.
Some basic encryption algorithms are as follows:
- Simple static encryption: Here, malware just uses usual operations such as xor, add, or rol:
Figure 23: Example of the rol operation
- Running key encryption: Here, malware can make key changes like this:
loop_start:
mov edx, <secret_key>
xor dword ptr [<data_to_encrypt> + eax], edx
add edx, 0x05 ;add 5 to the key,
inc eax
loop loop_start
- Substitutional key encryption: Malware can substitute bytes with each other or substitute each value with another...