In our last installment of the Shellcode Signature Series for security practitioners, we explored the x86 Countdown Encoder. This week, we will cover the x64 XOR, an encoder included in the Metasploit penetration testing framework. Encoders frequently rely on XOR instructions because they’re mathematically easy to reverse and don’t cause data loss. Because of these benefits, XOR operations, in combination with other mathematical instructions, will figure prominently in many of the encoders that we’ll discuss later in this series.
Understanding the x64 XOR Encoder
The loop of the encoder is hardcoded in as a byte string that includes the block count and the XOR key, as shown below: