The x64 XOR Encoder, Explained

In our last installment of the Shellcode Signature Series for security practitioners, we explored the x86 Countdown Encoder. This week, we will cover the x64 XOR, an encoder included in the Metasploit penetration testing framework. Encoders frequently rely on XOR instructions because they’re mathematically easy to reverse and don’t cause data loss. Because of these benefits, XOR operations, in combination with other mathematical instructions, will figure prominently in many of the encoders that we’ll discuss later in this series.

Understanding the x64 XOR Encoder

The loop of the encoder is hardcoded in as a byte string that includes the block count and the XOR key, as shown below:

      decoder = "\x48\x31\xC9" +           # xor rcx, rcx
      "\x48\x81\xE9" + block_count +       # sub ecx, block_count
      "\x48\x8D\x05\xEF\xFF\xFF\xFF" +     # lea rax, [rel 0x0]
      "\x48\xBBXXXXXXXX" +                 # mov rbx, 0x???????????
      "\x48\x31\x58\x27" +                 # xor [rax+0x27], rbx
      "\x48\x2D\xF8\xFF\xFF\xFF" +         # sub rax, -8
      "\xE2\xF4"                           # loop 0x1B

The block of code can be a bit dense to look at, so let’s break it down a bit:

The first instruction simply zeroes out the RCX register and loads it with the block_count. In our sample code, the block_count is 0xffffffffffffffc0, so the code reflects the following:

     rcx, rcx
     rcx, 0xffffffffffffffc0

RAX is then loaded with the relative offset: 

     rax, [0x00000000]

The XOR 8-byte key is loaded into RBX. This key is random and generated upon encoding:

     movabs rbx, 0x5ba8f682f98cbd3d

Then, the XOR operation is performed:

     qword [rax + 0x27], rbx

RAX is incremented by eight:

     rax, 0xfffffffffffffff8

Finally, LOOP continues for 40 iterations and stops when RCX equals zero:

     loop 0x1b

Yara Signature

Since most of the x64 XOR encoder’s code is static—aside from the block count and XOR key—a Yara signature for the encoder will largely resemble the encoder’s source code. For instance:

rule x64_xor_encoder
{
  meta:
    author = “Nick Hoffman / Jeremy Humble”
    description = “x64 xor encoder”
  strings:
    $op_decode = {48 31 c9
                  48 81 E9 [4]
                  48 8D 05 EF FF FF FF
                  48 BB [8]
                  48 31 58 27
                  48 2D F8 FF FF FF
                  E2 F4
                  }
    $ascii_decode = /4831c94881e9.{8}488d05efffffff48bb.{16}48315827482df8ffffffe2f4/ nocase
  condition:
    any of them
}

Conclusions and Suggestions

In the x64 XOR encoder, there are several different suggestions for making signature-based detection of the encoder more difficult:

• Swapping registers—Using a non-hardcoded register to store the XOR key may make it more challenging to write signatures for the encoder, since the MOV RBX instruction would no longer be a permanent component.
• Vary output of the encoding loop—Peppering in junk instructions of variable length between each iteration of the decode loop will make the encoder more difficult to detect. It will also force signature authors to apply variable length wildcards which hurt signature performance.
• Generate variable length instructions—Use a tool like the interactive redundant assembler to generate instructions of variable lengths and add variation to the encoder’s existing instructions. Varying instruction length and composition greatly complicates detection. This technique will be explored further in a later blog post.
• Modify the key at each pass—Our signature does not rely on the bytes of the key––it remains static through each iteration. However, modifying the key at each pass through a countdown or predictable modification could make it more difficult to determine the nature of the payload.