This blog is the sixth installment of our Shellcode Signature Series, where we disassemble shellcode and obfuscation techniques and write signatures to detect them. In this post, we will be looking at the x86_jmp_call_additive payload encoder that in found within Metasploit. The encoder itself is one of the older encoders in Metasploit, but it uses a novel approach to rotate an XOR key that can be helpful to understand from a defender's perspective.
Understanding the x86 Jmp Call Additive Encoder
The x86 Jmp Call Additive encoder contains a statically generated decoding stub, with a substitution for the key and payload. This is a great encoder for a beginning reverse engineer to study since it is not as complex as an encoder like Zutto Dekiru.
The primary logic of the module is contained within the init function as shown in the ruby source:
"\xfc" + # cld
"\xbbXORK" + # mov ebx, key
"\xeb\x0c" + # jmp short 0x14
"\x5e" + # pop esi
"\x56" + # push esi
"\x31\x1e" + # xor [esi], ebx
"\xad" + # lodsd
"\x01\xc3" + # add ebx, eax
"\x85\xc0" + # test eax, eax
"\x75\xf7" + # jnz 0xa
"\xc3" + # ret
"\xe8\xef\xff\xff\xff", # call 0x8
Let’s walk through each step of the encoder to understand its instructions. Viewing the init function in a disassembler breaks the chunk of assembly into two functions, making it easier to read. The first function at the entry point is: