Malware analysis: part 11. How to create your own mini-GPT for binary analysis.

﷽

Hello, cybersecurity enthusiasts and white hackers!

Everybody talks about GPT and large language models today. But a transformer is not magic - it is a small, understandable function trained with plain gradient descent. In this post we build our own mini-GPT completely from scratch in PyTorch, train it on raw bytes of real binaries, and turn it into a practical malware-analysis tool: a packing / encryption detector based on perplexity (how “surprised” the model is by the bytes it sees).

No black-box tools, no pretrained weights, no cloud API. Around 600K parameters, trained on a GPU in under half a minute. Everything is practice-oriented and reproducible.

the idea

A binary file is just a sequence of bytes. A GPT is a next-token predictor: given the previous bytes, it outputs a probability distribution over the next byte. If we train it only on normal machine code, it learns the “grammar” of real code - which byte usually follows e8 (a call), which patterns opcodes form, how padding looks.

Then we feed it new bytes and measure its surprise (the negative log-likelihood, a.k.a. perplexity):

normal code -> low surprise, the model predicts the next byte well;
packed / encrypted / compressed bytes -> high surprise, the model has no idea, because such bytes are close to random.

This is conceptually related to Shannon entropy from part 6, but it is smarter: entropy only counts byte frequencies, while the GPT models the order of bytes - the actual structure of code. That makes it a stronger signal for triage.

environment and data

You do not need a fancy setup - a Linux box with Python 3 is enough. A GPU is optional: this model is tiny and also trains on a plain CPU in a couple of minutes. If you have an NVIDIA GPU, the same script uses it automatically and trains in seconds.

We only need PyTorch, numpy and matplotlib. Start with a clean virtual environment:

python3 -m venv env && source env/bin/activate
pip install numpy matplotlib

Now install PyTorch. This is the one step that trips people up, so read it slowly. PyTorch ships in two flavors: a CPU-only build and a CUDA (GPU) build. If you just run pip install torch, on many systems you get the CPU-only one - it works, but it will never touch your GPU, and later you will wonder why training crawls.

To use an NVIDIA GPU (like the L20 I used here), do it in two steps:

# 1) check the GPU and its driver are visible to the OS:
nvidia-smi     # must print your card (e.g. "NVIDIA L20") and a driver version

# 2) install the CUDA build of PyTorch (match your CUDA version, e.g. 12.4 -> cu124):
pip install torch --index-url https://download.pytorch.org/whl/cu124

Copy the exact command for your CUDA version from the official page: pytorch.org/get-started. No NVIDIA GPU? Then just pip install torch and skip this - everything below still runs fine on CPU, only slower.

Now the important nuance - always verify that PyTorch really sees the GPU before you start training:

python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU')"
# GPU ready  ->  True NVIDIA L20
# CPU only   ->  False CPU

If it prints True and your GPU name - you are good. If it prints False but you do have an NVIDIA GPU, you almost certainly installed the CPU-only build: reinstall with the --index-url line above. Our script then picks the device automatically with a single line - dev = "cuda" if torch.cuda.is_available() else "cpu" - so the exact same code runs on your GPU or your laptop CPU with no changes.

practical example.

For training data we do not need any malware. We take real, legitimate ELF binaries from /usr/bin as our corpus of “normal code”. To get a packed sample for the detector, we simulate a packer the honest way - a small original header followed by a compressed body (real packers leave the same footprint: a low-entropy stub and a high-entropy compressed/encrypted body):

import os, glob, lzma, numpy as np

def collect(limit=160):
    out = []
    for p in sorted(glob.glob("/usr/bin/*")):
        try:
            if os.path.islink(p) or not (8000 < os.path.getsize(p) < 300000):
                continue
            b = open(p, "rb").read()
            if b[:4] == b"\x7fELF":
                out.append(b)
        except OSError:
            pass
        if len(out) >= limit:
            break
    return out

def pack_like(b, rng):                       # defensive packer simulation
    h = int(rng.integers(512, 4096))         # keep a small original header/stub
    body = b[h:]
    k = int(len(body) * rng.uniform(0.5, 1.0))
    return b[:h] + lzma.compress(body[:k], preset=int(rng.integers(1, 7))) + body[k:]

the model: a mini-GPT from scratch

Here is the whole transformer. It is a decoder-only GPT with causal self-attention - the same architecture as the big models, just tiny. Vocabulary is 256 (one token per byte).

# bin_gpt.py
# byte-level mini-GPT for binary analysis
# author: @cocomelonc
# https://cocomelonc.github.io/malware/2026/07/25/malware-analysis-11.html
import torch, torch.nn as nn
from torch.nn import functional as F

T, n_emb, n_head, n_layer, V = 64, 128, 4, 3, 256   # ctx, embed, heads, layers, vocab(bytes)

class Head(nn.Module):                        # one self-attention head
    def __init__(self, hs):
        super().__init__()
        self.k = nn.Linear(n_emb, hs, bias=False)
        self.q = nn.Linear(n_emb, hs, bias=False)
        self.v = nn.Linear(n_emb, hs, bias=False)
        self.register_buffer("mask", torch.tril(torch.ones(T, T)))
    def forward(self, x):
        b, t, c = x.shape
        att = (self.q(x) @ self.k(x).transpose(-2, -1)) * c ** -0.5   # scaled dot-product
        att = att.masked_fill(self.mask[:t, :t] == 0, float("-inf"))  # causal: no peeking ahead
        att = F.softmax(att, -1)
        return att @ self.v(x)

class Block(nn.Module):                        # attention + feed-forward with residuals
    def __init__(self):
        super().__init__()
        hs = n_emb // n_head
        self.h = nn.ModuleList([Head(hs) for _ in range(n_head)])
        self.proj = nn.Linear(n_emb, n_emb)
        self.ff = nn.Sequential(nn.Linear(n_emb, 4 * n_emb), nn.ReLU(), nn.Linear(4 * n_emb, n_emb))
        self.l1, self.l2 = nn.LayerNorm(n_emb), nn.LayerNorm(n_emb)
    def forward(self, x):
        x = x + self.proj(torch.cat([h(self.l1(x)) for h in self.h], -1))
        return x + self.ff(self.l2(x))

class GPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.tok = nn.Embedding(V, n_emb)      # byte -> vector
        self.pos = nn.Embedding(T, n_emb)      # position -> vector
        self.blocks = nn.Sequential(*[Block() for _ in range(n_layer)])
        self.lnf = nn.LayerNorm(n_emb)
        self.head = nn.Linear(n_emb, V)        # -> distribution over next byte
    def forward(self, idx, tgt=None):
        b, t = idx.shape
        x = self.tok(idx) + self.pos(torch.arange(t, device=idx.device))
        logits = self.head(self.lnf(self.blocks(x)))
        loss = None if tgt is None else F.cross_entropy(logits.view(-1, V), tgt.view(-1))
        return logits, loss
    @torch.no_grad()
    def generate(self, idx, k):
        for _ in range(k):
            l, _ = self(idx[:, -T:])
            p = F.softmax(l[:, -1, :], -1)
            idx = torch.cat([idx, torch.multinomial(p, 1)], 1)
        return idx

That is it - self-attention, residual connections, layer norm, a feed-forward block, and a linear head. The famous transformer, about 60 lines.

training

The training loop is the same three steps you use for any neural net: forward, loss.backward(), opt.step(). We build one long byte stream from normal binaries and sample random windows.

import numpy as np, time, math

dev = "cuda" if torch.cuda.is_available() else "cpu"
files = collect(160); rng = np.random.default_rng(0)
tr_files, te_files = files[:120], files[120:]
stream = np.frombuffer(b"".join(tr_files[:80]), np.uint8).astype(np.int64)[:3_000_000]
data = torch.tensor(stream, dtype=torch.long)
n = int(0.9 * len(data)); trd, vad = data[:n], data[n:]

def batch(split):
    d = trd if split == "tr" else vad
    ix = torch.randint(len(d) - T, (64,))
    x = torch.stack([d[i:i+T]   for i in ix])
    y = torch.stack([d[i+1:i+T+1] for i in ix])
    return x.to(dev), y.to(dev)

torch.manual_seed(0); model = GPT().to(dev)
opt = torch.optim.AdamW(model.parameters(), 3e-3)
for s in range(2001):
    xb, yb = batch("tr")
    _, loss = model(xb, yb)
    opt.zero_grad(); loss.backward(); opt.step()

Let’s run it. I trained on an NVIDIA L20, but the exact same script runs on any machine (it just falls back to CPU):

The validation perplexity drops from 287 down to ~14 bytes in about 25 seconds. Perplexity 14 means that, on average, the model narrows the next byte down to roughly 14 likely candidates out of 256 - it has genuinely learned the structure of machine code.

did it actually learn code?

Let’s ask the model to generate bytes from scratch and look at them as hex:

seed = torch.tensor([[0x7f]], device=dev)
print(model.generate(seed, 64)[0].cpu().numpy().astype("uint8").tobytes().hex(" "))

7f 92 00 f3 e8 a0 9c ff ff 58 e8 c2 49 ff ff 80 7c 24 02 74 2f 85 c9 0f 88 39 ff ff ff eb c0 b9

This is not random noise - it looks like real x86. We can literally read opcodes in it: e8 .. ff ff is a call rel32, 74 2f is je, 85 c9 is test ecx, ecx, 0f 88 is js, eb is a short jmp, b9 is mov ecx, imm32. Our tiny model learned the texture of compiled code purely from bytes, with no disassembler and no labels.

the payoff: packing detection by perplexity

Now the malware-analysis part. We measure how surprised the trained model is per file window. Normal code = low surprise; packed/encrypted body = high surprise.

@torch.no_grad()
def nll(byte_ints):                            # per-position negative log-likelihood
    model.eval()
    x = torch.tensor(byte_ints, dtype=torch.long, device=dev); out = []
    for i in range(0, len(x) - 1, T):
        ch = x[i:i+T+1]
        if len(ch) < 2: break
        lg, _ = model(ch[:-1].unsqueeze(0))
        lp = F.log_softmax(lg[0], -1); tg = ch[1:]
        out.append((-lp[torch.arange(len(tg)), tg]).cpu().numpy())
    return np.concatenate(out)

def winavg(a, w=512):                           # average surprise per 512-byte window
    return np.array([a[i:i+w].mean() for i in range(0, len(a) - w + 1, w)])

Plotting the per-window surprise for one normal binary and its packed version makes the effect obvious:

The blue line (normal binary) stays low in the code sections. The red line (packed version) has a huge plateau of high surprise exactly where the compressed body sits - the model screams “I have never seen bytes like this”. Average surprise was 1.88 on normal regions vs 3.31 on packed regions.

Turned into a simple detector - mean surprise over the first 60 KB of each file - and evaluated on 80 held-out files (each in a normal and a packed variant):

def score(b):
    return nll(np.frombuffer(b[:60000], np.uint8).astype(np.int64)).mean()

ROC-AUC = 0.994. A 600K-parameter transformer we wrote by hand, trained for 25 seconds, separates packed from unpacked binaries almost perfectly - and, unlike a plain entropy threshold, it reacts to unusual code order, not just byte frequency.

I also added some colors to final script:

summary

component	what it does
GPT (60 lines)	decoder-only transformer, 256-byte vocab, from scratch
training loop	next-byte prediction on normal code, ~25 s on a GPU
generate()	proves it learned x86 texture (readable opcodes)
nll() / perplexity	packing / anomaly detector, ROC-AUC 0.994

This is the whole point: a transformer is just a next-token predictor trained by gradient descent, and once you build one yourself it stops being a buzzword and becomes a tool. The exact same engine, scaled up, powers the LLM copilots appearing in SOCs today.

A few honest limitations. Perplexity flags unusual bytes, so legitimately compressed resources (icons, installers) will also look “surprising” - in production you combine this with PE/ELF section context and known-packer YARA rules. The model is also attackable: an adversary can pad a file with normal-looking code to lower average surprise. As always, one signal is never enough - use it as a fast lead, not a verdict.

In the next part we will go further and feed the model opcode sequences from a real disassembler (capstone) instead of raw bytes, which makes the “code grammar” even sharper for family classification.

I hope this post with practical examples is useful for malware researchers, reverse engineers and everyone interested in blue team and ML skills.

Malware analysis part 6: Shannon entropy
Malware analysis part 10: Practical PE parsing
Anti-DDoS: SYN flood detection
Attention Is All You Need (transformer paper)
Andrej Karpathy - nanoGPT
MalConv - malware detection from raw bytes
source code in github

This is a practical case for educational purposes only.

Thanks for your time happy hacking and good bye! PS. All drawings and screenshots are mine