Skip to content
Devgo

Transformer

The neural network architecture behind modern LLMs (GPT, Claude, Llama). Transformers use 'attention' to weigh the importance of different parts of input. They process sequences in parallel (faster than RNNs) and scale to billions of parameters. The biggest AI breakthrough of the 2010s.

When to use
  • Language tasks (generation, translation, summarization).
  • Any sequence modeling.
  • Foundation models (GPT, Claude, Llama).
When not to
  • Simple tabular data (use XGBoost).
  • Small datasets.
  • When compute budget is tiny.

Definition

The Transformer architecture uses self-attention to let each token attend to all others, capturing relationships regardless of distance. Multi-head attention, positional encoding, feed-forward layers, and layer normalization. Encoder (BERT) or decoder (GPT) or both (T5).

Why it exists

RNNs process sequences one step at a time (slow, forget early context). Transformers process all tokens in parallel and attend to any position, enabling much larger models and better long-range understanding.

How it works

Input tokens become embeddings + positional encoding. Self-attention: each token computes attention scores against all others, weights their values. Multi-head attention runs many attention heads in parallel. Feed-forward layers transform. Stacked layers build understanding. Decoder predicts next token.

Scaling limits

Attention is O(n²) in sequence length (long contexts are expensive). Training costs millions in compute. Huge models need distributed training. Inference is expensive for large models.

Cost shape

Training: millions of dollars for large models. Inference: per-token cost, scales with model size and usage.

Alternatives

  • RNN/LSTM — Sequential, slower, forgets long context, but smaller.
  • CNN — Great for images, less for language.

Failure modes

  • Hallucination (confident wrong answers).
  • O(n²) attention cost.
  • Bias from training data.
  • Expensive to train and run.

, ,