When a language model refuses a request, flatters a user, or confidently invents a citation, we can describe what it did but not why. We can read every one of its billions of weights and still learn almost nothing, the way listing every neuron in a brain does not explain a thought. Mechanistic interpretability is the effort to change that: to reverse engineer the computations inside neural networks into parts and rules that people can understand, test and, ideally, trust. This article explains the main ideas, the tools researchers use, and an honest picture of where the field stands.

Why Interpretability Matters

Behavior tests tell you what a model does on the inputs you tried. They cannot tell you what it will do on the ones you did not, or whether a good answer was produced for a good reason. That gap matters for several reasons:

  • Safety. If a model learned to behave well only when it detects it is being tested, behavior tests would miss it. Looking at internals offers a possible way to catch such cases. This ties directly into the concerns in AI alignment.
  • Debugging. When a model fails systematically, knowing which internal component is responsible lets you fix it more precisely than retraining blindly.
  • Trust and regulation. High-stakes domains often require some explanation of how a decision was reached.
  • Science. Understanding what networks actually learn is a research question in its own right.

Interpretability Is Not One Thing

The field has several approaches, and it helps to separate them.

Behavioral and attribution methods treat the model as a function and ask which input parts influenced the output. Saliency maps and feature attribution fall here. They are useful but describe correlation between input and output, not the internal mechanism.

Probing trains a small classifier on a layer’s activations to see whether some property (part of speech, truth value, a board position in a game) can be read out. Probing shows information is present, but not that the model actually uses it.

Mechanistic interpretability aims higher. It tries to identify the internal variables a model computes and the causal pathways connecting them, so that you can say “this component does X, and here is the experiment that proves it.”

The Basic Vocabulary

Activations and the residual stream. In a transformer, each layer reads from and writes to a running vector for every token, called the residual stream. You can think of it as a shared communication channel: attention heads and feed-forward layers read what they need and add their contribution.

Features. The central hypothesis is that networks represent concepts as directions in activation space. A “feature” might be a direction that is active when the text is about the Golden Gate Bridge, or when code contains a bug, or when the model is about to produce a quotation mark.

Circuits. A circuit is a subgraph of the network, a handful of heads and neurons connected by learned weights, that implements an identifiable algorithm, such as copying a name seen earlier in the context.

A Famous Circuit: Induction Heads

One of the best understood circuits explains a core behavior of language models: continuing repeated patterns. If the context contains ”… Harry Potter … Harry”, the model predicts “Potter”. Researchers found this is implemented by two cooperating attention heads in different layers. A first head copies information about the previous token into the position of the current one; a second head then looks for an earlier position whose “previous token” matches the current token, and copies what came after it.

This matters beyond the specific trick. It showed that parts of a transformer implement human-readable algorithms, that they can be located by experiment, and that their appearance during training coincides with a sharp improvement in in-context learning. It is the model case people point to when arguing the whole program is feasible.

The Superposition Problem

If each neuron stood for one concept, interpretation would be easy. Reality is messier. Individual neurons are frequently polysemantic: one neuron responds to a mixture of unrelated things.

The leading explanation is superposition. A layer has a limited number of dimensions, but the world contains far more concepts than that. Because most concepts are sparse (rarely active at the same time), the network can pack many more features than dimensions by assigning them almost-orthogonal directions and tolerating a little interference. The result is that features are not aligned with neurons, so inspecting neurons one by one is misleading.

This is the core obstacle: the natural unit of analysis, the neuron, is not the unit the model uses.

Sparse Autoencoders: Unpacking the Features

The most widely used response is the sparse autoencoder (SAE). You collect activations from a chosen layer across a large amount of text, then train a separate, wider network to reconstruct those activations through a bottleneck that is forced to be sparse, meaning only a few of its units fire for any input.

import torch
import torch.nn as nn

class SparseAutoencoder(nn.Module):
    def __init__(self, d_model, d_features):
        super().__init__()
        self.encoder = nn.Linear(d_model, d_features)
        self.decoder = nn.Linear(d_features, d_model, bias=False)

    def forward(self, x):
        features = torch.relu(self.encoder(x))   # sparse, non-negative
        recon = self.decoder(features)
        return recon, features

def loss_fn(x, recon, features, l1_coeff):
    mse = ((x - recon) ** 2).mean()
    sparsity = features.abs().mean()
    return mse + l1_coeff * sparsity

The dictionary has many more features than the model has dimensions. If training works, each learned feature tends to correspond to something more interpretable than a raw neuron: a specific topic, a syntactic role, a kind of code pattern. Researchers inspect each feature by looking at the text examples that activate it most strongly and at what it pushes the model to output.

Variants exist, including different sparsity penalties and architectures meant to reduce known artifacts, but the principle is the same: find a basis in which the model’s internal representation is sparse and readable.

Testing Causality

Finding a pretty feature is not enough; the claim must survive intervention. The standard toolkit:

  • Activation patching. Run the model on two inputs that differ in one key way, then swap an internal activation from one run into the other and see whether the behavior flips. If patching a specific head restores the correct answer, that head is causally involved.
  • Ablation. Remove or zero a component and measure the effect on behavior.
  • Steering. Add or amplify a feature direction during the forward pass and see whether the model’s output shifts accordingly. Amplifying a feature associated with a concept and watching the model start bringing it up is persuasive evidence the feature means what you think.

A small patching sketch using PyTorch hooks, written generically:

import torch

def run_with_patch(model, layer_module, clean_inputs, corrupt_inputs):
    """Cache a clean activation, then patch it into the corrupted run."""
    cache = {}

    def save_hook(module, inp, out):
        cache["act"] = out.detach()

    def patch_hook(module, inp, out):
        return cache["act"]

    handle = layer_module.register_forward_hook(save_hook)
    with torch.no_grad():
        model(**clean_inputs)
    handle.remove()

    handle = layer_module.register_forward_hook(patch_hook)
    with torch.no_grad():
        patched_out = model(**corrupt_inputs)
    handle.remove()
    return patched_out

In practice you patch at a specific token position or head rather than a whole layer, and you compare the change in output probabilities against a baseline, but the pattern is the same: save, swap, measure.

What Has Been Learned

Without leaning on any single result, several themes recur in published work:

  • Models contain interpretable features at many levels of abstraction, from token-level patterns to abstract notions.
  • Some features are multilingual or multimodal, suggesting shared internal concepts rather than separate per-language machinery.
  • Models sometimes appear to plan, for example by settling on a rhyming word before writing the line that leads to it, which is not obvious from the one-token-at-a-time training objective.
  • Models’ stated reasoning does not always match the computation that produced the answer, a reason to be careful with explanations a model gives of itself.

These are suggestive findings from a young field, and individual claims deserve scrutiny and replication.

Limits and Open Problems

An honest assessment includes what does not work yet.

  • Scale. Fully explaining even a modest circuit takes substantial human effort. Frontier models are far larger, and automation is only partially developed.
  • Faithfulness of SAEs. A sparse autoencoder reconstructs activations imperfectly, and the part it misses may matter. Features can also split or merge depending on dictionary size, so there is no single canonical set.
  • Completeness. Explaining a behavior is not the same as ruling out others. You cannot easily prove the absence of a dangerous capability.
  • Validation. There is little ground truth. How do you know an explanation is right rather than merely plausible? Causal interventions help but do not settle it.
  • Cherry-picking risk. Compelling examples are easy to show. Systematic, quantitative evaluation is harder and still maturing.

How It Connects to Safety Work

Interpretability is one tool among several, not a replacement for evaluation, red teaming and careful training. The hope is that it can eventually provide checks that behavior alone cannot: detecting deceptive or hidden objectives, verifying that a safety fix removed a capability rather than hiding it, and monitoring models in deployment for features that signal risk. Today it is more reliably useful for research and debugging than as a guarantee, and responsible teams treat any interpretability-based assurance as one piece of evidence.

Getting Hands-On

You do not need a large lab to start. Small open models are enough to reproduce classic results such as induction heads or to train a small SAE on one layer. A sensible path:

  1. Pick a small open model and a narrow behavior, such as predicting repeated tokens.
  2. Use hooks to cache activations and try activation patching to localize the behavior.
  3. Visualize attention patterns for the heads you find.
  4. Train a sparse autoencoder on one layer and read the top-activating examples of a few features.
  5. Try steering with one feature and check whether the effect matches your reading of it.

Keep notes on failed explanations as carefully as successful ones; the field’s biggest risk is believing a story too early.

Common Mistakes

  • Reading meaning into a single neuron without checking for polysemanticity.
  • Treating probe accuracy as proof the model uses the information.
  • Skipping causal tests and relying on top-activating examples alone.
  • Over-trusting a model’s own natural-language explanation of its reasoning.
  • Assuming an explanation of one prompt generalizes to all prompts.
  • Presenting interpretability results as safety guarantees.

Key Takeaways

  • Mechanistic interpretability tries to reverse engineer networks into features and circuits that are causally tested, not just correlated with outputs.
  • Superposition means neurons are polysemantic, so analysis must work with directions in activation space rather than single neurons.
  • Sparse autoencoders are a leading method for decomposing activations into more readable features, but they are imperfect and no feature set is canonical.
  • Activation patching, ablation and steering are the key tools for turning a hypothesis into a causal claim.
  • Induction heads show real algorithms can be found inside transformers, but scaling this to frontier models remains hard.
  • Treat interpretability as one piece of evidence within broader safety practice, not as a guarantee.