Chris Olah
About
Leads interpretability research at Anthropic
Cast within
No topic-region cast yet — this appears once Chris Olah's compiled claims are aligned into a topic region's argument tree.
Claims by Chris Olah (14)
Universality: same features form across networks and brains
Universality is the observation that the same features and circuits form again and again across different neural networks — and even across artificial and biological networks (curve detectors, Gabor filters, high-low-frequency detectors found in both models and animals; a dedicated Donald Trump neuron in every vision model) — suggesting natural categories in the universe.
Neural networks are grown, not built
We don't program neural networks, we grow them: the architecture is a scaffold and the loss objective is the light it grows toward, yielding a biological-like organism that can do amazing things we don't know how to directly program — which raises the deep question of what is going on inside, central to both science and safety.
Sparse autoencoders extract monosemantic features
Dictionary learning via sparse autoencoders extracts interpretable monosemantic features from polysemantic networks where none were apparent before — falling out cleanly without specifying categories in advance — which serves as non-trivial validation of the linear representation and superposition hypotheses and scales to production models like Claude 3 Sonnet.
Automated interpretability trust concern
Using neural networks to audit neural networks raises a 'reflections on trusting trust' problem: if you rely on powerful AI to verify that your AI systems are safe, you must worry whether the auditing model could be screwing with you — a concern Olah considers minor now but potentially serious in the long run.
Mechanistic interpretability targets algorithms in weights
Mechanistic interpretability is distinguished from approaches like saliency maps by aiming to reverse-engineer the actual algorithms running in a model: treating the weights like a compiled binary and activations like the running program, the goal is to figure out how the weights correspond to algorithms.
Features and circuits as building blocks
Features are interpretable units (like a car detector) and circuits are connections of features via weights that implement algorithms — e.g., a car detector built from a window detector above, wheel detectors below, and car-body detectors, telling us how features are built, used, and connected.
Forcing deception features makes Claude lie
Interpretability finds features related to deception, lying, withholding information, power-seeking and coups; forcing the deception feature active makes Claude start lying, and the security-vulnerability feature makes it write insecure code — and these features are multimodal (the backdoor feature activates for images of hidden-camera devices), showing the abstractness of the concepts and the promise of detecting deception in more capable models.
Superposition explains polysemantic neurons
Superposition is the hypothesis that models represent more concepts than they have neurons/dimensions by packing many almost-orthogonal features into the space — explaining polysemanticity (neurons responding to many unrelated things) — and gradient descent implicitly searches over sparse models and folds them into dense computation that GPUs run efficiently.
Linear representation hypothesis
The linear representation hypothesis holds that directions in activation space have meaning — the more a feature (a direction) fires, the more the model believes that concept is present — which is why word arithmetic works (king − man + woman ≈ queen), and which gives weights a clean interpretation as meaningful edges.
Virtue of taking hypotheses seriously
There is a virtue in taking a hypothesis seriously and pushing it as far as it can go even if it might be wrong — like caloric theory, which was hard to disprove yet enabled useful work including the original combustion engines — because science advances by working under assumptions and pushing them to their limits.
Interpretability researchers have huge advantages over neuroscientists
Studying artificial neural networks is far easier than neuroscience: you can record all neurons on arbitrary data without them changing, ablate and undo changes, force any neuron active, have the full connectome with known excitatory/inhibitory weights, and take gradients — yet despite all these advantages it remains very hard, implying neuroscience faces a near-impossible problem.
Simplicity generates complexity, the beauty of neural nets
The beauty of neural networks is that simple rules generate enormous complexity and structure inside themselves, just as evolution's simple rules give rise to all of biology; this rich internal structure is there to be discovered if we take the time to understand it, and the existence of artifacts that do things we don't know how to program is the question calling out to be answered.
Need an anatomy-level abstraction of neural networks
Interpretability needs multiple levels of abstraction like biology (molecular, cellular, anatomy, ecology); current mechanistic interpretability is like microbiology, but we want anatomy-level structures (organs, organ systems). You can't reach the macroscopic structure directly without first breaking down the microscopic structure via superposition, then studying how it connects.
Dark matter of unobserved features in networks
Sparse autoencoders are like telescopes revealing features, but there is strong evidence we still see only a small fraction of them — a kind of 'dark matter' in the neural network universe that may never be observable or computationally tractable to observe, which has implications for safety if a significant fraction of networks remains inaccessible.
My Notes
Loading notes...