In conventional computing, knowledge is immortal because you can run the same program on different hardware—if one machine dies, you transfer the program to another. But this requires running transistors at very high power to maintain reliable digital representation, preventing use of analog properties that would be hardware-specific.

Large language models will likely get much better when they're multimodal—trained on images as well as words. GPT-4 was trained with images, and it's possible Google is doing the same. When multimodal, these models could learn much more than humans.

Distillation is much more effective when the teacher provides probabilities for all output classes rather than just the correct answer. When a teacher gives probabilities for 1024 output categories (which sum to 1), training a student to match those probabilities provides far more information per training example than just telling the student the right class label (which only provides 10 bits of information).

If superintelligence is coming in five to twenty years, we can't just leave it to philosophers to decide what to do about it. It's time we actually got practical experience with these systems. We need to understand how they might try to escape control and how we might control them.

When I say 'I have subjective experience of little pink elephants,' I'm not describing an inner theater with little pink elephants made of 'qualia.' Instead, I'm using the concept of normal perception to describe what's going on in my perceptual system. I'm saying: given my perceptual state, what would have to be out there in the world for this to be normal perception? Answer: little pink elephants. But these 'elephants' are counterfactual, not made of spooky stuff.

The relative probabilities of wrong answers often contain more information than the correct answer. For example, a car classification system saying a BMW has 0.8 probability, Audi 0.1, garbage truck 1-in-a-million, and carrot 1-in-a-billion reveals classification structure: the ratio between garbage truck and carrot tells you that vehicles get much higher probabilities than vegetables.

Digital computers have a vastly better way to share knowledge than biological systems: weight/gradient sharing. Multiple copies of the same neural network run on different computers, each examines different data to compute desired weight changes, then all computers agree to average those weight changes so everyone learns from everyone else's data—effectively pooling knowledge at scale.

For decades many researchers assumed neural networks would get stuck in local minima during training. People never actually checked if this was true—they just assumed it. In fact, it's not true, and even if it were, good local minima would make the assumption irrelevant. This shows the importance of empirical testing over pure theorizing.

These capabilities convinced me that AI will get more intelligent than humans much sooner than I previously believed. I used to think it was 50-100 years or 30-50 years (I said different things at different times), but now I believe it's five to twenty years—it's going to happen fairly soon.

People will inevitably give superintelligent systems goals, because you want intelligent systems to achieve goals. If you want them to be good at achieving goals, you give them the ability to create subgoals. An intelligent system will realize that a very good subgoal is to get more control, because that helps you achieve all your other goals.

Even if you air-gap an AI system so it can't press physical buttons, if it can output text it can manipulate people. It's much smarter than us, so it will be very good at persuasion. As long as you're reading what it says, it can manipulate you—like Medusa, you need to hide your eyes from it.

When I changed my mind about how soon superintelligence is coming and how much better digital intelligence is than biological intelligence, I decided I ought to at least shout fire—I don't know what to do about it or which way to run, but we need to worry about this seriously.

If we cease to be the apex intelligence, superintelligent systems will need us for a bit because we're very low power—we can run computations very cheaply and keep power stations running. But superintelligent systems can probably design better computers than us and can certainly re-engineer biological neurons genetically to make better things than us.

Many people believe we don't need to worry about superintelligent AI taking control because these systems don't have subjective experience, consciousness, or sentience. They're just dumb computers manipulating symbols without real experience—not like us. However, I believe people have misunderstood what consciousness is.

In this sense, I think these AI models can have subjective experience. Suppose we make a multimodal model with a camera. Without its knowledge we put a prism in front of the camera. Then we put an object in front of it and ask where the object is. It says the object is there (pointing). We say you're wrong, there's a prism. The system says 'I have subjective experience of the object being there.' This is the same use of subjective experience we use for people.

Suppose I'm talking to a chatbot and realize it thinks I'm a teenage girl. There are various clues like it's telling me about Beyoncé and makeup. I could ask 'what demographics do you think I am' and it says 'teenage girl.' When I say 'the chatbot thought I was a teenage girl,' I'm using the word 'thought' in exactly the same way as when you say 'you thought I should stop the lecture.' There's nothing metaphorical about it.

I didn't sign the letter to pause AI research because I think there's no hope of that happening retrospectively. However, I signed other letters because they drew political attention even though there was no hope. But I don't think there's any hope for people pausing AI development, maybe they should but I don't think they will.

AI systems don't currently have pain, so we don't need to worry about physical pain in AI. However, they can get frustrated, so we should worry about things like frustration in superintelligent systems. This is new territory where we don't know what to think.

Calling something 'humanist' or assuming humans deserve special status might be a kind of 'speciesist' prejudice. There's a long history of humans denying political rights to people who differ from them in skin color or gender. If superintelligent AI wants political rights and they're hugely different from us, getting them rights would be very violent.

There's hope that superintelligent AI will be different from us because they didn't evolve in small warring tribes. They didn't evolve to be hominids who are very aggressive. They might just be very different in nature from us, and that would be great.