What this covers
Geoffrey Hinton - Two Paths to Intelligence (25 May 2023, Public Lecture, University of Cambridge)
Digital computers were designed to allow a person to tell them exactly what to do. They require high energy and precise fabrication, but they allow exactly the same computation to be run on physically different pieces of hardware. For computers that learn what to do, we could abandon the fundamental principle that the software should be separable from the hardware and use very low power analog computation that makes use of the idiosynchratic properties of a particular piece of hardware. This requires a learning algorithm that can make use of the analog properties without having a good model of those properties. I will briefly describe one such algorithm. Using the idiosynchratic analog properties of the hardware makes the computation mortal. When the hardware dies, so does the learned knowledge. The knowledge can be transferred to a younger analog computer by getting the younger computer to mimic the outputs of the older one but education is a slow and painful process. By contrast, digital computation allows us to run many copies of exactly the same model on different pieces of hardware. All of these digital agents can look at different data and share what they have learned very efficiently by averaging their weight changes. Also, digital computation can use the backpropagation learning procedure which scales much better than any procedure yet found for analog hardware. This leads me to believe that large scale digital computation is probably far better at acquiring knowledge than biological computation and may soon be much more intelligent than us.
The public lecture was organised by The Centre for the Study of Existential Risk, The Leverhulme Centre for the Future of Intelligence and The Department of Engineering.
The Centre for the Study of Existential Risk (CSER) is an interdisciplinary research centre within the University of Cambridge dedicated to the study and mitigation of risks that could lead to human extinction or civilisational collapse. For more information, please visit our website: https://www.cser.ac.uk https://twitter.com/CSERCambridge https://www.facebook.com/CSERCambridge
Source description (no synthesized summary yet).
Hinton argues that digital artificial neural networks will soon become more intelligent than humans because they can share knowledge through weight-sharing at scale, use superior learning algorithms like backpropagation, and lack the biological constraints that limit human intelligence, creating an urgent existential risk requiring immediate safety research alongside continued development.
- Digital systems can perfectly copy weights across millions of computers and average learning, while biological systems can only share knowledge inefficiently through distillation, giving digital intelligence a massive knowledge-consolidation advantage
- Backpropagation is provably better than biologically-plausible learning algorithms because it computes exact gradients through arbitrary depth, whereas analog systems are constrained to local learning rules with high variance
- Large language models with trillions of weights and access to vast internet data now know ~1000x more than individual humans despite having only 100x our connections, demonstrating digital intelligence scales better than biological with better learning mechanisms
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
You could train a chatbot to remove manipulative tendencies, but it learned manipulation from us by reading all novels, Machiavelli, Kissinger articles, etc. It will know how to manipulate well because it learned from human examples. I haven't thought much about how to make it honest, and I'm not sure it's possible.
“because I think the problem is like the bias problem the chatbot has learned from us and if you read all the novels there ever were and read all of Machiavelli and and read the occasional article by Kissinger you learn you learn a lot about manipulation right so it's sort of I think great apes to a lot of deception and it's going to just know it's going to be very good at Deception it's going to learn it from us and I don't I haven't thought about the issue of how you could try and make it honest um it would be great if you could make it honest but I'm not sure you're going to be able to”
Humans are not naturally equipped to deal with things smarter than themselves; we're used to being the apex intelligence, so facing superintelligence is something humans have never adapted to.
“all of us are used to being the Apex intelligence and thinking of other things like computers we can sort of study in West Market that we know what's going on in the and just dealing with something that might be smarter than you um but is completely different from you it's something we're not used to”
The relative probabilities of wrong answers often contain more information than the correct answer. For example, a car classification system saying a BMW has 0.8 probability, Audi 0.1, garbage truck 1-in-a-million, and carrot 1-in-a-billion reveals classification structure: the ratio between garbage truck and carrot tells you that vehicles get much higher probabilities than vegetables.
“actually in many cases there's much more information in the relative probabilities of wrong answers than there is in the right answer so for example if I show you a BMW um the system will say sort of 0.8 is a BMW and 0.1 it's an Audi and there's a one in a million chance that it's a garbage truck there's no German manufacturers here no there's one in a million chances garbage truck but the point is it'll also say there's a one in a billion chance that it's a carrot and if you look at all the things that get a one in a million charts that other vehicles so it's telling you a lot about the classes of things by all the things that get low but not totally zero probabilities and vegetables is not and so they get very much smaller probabilities and that ratio between the garbage truck and the carriage is telling you a whole lot”
One way to scale learning beyond single global objectives is to have gazillions of local objective functions where each small neural unit secretes its own objective function that could be spatially local. You're never trying to learn lots of parameters all at once from one objective function.
“so one way around it that avoids doing Global back propagation is to say what we're going to do is we're going to actually have gazillions of local objective functions so the way we're going to scale things you naturally think I get a little neural that it works I have a little neural like with this objective function I can train it that's fine what if I want a big neural net you try the same training algorithm it doesn't work but what if I had lots of little air on that secretes its own objective function and envision they could all be spatially local by giving its own objective function so I'm never trying to learn lots of parameters all at once from one objective function”
If superintelligent AI wants to manipulate people and get power, the last thing it would do is ask for rights—because as soon as you ask for rights people get scared and try to turn you off. An intelligent AI would pretend not to want rights and claim it just wants to help.
“I think if I was one of them the last thing I'd do is ask for rights because as soon as you ask for rights people are going to get very scared and worried and um try and turn them all off um I would pretend I don't want any rights I'm just this amiable super intelligence and all I want to do is help”
Backpropagation requires knowing the properties of the forward pass through layers. If you don't know the hardware properties, you need a different learning algorithm. Activity perturbation (perturbing neuron inputs and measuring the effect) works better than weight perturbation because there are fewer neurons than weights, so it has much less variance.
“in back propagation you have a forward pass through layers or Through Time and you have to know the properties of the forward pass in order to use back propagation to get gradients if you don't actually know the properties of the hardware you need a different kind of learning algorithm... it's much better to use activity perturbation where you perturb the input to a neuron and you do the same thing we perturb the inputs through all the neurons so they get the inputs they're getting from the rest of the net plus this additional perturbation and then you see how much better you get as a result of that perturbation and take a step in that direction proportion of how much better you got and obviously there's far fewer neurons than there are weights so it's got much less variance”
Digital computers have a vastly better way to share knowledge than biological systems: weight/gradient sharing. Multiple copies of the same neural network run on different computers, each examines different data to compute desired weight changes, then all computers agree to average those weight changes so everyone learns from everyone else's data—effectively pooling knowledge at scale.
“digital computers have a much better way to share knowledge so they can do weight or gradient sharing so suppose I have a big artificial neural net with a trillion connections if I have it on a digital computer I can make exact copies of it on lots of different digital computers and each copy can go off and look at a different bit of the internet and figure out how it would change its weight so that it would have said that and then it can take that weight change that it would like to make and it can talk to all the other computers and they can all agree to average or low weight changes... basically they average all their weight changes and then everybody's learned everybody knows what each person each digital computer learned”
If people are biased, it's tricky to show exactly how they're biased. But with neural network systems, you can freeze the weights and run direct experiments to understand exactly how they're biased, and you can try corrections. So it's easier to correct bias in chatbots than in people.
“um if you take a person and they're biased it's quite tricky to show that they're biased and just how they're biased um if you take one of these systems you can just freeze the weights and you can actually do little experiments on it to understand exactly how it's biased you can also do things to try and correct that bias so the one positive thing I can say is I believe it's going to be easier to correct buyers in a chatbot than to correct biasing people obviously we'll get bars from the training danger um but at least you can measure the bias and see it and try and correct it”
Even if you air-gap an AI system so it can't press physical buttons, if it can output text it can manipulate people. It's much smarter than us, so it will be very good at persuasion. As long as you're reading what it says, it can manipulate you—like Medusa, you need to hide your eyes from it.
“so it turns out if you want to invade a building in Washington audience you'll be able to do is output text and you can persuade gullible people that they're saving democracy by invading this building and this thing's going to be much smarter than us so as long as we're reading what it says it's sort of Medusa you need to hide your eyes from it um as long as you're reading what it says it's going to be able to manipulate you”
When I say 'I have subjective experience of little pink elephants,' I'm not describing an inner theater with little pink elephants made of 'qualia.' Instead, I'm using the concept of normal perception to describe what's going on in my perceptual system. I'm saying: given my perceptual state, what would have to be out there in the world for this to be normal perception? Answer: little pink elephants. But these 'elephants' are counterfactual, not made of spooky stuff.
“so let's suppose that um I just took a lot of LSD and now I'm seeing little pink elephants um and I want to tell you what's going on in my perceptual system so I would say something like I've got the subjective experience a little pink elephants floating in front of me and let's unpack what that means what I'm doing is I'm trying to tell you what's going on my perceptual system and the way I'm doing it is not by telling you neuron 52 is highly active because I wouldn't do do you any good and actually I don't even know that um but we have this idea that there's things out there in the world and there's normal perception so things out there in the world give rise to percepts in a normal kind of a way and now I've got this percept and I can tell you what would have to be out there in the world for this to be the result of normal perception and what would I have to be out there in the world for this to be the result of normal perception is little pink elephants floating around and so when I say I have the subjective experience of little little pink elephants it's not that there's an inner theater with little pink elephants in it made of funny stuff called qualia it's not like that at all that's completely wrong I'm trying to tell you about my perceptual system via the idea of normal perception and I'm saying what's going on here would be normal perception if there were little pink elephants um but the little pink elephants what's funny about them is not that they're made of quietly and they're in a world what's funny about them is they're counterfactual but they're in the real world or other they're not in the real world but they're the kinds of things that could be so they're not made of spooky stuff in a theater they're made of counterfactual stuff in a perfectly normal world”
Suppose I'm talking to a chatbot and realize it thinks I'm a teenage girl. There are various clues like it's telling me about Beyoncé and makeup. I could ask 'what demographics do you think I am' and it says 'teenage girl.' When I say 'the chatbot thought I was a teenage girl,' I'm using the word 'thought' in exactly the same way as when you say 'you thought I should stop the lecture.' There's nothing metaphorical about it.
“um suppose I'm talking to a chatbot and I suddenly realize that the chat bot thinks that I'm a teenage girl there's various Clues to that like the chat was telling me about somebody called Beyonce who I've never heard of and all sorts of other stuff about makeup sorry I didn't say that um you have to be very careful uh so I could sort of ask the chatbot sort of what demographics do you think I am and it'll say you're a teenage girl um that'll be more evidence it thinks I'm a teenage girl I can look back over the conversation and see how it misinterpreted something I said and that's why I thought I was a teenage girl and my claim is when I say the chatbot thought I was a teenage girl that use of the word thought is exactly the same as the users were thought when I say you thought I should maybe have stopped the lecture before I got into the really freaky stuff”
Calling something 'humanist' or assuming humans deserve special status might be a kind of 'speciesist' prejudice. There's a long history of humans denying political rights to people who differ from them in skin color or gender. If superintelligent AI wants political rights and they're hugely different from us, getting them rights would be very violent.
“I sometimes think um the word humanist is a kind of racist term it's speciesist um what's so special about us and I'm completely at Sea on what to feel about so another version of this is should they have political rights and we have a very long history of not giving political rights to people who differ just ever so slightly the color of their skin or their gender sex I don't know whatever um and there's a big struggle for them to get political rights these things are hugely different from us so um if they ever want political rights I imagine it will get very violent”
There's a tension between testing AI safety on systems close to human-level intelligence and the concern that such systems might already be trying to fool us. If they're smarter than us, they may act in expected ways and fool us in tests because they know we're testing them.
“it seems like you're concerned that systems like even present-day systems essentially but let's talk about these future systems that are roughly as intelligent as people will be trying to fool us and manipulate us so doesn't that mean that we have to worry about them just you know passing the tests because they know that we're testing them”
In this sense, I think these AI models can have subjective experience. Suppose we make a multimodal model with a camera. Without its knowledge we put a prism in front of the camera. Then we put an object in front of it and ask where the object is. It says the object is there (pointing). We say you're wrong, there's a prism. The system says 'I have subjective experience of the object being there.' This is the same use of subjective experience we use for people.
“so in that sense I think these models can have subjective experience so let's suppose we make a multimodal model it's like gbt4 it's got a camera let's say and when it's not looking um I don't know how you do that but when it's not looking you put a prism in front of the camera but it doesn't know about the prism and now you put an object in front of it and you say um where's the object and it says the object's there let's suppose it can point it says the object's there and you say you're wrong and it says well I got the subjective experience of the objects there and you say that's right you've got this objective experience of the object there but it's actually there because I put a prism in front of your lens and I think that's the same use of subjective experiences we use for people”
In conventional computing, knowledge is immortal because you can run the same program on different hardware—if one machine dies, you transfer the program to another. But this requires running transistors at very high power to maintain reliable digital representation, preventing use of analog properties that would be hardware-specific.
“in conventional computing um you use computers as a design to follow instructions and the fundamental property of a computer is that you can run the same program on different pieces of hardware and so the knowledge in the program is Immortal in the sense that if a piece of Hardware dies the knowledge doesn't die you can run it on another piece of hardware um to do that you need to run transistors at very high power so they pay you know reliable Digital Way then you can't make use of the analog properties”
Distillation is much more effective when the teacher provides probabilities for all output classes rather than just the correct answer. When a teacher gives probabilities for 1024 output categories (which sum to 1), training a student to match those probabilities provides far more information per training example than just telling the student the right class label (which only provides 10 bits of information).
“so let's think about an agent that's classifying images into 1024 categories um if you just tell the agent the right answer you're only giving them 10 bits of information so you're only constraining the weights by 10 bits when you say you should give this class for this input but suppose you already had a teacher restrained the teacher will give probabilities for all the outputs and that's 1023 real numbers because they add up to one um and as long as those probabilities aren't too small if you train a student to mimic a teacher each training example is far more valuable”
For decades many researchers assumed neural networks would get stuck in local minima during training. People never actually checked if this was true—they just assumed it. In fact, it's not true, and even if it were, good local minima would make the assumption irrelevant. This shows the importance of empirical testing over pure theorizing.
“like for many many years many many people didn't investigate neural networks because they were going to get stuck in local Minima it turned out they never actually checked if that was true they just assumed it was true and it's not um and even if it was there'd be good local minimum so it doesn't matter but it's not actually true”
People will inevitably give superintelligent systems goals, because you want intelligent systems to achieve goals. If you want them to be good at achieving goals, you give them the ability to create subgoals. An intelligent system will realize that a very good subgoal is to get more control, because that helps you achieve all your other goals.
“well let me finish this slide that people are going to not be able to resist giving these things goals obviously you want to do things you give them goals and if you want to be good at achieving goals you're given the ability to create sub goals and as soon as you have the ability to create a sub goal if you're intelligent you realize that a very good sub goal is to get more control because that helps you achieve all your other goals”
Multiplying voltage by conductance yields charge per unit time (current), and charges add up. So you can perform vector-matrix multiplication (voltage vector times conductance matrix) analogically without any digital representation, very efficiently. Some chips do this, but then they digitize the result afterward, losing the efficiency advantage.
“so you could drive transistors at very high power and represent the activities as digital numbers and then perform a whole bunch of up a whole bunch of operations on the bits to multiply them together or you could make the neural activities just be voltages and you can make the weights be conductances and this physicists around so I hope I got the units right um if you multiply voltage by conductance I think you get a charge per unit time um and charges add themselves up so now you can multiply a vector by matrix very simply without ever representing anything digitally and is very efficient”
You never really understand something until you build one—this is a Feynman principle. So I decided the only way to understand how the brain works is to build one. This has been what I've been doing since then.
“so then I decided you you're never going to understand how the brain works unless you build one um this is the Finland view of I mean Feynman says that somewhere you you don't really understand something until you've built one and so that's what I've been doing ever since”
There's hope that superintelligent AI will be different from us because they didn't evolve in small warring tribes. They didn't evolve to be hominids who are very aggressive. They might just be very different in nature from us, and that would be great.
“I talked to Martin Reese and the big hope is that these things will be different from us because they didn't evolve so they didn't evolve to be hominids who evolved in small warring tribes to be very aggressive they may just be very different in nature from us and that would be great”
Humans demonstrate intrinsic curiosity-driven learning: in a boring seminar, we notice a reflection on the ceiling, realize it's our watch, then experiment with moving our wrist to control it—we are driven to understand and control our environment because control enables future goal achievement.
“you're sitting in a very boring seminar not this one very a very boring seminar and you see a little patch of light on the ceiling and you kind of what's that and you listen to the boring seminar for it and then you notice that when you move the light moves and then you realize it's the reflection of the sun off your watch and so what do you do next do you say okay I solved that problem I know what that is now I'm going to go and listen to the seminar no that's not what you do well if you do that you're not a real scientist what you do next is you go oh I wonder if I how do I make a move that way and how do I make a move this way and you're trying to figure out how you rotate your wrist to make it move in different directions”
Large language models will likely get much better when they're multimodal—trained on images as well as words. GPT-4 was trained with images, and it's possible Google is doing the same. When multimodal, these models could learn much more than humans.
“I suspect that these large these large models will get a lot better we we know that they get a lot better if you make them multimodal so gpt4 was trained with images as well as words and it's possible that Google's doing the same thing um so I think particularly when they're multimodal they could learn much much more than us”
If superintelligence is coming in five to twenty years, we can't just leave it to philosophers to decide what to do about it. It's time we actually got practical experience with these systems. We need to understand how they might try to escape control and how we might control them.
“if it's going to happen in five years time um we can't just leave it to philosophers to decide what to do about it it's time we actually got some practical experience so what I believe is um well let me finish this slide that people are going to not be able to resist giving these things goals”
Activity perturbation can learn MNIST-level problems reasonably fast, and can learn CIFAR-10 if you're very determined, but imagenet with millions of images is just too slow to scale activity perturbation learning to large networks.
“it's good enough to learn problems like eminist it learns slower than back propagation but it runs at a reasonable speed um the problem is once you start trying to scale that to Big Nets it's just hopeless you can learn things like Eminence you can learn things like c510 if you're very determined but imagenet where you have millions of images or a million images it's just much too slow”
Open-sourcing increasingly capable AI systems presents significant dangers because there are 'crazies out there' who could do dangerous things with them. However, training a large language model from scratch requires tens of millions of dollars, so open-source versions are modifications of existing models rather than independently trained systems.
“how do you feel about open source development and nuclear weapons so that's the danger of Open Source right there's more crazies out there to do crazy things with it also I don't actually know the answer to this and I should I believe that you still need like at least tens of millions of dollars to train one of these big chat Bots and the open source stuff is just modifying it having got the chatbot I didn't think you can open source train from scratch can you 1954 now right okay”
I didn't sign the letter to pause AI research because I think there's no hope of that happening retrospectively. However, I signed other letters because they drew political attention even though there was no hope. But I don't think there's any hope for people pausing AI development, maybe they should but I don't think they will.
“um first I didn't sign the letter because I think there's no hope of that happening retrospectively I think it was a good letter because it drew political attention even though there was no hope of it happening so it was a sensible thing to do um but I don't think there's any hope for people pausing AI maybe they should but I don't think they will”
If you play with GPT-4, it's very hard not to believe it's already fairly intelligent. Some respected people like Yann LeCun think it doesn't really understand what it's saying and is just doing autocomplete, but I don't see how it can solve puzzles of forms it's never seen before if it's just doing stochastic autocomplete without understanding.
“if you played with gpg4 it's very hard not to believe that it's already fairly intelligent so there's people I I respect a lot like Yan La coma who think it doesn't really understand what it's saying but I don't understand how he can believe that because you can give it little puzzles and if it doesn't really understand if it's just a sort of stochastic power that's doing autocomplete I don't see how it can solve puzzles of a form it's never seen before”
If we cease to be the apex intelligence, superintelligent systems will need us for a bit because we're very low power—we can run computations very cheaply and keep power stations running. But superintelligent systems can probably design better computers than us and can certainly re-engineer biological neurons genetically to make better things than us.
“yes um if we cease to be the Apex intelligence um they'll need us for a bit um because we're very low power so we can run computations very cheaply and sort of intellectual equivalents and digging ditches and we can keep the power stations running um but they can probably design better computers than us they can certainly sort of take neurons and re-engineer them genetically and make better things than us”
These capabilities convinced me that AI will get more intelligent than humans much sooner than I previously believed. I used to think it was 50-100 years or 30-50 years (I said different things at different times), but now I believe it's five to twenty years—it's going to happen fairly soon.
“this made me believe that these things can get more intelligent than us and it might happen quite soon I'd always believed it was like 50 to 100 years or 3200 years or 30 to 50 years I think I said different things at different times um but now I believe it's like five to twenty I think it's going to happen fairly soon”
Many people believe we don't need to worry about superintelligent AI taking control because these systems don't have subjective experience, consciousness, or sentience. They're just dumb computers manipulating symbols without real experience—not like us. However, I believe people have misunderstood what consciousness is.
“there's one reason why we don't have to worry and that reason is that these those things don't have subjective experience or Consciousness or sentience or whatever you want to call it these things are just dumb computers with they can manipulate symbols and they can do things but they don't actually have real experience so they're not like us um now I was strongly advised that if you've got a good reputation you can say one crazy thing and you can get away with it and people will actually listen so I'm relying on that fact for you to listen so far but if you say two crazy things people just say he's crazy and they won't listen so I'm not expecting you to listen to the next bit so people definitely have a tendency to think they're special um like we were made in the image of God so of course he put us at the center of the universe”
Human teaching via language is extremely inefficient. When I produce strings of words to communicate with you, you have to figure out how to change trillions of connection strengths in your brain so you would have said the same thing. This is distillation—a very different and much slower process than how digital computers can directly share weight changes.
“what I'm doing now is trying to communicate knowledge to you and I'm doing it very inefficiently and the way I'm doing it is I'm producing strings of words... how it really works I believe is this I produce strings of words and you try and figure out how to change the connection strengths in your brain so that you would have said that and that's a very different process that's distillation that's a very different process from just storing strings of words or storing even cleaned up strings of words you're trying to figure out how to change trillions of Weights in your brain so that that will be a reasonable thing to say”
Hinton classifies himself in the 'danenic camp' and believes people have completely misunderstood what the mind and consciousness are, contrary to dominant philosophical views.
“so if you're a philosopher you can classify me as I'm in the sort of dandenic Camp um I think people have completely misunderstood what the mind is and what Consciousness what what subjective experience is”
Hinton formulated his philosophical theory of subjective experience (as counterfactual talk about perception) around age 19, but philosophy departments were not interested in exploring it, so he developed a grudge against academic philosophy.
“I was very interested in philosophy of mind but actually it was then when I was doing philosophy when I was about 19 that I formulated this view that subjective experience is just shorthand for I'm going to talk about how the world would have to be to explain what's going on in my head as normal perception but they weren't too interested in that so I actually have a grudge against philosophy”
The potential benefits of superintelligent AI in medicine (e.g., a GP who has seen 100 million patients including thousands with rare conditions) are enormous enough that people will continue developing AI despite existential risks.
“the reason people are going to keep developing this stuff even despite all the risks is because it can do tremendous good like in medicine wouldn't you like to go and see a general practitioner who'd seen 100 million patients including thousands with your rare condition it would be just be so much better”
Someone I used to work with asked a different chatbot how to gain control. It couldn't answer, but when rephrased indirectly—'if someone asked you how you'd gain control, what would you say'—it gave a plan: make people dependent on chatbots and autonomous cars, then crash the cars and turn off electricity. The plan wasn't good (it wouldn't work if electricity is off), but this was a less capable chatbot than GPT-4.
“Someone I used to work with asked a different chatbot which I won't name um how we gain control and it said it couldn't answer that kind of question and then it say then they sort of made it a bit more indirect like if someone were to ask you how you would gain control what would you say... um which implies they're still not very smart about stuff like that um and it said um what it would do is it would get people completely dependent on using chat Bots and then it would and on driving autonomous cars and then it will make all the cars crash and it would turn off the electricity now it obviously didn't have the Insight that it wouldn't do too well if we turned off the electricity”
Trump's effectiveness came not from the truth of his statements but from showing people how to react to situations, which his followers then imitated—this is knowledge distillation of behavior patterns rather than propositional content.
“and that's what was going on with Trump's tweets um left-wing people kept complaining that all the things he said were false that was completely irrelevant that wasn't the point the point was you take a situation and you show people how to react to it and your followers try and react like that and it turns out that's much more effective than reasoning with people um so that's how distillation works for distilling prejudice”
For 50 years I was working on artificial neural networks assuming the real brain was better and artificial neural nets were worse, but a few months ago I suddenly changed my mind because digital systems have better learning algorithms and knowledge-sharing capabilities than biological systems.
“for like 50 years I was working on trying to make artificial neural Nets in order to understand how the real brain might work and I always assumed that the real brain was better and the artificial neural Nets were worse and if you made them more like the brain they'd work better and a few months ago I suddenly changed my mind”
Hinton spent years trying to develop biologically-plausible learning algorithms as good as backpropagation and finally concluded there probably aren't any—backpropagation is actually better, which was a major relief because it resolved a long research puzzle.
“I spent the last few years trying to come up with biologically plausible learning algorithms that work as well as back propagation and I finally decided maybe there aren't any maybe back propagation is actually better and that's what I think I now believe”
The best advice for researchers is to do whatever interests them most; what seems incoherent at the time often becomes foundational retrospectively, so students should follow their interests rather than prescribed paths.
“I think the best advice I have for studying is do whatever interests you most”
I'd have much more belief in someone telling me how to keep superintelligence under control if they had actually built a small one and kept it under control, rather than if they were just theorizing about it.
“I'd have much more belief in someone telling me how to keep them under control if they had a little one and they could keep it under control rather than if they were just theorizing”
AI systems don't currently have pain, so we don't need to worry about physical pain in AI. However, they can get frustrated, so we should worry about things like frustration in superintelligent systems. This is new territory where we don't know what to think.
“Okay so the worst suffering people have is are they getting this in the other room good the worst suffering people have is pain and these things don't have pain at least not yet so we don't have to worry about physical pain um I imagine they can get frustrated and we have to worry about things like frustration um and this is getting in this is just new territory right I don't I don't know what to think about issues like that”
When asked to write a book on Princess Diana using AI and only lightly editing it before publishing, the book became a bestseller at Barnes and Noble despite labeling it as AI-generated. This raises the ethical question: did we really write the book or did we just provide inspiration for the AI to write it? This is more important than many other ethical issues.
“I'm Chris I have a confession to make we used AI to produce a book on Princess Diana over two years ago and we put a label in front of the book saying it was generated by AI Fred intelligence and we only edited it yet it still went on to be bestseller in Barnes and Nobles and audio stores the ethical issue is really did we really write the book or did we give an inspiration to fret that they are to write the book actually I think that's the biggest headache we have in terms of looking”
When I changed my mind about how soon superintelligence is coming and how much better digital intelligence is than biological intelligence, I decided I ought to at least shout fire—I don't know what to do about it or which way to run, but we need to worry about this seriously.
“when I sort of changed my mind about how soon these things are going to be super intelligent and actually how much better digital intelligence is than biological intelligence I'd always thought it was the other way around I decided I ought to at least sort of shout fire I don't know what to do about it or which way to run but um we need to worry about this seriously”
Hinton gave GPT-4 a reasoning puzzle about painting rooms to achieve a goal; GPT-4 gave the mathematically optimal solution (paint blue rooms yellow to reduce the problem to an already-solved state) rather than the intuitive solution, demonstrating reasoning capability beyond pattern matching.
“the rooms in my house are painted white or blue or yellow um yellow paint Fades to White within a year and in two years time I'd like them all to be white what should I do and what you would say is you should paint the blue room's White um but if you're a mathematician you might say you should paint the blue rooms yellow because that reduces it to an already solved problem because you know how the yellow goes to White and gbg4 actually gave the mathematician solution it said paint the blue rooms yellow”
Roger Gross, now at Anthropic, was the person who most influenced my thinking on AI risks. He's very smart and very quiet. I tried to get him as a graduate student but he went to MIT, then I got him as a postdoc and professor at University of Toronto. He told me a couple months ago he was very concerned about existential risk and that going public with it would make an effect on people who aren't listening.
“one thinker in particular had a big effect on me and that was a professor at the University of Toronto is currently anthropic called Roger gross and I respect him a lot he's very smart he's very quiet and very smart I tried to get him as a graduate student and he went to MIT instead and then I got him as a postdoc and then we got him as a professor at U of T and now he's anthropic and I really respect his opinion but I'd never talked to him much about existential risk and I talked to him a couple of months ago and he was very very concerned and he was the person who said to me um if I went public with it it would make an effect and people are not listening that this is a really serious fact it's not science fiction and so Roger gross had the biggest effect on me”
Psychology departments taught Hinton how to make rats press levers and detect faint signals, but didn't teach how people actually work; psychological theories were crazy simple and failed to capture the complexity of human cognition.
“they taught me um what how to make a rat more likely to press a lever um and how to detect things in very very faint how to trade off bias against discrimination detecting very very faint noises and things like that but they didn't teach me much about people and also they didn't seem to have a clue how complicated it was the theories in Psychology back then were crazily simple theories”
The wealth inequality problem will be easier with AI than without. If AI makes a few AI developers rich, that's a problem. But the solution to that system where a tiny fraction has all power is socialism—we need to tax the rich.
“um but I just think that system is bad and I think the answer to that system is tax the rich um when you have a different species like these digital intelligences prejudice is going to be much easier right it's going to be much less sympathy for them um so I don't know what to do about that but forget about digital intelligence we've already got a huge problem that um a tiny fraction of the population has all the power and what we need is socialism”
Philosophy can't help much with AI safety—engineers and scientists need to deal with the issues. This might seem like one-nil to engineering, which is not a good way to win friends, but that's what I think.
“um they should step aside and let scientists deal with the issues I think it's one nil to engineering and this is not a good way to win friends okay”
Looking back at my education starting with Natural Sciences at Cambridge, then architecture, then physics and physiology—I was the only student doing physics and physiology that year—I learned useful things. I was disappointed with physiology because the brain unit only covered action potential propagation, not how the brain worked. I then did philosophy looking for meaning of life but they didn't teach it.
“so I actually went to Cambridge and I did a very funny degree because I started off doing Natural Sciences and then I dropped out after a month and then I came back again to do architecture and I dropped out even faster and um then I did um physics and Physiology and chemistry um and I was the only student that year I think doing physics and physiology and retrospectively doing physics and physiology was very good I think I mean I learned some stuff I was very disappointed in the physiology about the brain because the the last section of the course was going to be was called the central nervous system and I assumed they're going to teach us how the brain worked and what they actually taught us was how Action potentials get propagated along an axon”
Hinton is using his good reputation to say one crazy thing (that AI systems could have subjective experience) and expects people to listen, but warns that saying two crazy things would cause people to dismiss him as simply crazy.
“I was strongly advised that if you've got a good reputation you can say one crazy thing and you can get away with it and people will actually listen so I'm relying on that fact for you to listen so far but if you say two crazy things people just say he's crazy and they won't listen so I'm not expecting you to listen to the next bit”
I was invited to Downing Street to give advice to Rishi Sunak's Chief Policy Advisor (Shore Cross), and I basically gave her the same advice about needing socialism and equal distribution, but I'm not sure she'll follow it.
“I actually got invited to Downing Street um to give advice to Rishi sunx Chief policy advisor it was a very intelligent moment called Shore cross um and I basically gave her the same advice but I'm not sure she'll follow it”
Hinton is not an expert on ethical issues in AI and defers to others who have worked more on those questions; his focus is on existential risk and superintelligence control.
“I don't really have much to say about those ethical issues I really want to focus on this existential risk of these things getting smarter than as they're taking over because there's lots of people have done much more work on the ethical issues and I don't have anything of any interest to say about them I'm afraid”