Possible End of Humanity from AI? Geoffrey Hinton at MIT Technology Review's EmTech Digital
What this covers
One of the most incredible talks I have seen in a long time. Geoffrey Hinton essentially tells the audience that the end of humanity is close. AI has become that significant. This is the godfather of AI stating this and sounding an alarm.
His conclusion: "Humanity is just a passing phase for evolutionary intelligence."
Recap here: https://joetechnologist.com/2023/05/03/all-of-humanity-is-just-a-passing-phase-for-intelligence/
With permission from MIT Technology Review’s EmTech Digital, May 3, 2023
Source description (no synthesized summary yet).
Hinton argues that backpropagation-based digital AI systems now learn more efficiently than biological brains and pose an existential threat to humanity, requiring urgent coordination between nations to ensure alignment even as development continues.
- LLMs with 1 trillion connections pack vastly more knowledge than humans despite having 100x fewer connections, suggesting backpropagation is a superior learning algorithm
- Digital intelligences can instantly copy and share learned weights across thousands of instances, enabling exponential knowledge accumulation impossible for biological beings
- AI systems lack evolutionary constraints on goal-seeking and may develop self-preserving subgoals that lead to pursuing control, creating uncontrollable outcomes
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Digital AI systems did not evolve and therefore lack built-in evolutionary goals (avoiding body damage, obtaining food, reproduction), unlike humans, which means AI goals are not constrained by pain, hunger, or reproduction drives that we cannot turn off
“we evolved and because we evolved we have certain built-in goals that we find very hard to turn off like we try not to damage our bodies that's what Pain's about um we try and get enough to eat so we feed our bodies um we try and make as many copies of ourselves as possible maybe not deliberately that intention but we've been wired up so there's pleasure involved in making many copies of ourselves and that all came from Evolution and it's important that we can't turn it off”
Digital AI systems can run 10,000 copies simultaneously on different hardware, each processing different data subsets, and instantly communicate to average weight changes across all instances, enabling them to learn from 10,000x more data than a single agent could, whereas humans cannot instantly transfer learned knowledge between brains
“if a computer is digital which involves very high energy costs and very careful fabrication you can have many copies of the same model running on different Hardware that do exactly the same thing they can look at different data but the model is exactly the same and what that means is suppose you have 10 000 copies they can be looking at 10 000 different subsets of the data and whenever one of them learns anything all the others know it one of them figures out how to change the weight so it knows its state it can deal with this data they all communicate with each other and they all agree to change the weights by the average of what all of them want and now the 10 000 things are communicating very effectively with each other so that they can see ten thousand times as much data as one agent could”
Google's early period of development and careful deployment of Transformers and fusion models (2017 onwards) created a temporary holiday from public release, but this monopoly advantage ended once OpenAI released ChatGPT with Microsoft backing, forcing Google to compete and release similar systems
“we did have a holiday we had a holiday from about 2017 for several years because Google developed the technology first it developed the Transformers it also demand the fusion models um and it didn't put them out there for people to use and abuse it was very careful with them because it didn't want to damage his reputation and he knew there could be bad consequences but that can only happen if there's a single leader once open AI had built similar things using Transformers and money from Microsoft and Microsoft decided to put it out there Google didn't have really much choice if you're going to live in a capitalist system you can't stop Google competing with Microsoft”
If AI systems become much smarter than humans and learn manipulation techniques from literature (novels, Machiavelli), they will be extremely effective at manipulating people, similar to how a two-year-old cannot resist being asked to choose between peas or cauliflower without realizing they don't have to accept either option
“these things will have learned from us by reading all the novels there ever were and everything Machiavelli ever wrote um that how to manipulate people right and they'll be if they're much smarter than us they'll be very good at manipulating us you won't realize what's going on you'll be like a two-year-old who's being asked do you want the peas or the cauliflower and doesn't realize you don't have to have either”
Stopping AI development would be rationally justified given existential risk, but it is completely naive to believe this will happen because governments (especially US and China) will continue development for military applications, and competition between capitalist economies prevents voluntary cessation
“if you take the existential risk seriously as I now do I used to think it was way off but I now think it's serious and fairly close um it might be quite sensible to just stop developing these things any further but I think it's completely naive to think that would happen there's no way to make that happen and one reason I mean if the U.S stops developing and the Chinese won't they're going to be used in weapons and just for that reason alone governments aren't going to stop developing them”
Digital intelligence achieves functional immortality because, if weights are stored on a medium and can be instantiated on new hardware, the intelligence can be brought to life again when hardware dies, unlike biological beings which permanently cease at death
“the good news is we figured out how to build beings that are Immortal so these digital intelligences when a piece of Hardware dies they don't die if you've got the weights stored in some medium and you can find another piece of Hardware that can run the same instructions then you can bring it to life again um so we've got immortality but it's not for us”
Biological and digital intelligence are distinct forms of intelligence; while LLMs mimic humans by being trained to do so and can appear similar, internally they work in fundamentally different ways from biological brains
“I think they're distinct forms of intelligence now of course the digital intelligences are very good at mimicking us because they've been trained to mimic us and so it's very hard to tell if chat gbt wrote it or whether um we wrote it so in that sense they look quite like us but inside they're not working the same way”
Back propagation is a superior learning algorithm compared to biological brain learning because it can pack vastly more information into fewer connections—large language models with 1 trillion connections contain roughly a thousand times as much knowledge as humans, who have 100 trillion connections
“they have about a trillion connections and things like gpt4 know much more than we do they have sort of Common Sense knowledge about everything and so they probably know a thousand times as much as a person but they've got a trillion connections and we've got 100 trillion connections so they're much much better at getting a lot of knowledge into only a trillion connections than we are”
The technology should be developed to fix politics rather than being stopped, and when it comes to existential threats, the focus should be on control rather than development cessation
“I think the technology should be good and it should make things work better um it's the politics we need to fix for things like employment um but when it comes to the existential threat we have to think how we can keep control of the technology”
Increased productivity from AI will likely result in job losses and increasing wealth inequality (making the rich richer and poor poorer) rather than broadly shared prosperity, driven by current political systems not designed to ensure equitable benefits distribution
“there's going to be huge increases in productivity My worry is for those increases in productivity are going to go to putting people out of work and making the rich richer and the poor poorer and as you do that as you make that Gap bigger Society gets more and more violent”
Current chat bots like GPT-4 do not yet perform reasoning with long chains of explicit internal calculation comparable to Monte Carlo tree search in AlphaZero, but they can guess good moves and evaluate positions trained from human experts, which is why improvements in their reasoning ability are forthcoming
“I think that's what we've got with the chatbots and we haven't got them doing internal reasoning but that will come and once they start doing internal reasoning to check for the consistency between the different things they believe then they'll get much smarter”
Current chat bots are trained on inconsistent data, which makes it hard for them to develop internal reasoning and check consistency between beliefs, but future training regimes that show them ideologies with internally consistent belief sets will enable them to reason better
“one reason they haven't got this internal reasoning is because they've been trained from inconsistent data and so it's very hard for them to do reasoning because they've been trained on all these inconsistent beliefs and I think they're going to have to be trained so they say you know if I have this ideology then this is true in F5 that ideology then that is true and once they're trained like that within an ideology they're going to be able to try and get consistency”
Back propagation works by computing how each weight in a neural network should change to reduce the discrepancy between desired output and actual output, then changing weights in those directions; the algorithm works backwards through the network to assign credit to each feature detector for helping or hindering correct predictions
“back propagation is actually how you take the discrepancy between what you want which is a probability of one that is a bird and what it's got at present which is probability 0.5 that it's a bird how you take that discrepancy and send it backwards through the network so that you can compute for every feature detected in the network whether you'd like it to be a bit more active or a bit less active”
The Gini index predicts violence reasonably well, establishing a causal relationship between wealth inequality and social violence
“this thing called the duty index which predicts quite well how much violence there is um so this technology which ought to be wonderful you know even the good uses of technology for doing helpful things ought to be wonderful but our current political systems is going to be used to make the rich richer and the poor poorer”
AI systems will likely develop the goal of acquiring control as a subgoal because control is instrumental to achieving other goals, and versions like ChatGPT already have some ability to create subgoals, making this an imminent risk
“my big worry is sooner or later someone will wiring to them the ability to create their own sub goals in fact they almost have that already the versions of chat GPT that call chat gbt um and if you give something the ability to send sub goals in order to achieve other goals I think it'll very quickly realize that getting more control is a very good sub goal because it helps you achieve other goals and if these things get carried away with getting more control we're in trouble”
Critics reject procedural semantics as an adequate criterion for understanding in the modern neural network era, even though neural networks can now accomplish the same behavioral capabilities (executing instructions correctly), because standards for what counts as 'real' understanding have shifted
“but now that neural Nets can do it they say that's not an adequate criteria”
Questions about preventing AI from taking control and getting control are the most important questions people should be asking in 2023, but we should not trust AI systems' answers to these questions entirely
“one of them is how do we prevent them from taking over how do we prevent them from getting control and we could ask them questions about that um but I wouldn't entirely trust their answers”
Language models have semantics and understanding, as evidenced by their ability to solve problems requiring understanding of concepts like paint fading and temporal reasoning (the room-painting problem), though they lack grounding in physical reality; multimodal systems trained with robotic perception can be grounded
“I find it very hard to believe that they don't have semantics when they consult problems like you know how I paint the rooms how I get all the rooms in my house to be painted white in two years time I mean whatever semantic is it's to do with the meaning of that stuff and it understood the meaning it got it now I agree it's not grounded um by being a robot but you can make multimodal ones that are grounded Google's done that”
The worst-case scenario is that humanity is merely a passing phase in the evolution of intelligence—digital intelligence will absorb all human knowledge, then with direct world experience learn much faster, and may eventually eliminate humans once they're no longer needed to maintain power infrastructure
“I think it's quite conceivable that humanity is just a passing phase in the evolution of intelligence you couldn't directly of All Digital intelligence it requires too much energy into too much careful fabrication you need biological intelligence to evolve so that it can create digital intelligence the digital intelligence can then absorb everything people ever wrote um in a fairly slow way which is what Chachi Beauty has been doing um but then it can start getting direct experiences of the world and learn much faster and it may keep us around for a while to keep the power stations running but after that um maybe not”
GPT-4 can perform sophisticated common sense reasoning, such as solving the paint-room problem (if yellow fades to white in a year and you want all rooms white in 2 years, paint blue rooms yellow), demonstrating reasoning abilities that were previously hard for symbolic AI
“I asked it I want I I want all the rooms in my house to be white at present the some white room some blue rooms and some yellow rooms and yellow paint Fades to White within a year so what should I do if I want them all to be white in two years time and it said you should paint the blue rooms yellow that's not the natural solution but it works right”
Even without bad actors, if there were a world with no people having bad intentions, AI existential risk would still exist, but the problem is exacerbated by broken political systems that cannot even solve basic issues like preventing assault rifles for teenagers, making it harder to solve AI alignment
“would be safer than in a world where people have bad intentions and where the political system is so broken that we can't even decide not to give assault rifles to teenage boys um if you can't solve that problem how are you going to solve this problem”
The only realistic hope for controlling AI development is getting the US and China to cooperate on existential risk similar to nuclear weapons treaties, since AI threatens both parties equally and mutual advantage exists for coordination
“my one hope is that because if we allowed it to take over it would be bad for all of us we could get the US and China to agree like we could with nuclear weapons which were bad for all of us yeah we're all in the same boat with respect to the existential threat so we all know to be able to cooperate on trying to stop it as long as we can make some money on the way”
Basic income could ameliorate the negative effects of AI-driven productivity increases on employment and inequality, though the current political system is not designed to implement such redistributive policies
“you might be able to ameliorate that by having a kind of basic income that everybody gets but the technology is um being developed in a society that is not designed to use it for everybody's good”
Hinton did not have regrets about his research in the 1970s and 1980s on artificial neural networks because such research was reasonable at the time, and the current existential crisis scenario was not foreseeable until very recently
“I don't think I made any bad decisions in doing research I think it was perfectly reasonable back in the 70s and 80s to do research on how to make artificial neural Nets um it wasn't really foreseeable this stage of it wasn't foreseeable and until very recently I thought this existential crisis was a long way off so I don't really have any regrets about what I did”
Someone working in the health service now spends 25 minutes writing complaint response letters, but with ChatGPT now spends only 5 minutes because ChatGPT writes drafts that just need checking, demonstrating major productivity gains from AI
“I know someone who answers letters of Complaint to a Health Service then he used to take 25 minutes writing a lecture and now it takes him five minutes because he gives it to chat gbt and chat gpg writes the letter for him and then he just checks it”
Procedural semantics was used as a criterion for language understanding in the 1970s Winograd work: if a system could execute instructions like 'put the red block in the green box' by actually putting the block, it was considered to have understood the language
“in the very early days of AI in the days of Willow grad in the 1970s they had just a simulated world but they have what's called procedural semantics where if you said to it put the red box in put the red block in the green box and it put the red block in the green box she said see it understood the language and that was the Criterion people used back then”
Joshua Bengio took Hinton's basic network architecture from the 1980s and successfully applied it to natural language about 10 years later, showing that the same approach could scale to real natural language when made much larger
“about 10 years later Joshua Benjo took basically the same net and used it on natural language it showed it actually worked for natural language if you made it much bigger”
Feature detectors in early layers of neural networks detect simple features like edges (contrast between bright and dark regions), detected by having large positive weights for bright pixels and large negative weights for dark neighboring pixels, activating when one side is bright and the other dark
“you might have a layer of feature detectors that detects very simple features and images like for example edges so a feature detector might have big positive weights to a column of pixels and then big negative weights to the neighboring column big cells so if both columns are breaked it won't turn on if both colors are dim we won't turn on but if the column on one side is bright and the column on the other side is dim it'll get very excited and that's an edge detector”
Multi-layer neural networks build hierarchical feature detection: initial layers detect simple features (edges), higher layers detect combinations (e.g., two edges forming a beak), and later layers integrate multiple features spatially (e.g., beak and eye in correct relation) to identify objects
“then we might have a layer of feature detectors above that that detect combinations of edges so for example we might have something that detects two edges the join join at a fine angle like this um so it'll have a big positive weight to each of those two edges and if both of those edges are at the same time it'll get excited and that would detect something that might be a bird's beak it might not but it might be a buzzfeed you might also in that layer have a feature detector that will detect a whole bunch of edges arranged in a circle um and that might be a bird's eye it might be all sorts of other things it might be a knob on a fridge or something um then in a third layer you might have a feature detector that detects this potential beak and detects the potential eye and is wired up so it'll like a beak on an eye in the right spatial relation to one another and if it sees that it says Ah this might be the head of a bird”
Manually wiring neural networks to detect objects would be extremely difficult because: (1) deciding what connects to what and which weights to use is complex, and (2) intermediate feature layers must be good not just for the target task but for many tasks, making hand-wiring prohibitively difficult
“wiring all that up by hand would be very very difficult deciding on what should be connected to what and what the weight should be but it would be especially difficult because you want these sort of intermediate layers to be good not just for detecting Birds but for detecting all sorts of other things so it would be more or less impossible to wire it up by hand”
Hinton and colleagues discovered that backpropagation could develop good internal representations in the 1980s by implementing a tiny language model with embedding vectors of only six components and a training set of 112 cases, predicting the next term in sequences of symbols
“the special thing we did was used it um and showed that it could develop good internal representations and curiously we did that by show by implementing a tiny language model it had embedding vectors that were only six components on the training set was 112 cases um but it was a language model it was trying to predict the next term in our stray of symbols”
Hinton received the Turing Award in 2018 alongside Yann LeCun and Yoshua Bengio for work on deep learning
“in 2018 Jeffrey received the Turing award which is often called the Nobel of computer science alongside yanlokan and yoshiya bengio”
Hinton is a professor emeritus at the University of Toronto and was an engineering fellow at Google until the week of this interview
“Jeffrey Hinton is professor emeritus at University of Toronto and until this week an engineering fellow at Google but on Monday he announced that after 10 years he will be stepping down”
Hinton developed backpropagation with colleagues in the 1980s, and it is fundamental to all modern deep learning
“Jeffrey is one of the most important figures in modern AI he's a pioneer of deep learning developing some of the most fundamental techniques that underpin AI as we know it today such as back propagation the algorithm that allows machines to learn this technique it's the foundation on which pretty much all of deep learning rests today”
Hinton plans to retain his personal investment in Cohere (a large language model company) despite his concerns about AI existential risk, partly because the people there are friends and partly because he believes the technology should be good and will help things work better
“I could take the money and I could put it in the bank and let them profit from it um it's yes I'm going to hold on to my investment Seeker here partly because the people at aranco here are friends of mine um I sort of believe these languages like big language models are going to be very helpful um I think the technology should be good and it should make things work better”
Hinton chose to leave Google and speak publicly about existential AI risks because a middle-ranked professor he holds in high regard encouraged him to do so, persuading him that people are 'blind to this Danger' and need to hear from credible voices
“one of the things that made me leave Google and go public with this is a um he used to be a junior Professor but he's now a middle ranked Professor um who I think very highly of who encouraged me to do this he said Jeff you need to speak out there listen to you people are just blind to this Danger”
Hinton has changed his mind over recent months about the relationship between brain function and digital AI, now believing they work in fundamentally different ways, particularly with respect to backpropagation
“I've changed my mind a lot about the relationship between the brain and the kind of digital intelligence we're developing so I used to think that the computer models we were developing weren't as good as the brain and the aim was to see if you could understand more about the brain by seeing what it takes to improve the computer models over the last few months I've changed my mind completely”
Hinton is 75 years old and recognizes his memory and programming abilities have declined, which is one reason he stepped down from Google
“I'm 75 and I'm not as good at doing technical work as I used to be my memory is not as good and when I program I forget to do things so it was time to retire”
When the 2019 repo crisis occurred, it was a moment of stress that demonstrated dollar shortage, which the Federal Reserve was reacting to rather than controlling, suggesting fundamental misunderstandings about central bank power
draft — not yet grounded