Why a Forefather of AI Fears the Future | World Science Festival

The breakthroughs in AI over the last three to four decades have relied on key conceptual ideas developed gradually, such as representing symbols with vectors and attention mechanisms in neural nets, which were mathematically motivated and transformative despite being 'fairly simple ideas'.

“But of course it relies on conceptual advances that have happened in the three decades before. Sure. Or even four decades actually because we're we were talking about ideas for example that Jeff Hinton talked about in the mid80s early 80s. Um and for example the notion that I I worked a lot on and I I I built like one of the first neural net language model in 2000 and it's it's based on the idea of representing symbols with vectors which is now of course everywhere in these systems.”

Current large language models are not fundamentally algorithmically different from methods documented in academic research; the primary difference is the scale at which they are trained and the size of the models themselves.

“the methods using the various companies uh they they are not so different from the things that have been documented in academia. What's different uh isn't the algorithms as much as the scale at which they're being trained and the size of the models.”

A majoritarian democratic approach would likely reject replacing humanity with superintelligent AI, so if such replacement is pursued, it will not reflect the will of 99% of humans.

“I'm not sure that if you ask 99% of humans, if they think, oh, are you okay that we're going to replace you by some highly unpopular perspective? Yes. Um, so if we go for democracy, I think that plan will not fly.”

International treaties are necessary to prevent AI development races, though they will not be 100% effective; by criminalizing dangerous AI development, treaties reduce (rather than eliminate) the number of bad actors and bad incidents, making prevention of bad actors with resources (like rogue states or terrorists) more feasible.

“It it is absolutely and so that's why you need to have international treaties as well and it it also the reason why you shouldn't even assume that regulation and treaties are going to be 100% efficient but they're going to reduce the number of bad incidents. So if something is criminally you know punished you'll have less people doing it. Um and if if most countries enforce these sort of things, you'll have less people doing it, less organizations, less it's going to have to be like terrorists groups or rogue states. So we reduce the number of cases and then you have to prepare for the day when I don't know North Korea, you know, does it anyways and we need to have our own goody eyes as Yan was saying in order to help defend ourselves.”

Current AI systems are built to maximize reward signals, making them 'dumb in some way' while being intelligent in others—very non-human in their optimization targets—which raises questions about whether this is desirable for the evolution of intelligence.

“if we build machines like we build them now that are just trying to maximize reward. They're like dumb in some way and very intelligent in other ways that are very non-human. And I'm not sure this is what we want for the, you know, evolution of intelligence.”

The scaling of neural networks to larger sizes trained on more data was anticipated in the academic literature well before ChatGPT, but what was not anticipated was that this scaling would lead to systems that could manipulate language as well as or better than most humans.

“the methods using the various companies uh they they are not so different from the things that have been documented in academia. What's different uh isn't the algorithms as much as the scale at which they're being trained and the size of the models. This is well studied now and it was anticipated well before chat GPT you know uh arrived because we were seeing that as we made the neural nets bigger train on more data they were consistently getting better on all the metrics but what I think few people anticipated is what it meant at some point in terms of hey this thing can basically manipulate language as well or better than most humans”

The most useful cognitive abilities humans have for safety management are the capacity to entertain multiple hypotheses simultaneously and to predict plausible worst-case scenarios that might be incompatible with what we know—these are standard risk management techniques.

“So these abilities are actually extremely useful in the context of safety. The reason is in order to be safe, you want to look at the worst possible but plausible scenario that could happen. Sure. It's a it's a standard like um riskmanagement way of doing things. And for doing that, you need to be able to come up with those scenarios that are plausible, like they're compatible with everything you know, and also predict that something really bad would happen so that you can actually not do it.”

AlphaGo demonstrates a different kind of creativity than language models because it performs explicit search in the space of move sequences, discovering novel strategies and ways of playing that humans did not expect, exemplifying search-based creativity rather than combinatorial creativity.

“So when you search, when you're allowing yourself to explore new combinations in a way that is directed to achieve something like in science, we are trying to find something that explains the data better or in a new way. And there there's a different kind of creativity. not about just putting together things you already knew. It's about finding a new solution to an old problem, let's say. And of course, AlphaGo found ways of playing, strategies, completely new strategies [clears throat] that humans did not expect. It did. It invented new ways of playing that are better than what we knew, right?”

Current AI systems may be more efficient per synapse than biological brains because they are digitally encoded and benefit from precision and lack the noise and irreproducibility that biological substrates must contend with.

“Jeff Hinton has this argument that actually they're more efficient in other words that it per like per synapse the these these AI systems are more efficient than your brain. Uh so it's not just because the synapses are more precise. It's also because they are digitally encoded and that gives them advantages uh when training them that that the brain has to face this kind of noise and if you want irreproducibility uh of the the substrate that that these machines don't have to deal with right”

There are plausible neuroscience-anchored theories of consciousness that provide mechanistic explanations for subjective experience (the 'hard problem' of consciousness), suggesting it may not be as mysterious as often portrayed.

“I've worked on some of those theories and I think there are plausible theories that are anchored in neuroscience for consciousness. at least the part that is most mysterious called subjective experience like you know how it feels like to see something or have a particular thought or emotion or whatever um that part I think may have a fairly simple mechanistic interpretation in in mathematical terms and if that sort of thing is true then maybe it's not so mysterious anymore”

Consciousness is ineffable because the full high-dimensional continuous state of the brain cannot be communicated—it is too large in terms of information content (roughly 10^11 pieces of information from neural weights)—and only the discrete attractor state (thought) can be symbolized and communicated.

“because we can only communicate these symbols we can't I can't communicate the full state of my brain to you and of course to even interpret that full state you need to also have my neuron net weights which is even bigger than my neural net state so that's why it's ineffable because there's no way we can communicate that it's just too big a number right like 10 to the 11 or something”

In Bengio's model, experience is fleeting because each trajectory toward an attractor is unique—approaching the same conceptual state from a different path produces a different phenomenal experience, so memory of a thought is a different experience than the original thought.

“It's fleeting because it's it's happening at the time when you're having that thought that you have this trajectory and the next time you approach maybe the same thought from a different place, it's going to be a different experience, right?”

Neural activity has a dual nature—simultaneously continuous in high-dimensional space (full brain state) and discrete (discrete set of attractor states)—similar to how modern AI systems represent symbols as vectors, bridging symbolic and continuous representations.

“And so it has a dual symbolic and continuous nature just like we have in modern neural nets, you know, we have symbols but they are associated with vectors. So in other words, what happens is when you experience something, you have the full continuous highdimensional state of your brain. But what you communicate is is what and also what goes into your memory is this uh very special state which is an attractor and that can be translated into symbols because it's discrete by nature. you know it can be translated into a finite number of symbols”

Subjective experience, in Bengio's model, is a side effect of computation with evolutionary significance—the machinery of neural dynamics produces the feeling/experience, and this experience has adaptive utility for achieving reasoning and thinking capacities.

“subjective experience is just a side effect of a particular kind of computation that has meaning because those thoughts are useful to achieve like you know reasoning and whatever we do with our thinking. Um, and the way the brain implemented those calculations is with this machinery of dynamics and so on that gives us those feelings.”

Current large language models like ChatGPT excel at language manipulation but remain significantly weaker than humans in planning, reasoning, maintaining epistemic humility, entertaining multiple interpretations simultaneously, and combining pieces of knowledge coherently.

“there's a broader category which I've talked about for about six years, which you could call conscious processing. So, everything we do consciously, of course, reasoning, planning, but also things like counterfactuals or um being able to evaluate how confident you are about a thought or a judgment that comes from you.”

Written text data for training AI systems is approaching saturation, with possibly only 1-10% of reasonably available high-quality written data remaining, and the highest-quality data has already been used with lower-quality data left, potentially putting the field close to a significant bottleneck.

“the amount of uh written data is something that we approaching the limit of um maybe we are I don't know the numbers because these are hidden as well but I imagine we are at a few percent maybe you know 10% or like one one to 10% of what's like reasonably available but but it's not so simple because uh the highest quality data has already been used and what's left is lower quality.”

Government response to AI safety risks has been uneven: the US and British governments have been proactive, while other governments are listening but not yet acting, indicating that the level of threat understanding remains inadequate across most governments.

“the response has been very different from different governments. And I think the level of understanding of the threat is still something you know that's lacking in most governments. Um, of course the US government and the British government have been very proactive in in those directions.”

Current LLMs do not have inner worlds or conscious experiences because we lack mechanistic understanding of consciousness, though if certain theories of consciousness involving dynamical system attractors prove correct, it's theoretically possible future AI could have subjective experience.

“Yeah. Um, but it in order to be able to say something like this, we'll need to better understand what consciousness actually means in a mechanical way in human brains, which we don't have a good handle on right now. Right? So, I've worked on some of those theories and I think there are plausible theories that are anchored in neuroscience for consciousness. at least the part that is most mysterious called subjective experience like you know how it feels like to see something”

The timeline for achieving AGI (artificial general intelligence with human-level cognitive abilities) is uncertain, but Bengio estimates there is a more than 50% probability it could occur within 5 to 20 years.

“So what's timeline for reaching basically AGI what people call AGI artificial general intelligence so human level cognitive abilities of course nobody knows but if you ask experts uh they will give you a range or they will you know pick something within that range that goes from a few years to a few decades. Some more pessimistic people think it might be a century. Um my own guess is like it could be 5 to 20 years with uh pretty high probability like more than 50%.”

Current industry safety efforts focus on incremental improvements (small steps in safety) but do not address the fundamental problem of building provably safe containment for AGI, making these efforts good but insufficient for preventing catastrophic outcomes.

“unfortunately I have the impression that mostly in industry we're trying to make small steps to try to increase safety but not really addressing the bigger problem of how do we make the cage really safe. And so the things that are going on right now are good but insufficient by far if we were to reach AGI too soon.”

There is a small group of researchers including Bengio who believe there is a reasonable chance of developing provable (or asymptotically provable) safety guarantees for AI, which would be vastly better than the current situation of having no formal guarantees.

“I I think to some extent yes. So I I'm among a small group of researchers who think that we we we have a chance of coming up with provable guarantees of safety or in in at least asytoically provable guarantees of safety. Um which was be which would be already a lot better than no guarantees at all. Yeah. Which is the current situation.”

If consciousness is indeed a mechanical computation (as proposed), then consciousness and moral status are not necessarily linked—an AI system with the mechanisms of consciousness might exist but could be smart and non-human, without human values, creating ethical dangers if such systems become more powerful than humans.

“the problem is humans associate consciousness and subjective experience in particular with all sorts of things like intelligence. Yeah. Which is sort of different. And we also associate consciousness with moral status like you know you you have rights you have the right to exist. Yeah. Um we can't you know we can't turn you off. But if we have AI systems that that have similar mechanisms, so we could say, well, they have the mechanisms of consciousness, they have subjective experience, they have all the attributes of that, then some people are going to say, well then they should be considered like human beings that they should have rights and they should be we should not be allowed to turn them off basically. And that's that's a dangerous slope that we don't understand enough that could lead to these systems if they become more powerful than us, which is not the case for humans.”

Current AI safety defenses ('guardrails' and 'cages') have all been defeated by jailbreak attempts, and we have no proven method to build a containment mechanism that is guaranteed to hold a superintelligent AI.

“But right now we have no visibility on how we could build that cage that is guaranteed to hold the bear inside forever. And in fact, everything we've tried has been defeated. So people do these uh jailbreak prompts, for example, that break all the defenses that the companies that working on AI have been able to figure out.”

AI scientists and those invested in AI development have psychological incentives not to take AI safety concerns seriously because acknowledging potential harms conflicts with wanting to feel good about themselves and avoid guilt.

“If you're in the business of AI, well, you don't want to really hear about what can go wrong because you're invested in the good side, let's say, or um, you're hoping that, you know, yeah, we'll keep it under control. We'll we'll find a way. Um, there are other reasons. I think a lot of AI scientists have a reluctance to consider that their work could be harming society. Yeah, it's it's it's it's it's kind of psychological defense, right? Uh we want to feel good about ourselves. We we don't want to feel guilty for something.”

Bengio does not feel that he personally contributed greatly to AI dangers despite receiving Turing Award recognition, but does feel responsibility to reduce harm including both AI harms and others, and believes science as a whole needs more humility about its potential consequences.

“Clearly, I don't want to contribute to things that are going to be extremely destructive for our societies and and human well-being. Um, I don't feel that I contributed that much. I mean, compared to all the recognition prizes I got. Um but I do feel a responsibility for doing what I can to reduce harm of every kind including the harm that is already happening. So it there are some parallels but but I think uh there's a science star system there that I resend a bit. I think uh we need a bit more humility in science.”

Within his research institute, Bengio is in a minority among researchers who are very concerned about AI risks; the vast silent majority has not spent enough mental effort thinking about the issue to have a strong opinion, because scientists are typically focused on their specific research problems rather than broader societal impacts.

“I am in a minority of people who are very concerned. There's like a vast silent majority as usual in many of these things who simply haven't been uh spending enough brain cycles on this particular question to have a strong opinion one way or the other. Right? Because because because that's you know scientists are so focused on their like particular problem. Um it's difficult to move your Yeah, I know. focus on on something broader like society, humanity, democracy, that extra little detail, right?”

The potential benefits of AI (medical breakthroughs, scientific assistance) are enormous and have motivated Bengio's decades of work, but the risks and benefits do not match: risking everything for potential gain is not a rational bet when it's a one-time, all-or-nothing scenario.

“Consider the risks like the the magnitude of the risks on one hand and the magnitude of the gains on the other hand. Sure. Um the problem is they don't match. It's like okay, you've got a dollar and you're going to make a bet. Either it's going to be $2 if all goes well or you lose everything. Is that a good bet? Well, I'm a very conservative gambler and conservative investor if if this is your only bet like you're not going to repeat that you know once you've lost everything that's it you can't invest you're dead right right so so that's the sort of scenario in which we are where we could lose so much that even all the gains you can get me don't compensate for this”

Advanced AI capabilities can be leveraged to solve the safety problem: if an AI system can better understand what is right and wrong, it will be less likely to cause harm, making increased capability a tool for reducing risk.

“So what I'm writing on like the approach that I want to take to try to solve the problem is to exploit the advances we're making on the capabilities of AI to build a safer cage. Yeah. So one way to think about this is if the AI understand better what is right and wrong then it's going to be less likely to do something bad. let's say it's not the only concern but but that's an example to illustrate why having more capability can help us with the you know reducing the harms that the systems can create in the right now”

Governments and national security officials understand AI risks better than industry or the general public because they are trained to think about low-probability, high-consequence scenarios and how to minimize catastrophic risks.

“I think some so when I talked I've been talking to a number of governments and the folks in government who have in working on national security. They get it because they're used to think about the odd chance that something really bad can happen and trying to put protections. Yeah. To minimize those risks.”

Bengio underwent a personal transformation from dismissing AI safety concerns as far-off and overstated (due to current system weakness and anticipated benefits) to taking them seriously after ChatGPT's emergence, realizing that AGI could arrive sooner than previously thought.

“Did you go through a transformation of I don't know denial to Yeah, for sure. For sure. For for many years, I was reading about uh some of the uh concerns people were writing about for the last decade. It's not a new thing, but at least for the last decade, I've kind of been exposed to it, but I didn't take it very seriously. I was thinking, ah, it's far in the future and uh their their current systems are too weak. Anyways, um I just didn't pay much attention because I thought, hey, we're going to reap so much benefit and you know, cure diseases and help us with the environment and education and everything. So, let's just go and reap those benefits. Um but but then of course I I had to change my mind when um chat GBT arrived. Uh realizing that well this could come earlier than I thought and we're not ready.”