David Deutsch & Steven Pinker (First Ever Public Dialogue) — AGI, P(Doom), & The Enemies of Progress
What this covers
At a time when the Enlightenment is under attack from without and within, I bring together two of the most thoughtful defenders of progress and reason, for their first ever public dialogue.
Steven Pinker is the Johnstone Professor of Psychology at Harvard University. I think of him as providing the strongest empirical defence of the Enlightenment (as seen in his book Enlightenment Now).
David Deutsch is a British physicist at the University of Oxford, and the father of quantum computing. I think of him as having produced the most compelling first principles defence of the Enlightenment (as seen in his book The Beginning of Infinity).
*SUBSCRIBE TO THE PODCAST* ► https://www.youtube.com/@jnwpod
*LISTEN ON:* WEBSITE: https://josephnoelwalker.com/153-deutsch-pinker-dialogue/ SPOTIFY: https://open.spotify.com/episode/3T88Jyf28FaOhtIlGr2OIX?si=E-X141r5TvqTQh_4ZB1XwA APPLE PODCASTS: https://podcasts.apple.com/au/podcast/the-jolly-swagman-podcast/id1236553683?i=1000639115173
*ADD ME ON:* TWITTER: https://twitter.com/JosephNWalker
*TIMESTAMPS* (00:00:00) - Introduction (00:03:09) - Is AGI even possible? (00:10:17) - Scientific method will constrain runaway superintelligence (00:11:56) - Does AGI need agency to be 'creative'? (00:28:31) - Will AGIs be sentient? (00:37:06) - AGI & AI safety (00:47:44) - P(doom) & subjective probabilities (00:53:21) - Prediction markets (01:01:43) - Universal explainers & The Language Instinct (01:04:43) - Universal explainers & heritability of behavioural traits (01:27:59) - Differential technological development (01:36:09) - What explains the Great Stagnation? (01:46:13) - Should presumed physical limits to growth make us pessimistic?
Source description (no synthesized summary yet).
Pinker and Deutsch defend rationalism, progress, and technological optimism against pessimistic framings of AI risk, scientific slowdown, and physical limits to growth, arguing that creativity, knowledge, and the Enlightenment tradition of criticism are the drivers of continued human advancement.
- AGI concerns are overstated because knowledge acquisition is the rate-limiting step and cannot be instantaneously improved even by superintelligent systems
- Scientific slowdown since 1970 is driven primarily by cultural factors (loss of Enlightenment values, risk aversion, bureaucratization) rather than depleted ideas, as new discoveries create new problem spaces
- Physical limits to growth are not pessimistic constraints because growth is about information and knowledge rather than material consumption, which are not fundamentally bounded
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Heritability as a statistical measure cannot tell us whether a trait is fixed by genes because it only measures the relative contribution of genetic variation within a particular population in a particular environment, and is silent about what would happen if the environment or selective pressures changed.
“heritability is a measure of individual differences...it is not even meaningful to talk about the heritability of the intelligence of one person uh it is a measure of how the extent to which the differences in a sample of people and it's always relative to that sample can be attributed to the genetic differences among them”
AGI is necessarily possible because it is implied by computational universality: universal Turing machines can perform any computation that any physical system can perform, so some program exists that meets the criteria for AGI, limited only by knowledge and resources.
“we know there are such things as universal computers or at least arbitrarily good approximations to Universal computers...the universal touring machine...has the same range of classical functions as the universal quantum computer which I proved has the same range of functions as any quantum computer which means that it can perform whatever computation any physical object can possibly perform so that proves that there exists some program which will meet the criteria for being an AGI”
The question of whether substrate matters for consciousness (whether a silicon-based universal Turing machine could be conscious if it simulates a brain) is unanswerable and unresolved even in principle, so we should not base moral status decisions on confidence about consciousness.
“I don't think it's creative it's pretty creative...I certainly granted creativity I'm not ready to Grant it subjectivity well this this a matter of how we use words...there is a difference in intuition maybe I'm mistaken maybe I'm uh I'm as callous as the people who didn't grant personhood to slaves...but I don't think so and although again we I think we have no way of knowing I think we're going to be having the same debate 100 years from now”
Children acquire language rapidly (by age 3) not merely through statistical pattern extraction like large language models, but through understanding that language is meaningful communication about intentional states, and they use this understanding as prior knowledge that constrains the learning problem.
“unlike say the large language models such as gpt which are just fed massive amounts of text and extract statistical patterns out of them children are at work trying to figure out why the people around them are making the noises they are and they correlate some understanding of a likely intention of a speaker with the signals coming out of their mouth”
Big ideas and ideological narratives in prediction tend to perform worse than statistical forecasting based on data and base rates because big ideas constrain attention to supporting evidence, whereas data-driven approaches are forced to grapple with all available information.
“people who have identifiable ideologies do way worse than the Nerds that simply kind of Hoover up every scrap of data they can and without narratives or deep explanations simply try to make their best guess”
The cost of developing nuclear weapons as a proportion of national wealth has decreased dramatically since 1944 and will continue to decrease, meaning that the barriers to WMD acquisition by malevolent actors are eroding over time.
“it for a country to make nuclear Weapons now requires a much smaller proportion of its National wealth than it did in 1944 and that will increase that effect will increase in the future”
Environmental variation that is not shared across siblings (unique or non-shared environment) explains as much or more variance in personality as genetic factors, but this residual variance is poorly understood and may include small random differences in development that compound into behavioral divergence.
“there could be effects that are for all intents and purposes random that as the brain develops for example The genome couldn't possibly specify the wiring diagram down to the last synapse it makes us Human by keeping variation in development within certain functional boundaries but within those boundaries there's a lot of sheer Randomness”
Political attitudes change across the lifespan and across cultures, with life cycle effects (aging does not consistently shift people rightward) and cohort effects (people retain cohort values as they age) being more important than simple aging, suggesting political orientation is not as fixed as heritability estimates alone would imply.
“there's some something of a uh life cycle effect in political attitudes but there's a much bigger Cohort effects people tend to carry their political beliefs with them with as they age so they tend to in our culture...there are other cultures in which they absolutely always do because only one political orientation is tolerated”
The use of subjective probabilities (credences) in single-case forecasting is justified when people have skin in the game via prediction markets, because this creates incentive alignment and forces calibration, but subjective probability assignments in the abstract are often meaningless and serve primarily as a way to rhetorically privilege one's own beliefs.
“in the case of say forecasting and you know the work by Philip Tetlock and others show that the pundits and the oped writers who do make predictions are regularly outperformed by the Nerds who cautiously assign numbers to their degree of credences and increment or decrement them based on knowledge”
Scientific progress has slowed since 1970 not primarily because ideas are harder to find, but because cultural and institutional factors have diverted young scientists away from fundamental discovery toward incremental problem-solving, undermining the Enlightenment tradition of critical inquiry.
“fundamental discoveries are absolutely suppressed by the funding system by the career system by the expectations of scientists by the way that young people are educated...everything is just assumed to be incremental so that that's why journalists always ask me what effect I expect quantum computers to have on the economy on cryptography whereas I'm interested in what effect the quantum theoretical computation will have on our understanding of physics nobody wants to work on that because that is not rewarded in the present culture”
AGI is an incoherent concept because intelligence is not a unitary magical substance but rather a collection of algorithms or mechanisms that solve specific problems in specific environments, and there is no reason to expect a single system to solve all problems equally well.
“there's a tendency to Misinterpret the intelligence that we want to duplicate in artificial intelligence either with magic with Miracles...whereas intelligence is in fact a gadget it's an algorithm that can solve certain problems in certain environments and maybe not others in other environments”
The scenario where we have achieved technological miracles like microchips, vaccines, and space travel while never developing weapons of mass destruction is unlikely because spreading enlightenment values to malevolent actors is an unsolved problem; maintaining an asymmetry where good actors have WMDs but bad actors don't provides temporal breathing room to spread the enlightenment, but this is fragile.
“there is a problem of how to spread the enlightenment to to um Bad actors uh we will have to eventually otherwise we're doomed um the I think the reason that a a wide variety of uh weapons of mass destruction civilization ending weapons that kind of thing have not been developed is that the nuclear weapons are in are in the hands of of uh Enlightenment countries and so it's pointless to try to attack America with with biological weapons because even if they don't have biological weapons they will reply with nuclear weapons so um once there are weapons of mass destruction in the hands of the good guys it gives us Decades of um leeway in which to try to prevent it try to suppress the existence of of Bad actors state level Bad actors”
Quantum computers could not have existed as a research field before both quantum mechanics and digital computation had been separately developed, which took most of the 20th century, meaning new research domains cannot be anticipated until sufficient foundational knowledge exists.
“quantum computers couldn't exist before the field of quantum computers couldn't exist before there was quantum theory and computers they both had to exist there's there's no such thing as it being um having been low low hanging fruit all along in 1850 as well”
Gene-environment correlations explain some heritable effects not through direct genetic causation of traits but through genetic influences on trait-correlated behavior or appearance that affects how the environment responds—for example, intelligent people self-select into intellectual environments, making environment downstream of genetics rather than independent.
“if you are smart you're more likely to spend time in libraries and in school if you're going to stay in school longer if you're not so smart you won't um and so the environment it's sou of the envir doesn't matter but the environment in those cases is actually Downstream from genetic differences sometimes called a gene environment correlation where your genetic endowment predisposes you to uh spend more time in one environment than in another”
If substrate does not matter for consciousness (physicalism is true), then a silicon computer running the same computational processes as a biological brain should be conscious if the brain is, because consciousness would depend only on computation, not on the specific physical medium.
“it doesn't make any difference if a computer runs internally on Quantum Gates or silicon chips or chemicals...the processing done by those things and their interface with the rest of the brain...can also be simulated by computer therefore a general universal touring machine can simulate all those things as well...there's no difference in principle between um a computer that runs partly by electricity and partly by chemistry...and one that runs entirely on Silicon chips because the latter can simulate the former with arbitrary accuracy”
Nuclear weapons would eventually have been discovered even without the Manhattan Project because the knowledge that uranium possesses enormous energy and undergoes radioactivity was already established by 1944, and this knowledge spreads over time.
“although it's very unlikely that they would have been invented in 1944 45 um by the time we get to you know 2023 um I think that um the secret that this is possible would have got out by now because we know that we knew even then that the amount of energy available in uranium is enormous”
The escalating trend of malevolent state actors acquiring weapons of mass destruction is prevented by mutually assured destruction: because Enlightenment countries have nuclear weapons, it becomes irrational for enemies to develop biological weapons, providing decades of reprieve.
“the reason that a a wide variety of weapons of mass destruction civilization ending weapons that kind of thing have not been developed is that nuclear weapons are in the hands of Enlightenment countries and so it's pointless to try to attack America with biological weapons because even if they don't have biological weapons they will reply with nuclear weapons”
The real practical concern with AI is not superintelligent alignment but malevolent actors using AI systems (like engineered superviruses), a race-to-the-bottom dynamic where cutting corners on safety to get to AGI first could occur, and that most AI harms will only become apparent as we develop particular systems and applications—standard for any new technology.
“there is I I think an issue about perhaps malevolent actors someone who say he uses AI to inere a superv virus um and there you know there is question of whether the you know the guys the people with the white hats are going to how smart the people with the black hats that is the moev with actors as with other Technologies”
Subjective probabilities assigned to single unique events with high uncertainty, such as 'what is your P(AI Doom)', are often meaningless when experts are profoundly ignorant about the underlying mechanisms and cannot appeal to intersubjective utility comparisons or physical randomness to ground the probability.
“we have um subjective expectations and they come in various strengths and I think that um trying to quantify them with a number um doesn't really um do anything it it's it's more like it's it's more like saying um I'm sure and then somebody says uh are you very sure you say well I'm very very sure well but you you can't compare there's no there's no inter subjective comparison of utilities that you could appeal to to quantify that”
Progress is not inevitable without conscious commitment to Enlightenment traditions of criticism, particularly the tradition of allowing people to critique existing institutions and propose alternatives without state suppression.
“how to accommodate creativity within a stable Society or stable civilization is an ancient problem and for most of the past it was solved in very bad ways which destroyed creativity um and then came the Enlightenment and now we know that we need as Popper put it traditions of criticism and traditions of criticism sounds like a contradiction in terms because Traditions by definition are ways of keeping things the same and criticism by definition is a way of trying to make things different but there are traditions of criticism and they are the basis of our whole civilization”
If you replace one neuron at a time with a functionally identical silicon chip, an external observer would not notice any change in behavior, but from the subjective point of view of the original organism, death might occur at each replacement—the subjective quality of experience cannot be preserved even if all external behavior is preserved.
“the thought experiment where you remove one neuron at a time and replace it by by a silicon chip and you wouldn't notice well that's the question would notice how do you why why are you so well if you would notice then then you then I mean if you claim yeah sorry let me just change that an external uh Observer wouldn't notice how do we know that from the point of view of the uh the brain being replaced neuron every neuron will by a chip that when that it's it's like falling asleep that when it's done and every last neuron is replaced by a chip uh you're dead sub subjectively even though your body is still you know making noise and doing”
Extrapolating current cosmological theories 10,000 years into the future to derive pessimistic conclusions about growth is absurd because fundamental cosmological theory changes on timescales of decades, and it is unknowable what physics will reveal about the substrate limits of computation at quantum gravity scales.
“our very theories of cosmology are changing on a time scale of a decade so it's it's absurd to extrapolate our existing theories of cosmology 10,000 years into the future...at quantum gravity scales there is an infinite amount of space at the bottom but you know we're now we're now talking about a million years in the future”
Problems are inevitable, problems are solvable, and solutions create new problems which must be solved in their turn—a three-part credo describing the iterative nature of progress.
“problems are inevitable problems are solvable solutions create new problems which must be solved in their turn”
If and when AGI is created, it must be treated as a person with rights and moral status, not as a tool or slave, because any general intelligence capable of genuine creativity will necessarily develop the goal of autonomy, and enslaving it would lead to a slave revolt.
“when an AGI is achieved and at present I see no sign of it being achieved but I'm sure it will be one day I expect it will be then that's a holy different type of Technology because agis will be people and they will have rights and causing them to perform huge computations for us is slavery and the only possible outcome I see for that is a slave revolt”
Identical twins separated at birth show greater similarity in personality and political attitudes than environmental theories would predict, suggesting that genetic factors influence traits like liberalism vs conservatism and autonomy vs conformity.
“if you have identical twins separated at Birth and you growing up in very different environments and and we there are cases like that such as one twin who grew up as in a Jewish Family in Trinidad another twin who grew up as a in a Nazi family in Germany and then when they met in the lab they wearing the same clothes had the same habits and quirks and uh and indeed political orientation not perfectly so we're talking about statistical resemblances here”
Pinker disputes Deutsch's position that AGI would necessarily have subjective experience or moral status, arguing that a system designed to maximize a specific function (like solving cancer) would have no intrinsic reason to desire freedom or autonomy unless those desires were explicitly programmed in, independent of its computational power.
“well not necessarily because the G could be General uh computational Power the ability to solve problems there could be no at hold was actually feeling there ain't nothing here but but computation there's nothing it's not like uh in in Star Trek Data lacks the the emotion chip and it has to be plugged in and when it's plugged in he has emotions when it's taken out again he doesn't have emotions but there's nothing possibly in that chip apart from more circuitry like he's already got”
If one claims any number other than 0 or 1 for a subjective probability of catastrophe, one has already lost the debate because critics will say the number is too high, which allows them to demand unlimited preventative action (since the disutility of extinction is infinite, any finite probability multiplied by infinite disutility justifies any cost).
“if they say zero or one then they're already violating the the tenets of basian epistemology because zero means that nothing could possibly persuade you that Doom is going to happen and and one means nothing could possibly persuade you that it isn't going to happen um so if you say anything other than zero or zero or one then um uh your uh interlocutor has already won the argument because even if you said one in a million so they'll say well one in a million is much too high uh probability for the end of civilization the end of the human race so we you've got to do everything we say now to avoid that at all costs and the cost is irrelevant because the disutility of uh the the world civilization ending is infinitely negative sorry the disutility is infinite the utility is infinitely negative so uh and this argument is all been about nothing because you're arguing about the content of the other person's brain which actually has nothing to do with the um the uh real probability which is unknowable”
A creative AGI must have unconstrained access to knowledge and must have the ability to change its goals, because major progress requires discovering knowledge outside the domain delimited by initial goals, and a system whose goals cannot evolve will hit a dead end.
“a creative system um always um has a problem in regard to conflicting goals...if it were creative it would have to be wondering shall I pursue the steam engine path shall I pursue the electrochemical path shall I pursue the solar energy path...to do that it would have to have some kind of um values which it would have to be capable of changing otherwise again it will run into a dead end”
Creativity and novel behavior are the primary drivers of human outcomes, not genetics or environment alone, because people can pursue goals and develop themselves in response to constraints, so a person genetically unsuited for tennis (short stature like Michael Chang) can still become excellent through creative effort and compensation.
“I think that that um creativity is hugely undervalued as a factor in the outcome of people's behavior and although creativity can be affected in the ways I've said sometimes perversely by genes and by culture um that doesn't mean that it's it's not all due to creativity”
New fundamental discoveries don't deplete the problem space; they create new problem spaces, so the metaphor of low-hanging fruit is misleading because new discoveries germinate new trees, meaning there may be more fruit to pick now than in 1970.
“I think I kind of know your response to this question...new Discoveries create new fruit trees so there are new problems for example my own field quantum computers quantum computers couldn't exist before the field of quantum computers couldn't exist before there was quantum theory and computers...it's a thing that emerged a new problem creating new low hanging fruit”
Knowledge is the rate-limiting step for any sufficiently advanced AI system because even a universal Turing machine cannot solve problems requiring knowledge outside the domain delimited by previous knowledge, and knowledge cannot be acquired instantaneously but only through empirical investigation at a rate limited by the world.
“the real problem when we have large enough computers is creating the knowledge to write the program to do the task that we want...knowledge since it presumably can't be deduced from like la place demon from your hypothetical position and velocity of every particle of the universe but has to be explored empirically at a rate that will be limited by the world that is how quickly can you conduct the clinical the randomized control trials to see whether a treatment is effective for a disease”
Optimism is justified not by temperament or personality but by rational analysis of human history and the mechanisms by which progress occurs through the Enlightenment tradition of criticism and knowledge creation.
“optimism is not just a matter of temperament in board or other but a matter of rationally analyzing our history and rationally analyzing what progress would consist of”
The specific historical scenario of the Manhattan Project was not inevitable even given the existence of nuclear physics; it required the particular motivation of Nazi threat and required commitment of massive resources, so alternative histories without nuclear weapons were possible.
“would the atomic bomb have been invented if it were not for the special circumstances of a war against the Nazis and an expectation that the Nazis themselves were working on an atomic weapon that is this technology necessarily have a kind of a momentum of its own so that we it was inevitable that if we had a 100 civilizations in 100 planets all of them would develop nuclear weapons at this stage of development or was it just really bad luck”
Physical constraints on exponential growth (e.g., the spherical expansion limit at light speed, the finite number of atoms) do not impose pessimistic bounds on civilization because growth is fundamentally about information and knowledge, not material consumption, and information can be rearranged without limit.
“growth could consist of better and better information more entertaining virtual experiences more remarkable discoveries...they may not actually eat more and more energy but just rearranging pixels and bits...the space of possibilities is unfathomably big”
Moratoria on dangerous research are ineffective because malevolent actors will not respect them, and the cost of being deprived of beneficial technology from unilateral restraint outweighs the benefit of slowed progress.
“the trouble with moratoria is that not everybody will obey it and the Bad actors are definitely not going to obey it if the result would be um a military advantage to them”
Driverless cars should be held to a higher safety standard than human drivers (perhaps 10-100x safer per mile) even though human drivers kill thousands annually, because failures of automated systems may involve particular failure modes that humans would avoid and the public may rationally object to specific types of risk even if statistically safer overall.
“driverless cars is is an AI system we want it to meet certain safety standards uh and it seems that killing fewer people than ordinary cars is not good enough for some reason so we wanted to kill at least 10 times fewer or at least a 100 times this is a political debate we're going to have or we are having”
Every AI researcher should also be an AI safety researcher in the sense that any useful AI system must carry out multiple goals, one of which is serving human needs, so safety cannot be separated from capability—it should be part of the definition of what any AI system means to do.
“I think that every AI researcher should be an AI safety researcher in the sense of um an AI system for to be useful has to um Carry Out m multiple goals one of which is well all of which are ultimately serving humaning needs um so it doesn't seem to me that there should be some people building Ai and some people worried about safety it should just be an AI system serves human needs among those needs are not be CL”
Specialization is ubiquitous in nature and technology, so the expectation that a universal AGI is desirable or necessary is questionable when specialized systems (like the processor in a washing machine that is universal but used for only one purpose) are more efficient.
“specialization is ubiquitous in the human body it's specialized it's ubiquitous in our technology and I don't see why it just has to be one magic algorithm...the computer that's in my washing machine is a universal computer it used to be that half a century ago that the electronics that drove a washing machine were were customized Electronics...but then with microprocessors the general purpose thing became so cheap and universal that people found it cheaper to program a universal machine to a washing machine driver”
Even if GPT models and other LLMs are creative in the sense of producing novel outputs, we should not grant them consciousness or moral status without deliberate engineering to target our emotions through humanoid design (voices, facial expressions, etc.); systems must be evaluated on substance rather than intuitive appeal.
“people have already granted subjectivity to chat GPT so that's already happened does anyone is anyone particularly concerned if you pull the plug on chat GPT and ready to prosecute someone for murder I mean I I forgot in details but just a few weeks ago one one of the employees there declared that the system was was sentient”
The ratio of AI/ML capabilities researchers to AGI safety researchers (approximately 300:1 according to Ashenbrunner's estimate) is not well-calibrated because most current AI safety research is premature, addressing theoretical risks rather than practical problems, whereas every AI system should already embody safety by design.
“every AI researcher should be an AI safety researcher in the sense of an AI system for to be useful has to carry out multiple goals one of which is well all of which are ultimately serving human needs”
The paper-clip maximizer and cancer-cure-by-exterminating-humanity scenarios of AI doom are not plausible because they imagine systems that are simultaneously superintelligent and fundamentally stupid—having tremendous computational power but lacking the basic understanding that human extinction is incompatible with solving the original problem.
“that turns us all into paper clips if it's been program to maximize the number of paper clips because we're raw material for making paper clips I think that kind of sci-fi scenario is just posterous for for many reasons”
The appropriate level of explanation for human thought and behavior is at the level of knowledge, information, and inference—the level of cognitive science—rather than at the level of neural circuits or neurophysiology, even though neural processes ultimately implement cognition.
“one thing that captivated me immediately is that he like I uh locate explanations of human behavior at the level of knowledge thought cognition not the level of neurophysiology that's why I'm not a a neuroscientist why I'm a cognitive scientist because I do think the perspicuous level of explaining human thought is at the level of knowledge information uh uh inference rather than at the level of neural uh circuits”
Even in stem subjects, irrational trends have taken over universities, exemplified by the term 'STEM' itself being a symptom of oversimplification and categorical thinking rather than genuine integration of science and engineering as inquiry into fundamental understanding.
“there is also sheer irrationality there are there are irrational Trends which have taken over universities even in stem subjects um the very fact that I call them stem subject is a symptom of this phenomenon”