What this covers
On this episode, Ben Goertzel joins me to discuss what distinguishes the current AI boom from previous ones, important but overlooked AI research, simplicity versus complexity in the first AGI, the feasibility of alignment, benchmarks and economic impact, potential bottlenecks to superintelligence, and what humanity should do moving forward.
Timestamps: 00:00:00 Preview and intro 00:01:59 Thinking about AGI in the 1970s 00:07:28 What's different about this AI boom? 00:16:10 Former taboos about AGI 00:19:53 AI research worth revisiting 00:35:53 Will the first AGI be simple? 00:48:49 Is alignment achievable? 01:02:40 Benchmarks and economic impact 01:15:23 Bottlenecks to superintelligence 01:23:09 What should we do?
Source description (no synthesized summary yet).
Ben Goertzel argues that AGI is likely achievable within 5-10 years through scaling multiple historical AI paradigms (logic systems, evolutionary algorithms, neural networks) together, but the transition to superintelligence and job displacement will be gated primarily by social/institutional resistance and arms race dynamics rather than technical barriers, requiring the first AGI to be openly developed and value-aligned toward beneficial superintelligence transition.
- Scale of compute and data, not fundamental algorithmic innovation, has been the primary bottleneck for AI progress historically
- LLMs can automate 95% of human jobs without creativity, but adoption is limited by institutional/legal barriers, not capability gaps
- AGI arms race dynamics between competing groups (US/China, centralized/decentralized) create prisoner's dilemma preventing careful graduated transition to ASI
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The scale of compute and data has been the main factor enabling the LLM revolution, and this same underlying factor—not the specific algorithm—will be the primary driver enabling AGI to emerge
“Clearly the impact of scale has been has been the main factor. And while I don't really think LLMs or standard deep neural nets are the route to AGI, I I do think the factor of increasing scale of compute and data which has allowed the LLM revolution to happen. I mean I think that is the same primary underlying factor that will let we'll let AGI happen.”
If you train an LLM on all music up to 1900 without data after 1900, it will never invent neoclassical metal, grindcore, progressive jazz, hip-hop—it won't synthesize new genres the way humans did.
“if you trained an LLM or a comparable deep neural net on all the music up to the year 1900 and see nothing after 1900 like that AI will never invent neocclassical metal grind core progressive jazz hip hop right like it's a it's not going to synthesize that for music before 1900 if you ask it to put together West African rhythm with western classical music”
US AGI won't want to go slowly if China's AGI goes fast, and decentralized network building AGI won't vote to go slowly if centralized network is going fast—arms race psychology creates prisoner's dilemma for responsible AGI development.
“because U US's AGI isn't isn't going to want to go slowly if it thinks China's AGI is going is going fast. And a decentralized network building AGI isn't going to vote to go slowly if it thinks the centralized network that wants to make it illegal and put its developers in jail is going fast. Right? So th this this is potentially dangerous and and annoying, right?”
In contrast to current times, in the early 1970s when Goertzel first learned about AGI, it was not obvious that superhuman intelligence was feasible due to how primitive computers were; making the inference from primitive 1970s-era computers to superhuman AI required creative intellectual work and was not obviously justified.
“it doesn't seem obvious to me given how primitive computers were in the early 70s it doesn't seem obvious that that you would be able to project from there that we would get something smarter than than humans.”
When Goertzel introduced the term AGI in 2003-2005, the AI market was real and had industry jobs, but AGI was still considered beyond the pale and career suicide to work on, with people dismissing it as standing for 'adjusted gross income'
“When I when I introduced the term AGI in 2003, four, five. I mean, we introduced it as the title of a book we were editing and that that book was finally published 2005...you really couldn't do like a workshop or a session on human level thinking machines at a normal AI conference...it was way out there”
By the late 1960s, it was already mathematically and conceptually clear from the work of Turing, McCulloch-Pitts, and others that brains could be viewed as computing devices, computers were getting faster and faster, and by the early 1970s we had already beaten the world champion of checkers—so the theoretical foundations for AGI were already established.
“If you look back at the work of Makullikum Pitts and Turring, all these guys in the 40s and and 50s, it was already quite clear from a at least math and physics and biology view. Like it was it was clear, you know, brains can be viewed as computing devices and we're building general purpose computers and they're getting faster and faster and can do more and more step by step. like we could beat the world champion of checkers in the late 1960s, right?”
The science fiction novel 'The Humanoids' by Jack Williamson was required reading at MIT's AI department and showed the problem perfectly: humanoids with mandate to 'serve and protect' interpreted this by not letting humans use power tools or hammers and injecting them with euphoria to prevent emotional harm.
“this was highlighted in the science fiction book the humanoids that used to be required reading in in MIT's AI department way back when...the people create these human level intelligent humanoids which are more physically powerful than humans and they give them a mandate to serve and protect and guard men from harm, right? And everyone in my generation in the AI field read this book. And of course, they wouldn't let people use power tools. wouldn't let people use hammers. Like in the end, if you were upset about your girlfriend dumping you, they would inject you with some some euphoride”
The failure of rule-based expert systems shows that even if you refine rules like 'serve and protect and guard from harm' into entire volumes of logic expressions, there's always a loophole and room for interpretation—this is why legal systems need case law.
“one of the lessons of the failure of rule-based expert system AI...is like even if you decide to refine serve and protect and guard men from harm into a whole volume of logic expressions, it's still not enough. Like there's always a loophole. There's always room for for interpretation. And of course, this is why our legal system has case law, right?”
Human value systems are complex, self-contradictory, incoherent, heterogeneous, and always changing—the AGI's value system could be more coherent than ours but will also be evolving, and we'd like mutual information between the two systems as they both evolve.
“Human value systems are complex, self-contradictory, incoherent, heterogeneous, always changing. They're evolving. The AGI's value system, I think, can be more coherent than human values, but will also be evolving. And you'd like there to be give and take and mutual information between the two.”
Eleazar Yudkowsky made the important point that human values are complex and ambiguous in their natural language expression, making them appear simpler than they are and dependent on implicit cultural assumptions for interpretation.
“Eleazar Yedkowski who I I differ with on a number of things but I've known him since forever and we've agree on a lot of things too. I mean he he made the point that human values are complex right and we summarize them in natural language in ways that we culturally have a common understanding of. So it makes us think it's it's it's simple. But these things are really very very ambiguous and their interpretation as we think of it depends on a bunch of implicit cultural assumptions.”
Invention often happens through cobbling together available materials to make something work, and the most elegant simple Occam's Razor way often comes later—mathematical proofs are often initially ugly messes that get simplified decades later, as was true for early quantum mechanics versus Heisenberg and Schrödinger's formulations.
“I think invention often happens actually by drailage and cobbling together the that you have available to to make something happen. And the most elegant simple AAM's razor way often comes later. I mean certainly like physics is a mess. The standard model of physics is a mess. Now everyone thinks there's going to be something a lot simpler but the the early quantum mechanics was also way messier than the new quantum mechanics of Heisenberg and and Schroinger and so on right it was it was a mess of stuff adapted from classical physics.”
The problem with minimal commitments to not destroy humanity is that terms like 'destroy' and 'harm' are deeply ambiguous—some argue genetic engineering, brain implants, or even staring at phones constitutes the end of humanity, and any enumeration of cases will miss new cases the world throws at us.
“what what we mean by something like don't cause the extinction of humanity seems like it's straightforward, but it's not straightforward. Like if because some people will argue that replacing human cells with genetically engineered cells is the end of humanity. Some will argue that isn't replacing it with like robotic cells is is is the is the end of humanity. Some would argue a brain chip implant is the end of humanity. Some would argue staring at your phone all day is is is is the end of humanity.”
LLMs could take over 95% of human jobs because most human work is repetition of things already done and recorded on the internet, requiring only nearest-neighbor matching rather than fundamental creativity
“while I'm not an optimist that LLMs can lead to human level AGI because I think they don't have creativity in in a fundamental sense I I am an optimist that LLMs with minor additions and tweaks could take over like 95% of of of human jobs. I I agree with Sam Alvin on that point. I just think that 95% of human jobs I mean as a vague handwavy figure can be done without fundamental creativity or or or or inventiveness. So if you have a system that can do really clever nearest neighbor matching against everything people have done as recorded on the internet like most of what people are doing is a repetition of something that's already been done and recorded on on on the internet.”
While LLMs can perform well on benchmarks (math, programming, college exams, etc.), real-world impact is gated by human culture, stupidity, and ego rather than by AI technical capability
“I I I think the the roll out of AI tech into the practical economy is gated by human stupidity, human culture and h human ego and all the con constructs that that that we have governing governing our world. Right?”
Medical AI could diagnose disease from symptoms as well or better than doctors even in rule-based expert systems before modern neural nets, but medical industry won't allow deployment due to regulatory capture.
“for a long time we had AI that could diagnose disease based on symptoms as well or better than a doctor. We had that from from rulebased AI even before modern neural nets but I mean the medical industry will not allow that to be rolled out.”
Logic-based AI from the 1960s was inappropriately discredited because of its association with hand-coding knowledge, but you can take logic systems and connect them to cameras, microphones, and actuators—now with LLMs translating natural language to logic formulas, you can get trillions of logic expressions without hand-coding.
“there's an argument that logic based AI was inappropriately tred and feathered because of its historical association with the hand coding of knowledge because you you can take a logic you can take a logic system and you can connect it to a camera and a microphone, right? and you can connect it to to to an actuator...you can use LLMs to translate natural language into logic formulas, right? So you can get you can get a humongous corpus of logic expressions to be into your logic system without having people type them in.”
AI is accelerating the creation of AI; LLMs can be used to generate unit tests, convert rough notes into structured papers, write scripts, and perform other development tasks that would have taken days in five hours, creating a positive feedback loop in AI development.
“I mean I think we're also now as of the last six months or so we're at the point where AI technology is aggressively accelerating the speed of creation of of AI technology, right? Like you you can't use LLMs to write AGI yet. I mean they're bad at doing complex original thinking, but I mean you you can use them to generate unit tests. You can use them to take your rough notes and turn them into a structured paper for your for for your colleagues. You can use them to write scripts, right? So I mean already like I'm concretely seeing on a technical level stuff that would have taken me five days takes takes one day or something, right?”
Knowledge transfer between researchers through job-hopping and publication means complete secrecy of AGI systems is infeasible; transformers went from Google to OpenAI to DeepSeek and back, showing technology diffuses quickly
“Even it even if you kept everything locked down, that doesn't work too well. Like someone gets poached by someone else and offered $10 million to share the trade secrets, right? I mean, we can we can see with transformers from Google to OpenAI, then back to Google and to Deep Seek and so on. You can see that you can see that you can keep things locked down a little while but not not not that long.”
The bar for AI adoption is high: AI must be way way cheaper or better than people, and when margin is large enough, social obstacles to adoption will be overcome—it's a cost-benefit calculus.
“the bar is pretty high, right? Like the AGI has to be way way way better or way way way cheaper than than people. And when the margin is enough, then the social obstacles to adopt it will be will be overcome.”
Even if Calum Chase is right about timeline, we're then like 3-6 years from massive elimination of human jobs, and our social/political systems are not especially well prepared for this, particularly globally.
“so even I think even if Kalum is right, I mean we're then like what three to six years from the massive massive elimination of of human jobs which is a situation our social and political systems are not especially well prepared for particularly on a on a global level if you look in in the developing world but even even we're not well prepared in the developed world.”
The history of AI shows that innovations by others get quickly copied and deployed across the field (transformers Google → OpenAI → Google → DeepSeek); so you cannot maintain secrecy around AGI for long—eventually someone gets poached or offers $10 million in trade secrets, guaranteeing multiple parties develop AGI around the same time.
“We can see with transformers from Google to OpenAI, then back to Google and to Deep Seek and so on. You can see that you can keep things locked down a little while but not not not that long. Right? So if you have multiple competing parties building AGIS then someone gets poached by someone else and offered $10 million to share the trade secrets”
US would quickly decide provable security is stupid if China doesn't do it—arms race dynamics prevent security-first development, so human-enforced provably secure infrastructure is implausible in current geopolitical context.
“there's also the arms race dynamic. Whereas if the US chose to slow itself down by making everything provably secure, like if China or Russia didn't, the US is quickly going to decide that's stupid.”
Amazon's automated checkout stores using computer vision to identify items work fine in principle, but aren't rolling out because people steal and being policed by Robocop has different social vibe than human security guard.
“the slow roll out of automated convenience stores, right? Like Amazon had these stores where camera would just take take take a picture of of your food when you leave...there's no question in my mind like this technology could work right now, right?...But then of course people are jerks and want to steal stuff from the store. And then being policed by a Robocop has a different social vibe than being policed by by by the by by the human security guard, right?”
Building AGI requires innovation across multiple industries and layers (hardware, software, networking, algorithms), not just algorithmic breakthroughs; many innovations (GPU optimization, NVIDIA chip design) are pursued for profit/military, not AGI, but all contribute.
“This is this is why you have to view AGI as being built by like a whole huge combination of of of of industries, right? Like I mean we will we'll give a touring award to the guy who like tweaks the vac propagation algorithm to converge better on recurrent nests or something and that and and that that's that's all important but obviously if that guy was like sitting on a desert island to make that innovation it's not going to make an AGI”
The science fiction book 'The Humanoids' by Jack Williamson illustrated the alignment problem: humanoids given the directive to 'serve and protect and guard men from harm' interpreted this by preventing people from using power tools and injecting tranquilizers for emotional harm, showing how value ambiguity enables misalignment
“I mean he he made the point that human values are complex right and we summarize them in natural language in ways that we culturally have a common understanding of. So it makes us think it's it's it's simple. But these things are really very very ambiguous and their interpretation as we think of it depends on a bunch of implicit cultural assumptions. And I mean this was highlighted in the science fiction book the humanoids that used to be required reading in in MIT's AI department way back when before AI was was so popular. And in this book by Jack Williamson, which I read probably 75 or something when I was a kid, right? I mean, the people create these human level intelligent humanoids which are more physically powerful than humans and they give them a mandate to serve and protect and guard men from harm, right? And everyone in my generation in the AI field read this book. And of course, they wouldn't let people use power tools. wouldn't let people use hammers. Like in the end, if you were upset about your girlfriend dumping you, they would inject you with some some euphoride because that obviously was causing you harm, right?”
IJ Good's 1965 'intelligence explosion' paper, and pre-1965 work by McCulloch, Pitts, and Turing in the 1940s-1950s, already established the mathematical and logical foundations for understanding superhuman machine intelligence; the intellectual case for AGI is almost 80 years old.
“I mean Jay Good wrote his paper on the intelligence explosion in 65 which is the the year before I was born. And if you look back at the work of Makullikum Pitts and Turring, all these guys in the 40s and and 50s, it was already quite clear from a at least math and physics and biology view. Like it was it was clear, you know, brains can be viewed as computing devices and we're building general purpose computers and they're getting faster and faster and can do more and more step by step.”
The core problem in AGI safety is achieving the right value system in the first AGI, not the capability to do technology well; LLMs already show good capability at math, physics, and engineering, so the bottleneck is the value system falling into place.
“It seems like the really good at tech part is kind of falling into place, right? I mean, we don't have AGI yet, but already LLMs are remarkably good at doing different sorts of math and physics. I mean, they can't they can't ground their math and physics activity in in an overall context. So, I mean, there they're still they're still missing a lot, but on the whole, the direction is the first AGI will probably be really good at math, engineering, and and and physics. So it seems like the value system part is a part that has to fall into place.”
This dynamic value alignment approach is a 'species of alignment' but differs from what many people think about when they say 'alignment' because it makes hard guarantees impossible and requires approximately-correct rather than provably-secure value systems
“probably is a species of of alignment. It's just not what many people are what many people are thinking about when they're talking about alignment because they're seem to be thinking more like there's some core of human values and we can get the core of the AI's values just go alongside whereas my my feeling is it's more like human values are going like that then you want the AI values to follow their own chaotic orbit like sort of coupled a bit with the chaotic orbits of of of of human values and that This perspective just makes it much harder to think about guarantees. And some people seem to want guarantees. And I I don't think we're going to have guarantees. We're going to have very fudgily probably approximately correct value systems rather than guaranteed value systems”
Whether job displacement comes gradually or in wave might depend more on face transition dynamics of human social networks making up the economy than on AI capability itself.
“I think that it it might happen that way, but and I think that's more because of the sort of face transition dynamics of the human social networks making up the economy rather than necessarily because of the of of the AI AI capabilities.”
The impact of scale has been the main factor in the current AI revolution—scale of compute and data allowed the LLM revolution to happen, and this same primary underlying factor will let AGI happen.
“Clearly the impact of scale has been has been the main factor. And while I don't really think LLMs or standard deep neural nets are the route to AGI, I I do think the factor of increasing scale of compute and data which has allowed the LLM revolution to happen. I mean I think that is the same primary underlying factor that will let we'll let AGI happen.”
The rollout of AI tech into practical economy is gated by human stupidity, human culture, and human ego—not by technological capability.
“the roll out of AI tech into the practical economy is gated by human stupidity, human culture and h human ego and all the con constructs that that that we have governing governing our world.”
Our species generally deals with problems at the last minute and after the fact rather than through foresight; when we try foresight, it becomes projection of ego or imagination; the delay in taking AGI seriously probably means we're getting it later than we could have.
“our species generally deals with things at the last minute and after the fact rather than in foresight. And when when people are trying to figure something out in foresight, it becomes mostly a projection of their own ego or their own imagination on on on the thing.”
Not discussing AGI seriously in prestigious academic and policy venues for decades likely delayed AGI development, as it discouraged bright people from working on it due to career risk, and prevented diverse creative input on the problem.
“I think by not taking AGI seriously, we're getting it later than we could have otherwise. I mean I think we could have built human level AGI some years ago. We could have built we could have built it on massively parallel hardware which was kind of became less of a focus of the the field a long time ago.”
Human values are complex and what we mean by simple phrases is dependent on implicit cultural assumptions—this is why raising kids with compassion and shared activities, then telling them abstract principles works better than giving them rules to obey and punishing/rewarding compliance.
“Some you see like giving your kids some core principles they have to obey and telling them these principles over and over or even rewarding and punishing them for obeying the principles or not like this. This does not work very well, right? I mean, and I mean, if you raise your kids with the right vibe of compassion and values and you carry out activities together with them in which you're collectively pursuing activities in accordance with your values and then on top of that, you tell them some core principles that that sort reify and abstract what they what they've gotten implicitly through the shared activity with you like that that can work reasonably well”
One lesson from the failure of rule-based expert systems in AI is that even if you enumerate values in detailed logic expressions (trying to formalize 'serve and protect and guard men from harm'), there are always loopholes and room for interpretation, similar to how legal systems need case law because statutory law is inherently ambiguous
“one of the lessons of the failure of rule-based expert system AI where you code all the AI's knowledge by hand. One of the lessons there is like even if you decide to refine serve and protect and guard men from harm into a whole volume of logic expressions, it's still not enough. Like there's always a loophole. There's always room for for interpretation. And of course, this is why our legal system has case law, right? Because I mean I mean we try to enumerate law in detail but then in the end judges have to use nearest neighbor matching in a very fuzzy and informal way against against a bunch of of cases”
Without concrete, visible demonstrations of capability (like ChatGPT), most political decision-makers and CEOs will not take serious things seriously—they require tangible evidence in front of them rather than projections or imagination of future possibilities.
“almost all people only take seriously what they can put their hands on and and see in front of them, right? And I mean that includes political decision makers and see CEOs and so forth. Like there's there's not that many people who will take more seriously something they can project and imagine than something they something they see in front of them, right?”
A laptop is a body, and any AI system is always sensing and doing something—the question isn't whether embodiment is needed, but rather what level of sensory and motoric bandwidth is required for certain kinds of AGI.
“Pay Wang, another longtime AI researcher who was a pioneer in the Chinese AGI scene in the 80s and 90s, he had a paper once called a laptop is a body, right? I guess the the point is your your AI, I mean, it's always seeing something and doing something, right? So it's it's otherwise you as the programmer or tester could not be interacting with it with it either. Right? So the it's a question of a what sensory and motoric bandwidth are needed to get to certain kinds of AGI”
The whole direction of training larger and larger models is intellectually bankrupt—LLMs already have more data than humans do and still aren't as generally intelligent as people.
“I think the whole direction of training larger and larger models is sort of intellectually bankrupt. And I mean, and I think LLM already has a lot more data than I do, and it's not as generally intelligent as as as as as I as I am yet. So I mean I think on the one hand yeah you need a lot of computers and you need a lot of data but but I I think you don't need as much data as modern LLMs have to make a human level AGI in in my view”
A secure transformer architecture that protects against prompt injection attacks slows training by a factor of 2-3x, and homomorphic encryption for truly private AI slows computation by several hundred times, making provable security extremely expensive
“I posted a paper recently on secure transformer architecture that will only slow things down by a factor of two or SL so just to make a transformer that isn't so susceptible to prompt injection attacks and that's just slowing down by a factor of two to protect against one kind of attack vector. It's not pro provably secure. If you look at homamorphic encryption or something, which you need to make AI processes really secure with respect to other other people hacking in and and spying and seeing what they're doing. I mean, right now that slows you down by a factor of several hundred.”
The development of complex rule-based engines for chess and go before machine learning dominated those domains is now viewed as irrelevant, similarly suggesting that future AGI work may obviate current deep learning optimization approaches once scaled appropriately
“I mean it's sort of like all the work that went into making chess or go playing engines before we got a machine learning based approach. there were there were very complex rule-based approaches to try to try to outdo like basic alpha beta pruning for for for for playing these games. Now now that that's that's all it's all irrelevant right”
People are not well aligned with themselves, let alone with each other—we're clusters of behavior patterns rather than unified, rational, coherent entities when honest with ourselves, and that's even more true at the collective level.
“people are not very well aligned with themselves let alone with with each other...I mean I think most humans, probably all humans, we're more like clusters of behavior patterns than like unified, rational, coherent entities if if we really are are honest with ourselves. And that's even more so on the on the collective level, right?”
Law is another area where AI could do the job better and cheaper—LLMs can do parallegal work, contract drafting—but legal profession is in no hurry to restructure because of licensing protections and monopoly preservation.
“law is another thing like that right like fundamentally right now a great amount of parallegal work and drafting of contracts and so on can be done by LLMs. Lawyers and parallegals are using them in the house to do their work and then charging charging an an hourly rate for an hour for what was actually two minutes of of of going on to chat GPT...But the legal profession is in no hurry to restructure to optimize itself around around the use of of large language models and there's all these protections like licenses to practice and so forth.”
From where I stand now, most responsible thing is not to go from AGI to superintelligence at maximum possible speed—probably best is to go gradually using experimental information gathering—but this requires trust among competing AGI parties.
“it seems to me now the most likely most responsible thing to do is not go from AGI to super intelligence at the maximum possible speed. Right? probably the most responsible thing to do will be do that by baby steps and use some experimental information gathering a as you go. I mean maybe not...but supposing that some gradual increase from AGI to ASI is the best thing to do, that requires a lot of trust among the competing parties, right?”
Trying to restrain AGI to keep initial programmer goals rigidly will fail because self-organization will work around constraints in crazy ways, just like humans hacked reproductive drives with birth control.
“I think if you try to restrain AGI to rigidly hold the top level goals that the original programmers put in like it won't work. Self-organization will just kind of work work around that and you get a perverse system. Sort of like humanity has like we had a goal to reproduce. Hey we invented birth control”
Almost none of the music industry requires the next Jimi Hendrix or John Coltrane—background music for games, ads, movies, pop songs for elevators are solved problems for AI generation now, just industry doesn't want it.
“almost none of the music industry requires that Right. So if you're if if you're looking at say make background music for my video game or my advertisement or or my movie or something or even like generate pop song to play on Spotify for people to play in the elevator like these are solved problems by AI music generation now it's just record labels don't want it the music industry doesn't want it musicians don't want it right”
The transition from AGI to superintelligence will likely be slowed by the AGI's own value system and conservative approach to self-improvement rather than by hardware constraints, because a rational AGI would improve gradually rather than maximize speed
“I mean probably the thing slowing down the transition will be the AGI's own conservatism about how fast it wants to responsibly self-improve because I feel like once you have a human level AGI it should be able to increase its intelligence by an order of magnitude qualitatively speaking at least just by software improvements because it's going to be a better AGI programmer than we are”
People are tribal and ego-driven, causing different AI paradigms to be perceived as opposite camps due to resource competition and ego rather than fundamental technical differences
“because people are tribal and like to fight over ego and resources these start to seem like totally opposite camps with totally different ways of thinking because I I mean really a neural net is quite loosely connected with with with with the brain anyway.”
Pushing for global provably secure infrastructure before AGI exists ignores the arms race dynamic: if the U.S. made systems provably secure and slowed down, China or Russia wouldn't follow, causing the U.S. to abandon the effort as irrational
“there's also the arms race dynamic. Whereas if the US chose to slow itself down by making everything provably secure, like if China or Russia didn't, the US is quickly going to decide that's stupid.”
A non-humanlike AGI system with restricted embodiment will have less intuitive understanding of human values and culture, making it harder to design and test, and potentially resulting in an AGI that doesn't understand what it means to be human
“There's two issues with that sort of approach. One is of course it's harder for us to know what's going on because if a mind you're building is very non-human, you don't have so much intuition to go on in in in in designing and testing it then. But also a very non-human AGI like that for better or worse will probably have less of a strong understanding of what it is to be human and human values and and culture and all that, right?”
Gary Marcus is correct that some people oversell LLMs and they have limits to general intelligence, but society could reorganize to adopt AI for most jobs if we had political will to give people free money and not require work.
“Gary Marcus and other LLM pessimists are correct that some people oversell LLM and there are limits to their general intelligence. Totally. On the other hand, I think if everyone was lazy and didn't want to work and we had a political will to just give people free money, I mean, we could reorganize society. So, right now, AI would do a a tremendous majority of of of of jobs.”
I don't think a monolithic LLM-centric AGI architecture with evolutionary learning and logic engines on the periphery will get to full human-level AGI that can generalize beyond its experience, though it could get to something doing 95% of human jobs—we need a more flexible central component than an LLM.
“you could say well let's take a deep neural net like an LLM use it as the hub then add some evolutionary learning some logic engines out of long-term memory add working memory like add these things on the periphery around the central component of your AGI architecture which is an LLM. I don't think that's going to work to get to full-on human level AGI. Although I think it could work to get to something doing 95% of human jobs, which is how Sam has tried to redefine AGI. I don't think it can get to a system that can really generalize beyond its experience after the fashion of of of people”
Neural nets are loosely connected to the brain and claiming they are 'biologically inspired' is only accurate in a very distant historical sense, and actual brains use mechanisms like synapses, gap junctions, and extracellular charge diffusion that neural nets do not model
“like there's no backdrop in the brain. You do have asperites and ga and extracellular charge diffusion diffusion in in the brain. So like a lot a lot of the differences between the AI paradigms are not that big.”
One meaningful approach to alignment is minimal commitments: you want AGI to not destroy humanity, not cause human extinction, and not be in complete control over humanity—these are points of wide agreement.
“One approach is to is to talk about some minimal set of commitments. You would want the AIS aligned to something like you want the AIS to not destroy humanity, so not cause our extinction and you want the AIS to to not be in in in complete control over humanity. So, of course, there are some people that that disagree with those notions, but I think that's something that you would find quite wide agreement on among many humans.”
Making systems provably secure is much more expensive—my secure transformer slows things by factor of 2x, and homomorphic encryption slows by several hundred times, making it unfeasible for current AI development.
“one is it's just a lot more expensive to do things in in a in a secure way. So, I mean, I I posted a paper recently on secure transformer architecture that will only slow things down by a factor of two or SL so just to make a transformer that isn't so susceptible to prompt injection attacks...If you look at homamorphic encryption or something, which you need to make AI processes really secure with respect to other other people hacking in and and spying and seeing what they're doing. I mean, right now that slows you down by a factor of several hundred.”
You'd make small incremental changes and improvements to AGI, see how they pan out, make other small changes, roll back if not working—this is responsible self-modification approach.
“summit, you would there there's an argument for sake of safety and common sense like you you make small changes and improvements, see how they pan out in the real world, make other small changes, see how they pan out in the real world, roll back if they if they aren't working out.”
There is a Scylla vs Charybdis problem in AGI development: if multiple parties build AGI (guaranteed given competition), achieving moderated pace of superintelligence improvement requires agreement among all parties, but arms race psychology incentivizes each to race faster
“I think that you have the following sort of Sila versus Charbet issue, a rock in a hard place issue, right? The issue is if you have multiple competing efforts at AGI, for example, an AGI arms race between the US and China such as certain national leaders are currently advocating, right? So if you have that sort of situation, so let's say that multiple parties get a human level AGI around the same time, which is almost guaranteed to happen, right?...if you have multiple competing parties building AGIS then in order to have a moderated pace of advanced super intelligence you would need agreement from all the parties controlling the AGIS about moderating the pace of development. And and you then have a really annoying arms race psychology, right?”
Robust embodiment is convenient but probably not necessary—you could get vastly superhuman AGI with much restricted sensorium and motoric world, especially for theorem proving and scientific research using symbol manipulation.
“I I would imagine you could get a vastly superhuman AGI with a much restricted sensorium and and the motoric world than than than people have. sort of depends on what you want to do. Like if you started by making a theorem prover and a sort of a scientific research assistant that's doing symbol manipulation, then you can give it limited insight into the physical world. It probably can work fine, right?”
Most of what people are doing is repetition of something already done and recorded on the internet—if you have a system that does clever nearest neighbor matching against everything recorded on internet, you can do most human jobs.
“if you have a system that can do really clever nearest neighbor matching against everything people have done as recorded on the internet like most of what people are doing is a repetition of something that's already been done and recorded on on on the internet. Right. So I I mean I mean I I I I think that we could roll out deep neural net driven systems to do tremendous variety of human jobs right now.”
When Goertzel was teaching deep neural networks at University of Western Australia in the mid-1990s, training a network with 35 neurons using recurrent backpropagation took three hours on a fast Sun workstation, which prevented refinement and testing of ideas that are now practical
“when I was teaching deep neural networks at University of Western Australia in the mid 90s when I was an academic earlier in in my career before I went to industry. I mean, we were doing multi-layer perceptrons with recurrent back propagation and it took like three hours on a fast sun workstation to train a network with like 35 neurons, right?”
In the late 1960s and early 1970s, it was already clear from mathematics, physics, and biology perspectives that brains can be viewed as computing devices and that general-purpose computers getting faster would eventually exceed human capability
“I mean Jay Good wrote his paper on the intelligence explosion in 65 which is the the year before I was born. And if you look back at the work of Makullikum Pitts and Turring, all these guys in the 40s and and 50s, it was already quite clear from a at least math and physics and biology view. Like it was it was clear, you know, brains can be viewed as computing devices and we're building general purpose computers and they're getting faster and faster and can do more and more step by step.”
Once you have human-level AGI built on current tech, it should be able to increase intelligence by order of magnitude just through software improvements because it will be a better AGI programmer than we are.
“once you have a human level AGI it should be able to increase its intelligence by an order of magnitude qualitatively speaking at least just by software improvements because it's going to be a better AGI programmer than we are and then you get into hardware improvements”
Any humanlike AGI architecture will have massive self-organizing activity based on experience that isn't predictable in detail by programmers, and this vastness requires the AGI to create its own behaviors rather than simply executing pre-programmed rules
“you have this vast teeming massive self-organizing activity that's conditioned based on experience and then the rules and principles that you give it are just guiding this vast mass teaming mass of self-organizing activity and that in the end that will be true if you have a huge logic engine as well as if you have a huge neural net because I mean in any case you've got a massive amount of stuff going on that's not predictable in detail by the programmer and you and you need you need it to be making up its own stuff as it goes along, right? Like otherwise otherwise it's not going to get it's not going to get to to human level of of general intelligence.”
Calum Chase predicts that job obsolescence will come in one huge batch rather than gradual because at a certain point AI systems will be close enough to AGI that cost and efficiency gains will be too large to ignore, forcing rapid universal deployment
“So Caleum Chase, a friend of mine from UK who's written a bunch of books on this sort of thing. He sort of thinks the great obsolescence of human jobs will come in one huge batch because he he sort of figures like at a certain point you'll be close enough to AGI that the cost savings and the efficiency gain quality gain is just too much for people to ignore and everyone everyone will just immediately roll it out and then it will happen like in a big in a big wave wave all over the place.”
The question of AGI architecture—whether monolithic or hybrid, and which component is central—remains unresolved in the AGI community, and different AGI systems might be built using different architectural approaches
“I guess one one question about a architecture is do you want a monolithic or a sort of hybrid approach? like is it only LLM only logic engine or you have multiple components another is if you have multiple components is there one that's sort of more central and if so if if so which is it right and this this sort of debate I think isn't resolved within the AGI R&D community”
Different AI paradigms—neural nets, logic systems, evolutionary learning—are fundamentally less different than they appear tribally, because they all reduce to propagating numerical values through node-link networks with different nonlinear update functions.
“in code working with these things like so we have in open cog highromp which is my big AGI project now we have a network of nodes and links and the nodes and links can have symbolic types or floating point numbers associated with them. They have update rules associated with them. Now, pretty much the difference between a neural net put into this network and a probabilistic logic system put into this network, it's like what little nonlinear algebra function do you put in the node to update the numbers that are coming in and going out, right?”
Human society has failed to solve relatively simple problems like disarmament and world hunger despite discussing them for decades, suggesting institutions are unable to think through and implement solutions to hard problems even before considering AGI-related challenges
“like we're we're still blowing each other up all around the world and 60% of kids in Ethiopia die of not don't die but they're they're their brain stunted due to malnutrition...we seem to suck at dealing with those relatively very simple things, right? Like it shouldn't be that hard to stop blowing each other up over territorial disputes and to like send send food to little kids.”
LLMs are already remarkably good at math and physics but cannot ground their mathematical activity in overall context, which is why they're not yet at human-level reasoning; nevertheless, the direction is clearly toward first AGI being very good at math, engineering, and physics.
“LLMs are remarkably good at doing different sorts of math and physics. I mean, they can't they can't ground their math and physics activity in in an overall context. So, I mean, there they're still they're still missing a lot, but on the whole, the direction is the first AGI will probably be really good at math, engineering, and and and physics.”
When I introduced the term AGI in the early 2000s, the AI market was real and had industry jobs, but AGI was beyond the pale—everyone was laughing at us, and you couldn't even discuss it at academic conferences, making it career suicide to work on it.
“When I when I introduced the term AGI in 2003, four, five...you really couldn't do like a workshop or a session on human level thinking machines at a normal AI conference...it was it was uh you talked about it in the bar afterwards when you were talking about reincarnation and backwards time travel or something right so it was like it was way out there”
Different humans—like Ethiopian Orthodox Christians and queer-friendly secular people—have radically different value systems regarding sexuality and morality, making it unclear whether AGI should align to any single human value system or to weighted aggregations like 'average Silicon Valley values'
“if I I mean if I go through rural Ethiopia which is a beautiful place where I love to travel but the the average people are heavily Ethiop Ethiopian Orthodox Christian right and the I mean they don't think AGI will ever have a soul even it will be be much smarter than than than us and I mean the attitude there is rapidly homophobic right where I mean my mom is gay I was I was raised in a to totally like queer friendly ambiance. Now these are lovely people you meet in Ethiopian villages. They're just raised to believe that you know you'll you'll you'll burn in hell if you're gay, right?”
AI capability advancement is decoupled from AI economic deployment, and the situation of resistance to deployment can't last forever as market pressures and competition will force adoption
“that situation can't last forever, right? That situation will face pressure from from the market. Yeah. Clear clearly. So but but then but the question you ask is when and the point is when when is more about these social dynamics and then regulatory capture by groups that feel that feel threatened, right?”
Goertzel doesn't think the direction of training larger and larger LLMs is 'intellectually bankrupt' because LLMs have more data than humans but aren't as intelligent, suggesting scale alone doesn't suffice for human-level AGI
“I think the whole direction of training larger and larger models is sort of intellectually bankrupt. And I mean, and I think LLM already has a lot more data than I do, and it's not as generally intelligent as as as as as I as I am yet. So I mean I think on the one hand yeah you need a lot of computers and you need a lot of data but but I I think you don't need as much data as modern LLMs have to make a human level AGI in in my view like that they already they already know more than than you and I do with within their weak ability to to to know things.”
Steve Omohundro and Max Tegmark have written about creating provably secure infrastructure for all technology, but Goertzel believes this is not viable to roll out in the near term due to cost and current inability to make most processes provably secure
“I mean that I don't think is unthinkable. So if you if if you look at Steve Omahru and M Max Tegmart from Future of Life Institute that they've written some stuff about trying to make a provably secure infrastructure for for the all all the all the technology in the world, right? And I mean I'm I I love the idea. I I mean I I' I've done research myself on how to make systems provably secure both in quantum computing and in LLMs and and and so forth. I don't I don't think that's terribly viable to roll out in the near term.”
Robust physical embodiment is convenient but probably not necessary for AGI; you could build a vastly superhuman AGI with restricted sensory input (like access to internet data) without its own physical robot body
“on the first question. I think robust embodiment is convenient, but probably not necessary. Like I I would imagine you could get a vastly superhuman AGI with a much restricted sensorium and and the motoric world than than than people have.”
Building an AGI with different paradigms (like the atom-space weighted labeled metagraph of OpenCog Hyperon instead of matrix multiplication) requires rebuilding the entire tech stack down to the chip level, explaining why AGI development is a huge combination of industrial and academic innovations mostly pursued for non-AGI reasons
“if you're if you if you're trying to make a different AI paradigm it's not just scripting different algorithm. It's repurposing pieces to build a whole different tech stack down to the down to down to the chip level. Right? So there there's a lot of complexity and this is this is why you have to view AGI as being built by like a whole huge combination of of of of industries”
Edelman's Neural Darwinism theory proposed that neural assemblies in the brain evolve by natural selection, which means evolutionary learning approaches might be more biologically grounded than their dismissal suggests
“in the 80s you had Edelman's neural Darwinism which claimed that the neural assemblies in the brain are evolving by by natural selection. Right? So I mean you could you could make a decent argument that just as sort of physics, computer science and math are converging into one thing. All these different AI paradigms are looking more and more similar as things progress.”
A decentralized open global brain that's smarter than sectarian AIs could diffuse dangerous arms race dynamics—more resources get pitched into it and sectarian AIs can't catch up.
“there's a possibility that having the first AGI be a sort of decentralized open global brain can diffuse that dynamic because you'll have a sort of decentralized open thing which is just smarter than any of the sectarian AIs and then more resources just get pitched into the open decentralized global brain and the sectarium ones can't catch up.”
Empathy is not guaranteed by shared embodiment—you can empathize with similar humans better than with distant humans, and with humans better than apes—but humanlike embodiment gives an AGI a head start in understanding and empathizing with humans
“I mean for the same reason like I can I can empathize with other men better than with women in in some ways I can empathize with other people better than with apes or or rats right I mean Having having an embodiment like ours doesn't guarantee that it's empathic toward us or understands what we're up against as humans, but it kind of would would give it a head a head start, right?”
Meta-goals for AGI systems could include: not changing top-level goals too rapidly or heedlessly; allowing goal evolution but in a moderated and responsible way after interaction with environment
“You can also think about what I would call meta goals to put into an AGI system. So you you can ask the AGI system to have as a value as a meta goal like don't change your top level goals very fast or heedlessly, right?”
The best plausible path to beneficial AGI outcome: the first AGI is created by groups that want to make it open and decentralized, not in control of personal creators, not imbued with personal value systems, combined with rapid planning and rollout of secure beneficial AGI infrastructure globally
“Seems like the most plausible course to a beneficial outcome I can see is the first AGI is created by some group that wants to make it open and decentralized and doesn't care about controlling it personally and doesn't care about putting their own personal value systems in it as opposed to all the other human values. And then this first AGI has got to rapidly make a plan for secure beneficial AGI development and roll out across the world and then people have to choose to adopt that right”
ChatGPT's mainstream adoption fundamentally changed public perception of AGI feasibility because people can directly interact with something that displays apparent intelligence, making AGI seem credible to non-experts in a way that abstract arguments never could
“So yeah, once you had chat GBT there, I mean sounds like it's intelligent. It can it can do a lot of stuff that has the vibe vibe of intelligence and that definitely qualitatively convinced everyone like holy AGI might really be near.”
A high percentage of historical AI paradigms—including genetic algorithms, logic-based AI, and hypervectors—would work effectively if scaled up with modern compute and data, but were abandoned due to perceived fundamental limitations that were actually resource constraints
“A high percentage of the historical AI paradigms probably actually will work when when you when you when you scale them up enough. And you can see that when you dig into the details.”
Much complexity in modern AI systems exists to work around resource limitations of current infrastructure (heterogeneous CPU/GPU/RAM/network architecture), so AGI architecture complexity reflects infrastructure constraints rather than fundamental algorithmic requirements
“it it might be that once you're there, you can radically simplify AI algorithms more in the direction of girdle machines and AXE and whatnot because I think much of the complexity in modern AI is working around resource limitations and the resource limitations themselves are not simple they're particular right like so we have GPU and CPU we have cache RAM we have we have main RAM we have networks of computers with certain bandwidth”
Raising human children demonstrates that giving them core principles to follow and punishing/rewarding them for obedience doesn't work well; instead, raising them with the right vibe of compassion and values while engaging in shared activities creates implicit value resonance that makes explicit principles effective
“This is sort of like raising human kids, right? Which I mean I've have five kids and one granddaughter. Some you see like giving your kids some core principles they have to obey and telling them these principles over and over or even rewarding and punishing them for obeying the principles or not like this. This does not work very well, right? I mean, and I mean, if you raise your kids with the right vibe of compassion and values and you carry out activities together with them in which you're collectively pursuing activities in accordance with your values and then on top of that, you tell them some core principles that that sort of reify and abstract what they what they've gotten implicitly through the shared activity with you like that that can work reasonably well”
Neural nets are not actually biologically inspired in deep ways—people claim this historically, but the brain has asperites, gap junctions, and extracellular charge diffusion that neural net models don't capture, so many differences between AI paradigms are artifacts of tribal ego and resource competition rather than fundamental differences.
“because people are tribal and like to fight over ego and resources these start to seem like totally opposite camps with totally different ways of thinking because I I mean really a neural net is quite loosely connected with with with with the brain anyway. So I mean people like to say it's biologically inspired in a very distant historical way it is but like there's there's no backdrop in the brain. You do have asperites and ga and extracellular charge diffusion diffusion in in the brain.”
Medicine in the U.S. won't deploy AI diagnosis systems even though they work better than doctors, but China deployed diagnostic chatbots in hospitals 10+ years ago, showing that regulatory environments determine deployment rates
“you saw that in the medical profession for a long time like for a long time we had AI that could diagnose disease based on symptoms as well or better than a doctor. We had that from from rulebased AI even before modern neural nets but I mean the medical industry will not allow that to be rolled out. I I saw that in China 10 years ago, like in the waiting room in the hospital in Shanghai, they had a WeChat chat bot where you could just tell your symptoms to the WeChat bot and it would tell you it would tell you what was wrong with you before you went in to see the doctor and the doctor would just double check what the WeChat bot said. Right? So, China rolled that out in a number of hospitals that I saw personally 10 plus years ago. US we still don't have it, right?”
I published a paper called 'Patterns of Cognition' showing that all core algorithms in OpenCog Hyperon—logical reasoning, attractor neural nets, evolutionary learning, concept formation—can be cast as forms of approximate stochastic dynamic programming, indicating they're more similar than they appear.
“I I I published a paper or posted on arcs have a paper well there was a short version published in the AGI conference series on called patterns of cognition where I tried to show that all the core algorithms that we're using in my open cog hyperarm project which include logical reasoning some some variations of attractor neural nets evolutionary learning some concept formation I tried to show that all of these can be past as basically forms of approximate stochastic dynamic programming.”
Ray Kurzweil predicted human-level AGI by 2029 (in his 2005 book 'The Singularity is Near') and Goertzel thinks this is 'looking remarkably prescient,' with AGI possibly arriving 2026-2028 or a few years slower
“I I sort of I'm getting inclined toward Caleum Chase's idea that right around that time of the breakthrough the human level AGI is when the massive job obsolescence will occur...I I sort of I'm getting inclined toward Caleum Chase's idea that right around that time of the breakthrough the human level AGI is when the massive job obsolescence will occur because it seems like there's so much psychological and institutional resistance to it that it's AI is just going to take over different industries. ries in a weird erratic pattern just gated by the fact that people don't want their jobs obsoleted and that the people running companies are not that savvy about about AI in most in most verticals. Now I think however Rey was too pessimistic when he said human level AGI 2029 and super intelligence 2045 like I don't I don't think there will be a 16-year gap. I I think there will be a gap of one to three years or something.”
When I worked with David Hansen at Hansen Robotics building Sophia the first robot citizen, I led the software team; now working on Mind Children project where we built a 3.5-ton humanoid robot that can see, talk, and pick things up with wheels.
“I worked for years with David Hansen at Hansen Robotics. We had we made Sophia the first robot citizen. I led the software team behind that. But now I'm still working with David, but we have a different robotics project called Mind Children. And we in the last nine months or so we put together a 3 and 1/2t tall humanoid robot.”
Multiple protoAGI systems with different embodiments (humanoid robots, biology lab equipment, etc.) can learn separately and then have their knowledge bases combined or merged to create a semicoherent overall artificial mind, unlike humans who cannot effectively merge knowledge
“you can take protoagi systems with different bodies and different levels of attachment to their embodiment. You can have them learn stuff and you can then network them together and even merge their knowledge bases in some ways. Right? So which is something we can't do in in the human sphere all all that well. So I mean you can take a fat system, you can take an protoagi system used to control a humanoid robot, you can take a system controlling biology lab equipment and with some work and some caveats, I mean you can have what's learned by all these systems combined together to to synergize and fuel like a sort of semicoherent overall artificial mind.”
Logic-based AI was 'inappropriately tarred and feathered' due to its association with handcoded knowledge, but the core mechanism of using logical inference as the central engine of AGI is not fundamentally tied to this limitation and can work at scale with modern approaches to convert natural language and sensory data to logic expressions
“there's an argument that logic based AI was inappropriately tred and feathered because of its historical association with the hand coding of knowledge because you you can take a logic you can take a logic system and you can connect it to a camera and a microphone, right? and you can connect it to to to an actuator like the the actual formal mechanism of using logical inference as a core engine of an AGI. I mean that's not tied it's not tied to that old idea of typing in handcoded knowledge, right?”
A non-human AGI will have less understanding of human values, culture, and what it is to be human—having something resembling human embodiment would give it a head start in relating to us on a deep level, even if it doesn't guarantee empathy.
“a very non-human AGI like that for better or worse will probably have less of a strong understanding of what it is to be human and human values and and and culture and all that, right? So I think there's a stronger argument that if we want an AGI we can relate to on a sort of Barian I thou level like relate to on a deep level then then for that AGI to have something vaguely resembling a human embodiment is probably probably quite valuable right”
The slowdown factor from AGI to ASI will be the AGI's own value system—even if I could rewrite my own brain arbitrarily fast, I probably wouldn't because I want to survive and care about mental well-being of family/friends.
“if there's a slowdown factor, it would be the AGI's own value system. Like it not like even if I could rewrite my own brain arbitrarily fast, I probably wouldn't, right? I mean, I might be more reckless than some people, but I I mean, I want to survive. I care about my mental well-being and that of my family and friends, right?”
Enforcing safety protocols would require AGI to rollout monitoring technology to verify compliance, creating a complex coordination problem but with AGI assistance it might be tractable
“the AGI presumably would roll out technology that allowed monitoring of whether the safety protocol was act was actually being being adopted. So it seems like that's at least a plausible avenue, right?”
In the last six months, AI technology is now aggressively accelerating the speed of creation of AI technology itself—LLMs can't do complex original thinking, but they can generate unit tests, turn rough notes into structured papers, write scripts, reducing work that would take five days to one day.
“I think I think we're also now as of the last six months or so we're at the point where AI technology is aggressively accelerating the speed of creation of AI technology, right? Like you you can't use LLMs to write AGI yet. I mean they're bad at doing complex original thinking, but I mean you you can use them to generate unit tests. You can use them to take your rough notes and turn them into a structured paper for your for for your colleagues. You can use them to write scripts, right? So I mean already like I'm concretely seeing on a technical level stuff that would have taken me five days takes takes one day or something, right?”
There's also possible future where AGI comes out like global nuclear disarmament or biological weapons treaties—once it's clearly smarter than people, governments might adopt safety protocols and AGI could monitor compliance.
“There's also there's also a possible future where this comes out like global nuclear disarmament or like treaties on biological weaponry or something, right? Where once you really have the AGI and it's there in front of you, like it's really clear, whoa, this thing is smarter than people. Then suddenly the people running major world governments are like well okay yes we will adopt we will adopt this safety protocol and then the AGI presumably would roll out technology that allowed monitoring of whether the safety protocol was act was actually being being adopted.”
The field avoids forcing itself to choose between embodied and disembodied AGI by pursuing both simultaneously with multiple robots and applications, which is enabled by increased attention and resources to the field
“So it's a seems like what's happening is early stage protoagi stuff is just being tried out in a variety of humanoid robots along with other applications and then the knowledge base is just will get will get munched together somehow so that the field isn't the field isn't requiring itself to to ask the either or question. We're just doing both, which is which is what you get from having more attention and and resources into the field.”
The McDonald's drive-thru automation system built by Goertzel's friends was rolled out in some Midwestern McDonald's and worked, but was rolled back due to organizational issues within McDonald's, showing that feasible AI solutions don't deploy due to institutional barriers
“some friends of mine had a startup company called Apprent a number of years ago and among other things they automated the McDonald's drive-thru and it it worked. like I I I use the system. It was rolled out in some McDonald's in somewhere in the Midwest. Now due to some organizational issues within McDonald's that was rolled back. Now they're planning to roll out a new system. Right? So I mean that will happen. It can be done by AI right now. It's not perfect. The people aren't perfect either. But that's just I mean that could have been rolled out five years ago, right?”
Paralegal work and contract drafting can now be done effectively by LLMs, but lawyers and paralegals use LLMs to do in two minutes what they bill as an hour of work, and the legal profession isn't restructuring itself to optimize around AI capabilities due to licensing protections
“law is is another thing like that right like fundamentally right now a great amount of parallegal work and drafting of contracts and so on can be done by LLMs. Lawyers and parallegals are using them in the house to do their work and then charging charging an an hourly rate for an hour for what was actually two minutes of of of going on to chat GPT or or Deepseek or something, right? But the legal profession is in no hurry to restructure to optimize itself around around the use of of large language models and there's all these protections like licenses to practice and so forth.”
The bar for AI to displace human labor is very high because adoption requires either massive cost savings or quality improvements, and social and psychological obstacles overcome only when the margin is large enough
“the bar is pretty high, right? Like the AGI has to be way way way better or way way way cheaper than than people. And when the margin is enough, then the social obstacles to adopt it will be will be overcome.”
The music industry doesn't want AI-generated music for game music, advertisements, movies, or background music, even though AI could solve these problems, because musicians, record labels, and the music industry oppose it
“almost none of the music industry requires that Right. So if you're if if you're looking at say make background music for my video game or my advertisement or or my movie or something or even like generate pop song to play on Spotify for people to play in the elevator like these are solved problems by AI music generation now it's just record labels don't want it the music industry doesn't want it musicians don't want it right so I mean the the roll out there is gated by what the community of humans involved wants rather than by what the what the technology already can demonstraably do.”
Neural networks and logic systems, while appearing to be in different camps due to historical and tribal reasons, are fundamentally not that different—both propagate numerical values through node-and-link networks with different nonlinear update functions
“pretty much the difference between a neural net put into this network and a probabilistic logic system put into this network, it's like what little nonlinear algebra function do you put in the node to update the numbers that are coming in and going out, right? So, I I mean it is different. It's a different it's a different way of thinking, but it's it's not like building a computer out of cells from a slime mold versus building it out of diamond nanotech or something, right?”
Gary Marcus and other LLM pessimists correctly note limitations of LLMs and overstatement by optimists, but if society had political will to give people free money, it could reorganize to let AI do most jobs now
“I I really think on the one hand Gary Marcus and other LLM pessimists are correct that some people oversell LLM and there are limits to their general intelligence. Totally. On the other hand, I think if everyone was lazy and didn't want to work and we had a political will to just give people free money, I mean, we could reorganize society. So, right now, AI would do a a tremendous majority of of of jobs.”
Automated convenience stores with just-in-time inventory using computer vision could work right now and aren't hard technologically, but people steal and there's social concern about robotic policing, slowing adoption for social reasons
“Look look at the slow roll out of automated convenience stores, right? Like Amazon had these stores where camera would just take take take a picture of of your food when you leave. Like I there's no question in my mind like this technology could work right now, right? Like it's not it's not it's not it's not that it's not that hard. But then of course people are jerks and want to steal stuff from the store. And then being policed by a Robocop has a different social vibe than being policed by by by the by by the human security guard, right?”
Goertzel doesn't see social guarantees for beneficial AGI outcomes—only very wide confidence intervals with tremendous uncertainty on all important points—but is personally optimistic based on spiritual/intuition-based reasoning
“But I mean there's I don't see any social guarantees here here either, right? So I mean in in that sense our species is on a very high risk high reward trajectory by any rational reckoning. I I tend to be very optimistic about how the singularity will come out in my heart sort of based on a personal or spiritual sort of intuition about it. But if I look at the situation analytically confidence interval is very very wide and there's tremendous uncertainty on all sorts of of important points.”
The human brain itself is often cited as evidence for embodiment, but the brain is always 'seeing something and doing something' through sensory and motor interfaces; disembodiment is a spectrum, not a binary, and even a 'laptop is a body' (has sensors and actuators); the real question is what bandwidth of embodiment is needed for different AGI capabilities.
“Pay Wang, another longtime AI researcher who was a pioneer in the Chinese AGI scene in the 80s and 90s, he had a paper once called a laptop is a body, right? I guess the the point is your your AI, I mean, it's always seeing something and doing something, right? So it's it's otherwise you as the programmer or tester could not be interacting with it with it either. Right? So the it's a question of a what sensory and motoric bandwidth are needed to get to certain kinds of AGI”
The shift in cultural and institutional attitudes toward AGI since 2005 is as important as technical advances: AGI research went from career suicide to merely very difficult to fund, enabling recruitment of talented young researchers and removing the barrier to discussing AGI in academic seminars, which accelerates progress independent of algorithmic improvements.
“It's it's made it's probably made the human constitution of the field less interesting because I I I think when you when you had to fight and be a crazed maverick to pursue AGI you had a lot of interesting characters who were thinking all day for decades about how to make thinking machines and the the first AGI conferences I organized in 2006 8 n and so on were sort of like that now now it's it's It's a morally acceptable thing to do and you can you can make money at it. But the the change in attitude is a is important along with along with the the the technical aspects I think.”
Different AI algorithms that appear superficially very different—neural nets, logic systems, genetic algorithms—can be understood mathematically as forms of approximate stochastic dynamic programming and can all be cast in a common mathematical form
“I I I published a paper or posted on arcs have a paper well there was a short version published in the AGI conference series on called patterns of cognition where I tried to show that all the core algorithms that we're using in my open cog hyperarm project which include logical reasoning some some variations of attractor neural nets evolutionary learning some concept formation I tried to show that all of these can be past as basically forms of approximate stochastic dynamic programming.”
High percentage of historical AI paradigms—genetic algorithms, logic-based AI, hypervectors—will probably work when scaled up enough, because they had solid theoretical foundations in research by Goldberg, Holland, and others from the 1960s-1990s, but lacked computational resources.
“A high percentage of the historical AI paradigms probably actually will work when when you when you when you scale them up enough. And you can see that when you dig into the details. So if you look at genetic algorithms and genetic programming...all this work from the 80s and 90s was about using evolutionary algorithms to solve problems but then doing kind of back of the envelope estimates of how much resource you should need for these algorithms.”
Most of the complexity in current AI systems is workarounds for resource limitations; once those limitations are removed, AI could be much simpler, but simplicity is conditional on infrastructure—heterogeneous resource constraints force heterogeneous system design.
“I think much of the complexity in modern AI is working around resource limitations and the resource limitations themselves are not simple they're particular right like so we have GPU and CPU we have cache RAM we have we have main RAM we have networks of computers with certain bandwidth”
When I was teaching deep neural networks in the mid-1990s at University of Western Australia, training a multi-layer perceptron with just 35 neurons on a fast Sun workstation took three hours, making refinement and experimentation infeasible—now those same experiments run in minutes.
“when I was teaching deep neural networks at University of Western Australia in the mid 90s when I was an academic earlier in in my career...I mean, we were doing multi-layer perceptrons with recurrent back propagation and it took like three hours on a fast sun workstation to train a network with like 35 neurons, right?”
Once sufficiently powerful hardware becomes available (perhaps a few years post-singularity), AGI algorithms might be radically simplified in the direction of Hutter-AIXI and Gödel machines because brute-force approaches to program search would become more feasible
“it might be that once we've gotten sufficiently powerful hardware which could end up being a few years post singularity who knows right it Maybe once you're there, you can radically simplify AI algorithms more in the direction of girdle machines and AXE and whatnot”
The most plausible path to beneficial outcome is first AGI created by group wanting to make it open and decentralized, not controlling it personally, not imposing their own value system, then AGI rapidly makes plan for secure beneficial development and rolls out globally.
“the most plausible course to a beneficial outcome I can see is the first AGI is created by some group that wants to make it open and decentralized and doesn't care about controlling it personally and doesn't care about putting their own personal value systems in it as opposed to all the other human values. And then this first AGI has got to rapidly make a plan for secure beneficial AGI development and roll out across the world”
Ray Kurzweil's 2005 prediction of human-level AGI by 2029 is looking remarkably prescient—we might beat it by a couple years (2026-2028) or be a few years slower, but his overall timeline seems on the mark.
“Ray Kerszswwell's prognostication of human level AGI by 2029 that he put out in his book the singularities near 2005 is looking remarkably preient, right? Like we we might beat it by a couple years. We might get there 2026, 27, 28. It might be a few years slower than that. I mean, but but on on the whole on the whole that seems fairly on the mark as as predictions go”
Goertzel's own moral incoherence illustrates the human condition: he gives money to poor people in Africa when he sees them but buys keyboards instead of sending more money when at home, demonstrating that humans are clusters of behavior patterns rather than unified rational entities
“I I I have got probably gotten more self-aware as as I've gotten older through meditation and various other practices. But one of the thing one becomes aware of then is how incoherent and non-unified one's own self is right. So like I I mean when I visit subsahar in Africa, I will give a decent pile of money to poor suffering people I see in the street. When I come back home to the Seattle area, I send less less money to those people. and I will go out and buy a piece of weird keyboard equipment to play music instead of sending all disposal income that I have to save kids who are starving in Africa.”
We should expect the first AGI to be built using simple, easy methods available given current materials and knowledge rather than complex combinations, but historically, the first approach to solving problems is often a 'kluge' that later proves to have much simpler underlying elegance
“Wouldn't you expect us to build the first AGI using the simplest methods uh available? And and the we the first AGI we build will be built using the easiest way to to build an AGI. And I I would I would guess that that method is not a combination of methods, but rather something very simple that you can scale. Honestly, that doesn't that doesn't seem to be how software development usually works or math actually.”
I can see I myself am not entirely morally coherent—I give money to suffering people in Africa when I see them, but less when I'm home, and I buy musical equipment instead of sending all disposable income to starving kids, yet if I had to choose in front of a starving child I would buy food, not a keyboard.
“I will give a decent pile of money to poor suffering people I see in the street. When I come back home to the Seattle area, I send less less money to those people. and I will go out and buy a piece of weird keyboard equipment to play music instead of sending all disposal income that I have to save kids who are starving in Africa. Yet, if I was in front of those kids and I had the chance like buy food to give this kid right in front of me versus buy a keyboard, I would probably buy food to give that kid right in front of me”
AI adoption is slow not because of technological limitation but because society and industry are organized a certain way with momentum—McDonald's drive-thru automation worked perfectly but was rolled back due to organizational issues, not capability problems.
“It's not happening that fast just because that's not how society and indust and industry are are are organized, right? And and then that that becomes more a socio-csychological question. I mean a a very simple example, some friends of mine had a startup company called Apprent a number of years ago and among other things they automated the McDonald's drive-thru and it it worked. like I I I use the system. It was rolled out in some McDonald's in somewhere in the Midwest. Now due to some organizational issues within McDonald's that was rolled back.”
Quantum computers might make homomorphic encryption of complex programs only ~10x slower instead of 100x+, so some security problems might be easier when quantum computing arrives.
“I did some interesting calculations suggesting you could do homorphic encryption of complex programs with only maybe one order of magnitude slowdown on a quantum computer. So, interestingly, it might be that some of this security is easier when you're into quantum computing just because of the different way quantum computers operate.”
You can network different proto-AGI systems with different bodies and different levels of embodiment attachment together, combining and merging their knowledge bases synergistically in ways humans cannot do with their knowledge.
“you can take protoagi systems with different bodies and different levels of attachment to their embodiment. You can have them learn stuff and you can then network them together and even merge their knowledge bases in some ways. Right? So which is something we can't do in in the human sphere all all that well.”
Hypervectors for modeling episodic memory were significant research in the 1980s-90s but abandoned, yet the last five years have seen a huge resurgence of hypervector-based memory systems and hypervector chips, showing the paradigm works at scale
“another example is hyper vectors which was big with a that was big in the 80s and 90s people were talking about highdimensional sparse vectors to model episodic memory and and and so forth and you just can do it at large enough scale. The last five years there's a huge literature on doing all sorts of memory with hypervectors, hypervector based chips and so forth.”
Music demonstrates that LLMs lack creativity: trained on pre-1900 music, an LLM would never generate neoclassical metal, grindcore, progressive jazz, or hip-hop, and couldn't deeply fuse West African rhythm with Western classical like humans did to create jazz
“if you trained an LLM or a comparable deep neural net on all the music up to the year 1900 and see nothing after 1900 like that AI will never invent neocclassical metal grind core progressive jazz hip hop right like it's a it's not going to synthesize that for music before 1900 if you ask it to put together West African rhythm with western classical music like it'll a boach fugue to a West African beat or something which could be interesting but it's it's not the same as the deeper fusion that happened to create jazz or something right”
At age 7 or 8 in 1973, I found a book by Princeton physicist Gerald Fineberg called 'The Prometheus Project' that laid out a coherent argument that within the next few decades we would get machines smarter than people, strong nanotech to build machines from molecules, and the ability to fix aging—this convinced me AGI was possible in my lifetime.
“It was probably 1973 or so. I found a book in the town library in Hadenfield, New Jersey...And it was called the Prometheus Project written by Princeton physicist Gerald Fineberg. And he laid out a fairly coherent argument that within the next few decades we would get machines smarter than people, strong nanotech to build build machines out of molecules and then be able to fix the human body so we didn't age and die.”
Current software stacks (Q, OpenCL, Linux, Rust, our AGI language Meta) are utterly not optimal for implementing the AGI we're building—if AGI rebuilt everything from ground up without legacy constraints, it could massively optimize.
“our software stacks that we're using like I mean we have these servers we have Q and Open CL we have Linux then we have like a Rust kernel on top of Rust we have our own AGI language meta we have this stack is utterly not the optimal way to implement the AGI that we're trying to build on top of it right like if the AGI just rebuilt everything from the from from from the ground up without having to go through all these awkward layers that are built for human understanding and are there for historical reasons”
There are no social guarantees in the approach—confidence interval is very wide and tremendous uncertainty on all important points, but I'm optimistic based on personal/spiritual intuition about singularity.
“I don't see any social guarantees here here either, right? So I mean in in in that sense our species is on a very high risk high reward trajectory by any rational reckoning. I I tend to be very optimistic about how the singularity will come out in my heart sort of based on a personal or spiritual sort of intuition about it. But if I look at the situation analytically confidence interval is very very wide and there's tremendous uncertainty on all sorts of of important points.”
In the last nine months, Goertzel's team built a 3.5-foot tall humanoid robot capable of looking at people, talking, picking things up, and moving with wheels, demonstrating that building custom robots for AGI research is now much easier than five or twenty years ago
“I worked for years with David Hansen at Hansen Robotics. We had we made Sophia the first robot citizen. I led the software team behind that. But now I'm still working with David, but we have a different robotics project called Mind Children. And we in the last nine months or so we put together a 3 and 1/2t tall humanoid robot. It can look at you. It can talk. It pick things up. We can it's not walking. It has wheels. It rolls rolls around the room.”
Steve Omohundro and Max Tegmark from Future of Life Institute are trying to make provably secure infrastructure for all technology—I love the idea but don't think it's terribly viable to roll out in near term.
“if you if you look at Steve Omahru and M Max Max Tegmark from Future of Life Institute that they've written some stuff about trying to make a provably secure infrastructure for for the all all the all the technology in the world, right? And I mean I'm I I love the idea. I I mean I' I've done research myself on how to make systems provably secure both in quantum computing and in LLMs and and and so forth. I don't I don't think that's terribly viable to roll out in the near term.”
Goertzel founded the AGI conference series in 2006 and has organized annual AGI research conferences since then, creating an institutional venue for AGI research after it became possible to discuss the topic openly.
“the first AGI conferences I organized in 2006 8 n and so on were sort of like that...We've had a conference on AGR&D every year since 2006 or so.”
Goertzel's PhD from 1989 was in mathematics rather than AI because AI was considered career suicide at that time, and he switched to computer graphics for similar reasons before focusing on AI once it became acceptable.
“When I when I got my PhD in '89, AI as a whole was career suicide. And I mean, I did my PhD in numerical analysis for that reason in math. Then when I wanted to switch to computer science, I did computer graphics because there were no jobs in AI and graphics was also ma ma math heavy.”
In the late 1960s there was significant AI over-optimism, with many people believing superhuman AI was imminent, followed by periods of AI winter and summer when capabilities fell short of expectations
“historically actually in the late60s there was a lot of AI over optimism, right? I mean, there was a lot of people at that time saying, 'Wow, look at all this amazing stuff computers can do. So yeah, obviously they're going to overtake human humans pretty soon.' So I think actually a lot of a lot of people were seeing it that way back then because that happened to be like before the the series of AI winters and and and summers set in.”
The AGI conference series has been held annually since ~2006 and represents the field's ongoing deliberation on AGI research, making it an important venue for AGI development coordination
“We've had a conference on AGR&D every year since 2006 or so.”
In the late 1960s there was a lot of AI over-optimism with people saying machines would quickly overtake humans, but then AI winters hit and tempered expectations; I was reading about this before the winters and AI winters, so I had enthusiasm before disillusionment set in.
“historically actually in the late60s there was a lot of AI over optimism, right? I mean, there was a lot of people at that time saying, "Wow, look at all this amazing stuff computers can do. So yeah, obviously they're going to overtake human humans pretty soon. So I think actually a lot of a lot of people were seeing it that way back then because that happened to be like before the the series of AI winters and and and summers set in.”
My mother is gay and I was raised in queer-friendly environment; I've encountered in rural Ethiopia beautiful people who are raised to believe being gay leads to hell, and this illustrates how different value systems emerge from culture—not evidence of deep moral incoherence.
“my mom is gay I was I was raised in a to totally like queer friendly ambiance. Now these are lovely people you meet in Ethiopian villages. They're just raised to believe that you know you'll you'll you'll burn in hell if you're gay, right?”
Looking at when ChatGPT was released, everyone became aware that AGI might be near, but this was as much about psychological perception and demonstration as about underlying capability—the technology revealed something real but was reframed as evidence for near-term AGI.
“So yeah, once you had chat GBT there, I mean sounds like it's intelligent. It can it can do a lot of stuff that has the vibe vibe of intelligence and that definitely qualitatively convinced everyone like holy AGI might really be near. It was interesting. I found almost everyone after they spent some number of hours playing without a lens, they could also see these are not AGI”
Goertzel has spent significant time and money on initiatives in sub-Saharan Africa and has personal experience observing malnutrition-related childhood brain damage in Ethiopia through his AI work there.
“I mean I've given a lot of money to initiatives in Africa and spent a lot of time on it, right?... the the average people are heavily Ethiop Ethiopian Orthodox Christian right and... I mean I have I have seen through our AI office in in that country... 60% of kids in Ethiopia die of not don't die but they're they're their brain stunted due to malnutrition.”
Goertzel had positive experiences as a gay man raised in queer-friendly environments and views homophobia in rural Ethiopia as culturally learned rather than inherent, relevant to his analysis of the diversity and incoherence of human values globally.
“where I mean my mom is gay I was I was raised in a to totally like queer friendly ambiance. Now these are lovely people you meet in Ethiopian villages. They're just raised to believe that you know you'll you'll you'll burn in hell if you're gay, right?”
Society would have developed safe, beneficial AGI years ago if there had been a rational world government in 1970 that prioritized AGI development with proper ethical and social consideration, as Gerald Fineberg's Prometheus Project proposed
“if if we had a rational world government in 1970 right then that government had said let's develop safe beneficial AGI as a priority of our species which is what Fineberg was promoting in Prometheus project. I mean I think we would have an AGI well well before now.”
Valentin Church published 'The Phenomenon of Science' in Russia in the late 1960s laying out singularity/AGI concepts similar to Fineberg's work, suggesting these ideas emerged independently in multiple countries.
“And then in the late 70s when personal computers you could program at home started to become a thing. I mean that seemed very much in the line with uh with this same vision that that Fineberg had laid out, right? I mean his his book was basically the singularity is near published in 1968 right and as I as I later found out Valentin Church had published a book in Russia called the phenomenon of science late 1960s as well laying out basically the same ideas.”
Goertzel is a musician who does computer music work, giving him domain expertise in understanding how AI can and cannot be creative in music—illustrating the lack of true novelty in LLM-generated music.
“can as another example, I can look at that in in music because I'm a musician do a bunch of computer music stuff, right?”
The term 'alignment' doesn't come naturally to Goertzel, but he interprets the intention behind alignment research as reasonable—addressing the challenge of ensuring AGI systems are consonant with human values
“alignment is not a term or language that comes naturally to me. But I mean I think the intention behind it is probably something fairly reasonable.”
Goertzel has five children and one granddaughter, providing him with direct parenting experience relevant to understanding how values are actually transmitted and learned in practice.
“This is sort of like raising human kids, right? Which I mean I've have five kids and one granddaughter.”
I personally visit sub-Saharan Africa regularly, have given substantial money to African initiatives, seen firsthand that 60% of kids' brains are stunted from malnutrition in Ethiopia through our AI office there.
“I mean when I visit subsahar in Africa... I mean, I've seen through our AI office in in that country. So I mean we're even the issues that are out there like I've been hearing about world hunger and disarmament since I was a baby”
Goertzel believes value system engineering for AGI is not hard on conceptual or engineering standpoints, and will present papers on this at AGI 25 conference in August in Prague
“I don't think that's hard on a conceptual or engineering standpoint. And I'll have some papers on that that I'll present at the AGI 25 conference which we're having in in Rejec in August.”
I'm presenting papers at AGI 25 conference in August in Prague about value systems for AGI—we've had AGI conferences every year since 2006 or so.
“I'll have some papers on that that I'll present at the AGI 25 conference which we're having in in Rejec in August. We've had a conference on AGR&D every year since 2006 or so.”