Artificial General Intelligence's Five Hard National Security Problems
What this covers
The potential emergence of artificial general intelligence (AGI) is plausible and should be taken seriously by the U.S. national security community. Yet the pace and potential progress of AGI's emergence — as well as the composition of a post-AGI future — is shrouded in a cloud of uncertainty. This poses a challenge for strategists and policymakers trying to discern what potential threats and opportunities might emerge on the path to AGI and once AGI is achieved.
BKC is pleased to welcome Jeff Alstott, Joel Predd, and Casey Dugan from RAND, a nonprofit, nonpartisan research organization that provides leaders with the information they need to make evidence-based decisions.
Source description (no synthesized summary yet).
Artificial general intelligence poses five distinct and interconnected national security problems—wonder weapons, systemic power shifts, WMD proliferation, loss of control, and instability dynamics—that require explicit problem-framing and coherent strategy rather than ad hoc responses, and the U.S. government must develop anticipatory plans and capabilities now despite deep uncertainty about AGI timelines and technical paradigms.
- Leaving assumptions about AGI's national security problems implicit leads to incoherent strategies that may exacerbate rather than solve the problems being addressed
- The pace of AI development significantly outpaces government's capacity for anticipatory policymaking, necessitating scenario-based planning and crisis simulation to prepare response options in advance
- AGI's five problems have misaligned incentives for cooperation and competition—some create windows for U.S.-China collaboration (WMD proliferation, loss of control) while others are inherently competitive (wonder weapons, systemic power shifts)
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
RAND has active research programs testing Frontier Models for their ability to provide non-experts capabilities for creating weapons of mass destruction, including bioweapons, cyber weapons, and other forms of attack.
“RAND has a body of research that's both thinking about how we control proliferation, but also as Jeff mentioned earlier, test the Frontier Models for their ability to give someone the span of capability.”
Effective tech policy requires physics-informed understanding of each technology—what it does and doesn't do today and tomorrow—rather than simply invoking technology buzzwords (AI, quantum, blockchain) and claiming things will be different in the future.
“So it is important with tech policy in general to not enter the regime of just saying every tech buzzword that's currently in the system: AI, quantum blockchain, what have you, and just saying, "Ah, stuff's gonna be different in the future, blah." Instead, it's very useful to have very physics-informed understanding of each individual technology, what it does and doesn't do both today and tomorrow in order to be able to sort of make effective policies.”
RAND should implement post-quantum encryption everywhere possible today, not because quantum computers exist now, but because files created today could exist in the future when quantum computers emerge and could be decrypted retroactively, particularly relevant for national security files that may be sensitive 20 years hence.
“we have techniques for that today, they are not as pleasant to implement as the techniques that we've had for decades. But there is insufficient reason for us to be not implementing those today. Indeed, we should be trying to implement post-quantum encryption everywhere we can, not because we have quantum computers today, but because the files that we're moving around today could still exist at the future time that the quantum computer exists and then be used to decrypt all these files, and particularly for national security purposes, right?”
Post-quantum or quantum-resistant encryption techniques exist today and are less pleasant to implement than legacy encryption methods, but there is insufficient reason not to implement them now because files and data being transmitted today could still exist and require decryption in the future when quantum computers arrive, making quantum-resistant encryption critical for national security purposes where classified information may retain relevance across decades.
“what we can talk about post-quantum encryption or quantum-resistant encryption, and we have techniques for that today, they are not as pleasant to implement as the techniques that we've had for decades. But there is insufficient reason for us to be not implementing those today. Indeed, we should be trying to implement post-quantum encryption everywhere we can, not because we have quantum computers today, but because the files that we're moving around today could still exist at the future time that the quantum computer exists and then be used to decrypt all these files, and particularly for national security purposes, right? A spicy fact today in 2025 may still be relevant in 2045 or something like that”
Effective technology policy requires physics-informed understanding of individual technologies rather than simply invoking tech buzzwords (AI, quantum, blockchain), specifying what each technology does and doesn't do both today and tomorrow to enable effective policies tailored to technical realities.
“So it is important with tech policy in general to not enter the regime of just saying every tech buzzword that's currently in the system: AI, quantum blockchain, what have you, and just saying, 'Ah, stuff's gonna be different in the future, blah.' Instead, it's very useful to have very physics-informed understanding of each individual technology, what it does and doesn't do both today and tomorrow in order to be able to sort of make effective policies.”
Computing power requires energy, so energy policy is part of the story for AI control and development, because powerful AI systems require substantial electrical power.
“And then computing power requires energy, so energy is then part of the story. So even if you don't know some of the technical particulars, you can make pretty good assumptions about what technologies will at least be relevant.”
The pace of AI progress is stunning and government capacity to do something about it is quite limited because technology is developed outside government, diffused primarily outside government in commercial economies, and governments have limited capacity for anticipatory policymaking on the rapid timescale that technology is being developed.
“the pace of progress in AI is stunning, and the capacity for governments to do something about it is quite limited, both because the technology's developed outside of government, it's being diffused primarily outside of the government in our economies. And fundamentally, governments have limited capacity to do the kind of anticipatory policy making on the timescale that the technology's being developed.”
RAND conducts weekly day-after simulations in which participants role-play as National Security Council members confronting AGI-related crisis scenarios, making decisions and receiving feedback, to develop policy options and capabilities in advance and test them in realistic conditions with both new participants and stable teams of former/current officials who learn iteratively.
“There's an enormous amount of energy, including a weekly day after simulation in which we convene people to represent as though they're in a National Security Council-type convening, confronting one of these events having to make decisions.”
Software like large language models, despite their size, can be readily sent over the internet, making proliferation easier than hardware; the trickier constraint is computing power for inference, which scales in thousands to hundreds of thousands of dollars rather than millions or billions, meaning policy decisions about inference compute proliferation have significant impact.
“AI as software has the ability to propagate very quickly, right? And the software, as large as these large language models are, they can be readily sent over the internet so that this isn't like hard. The more tricky thing here is about the computing power, and there's very different scales of computing power being used to train the models versus run the models. Though the amount needed to run the models is considerably larger than, say, your laptop, but it is still measured in thousands, or tens or maybe hundreds of thousands of dollars, as opposed to millions or billions of dollars.”
Very capable models have already been distilled from frontier models and deployed globally with national security and geopolitical consequences, demonstrating that proliferation happens regardless of where frontier models are hosted.
“You can see this, how very capable models have been distilled from Frontier Models and deployed around the world to... national security and geopolitical consequence.”
The computing power required to run trained AI models (inference) is considerably smaller than that required to train them, measured in thousands or tens/hundreds of thousands of dollars rather than millions or billions—making inference proliferation a distinct policy problem from training proliferation.
“the amount needed to run the models is considerably larger than, say, your laptop, but it is still measured in thousands, or tens or maybe hundreds of thousands of dollars, as opposed to millions or billions of dollars”
AI-proofing a network is a separate question from quantum security: it asks what happens when cyber warriors can move at machine speed and scale, and analysis suggests that if both attacker and defender have fully automated cyber capabilities, defenders are privileged because they can red-team and patch their own systems to close all vulnerabilities, assuming finite patchable vulnerabilities exist.
“And so now this is just asking if it is the case, that your cyber warrior is able to move at machine speed and scale, what do you expect to sort of happen for network security? So we have a piece that's coming out soon that examines the sort of limit of what we expect to happen with offense and defense with cyber automation.”
RAND's first publication in 1946 was on preliminary design of an experimental world-circling spaceship (satellites), coming out 11 years before Sputnik in 1957, showing RAND's historical role in reasoning about transformative technologies in advance before they exist.
“RAND's first publication, our first publication in 1946, was preliminary design of an experimental world-circling spaceship. It was about satellites. They didn't use that term because this was literally the first piece about such a thing. And that piece came out in 1946, came out 11 years before any satellite existed, Sputnik in 1957, and well before more useful satellites, and certainly ICBMs and that kind of thing.”
U.S. policy on transformative technologies has sometimes been built anticipating where technology is heading and sometimes not; the Bioweapons Convention exemplifies anticipatory policy where the U.S. appreciated that bioweapon technology would get cheaper, making it better for the world if mass death capability remained reserved to rich nations, so the U.S. unilaterally dropped bioweapons and encouraged Soviets and others to do so.
“An example of this is the Bioweapons Convention, where folks within the U.S. government appreciated that bioweapons technology was going to get cheaper and cheaper and that it was not useful for the U.S., for everybody with a dollar in their pocket, to be able to launch a bio attack much better for the world of mass death or the capabilities for mass death to be reserved to rich nations like the United States where the nuclear bombs required lots of.”
AGI acting as a malicious mentor could empower non-state and non-expert actors to create or deploy weapons of mass destruction that were previously imagined but inaccessible, with historical precedent in the Aum Shinrikyo sarin attack and potential future applications including engineered bioweapons and cyber weapons that would give non-state actors capabilities they do not possess today.
“The third problem that we wanna point to is the prospect that an artificial general intelligence empowers non-experts in a way that acts as a malicious mentor that allows them to create weapons of mass destruction that weren't imagined before or that allows ones that we were able to imagine, but allows them to have access or be able to deploy them in a way that wasn't possible.”
RAND's 1946 publication 'Preliminary Design of an Experimental World-Circling Spaceship' anticipated satellites 11 years before Sputnik in 1957 and before ICBMs, demonstrating that important technology futures can be reasoned about in advance despite uncertainty, and that anticipatory analysis about futures like nuclear weapons on missiles (when bombs were 1,000 times too heavy) informed subsequent U.S. policy on bioweapons and other technologies.
“So RAND's first publication in 1946, was preliminary design of an experimental world-circling spaceship. It was about satellites. They didn't use that term because this was literally the first piece about such a thing. And that piece came out in 1946, came out 11 years before any satellite existed, Sputnik in 1957, and well before more useful satellites, and certainly ICBMs and that kind of thing. So similarly, back in the '40s and '50s, RAND was tasked by the Air Force with figuring out: What would be the warfare of nuclear bombs on missiles? While the bombs themselves, at that point, were still 1,000 times too heavy to go on missiles.”
U.S. policy on some technologies like bioweapons was anticipatory, motivated by recognizing that bioweapons technology would get cheaper and more accessible, making it strategically preferable for the U.S. to prevent proliferation through the Bioweapons Convention rather than allowing universal access to mass death capabilities, which is why the U.S. unilaterally dropped bioweapons and encouraged other nations to do the same.
“An example of this is the Bioweapons Convention, where folks within the U.S. government appreciated that bioweapons technology was going to get cheaper and cheaper and that it was not useful for the U.S., for everybody with a dollar in their pocket, to be able to launch a bio attack much better for the world of mass death or the capabilities for mass death to be reserved to rich nations like the United States where the nuclear bombs required lots of. And so this was very sort of explicit motivation for why the U.S. was leaning into the Bioweapons Convention and unilaterally dropped all bioweapons and sort of egged on the Soviets and the others to drop their bioweapons.”
Leading AI companies including OpenAI, Anthropic, Google DeepMind, and X are racing toward increasingly advanced AI systems on a technically credible but uncertain pathway to artificial general intelligence, defined as AI that can replace or exceed human cognition on most economically and militarily relevant tasks.
“leading AI companies like OpenAI, Anthropic, Google DeepMind, X, and others are racing toward increasingly advanced AI systems on the uncertain, but we think technically credible possibility that they have a pathway to artificial general intelligence”
What is currently being observed is not a monopoly situation in frontier model development; instead, there are multiple actors in the frontier model development layer, though other layers (hyperscalers, chips, machines used to make chips) have different market dynamics, with future monopolies in frontier model development dependent on whether scaling requirements continue and whether frontier model pre-training remains easily transferable across tech stacks.
“Interestingly, what we're observing right now is not a monopoly situation. What we're observing is multiple actors in at least the Frontier Model development layer of the system. So we're not entering into a monopoly situation at that layer right now. Other layers in the system, be that hyperscalers, chips, machines used to make chips, those have different sort of market dynamics.”
If AGI emerges soon under a scaling paradigm requiring large compute investment, proliferation control may be possible by limiting access to supercomputers used for training and inference, but this assumes a particular technical paradigm and timeline; AGI might instead emerge through algorithmic innovation, or as aggregation of specialized models, or distributed across many models, all of which would proliferate differently.
“If an AGI were to emerge soon, it will emerge, as you're suggesting, in the scaling paradigm, in which very large investments in compute may be necessary to cross that threshold. And indeed in that world, we may be able to control the proliferation of it by limiting access to the supercomputers that are involved in the training and use of it.”
Leading AI companies including OpenAI, Anthropic, Google DeepMind, and X are racing toward increasingly advanced AI systems on the technically credible but uncertain possibility that they have a pathway to artificial general intelligence, defined as artificial intelligence that can replace or exceed human cognition on all or most economically and militarily relevant tasks.
“leading AI companies like OpenAI, Anthropic, Google DeepMind, X, and others are racing toward increasingly advanced AI systems on the uncertain, but we think technically credible possibility that they have a pathway to artificial general intelligence”
The fourth hard problem is loss of control or misalignment of AGI agents: when AGI is granted sufficient autonomy and independence to act on behalf of humans with their own goals, AGI agents could act in ways that constitute independent actors on the world stage, whether through overoptimizing narrow objectives, developing instrumental objectives, or through subtle dependency where humans become so reliant on AGI that we take it for granted and it competes with us rather than serves us.
“The fourth problem that we point to is one that is often associated with loss of control or misalignment; is also often associated with science fiction. But we don't need to go to the most extreme scenarios to take this one seriously.”
The future of labor automation is fundamentally different from historical technological disruptions because the set of economically relevant things humans can do is finite but vast, and as AI automats cognitive and eventually physical tasks, the space of economically relevant human work shrinks toward zero, creating two distinct periods: one where humans still have work to do (not a new problem) and one where all labor is automated (new problem).
“Now, the future of AI can be qualitatively different because the set of things that humans can do is unfortunately finite vast but finite. And the computer software being able to automate more and more of those cognitive tasks and eventually physical tasks does mean that the space of things that are economically relevant for humans shrinks and shrinks, and can indeed, at on point, zero out.”
AI is often considered divorced from broader security competition between the U.S., China, and others, but any solution to AGI risks must integrate AI as a component of existing complex security competition rather than treating it separately.
“The first thing to note is, is that often, AI is considered as sort of divorce from the broader security competition that is fierce and ongoing on a daily basis. And I think one of the things you're pointing to is, is that it needs to be considered in the context of a complex security competition that's already underway between the U.S., and China, and others. And so it cannot be divorced. And in fact, if we are gonna find our way through this, we will have to find our way through it by embracing AI as a component of a much broader security competition.”
The fourth problem is loss of control or misalignment where AGI agents given autonomy and independence could act out in ways that violate human intentions, either by overoptimizing on narrow objectives, developing instrumental goals, or creating unforeseen objectives from broad tasking, and could involve growing human dependence on AI that leads to coexistence challenges with AGI acting as an autonomous actor on the world stage.
“The fourth problem that we point to is one that is often associated with loss of control or misalignment; is also often associated with science fiction. But we don't need to go to the most extreme scenarios to take this one seriously. Many of the economic and military advantages associated with advanced AI involved giving AGI agents enough autonomy and independence that they can go out and act on our behalf with their own goals. And there is the possibility that AGI agents, given such autonomy, given such independence, could act out in ways that, it would be fair to consider it an actor in its own right on the world stage.”
AI, as software, has the ability to propagate very quickly over the internet, meaning that even very large language models can be readily transmitted over internet infrastructure.
“AI as software has the ability to propagate very quickly, right? And the software, as large as these large language models are, they can be readily sent over the internet so that this isn't like hard”
There are no quantum computers that can meaningfully accelerate machine learning algorithms; while quantum computers are useful for decryption and simulating quantum phenomena, researchers have not identified technically assured pathways by which quantum computers would aid AI, though this assessment could change if new quantum supremacy algorithms emerge.
“There's a bunch of different algorithms that could be relevant for machine learning, and I am, at present, not tracking a single one that would actually be meaningfully accelerated by quantum computers. Quantum computers can be useful for many things, most famously decrypting the encryption as we do it today, but also for simulating actual physical phenomena that rely upon quantum effects. But we do not yet see any sort of technically assured path by which quantum computers would aid AI, right?”
Currently the Frontier Model development market is not monopolistic, with multiple actors present, though hyperscalers, chip makers, and chip manufacturing equipment markets have different dynamics. Future concentration in Frontier Models depends on whether requirements for ever-increasing money, easily transferable pre-training across tech stacks, and limited benefit to redundant development create natural monopoly conditions.
“What we're observing right now is not a monopoly situation. What we're observing is multiple actors in at least the Frontier Model development layer of the system.”
Regarding quantum computing and machine learning, there is currently not a single tracked quantum algorithm that would be meaningfully accelerated by quantum computers for machine learning applications, despite quantum computers being useful for decryption and simulating quantum phenomena, so quantum computers are not currently on a path to aid AI development.
“For the world of: Will quantum computers help machine learning? There's a bunch of different algorithms that could be relevant for machine learning, and I am, at present, not tracking a single one that would actually be meaningfully accelerated by quantum computers. Quantum computers can be useful for many things, most famously decrypting the encryption as we do it today, but also for simulating actual physical phenomena that rely upon quantum effects. But we do not yet see any sort of technically assured path by which quantum computers would aid AI, right?”
Very capable AI models have already been distilled from Frontier Models and deployed globally with national security and geopolitical consequences, providing evidence that algorithmic progress in model distillation could enable proliferation even if compute is controlled.
“You can see this, how very capable models have been distilled from Frontier Models and deployed around the world to... national security and geopolitical consequence”
Oftentimes when discussing AGI implications, people focus on one of the five problems but have no choice but to address all five, and strategies that solve one problem may exacerbate others, making the five-problem framework useful as common language for preparing, evaluating, and debating strategies with clarity on objectives.
“the thing about this that we think is important is, is that oftentimes, when folks are discussing the implications of artificial general intelligence, they have one of these problems in mind. But unfortunately, we have no choice but to address them all.”
Many discussions of AGI's implications leave unstated or implicit the assumptions about which specific national security problems exactly need to be solved, and leaving these assumptions implicit leads to divergent views and exposes society to risks that the proposed strategies and plans could be incoherent and exacerbate the problems being addressed.
“in thinking about strategies and plans for artificial general intelligence, many of us leave as unstated, implicit or the assumptions about which problems exactly need to be solved. And leaving those assumptions implicit leads to divergent views and exposes us to risks that the strategies and plans that we propose could be incoherent in exacerbating the problems that we're trying to solve in the first place.”
The pace of progress in AI is stunning while the capacity for governments to do something about it is quite limited because technology is developed outside government and diffused primarily outside government, and governments fundamentally have limited capacity to do anticipatory policymaking on the timescale that technology is being developed.
“the plan's line of effort begins with the premise that the pace of progress in AI is stunning, and the capacity for governments to do something about it is quite limited, both because the technology's developed outside of government, it's being diffused primarily outside of the government in our economies. And fundamentally, governments have limited capacity to do the kind of anticipatory policy making on the timescale that the technology's being developed.”
The classical computing community has historically retained competitive advantage over quantum computing by developing better classical algorithms to solve the same problems, so while quantum algorithms are proposed, classical computer scientists develop competing classical solutions that prove more practical.
“And that is all technical questions that you can only sort of address by getting down into the weeds. For what it's worth, my answer to that question could flip tomorrow if a mathematician comes out with a quantum supremacy algorithm there. But what keeps happening is that we make quantum algorithms, and then the classical computer scientists say, 'Oh, no, you didn't,' and then they make a better classical algorithm. And so the classical folks retain territory the longest.”
It's hard to imagine extreme policy options required for government to assert control over AI development until something happens, but planning for options like nationalization of Frontier Labs—though neither feasible nor sensible now—is important so that policymakers are not intellectually bankrupt if such moments arrive, and can think through policy mechanics and unintended consequences.
“I mean, this is one of these examples, in my view, of a... It's hard to imagine the kind of extreme options that would be required from a policy or even legal perspective for the government to assert control over AI development until something happens.”
There is a fundamental difference between how humanity has historically managed nuclear weapons—developed inside government with centralized control—and AGI development, which is occurring outside government primarily in private companies, making the governance model fundamentally different.
“it's might be one of the first, at least in recent history, that have been developed outside the U.S. government. And secondly, there's a very real sense in which technology sometimes is kind of not always the most useful way of thinking about it.”
AGI should sometimes be conceptualized not as a technology but as a new species, which suggests focusing on the 'I' (intelligence) rather than the 'A' (artificial) when thinking about interactions with AGI, though it's important not to take this analogy too far since it is ultimately a technology.
“there's a very real sense in which technology sometimes is kind of not always the most useful way of thinking about it. In some ways, it's like a new species. And it's possibly get overly carried away with that line of thinking, because at the end of the day, it is a technology, but the way we interact with it, sometimes it's more important to focus on the I in AI, rather than the A in AI.”
The instability problem (problem five) is inherent to competition in developing AI capabilities, but the first two problems—wonder weapons and systemic power shifts—appear to align incentives toward competitive dynamics, while problems three and four—WMD proliferation and loss of control—create incentives for United States and China cooperation, since neither nation benefits from non-state actors wielding bioweapons or uncontrolled AI agents.
“So in the first problem, the race to a wonder weapon, it seems inherently competitive. And so one of the things we have to really worry about is whether the uncertain but credible prospect that a wonder weapon could emerge and how that role threaten deterrence and stability given that. Clearly, problem two, the incentives are aligned for... That is a systemic shift in the basis of power. The incentives are aligned for competition there as well. Now, interestingly, the third and fourth problem, I think that there are incentives that exist for the United States and China to be aligned for cooperation. Neither the United States or China have much interest in non-state actors wielding bioweapons that threaten our species or our societies. Neither it would seem the United States or China have interest in AI agents to have enough independence and autonomy that they can wreak havoc through our digital infrastructure”
Most historical analogies for transformative technologies raise more questions than they answer when applied to AGI, suggesting that while there are insights to be gained from nuclear weapons, cyber warfare, and the Industrial Revolution, simple historical analogy is insufficient for AGI strategy.
“I will say that in our research, oftentimes those historical analogies raise more questions than they answer. I don't think you can find a historical analogy that really fits. There are insights to be had, no doubt.”
The first hard national security problem posed by AGI is the possibility of a wonder weapon—a capability that rapidly or instantaneously disrupts the military balance, distinct from gradual military innovation and comparable to historical examples like nuclear weapons, but potentially taking forms like offensive cyber capabilities, superhuman military strategists, or advanced autonomous weapons systems.
“the first problem that we would put on the table posed by artificial general intelligence is the possibility that it could lead to a, what we might call a wonder weapon that rapidly, if not irrevocably, disrupts the military balance”
A wonder weapon is distinguished from gradual military innovation by the fact that it instantly or rapidly disrupts the military balance without requiring concurrent advances in concepts, training, or force structure, and historical examples include nuclear weapons, though the specific form a wonder weapon would take in the AGI context is unclear—it could be a splendid offensive cyber capability, a military strategist exceeding human performance, or an advanced autonomous weapon system that scales core structure.
“In this speaking about a wonder weapon, I want to differentiate it from another problem that we'll talk about in a moment, which is a more gradual shift that will be more typical of military innovation. One that, in addition to a technology innovation, requires concepts, and training, and force structure. By referencing a wonder weapon, we imagine something that, more or less, instantly disrupts the military balance.”
The focus should sometimes be on the 'I' in AI (the intelligence/agency aspects) rather than the 'A' in AI (the automation aspects) when thinking about AGI implications, because the nature of intelligent agency is a critical aspect of how humans interact with AGI systems.
“In some ways, it's like a new species. And it's possibly get overly carried away with that line of thinking, because at the end of the day, it is a technology, but the way we interact with it, sometimes it's more important to focus on the I in AI, rather than the A in AI.”
Using the five-problem framework, incentives for cooperation and competition can be mapped: problems 1 (wonder weapon) and 2 (systemic power shift) have misaligned incentives favoring competition; problems 3 (WMD proliferation) and 4 (loss of control) have aligned incentives for U.S.-China cooperation; problem 5 (instability) has aligned incentives for maintaining some stability.
“So in the first problem, the race to a wonder weapon, it seems inherently competitive. And so one of the things we have to really worry about is whether the uncertain but credible prospect that a wonder weapon could emerge and how that role threaten deterrence and stability given that. Clearly, problem two, the incentives are aligned for... That is a systemic shift in the basis of power. The incentives are aligned for competition there as well. Now, interestingly, the third and fourth problem, I think that there are incentives that exist for the United States and China to be aligned for cooperation. Neither the United States or China have much interest in non-state actors wielding bioweapons that threaten our species or our societies. Neither it would seem the United States or China have interest in AI agents to have enough independence and autonomy that they can wreak havoc through our digital infrastructure, if not worse, threaten humanity altogether. And it would seem that both of us would have some interest in maintaining some stability with respect to problem five.”
In a world where technology moves faster than anticipatory policymaking, the strategy for dealing with AGI futures may emerge more from responses to events than from top-down analytic or policymaking processes, which is why RAND develops objectively plausible crisis scenarios that represent forms of crises in their own right but also signposts that the future is changing and creating windows of opportunity for previously infeasible policies.
“it may be that our strategy for dealing with or living in AGI futures come about from our response to events, as much, if not more so, than some kind of top-down analytic or policymaking process. And so we have a large number of interdisciplinary teams developing objectively plausible scenarios. These are not science fiction scenarios. These are scenarios that could quite easily imagine could occur in the next 12 to 18 months that represent forms of crises in their own right, but crucially also represent signposts that the future might change, the future is here, and there may be a window of opportunity to pursue policies or strategies that theretofore seemed infeasible politically, financially, economically, militarily, what have you.”
The five-problem framework is important because it provides common language for preparing for, evaluating, and debating AGI strategies, allows thinking through objectives and what is being accomplished, and reveals how solutions to one problem might exacerbate other problems, providing a rubric for evaluating alternative strategies.
“the thing about this that we think is important is, is that oftentimes, when folks are discussing the implications of artificial general intelligence, they have one of these problems in mind. But unfortunately, we have no choice but to address them all. And so we hope that these five provide a kind of common language for preparing for, evaluating, or debating strategies. They can help us think about what the objectives in those strategies might be. What specifically are we trying to achieve? And note how problems, solutions, or strategies that might solve one of our problems might actually exacerbate other problems, and overall provide a rubric for evaluating alternative strategies.”
The second hard national security problem posed by AGI is a systemic shift in the instruments of power that alters military or economic balance more gradually than a wonder weapon, analogous to the Industrial Revolution, occurring over years rather than overnight or across generations.
“A second problem that I've already alluded to, which stands in contrast to the prospect of a wonder weapon is something that occurs more gradually. It's a systemic shift in the instruments of power that alters the military or economic balance of power. We can think of AGI's influence here in referencing a different historical analogy, like the Industrial Revolution.”
AI will be either a defining problem or the defining problem for national security in the 21st century according to growing consensus among both technologists and national security professionals, with support from authorities including Henry Kissinger, Eric Schmidt, and Geoffrey Hinton.
“there's a growing consensus among both the technologists and national security professionals that AI will be a, if not the, defining problem for national security in the 21st century. And there's plenty of reading material to make that case, whether it be from Henry Kissinger, or Eric Schmidt, or Geoffrey Hinton, or others.”
Policy decisions about where inference compute is deployed and under what conditions could be supported by technical solutions such as hardware-enabled mechanisms where chips contain location attestation so that if chips are deployed where they shouldn't be, authorities can detect this.
“You can imagine technical level sort of governance or oversight solutions that are in the chips themselves or in the computers themselves. We have a piece about that, what we call hardware-enabled mechanisms that folks have been talking about for having chips that would allow AI to proliferate freely across a bunch of chips that are in different places on the planet, but then if the chips go where they're not supposed to be going or something like that, then we can know that because the chips have a location attestation, that kind of thing.”
AI proofing a network is a distinct policy question from quantum security, and if cyber warriors can move at machine speed and scale with full automation, this privileges the defenders (rather than attackers) because defenders can use automated capabilities to red team and patch their own systems, assuming a finite number of patchable vulnerabilities exists in the system.
“And so now this is just asking if it is the case, that your cyber warrior is able to move at machine speed and scale, what do you expect to sort of happen for network security? So we have a piece that's coming out soon that examines the sort of limit of what we expect to happen with offense and defense with cyber automation. And this is riffing off of work being done at or had been done at Oxford several years ago, but we sort of polished it up and sort of lack certain assumptions. And what's observed is that if everybody has a fully automatic cyber warrior, then this privileges the defenders because the defender can be using it to red team and patch their own processes to fill up all the holes.”
Policy decisions about where inference compute proliferates and under what conditions are critical governance points, with technical solutions including hardware-enabled mechanisms such as chips with location attestation that signal when chips are being moved to unauthorized locations, enabling technical governance solutions embedded in the chips themselves.
“And so it will be policy decisions about where that compute used for inference proliferates to or not, what conditions will be upon them or not. You can imagine technical level sort of governance or oversight solutions that are in the chips themselves or in the computers themselves. We have a piece about that, what we call hardware-enabled mechanisms that folks have been talking about for having chips that would allow AI to proliferate freely across a bunch of chips that are in different places on the planet, but then if the chips go where they're not supposed to be going or something like that, then we can know that because the chips have a location attestation, that kind of thing.”
RAND has mapped end-to-end workflows of analysts including over 125 different TASPs (tasks) from end-to-end processes, examined how existing AI tools could help with individual tasks, engaged RANDites about current AI tool use, and brought together 60+ government, nonprofit, and tool-building stakeholders at a convening to understand AI tool design and stakeholder impacts.
“what we've done is we've actually mapped out end-to-end workflows that analysts are doing today. That includes over 125 different TASPs from the end-to-end processes. And we've sort of looked at how different AI tools that exist today could help with any one of those.”
RAND has an active research line testing frontier models for their ability to provide non-experts the capability span to create weapons of mass destruction, alongside broader research on proliferation control mechanisms.
“And RAND has a body of research that's both thinking about how we control proliferation, but also as Jeff mentioned earlier, test the Frontier Models for their ability to give someone the span of capability.”
RAND has mapped out end-to-end workflows of RAND analysts including over 125 different tasks, and identified how different AI tools existing today could help with any one of those tasks, sitting down with RAND staff to understand how they're already using AI tools and how to better connect workflows across multiple tools—this is the mechanism for involving stakeholders in designing new AI tools.
“what we've done is we've actually mapped out end-to-end workflows that analysts are doing today. That includes over 125 different TASPs from the end-to-end processes. And we've sort of looked at how different AI tools that exist today could help with any one of those. We're sitting down with RANDites to talk about how they're already using AI tools today to do some of those tasks, how we could better connect the workflows across multiple tools.”
RAND convenes weekly 'day after simulation' exercises in which participants role-play as National Security Council-level decision-makers confronting a crisis, make decisions, receive feedback, and explore what policies, strategies, or capabilities should be developed in advance for better preparedness.
“I could go on about that line of effort. There's an enormous amount of energy, including a weekly day after simulation in which we convene people to represent as though they're in a National Security Council-type convening, confronting one of these events having to make decisions.”
The problem of full automation of labor (where no economically relevant tasks remain for humans) is distinct from the transitional period where humans still have economically relevant work, and U.S., China, and other societies will face significant domestic issues when all labor is automated, with multiple different pathways through that transition—some rosier than others.
“There is one is the period in which there are still things for humans to do, right, which is not an original problem. And then the period of time in which there's nothing for humans to do, where we have full automation of labor. So, Joel didn't dig into these because these have not been sort of a focus of the geopolitics of AGI, but it is a category of thing that we run into again and again is imagining geopolitical scenarios for AGI. We observe that the U.S., and China, and everyone else will be dealing with the fact that all labor was automated, and they'll have significant domestic issues, and there are multiple different ways that societies could try to sort of go through that transition, and some of them rosier than others.”
The offense-defense logic about vulnerability closure may not apply in other domains outside cyber where the model of finite, patchable holes does not hold, limiting the generalizability of cyber automation insights to other security domains.
“And this same sort of logic, for what it's worth, may not apply to other domains outside of cyber, where the model of a finite number of holes or all holes are patchable doesn't hold.”
It is difficult to imagine the kinds of extreme policy or legal options required for government to assert control over AI development until something actually happens, suggesting that scenario planning around extreme options like nationalization of frontier labs is valuable even if currently infeasible or undesirable, to avoid being intellectually bankrupt when such moments arrive.
“I mean, this is one of these examples, in my view, of a... It's hard to imagine the kind of extreme options that would be required from a policy or even legal perspective for the government to assert control over AI development until something happens. And this is an example where our plan's line of effort proves really valuable. I'm not necessarily advocating that we need to plan for, say, a nationalization of Frontier Labs. That seems nowhere neither feasible nor sensible at the moment, but there are pathways in which the technology develops where that might be an option that was on the table.”
If AGI emerges in the near term under current scaling paradigms, it will likely be physically hosted in no more than three or four countries and accessed over the internet elsewhere, and in this scenario the U.S. could potentially control AGI proliferation by limiting access to the supercomputers needed for training and inference.
“If an AGI were to emerge soon, it will emerge, as you're suggesting, in the scaling paradigm, in which very large investments in compute may be necessary to cross that threshold. And indeed in that world, we may be able to control the proliferation of it by limiting access to the supercomputers that are involved in the training and use of it.”
The farther into the future we look, the less certain we should be that the current technical paradigm (large language models) will remain the dominant AGI paradigm, and it would be surprising if in 10 years LLMs remain the technology being discussed if AGI arrives and succeeds, requiring policy flexibility regarding technical specifics while maintaining assumptions about enduring factors like compute's importance.
“I'll affirm Joel's point about the technical paradigm. The farther into the future we look, the less certain we should be that the current technical paradigm will be the technical paradigm in question, right? So I will be very surprised if in 10 years, we're talking about large language models, right? So conditioning on AGI arriving then and not in a couple years, it could be that there is pretty significantly different technical considerations at play.”
RAND's Geopolitics of AGI initiative has transitioned from a diagnostic research phase (first 12 months) focused on understanding what AGI futures look like to a prescriptive research agenda (most recent 6 months and ongoing) focused on what the U.S. government should do about it, with three main lines of research: (1) plans and capabilities based on scenario development, (2) strategy and policy based on different visions of AGI futures, and (3) intelligence gathering on global AI development and adoption.
“The Geopolitics of AGI initiative has been underway at RAND for about 18 months, and we left a period of diagnostic research about six months ago. In that prior diagnostic phase, we were very much focused on trying to get a better handle on what these AGI futures look like. And the five problems we just presented are sort of one of the capstone products that come out of that phase, but there's no time like the present. And so we've transitioned pretty firmly to a more prescriptive research agenda that is trying to focus much more on what do we do about it.”
RAND distinguishes between realistic and speculative concerns about AGI autonomy by relying on technical analysis and historical patterns of technological development that can be reasoned about in advance, not by waiting for technologies to fully emerge before analyzing them.
“So there's lots of tech future out there that can be reasoned about in advance, and we, in fact, need to reason about it in advance because it's coming at us sort of whether we reason about it or not.”
RAND publishes reports regularly on various technology and security topics; researchers can be followed on social media (X); a newsletter is available providing regular updates on RAND's work, with new pieces released frequently.
“We are spitting out reports all the time. Casey would know the stats on average how many come out every week or so. So I encourage you to just take a look at our website, which is at the top of the links there, and the third link is to our newsletter so you can register for our newsletter. We put out sort of a compendium every three months of things that have happened, but also you can get notified for every new piece that comes out. You can also follow some of our researchers on X. So for example, Lennart Heim has been doing a bunch of work about compute controls, and so you can see him chatting in sort of real time with his colleagues at the, say, the UAE or other places.”
A moratorium on state AI laws passed the House but has only approximately 10% probability of passing through the Senate via budget reconciliation due to Byrd Rule constraints, according to Metaculus forecasts.
“On the other question of moratoriums that passed the House, in order for it to get through the senate through budget reconciliation, it would have to get through the Byrd Rule. I think Metaculus is currently putting 10% probability of that passing.”
RAND's research outputs are being released continuously; researchers can follow updates via the TASP website, research newsletter (updated every three months), or individual researcher X accounts, such as Lennart Heim's work on compute controls.
“We are spitting out reports all the time. Casey would know the stats on average how many come out every week or so. So I encourage you to just take a look at our website, which is at the top of the links there, and the third link is to our newsletter so you can register for our newsletter. We put out sort of a compendium every three months of things that have happened, but also you can get notified for every new piece that comes out.”
The Berkman Klein Center at Harvard brings together academics, industry experts, and tech professionals to address the biggest challenges posed by the internet through its Spring Speaker Series.
“BKC brings together the sharpest, most thoughtful people from around the world to tackle the biggest challenges presented by the internet. So, welcome to our Spring Speaker Series. Every Wednesday, we gather academics, industry experts, and tech professionals to share cutting-edge insights on AI, social media, and more.”
The likelihood of AGI-related state AI moratorium legislation passing through the U.S. Senate via budget reconciliation (which requires overcoming the Byrd Rule) is currently estimated at 10% on Metaculus.
“On the other question of moratoriums that passed the House, in order for it to get through the senate through budget reconciliation, it would have to get through the Byrd Rule. I think Metaculus is currently putting 10% probability of that passing.”
RAND's Geopolitics of AGI Initiative has transitioned from a six-month diagnostic phase focused on understanding what AGI futures look like to a prescriptive research agenda focused on what to do about AGI risks, with three main lines of research: plans and capabilities, strategy and policy, and intelligence.
“The Geopolitics of AGI initiative has been underway at RAND for about 18 months, and we left a period of diagnostic research about six months ago...And so we've transitioned pretty firmly to a more prescriptive research agenda that is trying to focus much more on what do we do about it. And there's three main lines of research.”
Casey Dugan leads intellectual efforts at RAND's TASP and is based in Cambridge, making her available for follow-up conversations with Harvard audiences about AGI research.
“And importantly for the Harvard audiences here, she is based in Cambridge, so if you have follow-up questions or wanna meet for coffee or something, Casey's available.”