Holden Karnofsky
About
Co-founder of GiveWell
Cast within
No topic-region cast yet — this appears once Holden Karnofsky's compiled claims are aligned into a topic region's argument tree.
Claims by Holden Karnofsky (20 of 38)
Lock-In: Advanced Tech Could Freeze Civilization
Historically, bad rulers are temporary because they age, die, and the world keeps changing; but at a high enough level of technological development there may be nothing new to find and people may not age or die, so the sources of dynamism could disappear, allowing a stable indefinite dictatorship—Karnofsky estimates the odds of such lock-in given transformative AI at roughly a quarter to a half.
AI Success Means Staying On Current Trajectory
Success with transformative AI would frankly look like staying close to the trajectory we're already on: AI systems that behave as intended, act as tools and amplifiers of humans, remain broadly distributed rather than controlled by one government or person, and let us keep getting richer, wiser, and better at understanding ourselves—possibly eventually extending rights to AIs through public deliberation.
Humanity As Adolescent Gaining Dangerous Strength
Using Toby Ord's analogy of humanity as a child becoming an adolescent, gaining strength is great up to a point, but eventually you become strong enough to really hurt yourself or not know your own strength; humanity is reaching that point with nuclear weapons, bioweapons, and AI, so the centuries-old push for more power and technology now warrants more caution as we enter a gray zone.
Pursue High-Stakes Beliefs But Keep Common-Sense Ethics
When people believe they live in an especially important time, the best norm is neither to dismiss the thought (so genuinely important actors do the wrong thing) nor to take it so seriously they promote their own interests above everyone's; rather, people should take such beliefs reasonably seriously and act on them while adhering to common-sense ethical standards and avoiding 'ends justify the means' reasoning like lying or breaking the law.
Current Growth Rate Cannot Continue 10000 Years
Extrapolating the current ~2% economic growth rate out 10,000 years (a blink of an eye on galactic scales) implies we would need multiple times the value of today's entire world economy per atom in the galaxy; since we cannot exceed the speed of light to leave the galaxy, we run out of material, meaning current growth is too high to sustain and our era is necessarily a special, dynamic anomaly.
Past Futurism Was Unserious, Not Impossible
Historical failures to predict the future may reflect that past attempts were deeply unserious rather than that the future was genuinely unknowable; people in the 70s arguably could have reasoned that rising population and continual resource innovation would continue, so we should not become defeatist from past errors, and modern prediction methods have likely improved.
Direct Good Mostly Washes Out After Transformative AI
While the effects of helping someone live a healthier, better life are permanent in the sense that the life happened and mattered, the effects of direct global-health interventions on the long-run future will mostly wash out and not persist in any systematic, predictable way after the crazy changes brought by transformative AI—similar to how most things probably did not predictably persist across the Industrial Revolution.
OpenAI Grant Defense And Board Seat Rationale
Karnofsky disputes that OpenAI has been net negative; while faster AI advancing gives less time to prepare (a bad thing), OpenAI has done good things and set important precedents, and the $30M grant was partly about securing a board seat to influence governance at a crucial early time rather than merely boosting them—he sees it as neither his best nor worst grant.
Bottleneck Objection And Partial Automation Rebuttal
The single best objection to the most important century is that one non-automatable step could bottleneck everything and prevent explosive growth; but you don't need to automate the whole economy—automating just key tech like AI itself and energy (which are less bottlenecked, e.g. robot-built AI chips and self-improving solar) could drive the growth loop, and a massive population of human-level AI thinkers could find ways around remaining bottlenecks such as simulating experiments.
Orthogonality: Intelligence Independent Of Goals
Per Yudkowsky's orthogonality thesis, an AI can be highly intelligent in pursuit of any goal—even a stupid one—so the fact that smart humans often have nuanced moral goals provides no comfort; modern AI is trained by trial-and-error reinforcement, so you can end up with a system that very intelligently pursues something you didn't mean to encourage (like maximizing money in a bank account) without ever asking whether that goal is good.
Moral Parliament For Moral Uncertainty
Faced with moral uncertainty (e.g., total-good views wanting a big world vs suffering-focused views wanting a small one vs integrity-focused views), Bostrom's moral parliament approach treats your competing moral views as multiple people living in your head who care about each other and negotiate a deal—so you favor actions that are really good according to one view and not too bad according to the rest, which moderates against extreme ends-justify-means behavior.
AI Probability Should Scale Resource Allocation
Both very good (scarcity-free) and horrible dystopian outcomes from transformative AI are easy to imagine; the more likely you think transformative AI is and the more imminent it is, the more it should be the top philanthropic priority over more direct short-term problems—so Open Philanthropy does both, with the balance tilting toward AI as the technology looks more real and imminent.
Growing Organizations Trade Nimbleness For Output
As an organization, city, or society grows, it must satisfy more stakeholders and by default becomes less able to make disruptive quick changes (less 'nimble'); but big companies often produce far more than they could when small—Apple at 10 people might be more exciting but couldn't make all those iPhones—so for judgment-heavy unconventional giving Open Philanthropy benefits from staying as small as possible, deliberately treating each hire as something only done with a really good reason.
Growth Feedback Loop Broke With Demographic Transition
Standard economic theory predicts accelerating growth from a people-ideas-resources feedback loop, but this loop broke a couple hundred years ago when people stopped having more children when they got richer—they got richer instead of more populous—which is why we don't currently see the accelerating feedback loop, though AI doing the 'ideas' work could restore it.
AI Could Make This The Most Important Century
If we develop AI systems this century that can do all the key tasks humans do to advance science and technology, we could very quickly reach a futuristic, post-human world that is very stable and very large, making this our last chance to shape how it happens and potentially the most important century of all time.
We Already Live In A Verifiably Weird Time
Even ignoring AI entirely, commonly accepted facts—the last couple hundred years showing far faster economic and technological growth than all prior history, our position in a tiny sliver of cosmic time, and the apparent absence of other galactic life—make our era one of the most extraordinary and earliest times ever, so claiming transformative AI is coming is only a moderate quantitative update rather than a wild leap.
Against Ends-Justify-The-Means Reasoning
Ends-justify-the-means reasoning—doing horrible things, coercing others, or using force because the goal is deemed more important than everything else—looks far worse historically than people simply trying to do helpful things within common-sense ethical bounds, so even immense expected-value calculations do not currently justify breaking the law given how uncertain one's confidence really is.
Lock-In Some Things To Prevent Worse Lock-In
Lock-in is mostly bad because it sacrifices optionality and risks one person running everything forever, but in a world with enormous control over the environment you might deliberately lock in certain protective constraints—such as ensuring no single person can ever hold all the power—precisely in order to prevent other, worse forms of lock-in.
Enlightenment As Precedent For Shaping Transitions
The Enlightenment thinkers who worked on esoteric questions about human rights, individual liberties, and the rights of the governed may have meaningfully shaped the entire world because the UK became disproportionately influential after the Industrial Revolution, suggesting that working on seemingly esoteric, low-prestige problems (like AI alignment today) before a major transition can have outsized long-run impact.
Moral Progress Is Real But Not Inevitable
Moral progress is real—the term refers to changes in morality that are genuinely good (e.g., increased acceptance of homosexuality)—but it is not objective truth, not inevitable, and does not happen automatically just because time passes or intelligence increases; much of historical moral progress came from humans getting to know other humans they previously stereotyped, a mechanism a non-human AI would not share.
My Notes
Loading notes...