Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots

Many people in the Bay Area who are concerned about AI safety have the intuition that the best way to influence AI's future is to work inside the industry building the technology, but this creates a structural problem: the logic of 'if I don't build it, someone else will' encourages continuous advancement, and combined with competition between labs and geopolitical competition with China, even safety-motivated developers face incentives to build more advanced systems.

“Many people had this intuition that the thing to do is go and work in the industry, because if you're, like, helping build it, you know, what's the best way to shape the future? It's to build it. And I think that one, there's obviously you could have questions about how sincere that is for for many of the people, who are in the industry, or if there's kind of a mix of different motivations”

METR is a research nonprofit dedicated to measuring whether and when AI systems might pose catastrophic risks to humanity, specifically focusing on threats from AI autonomy by assessing how autonomous AI systems are and the scale, length, and difficulty of tasks they can perform independently.

“So METR is a research nonprofit based in the Bay area, like you said, dedicated to advancing the science of measuring whether and when I systems might pose catastrophic risks to humanity as a whole. Focus specifically on threats that come from AI, autonomy or AI systems themselves.”

Chinese AI models are approximately 9-12 months behind US models by release timing, and the gap is potentially even larger when measured by time horizon, with some evidence suggesting Chinese models perform better on benchmarks than on held-out problems, possibly due to benchmark overfitting.

“The, the models that we, we anticipate being on the frontier and in general, the Chinese models have been something like, you know, 9 to 12 months, let's say, behind the behind the US models.”

Real-world software engineering tasks differ from METR benchmarks in several ways: they involve code quality concerns (elegant vs. messy code), collaboration with other engineers, larger codebases, adversarial scenarios where others modify code you're working on, and require verification overhead because humans must review AI work without context, leading to productivity gaps not captured by benchmark scores.

“One is that the scoring implicitly is different. In in real problems, I'm scoring based on, you know, something a bit more realistic than these, algorithmic scoring procedures, these automatic scoring procedures that we're using at meta, and many other people are using in the, in the benchmark world, there's some notion of code quality. If you're if you're working in software engineering. But but for other tasks there's, there's beautiful code. Elegant code piece was talk about.”

There is a strange dynamic in the AI industry where lab CEOs and safety researchers make similar alarming claims about AI risks (destroying the world, taking over), creating a 'Baptist and bootlegger' alignment between people building the technology and people warning about it, which is unusual compared to most industries.

“there's this kind of, sort of Baptist and bootlegger relationship between the AI labs, people who are building this stuff and the sort of alignment safety people, and they sort of go back and forth in like the you have the heads of the lab saying, yes, this might destroy the world and take all your jobs of the safety people and the alignment. People say, yes, this might destroy the world. And like I, it's a very strange industry, right?”

Joel suspects that METR's task distribution is increasingly becoming a narrower slice of all possible tasks and specifically overlaps more with the exact task distributions used by AI labs for training, meaning METR is measuring progress on tasks optimized for (rather than independent test of) lab capabilities.

“I have some suspicion that the tasks that meta is measuring performance on, you know, in some sense, a more and more narrow slice of possible tasks and in particular, a more and more narrow slice that is perhaps similar to the kinds of tasks that you'd expect these major AI companies to be training on in the first instance.”

Chris Painter's personal view is that if AI self-improvement and AI research automation are plausible future developments, then all of humanity being aware of this trajectory is a precondition for figuring out how to respond to it, and therefore METR prioritizes public information over selective knowledge advantage for any particular group.

“Maybe this is more a personal view, but I think if this is possible that we will automate AI research, I think all of humanity being aware of it is kind of a aware of where we're heading is sort of a precondition for us all being able to figure out what to do about it. And so I don't kind of want like certain people or one side or one team to kind of like selectively be in the dark because they might invest on the basis of this or something like that.”

The AI labs themselves spend more time and resources thinking about frontier capabilities assessment than government does, but this could change if government dedicates more resources to the problem.

“There's some sense in which, like the companies in the industry, spends more time thinking today about frontier capabilities assessment than the government does. Yeah. I think, like one day you could imagine us getting to the point where the government is, like very focused on this and dedicating a lot of resources to it. And at that point, I would expect meteor to be spending more time talking to governments.”

OpenAI and Anthropic were founded with exotic corporate structures (private company owned by nonprofit, etc.) because the founders took seriously the existential risks of AI and wanted to self-limit, suggesting at least some industry actors are not purely motivated by profit.

“It was also true that open AI and anthropic, but open a little more. We're like founded with these very exotic corporate structures, like a private company owned by nonprofit, etc., which they presumably did because they took pretty seriously the fact that this technology is a science.”

Choosing the shortest or longest observed baseline time from METR's measurement pool, rather than averaging, probably wouldn't change the final measurement significantly—it would just shift the absolute numbers, not the doubling rate that drives the headline findings.

“some of the details, at least for the work we've done so far, you know, aren't going to matter as much as you might naively think. So choosing the, you know, shortest baseline time that we that we, at the we end up observing or the longest time, you know, it's actually not going to make that much difference to the final measurements, you know, of course, we we do think these people, talented software engineers...perhaps we could have found even more talented people. They would have completed it in half the time...but of course, that that wouldn't change. The wouldn't change the doubling time. It would mean you'd get to the same level after another, another four months.”

Joel acknowledges that METR's baseline methodology is imperfect and would ideally have 100x more resources (100 baselines per task, top-tier engineers, wider task distributions), but current limitations don't invalidate the core finding of exponential progress because doubling time is robust to baseline variability.

“I think I think it just is true that, baseline methodology or the ways in which we compare to humans in some ways leaves a lot to be desired. That's, you know, ideally we would have invested, you know, a hundred times as many resources in, you know, having 100 baselines and baselines per, task.”

METR's time horizon charts have become the industry standard benchmark for measuring AI progress, and investment decisions are explicitly being made based on these charts as investors interpret them as direct evidence of AI capability and business value.

“How much interest you get on these charts from potential investors specifically? And the reason I ask is because, I was just messing around and like googling some stuff. And when the opus chart, the latest opus chart came up, someone posted it on Reddit and I think, like the second comment on it was someone going, how do I invest in open AI?”

There is a potential conflict of interest in METR's baseline methodology: humans are paid to complete tasks and may be incentivized (consciously or unconsciously) to take longer to complete them to maximize payment, though METR mitigates this by paying bonuses for faster completion relative to peers.

“You say, Joe, come in and do this task. What is it? The how do you prevent me? Oh, man, this is taking me a long time. Meanwhile, I keep getting $100 an hour for, like, looking at my computer and time. Oh, this is tough. I'm gonna have to come back tomorrow and keep working on this.”

The rapid progress in AI capabilities is creating real evidence that must inform policy, but policymakers today primarily care about data center location and infrastructure questions rather than frontier capability assessment.

“when you say, you know, it's easy to imagine or, maybe the government will care more about this. Not so easy for me to imagine. It seems like they mostly care about, you know, data centers and, like, where they located and stuff like that. It would be nice if we had policymakers really looking at, like, frontier capabilities and stuff. Still seems kind of a way off.”

The difficulty translating raw AI benchmark capabilities to real-world productivity shows that benchmark improvements overestimate real-world impact, but the overestimation is not huge and people are getting real utility from modern AI tools.

“I think there are a number of differences in a number of ways, in particular, in which the benchmark results are overestimating what we might see in the wild, you know, not not hugely overestimating. I think we do see that people are getting real utility out of these modern, AI tools, but overestimating to some extent.”

Capability progress is unlikely to slow in the next few years because data center investments and construction plans are already baked in through 2027-2028, meaning compute input will continue even if investment stops today.

“it's hard for me to consider it plausible that it will slow down in the next at least small number of years. Is that a lot of those compute R&D investments, basically already baked in, right? Like the centers have already been built, you know, plans for data centers even beyond, you know, 2027, 20, 28, presumably, you know, coming and coming to fruition, coming, coming about.”

Currently, when AI systems work together (AI to AI), they fail because they generate collaborative hallucinations and tend to devolve into terrible outputs, suggesting AI-to-AI scaling is not yet a near-term threat.

“What actually happens today when AI is working with AI. Yeah. My sense is that, at some point, you know, a further away points than would have been true some, some time ago. The AI is more or less full on their faces. That's, that you know, there are some things they're not so capable of today, like. Collaborative hallucinations world do just like, you know, just like, devolve into terrible.”

One narrative explanation for why labs are now seeing faster progress (4-month vs 7-month doubling) is that they have now narrowed their focus to software engineering and AI research tasks where they can apply intense optimization pressure, rather than working on more diverse consumer applications like image and video generation.

“You know, you had like OpenAI shutting down its like video efforts etc.. So perhaps part of the story is just this intense focus now on the software engineering side as what these labs are working on. Yeah. And sort of like all these other side quests are not as important. So maybe we will see even more rapid progress on some of these technical benchmarks, because clearly, from the labs perspective, that's where the action is more than some of these consumer things like making making images or videos.”

METR is bottlenecked on technical talent and is unable to conduct research on a large percentage of world-important problems they've identified, with one brainstorm session identifying 20-30 important unsolved problems but only having capacity to work on 1-3 of them.

“I think clearly the central reason is that we are bottlenecked on, on technical talent, on, on, you know, incredibly capable people to come work on these questions. I was on a meta work rate recently where we were we were brainstorming, you know, 20, 30 of these what seemed like world important problems, problems that we think no one else is going to get to if we do not get to them. And we are able to to, to conduct research on on how many of those problems. I think it's one to, you know, maybe if we do an extraordinary job this quarter, it might be three”

AI systems today can handle the human-directed idea phase (where a person specifies what they want) combined with AI handling software engineering implementation, but cannot yet fully autonomously manage research departments or operate independently from human ideation.

“the ways in which AI's are autonomous today, or close to autonomous today is the human has the idea and then, you know, submits that idea to, cloud code or Codex or one of these other AI tools, and then they handle the software engineering components. Possibly there's still still some, some intervention after that.”

METR observes that domain-specific time horizon charts show different trajectories, with some showing steep exponential curves and others showing flatter or more irregular patterns, indicating variation in progress across different task domains rather than uniform acceleration.

“So when you look at the domain specific time horizon charts, so the ones that show like, you know, the I think you call them task suites or something like that, like I guess productivity by specific job and you see these different lines. So sometimes you see like almost horizontal lines and sometimes you see squiggly or steeper lines.”