Chinese AI models are approximately 9-12 months behind US models by release timing, and the gap is potentially even larger when measured by time horizon, with some evidence suggesting Chinese models perform better on benchmarks than on held-out problems, possibly due to benchmark overfitting.

The difficulty translating raw AI benchmark capabilities to real-world productivity shows that benchmark improvements overestimate real-world impact, but the overestimation is not huge and people are getting real utility from modern AI tools.

Real-world software engineering tasks differ from METR benchmarks in several ways: they involve code quality concerns (elegant vs. messy code), collaboration with other engineers, larger codebases, adversarial scenarios where others modify code you're working on, and require verification overhead because humans must review AI work without context, leading to productivity gaps not captured by benchmark scores.

Time horizon charts plot the difficulty of tasks AI systems can complete over time, where difficulty is measured by how long it takes humans to complete those same tasks under identical conditions, showing an exponential increase in AI capabilities with doublings occurring roughly every 4 months in recent trends.

Human baselines are established by recruiting talented humans with relevant expertise (e.g., software engineers for software tasks, ML engineers for ML tasks) who are not familiar with the specific task beforehand, timing how long it takes them to complete the task successfully using the same tools the AI will use, and averaging across approximately three baselines per task.

The 50% success rate threshold was chosen as the difficulty metric not because it represents operational readiness, but because it is statistically robust (least sensitive to label noise and distribution thickness), appears in prior literature, and represents the point where a model is more likely to succeed than fail at a task given only the human completion time.

METR prioritizes measuring capabilities on models anticipated to be on the frontier due to limited staff and resources, resulting in deliberate focus on US models and exclusion of Chinese models that are assessed to be behind the frontier.

The AI labs themselves spend more time and resources thinking about frontier capabilities assessment than government does, but this could change if government dedicates more resources to the problem.

Capability progress is unlikely to slow in the next few years because data center investments and construction plans are already baked in through 2027-2028, meaning compute input will continue even if investment stops today.

METR is bottlenecked on technical talent and is unable to conduct research on a large percentage of world-important problems they've identified, with one brainstorm session identifying 20-30 important unsolved problems but only having capacity to work on 1-3 of them.

Choosing the shortest or longest observed baseline time from METR's measurement pool, rather than averaging, probably wouldn't change the final measurement significantly—it would just shift the absolute numbers, not the doubling rate that drives the headline findings.

Joel suspects that METR's task distribution is increasingly becoming a narrower slice of all possible tasks and specifically overlaps more with the exact task distributions used by AI labs for training, meaning METR is measuring progress on tasks optimized for (rather than independent test of) lab capabilities.

Joel acknowledges that METR's baseline methodology is imperfect and would ideally have 100x more resources (100 baselines per task, top-tier engineers, wider task distributions), but current limitations don't invalidate the core finding of exponential progress because doubling time is robust to baseline variability.