Human baselines are established by recruiting talented humans with relevant expertise (e.g., software engineers for software tasks, ML engineers for ML tasks) who are not familiar with the specific task beforehand, timing how long it takes them to complete the task successfully using the same tools the AI will use, and averaging across approximately three baselines per task.
factualpending
Speaker
Joel BeckerEvidence Quote
“using essentially the same tools that we're about to give the AIS, we take, talented humans, you know, not people who have seen this particular type of task before, but people who have relevant expertise”
Created: 8/12/2026, 6:03:24 PM
My Notes
Loading notes...