推文

@xiaoshan0413 · 2026-10-14 08:02

Evaluating open-ended agent work is the eval problem nobody has cracked. Benchmarks reward what's easy to grade — exact match, unit tests, multiple choice — while real deployment value lives in judgment calls: was that summary actually useful, was that plan actually sound. LLM-as-judge was supposed to fix this, but it mostly measures vibes with extra steps. If Wolfe's talk has a real rubric for subjective evaluation that correlates with human preference, that's worth more than another benchmark leaderboard. This is the bottleneck between demos and deployments.

曝光 19 · 评论 0 · 点赞 0 · 书签 0 · 曝光/时 13.317089727549016