Rendered at 05:15:31 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Good4boothee 21 hours ago [-]
I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.
vintermann 20 hours ago [-]
You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.
But wow, they found a lot of issues with the benchmark.
scotty79 19 hours ago [-]
I really like the benchmarks where frontier models score each other's solutions to a problem.
But wow, they found a lot of issues with the benchmark.