Benchmarks
Benchmarks coverage belongs in AI Development. The constraints that determine whether AI systems work in production.
Runtime and evaluationSearch
AI intelligence results for "Benchmarks", including topic guides, current stories, and graph profiles.
Topic guides
Pagish coverage for Benchmarks
Benchmarks
Benchmarks coverage belongs in AI Reviews. A repeatable review format for decision support.
Review criteriaRelevant AI stories
Policy and SafetySep 23, 2026
OpenAI's MentalHealthBench puts pressure on AI's most sensitive use case
OpenAI's MentalHealthBench arrives because people are already bringing emotional distress, crisis language, and therapy-like conversations to AI systems. That makes mental health one of the highest-stakes product surfaces in consumer AI.
ModelsSep 22, 2026
Claude Opus 5.5 turns the model race into a margin fight
The Decoder's coverage of Claude Opus 5.5 matching a rival model at lower cost shows how quickly AI competition is becoming a margin fight. The story is not only who tops a leaderboard, but who can deliver comparable capability at a price developers can actually use.
ResearchSep 22, 2026
UK AISI and EvalEval are attacking the quiet problem of benchmark trust
The Hugging Face post on UK AISI and EvalEval is about a less glamorous but essential AI problem: benchmark results have to be reproducible before they can guide safety or procurement decisions.
AgentsSep 14, 2026
Agent benchmarks are starting to look more like security tests
Recent arXiv work on software-agent evaluation points to a shift in how the industry should judge agents. The important question is no longer only whether an agent can finish a task, but whether it can do so without creating security, reliability, or permission problems.
ResearchSep 4, 2026
BenchMIRT asks whether AI benchmarks measure what users need
Benchmarks are supposed to turn model quality into something comparable. The problem is that a high score can hide what a model is actually good at, where it fails, and whether the test resembles the work users care about.
ResearchSep 4, 2026
Translation benchmarks are being rebuilt for a multilingual AI world
Global AI will fail quietly if translation quality is measured badly. A model can look strong in aggregate while still mishandling low-resource languages, domain-specific terms, dialect, or culturally loaded phrasing.
ModelsAug 28, 2026
Protected benchmarks are becoming necessary for model trust
AI benchmarks are supposed to clarify model quality, but the market has learned how easily a score can become launch theater. Google DeepMind's use of protected testing for Gemini points at a more serious standard: evaluations need to be harder to leak, game, or tailor around.
ResearchAug 28, 2026
Speech AI benchmarks are expanding beyond the usual language map
AI benchmarks often reflect the languages and markets with the most data. Hugging Face adding a Global South language to its open ASR leaderboard is a reminder that speech AI quality is not evenly distributed around the world.
ResearchAug 28, 2026
Google wants AI benchmarks to prove more than leaderboard scores
AI benchmarks are supposed to settle arguments, but the industry has learned how quickly they can become part of the marketing machine. When a model launch depends on a chart, everyone has an incentive to understand the test, optimize around it, and frame the result in the most flattering way.
ResearchAug 25, 2026
A Bayesian RAG evaluation paper targets the messy part of retrieval systems
RAG systems often look good in demos and then break in production for frustrating reasons: the retriever missed the right document, the answer used the wrong passage, or the evaluation hid both problems. This paper focuses on that messy middle.
Developer ToolsAug 24, 2026
SWE Refactor Bench tests whether coding agents can complete repository migrations
A benchmark focused on large-scale refactoring targets a practical question: can coding agents preserve behavior while changing many files?
AI in PracticeAug 24, 2026
Thomson Reuters chooses owned AI over rented frontier models
Thomson Reuters is a useful enterprise signal because its business depends on trusted information. If a company like that leans toward owning more of its AI capability, it suggests some workloads may be too sensitive, specialized, or valuable to leave entirely to rented APIs.
ModelsAug 23, 2026
DeepSeek Flash vision model pressures agent benchmarks
The Decoder reports that DeepSeek released an experimental Flash vision model positioned against strong agent-benchmark results, adding momentum to multimodal agent competition.
ResearchAug 23, 2026
Hugging Face examines benchmark optimization in speech recognition
Hugging Face published a technical analysis of benchmark optimization in speech recognition, raising practical questions about how audio AI progress is measured.
ResearchAug 23, 2026
Inter-X++ benchmark targets multimodal human interaction understanding
A recent arXiv paper introduces Inter-X++, a benchmark for multimodal human-human interaction analysis across perception and synthesis tasks.