Adaptive harnesses, concept audits, and the measurement of agent research

Sahil Kale, Ian Harris

Effective unlearning must remove harmful uses of dual-use concepts while preserving benign ones, yet existing benchmarks treat forget and retain sets as disjoint facts and score only direct recall. ConceptGuard constructs complementary forget and retain sets around dual-use concepts and evaluates intent-sensitive contextual separation. Current methods show weak contextual separation, strong forgetting-utility trade-offs, and poor concept-level consistency.

Dingzirui Wang, Xuanliang Zhang, Keyan Xu, Qingfu Zhu, Wanxiang Che

The benchmark draws 175 expert-validated instances from 2025-2026 STOC, FOCS, SODA, and COLT papers, preserving definitions, assumptions, and proof dependencies with Lean formalizations. Autoformalization is the sharpest bottleneck: the best model scores only 11.5 on translating claims into formal statements versus 28.6 Pass@8 on human-provided statements. An automated research pipeline generates 64 claims of which only 6 survive expert evaluation and proof verification, highlighting limited research taste.

Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki

Harness optimization rewrites agent scaffolding code from validation performance without weight updates, but full fixed validation sets waste compute on non-discriminative tasks. Task-CoEvolve selects tasks on which candidate harnesses disagree via variance-weighted sampling that tracks the capability frontier, then estimates full-set scores from the sampled subset. It matches full-set final performance while cutting evaluations by 80 percent on text classification and Terminal-Bench 2.1.

Qian Kou, Xiaofeng Shi, Xiaosong Qiu, Hua Zhou

IAR converts a fixed corpus into parametric knowledge via three stages: structured injection through continuation, rewrite, and reconstruction objectives; QA alignment with answer-only supervision; and recovery by merging with the base instruction model. Across multiple model families and corpora it improves the domain-general frontier, gaining 3.6 points domain QA accuracy and 12.1 points mean general performance over vanilla SFT in most settings. Among methods that reach strong domain internalization, IAR retains one of the strongest general-capability profiles.

Chengsong Huang, Zifeng Wang, Rujun Han, Jun Yan, Yanfei Chen, Zoey CuiZhu, Ke Jiang, Peng Xia, Han Yu, Yufan Zhuang, Yifei Ming, Jiaqi Pan, Bhavana Dalvi Mishra, Jiaxin Huang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

Static hand-built environments cannot track agent weaknesses or improve with the policy. EnvHarness wraps any static environment with programmable plug-in components that reshape behavior while preserving the original verifier; EnvRigger diagnoses trajectory failures and synthesizes the components automatically. Across five benchmarks it yields up to 9-point gains on held-out instances with fewer steps and supplies a stronger RL co-evolution signal.

Mattia Carletti, Edward Phillips, Fredrik K. Gustafsson, Patitapaban Palo, Lei Clifton, Danielle Belgrave, Xiao Gu, David A. Clifton

A controlled benchmark generates latent risk trajectories that produce both numerical series and text summaries so that exactly one modality matches ground truth. Models show systematic rather than random arbitration: distinct text-versus-number preferences, stronger reliance on temporal recency than on explicit reliability cues, and over-reliance on external forecasts even when they contradict direct evidence. These heuristics pose a clear failure mode for tool-augmented decision systems.

Matthias Seeger, Zeyu Zhang, Vihang Patil, Konstantinos Benidis, Sebastian Schelter

The method fine-tunes models under any KV-cache sparse-attention policy on modest hardware so the model co-adapts to the selection rule, often outperforming exact-attention training. An efficient H2O implementation with custom kernels is released inside the KeysAndValues library. Co-adaptation lets sparse policies deliver long-context performance without excessive memory budgets.

Bin Zhu, Yi Xie, Yanghui Rao

Judge-panel design is cast as role-conditioned allocation: from a small audit set the method classifies judges as copies, complements, or specialists and produces a policy that drops redundant copies, routes specialists conditionally, and stops when marginal validation gain falls below cost. Across reasoning, code, safety, preference, and math audits the resulting call plans dominate single judges, flat panels, and fixed cascades. The output is a reusable, auditable routing plan for subsequent evaluation batches.

Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu

The benchmark contains 119 repository-level tasks from 98 scientific GitHub projects across 20 domains, organized into issue-driven, expert-exploratory, and engineering-integration paradigms. Even the strongest agent stays below 50 percent pass@1. Four recurring failure modes emerge: missing scientific abstraction, surface-level repair, incomplete system integration, and failure to generalize knowledge; a paired ablation shows that poorly aligned scientific guidance can induce anchoring.

Yansen Han, Shengyi Liao, Yuanxing Zhang, Pengfei Wan, Tao Lin

Preference updates on flow-matching models can displace terminal samples off the pretrained data manifold whenever the induced displacement has a nonzero normal component. ThermoDPO anchors pairwise preference optimization on preferred samples via a temperature-controlled objective that recovers a continuum between rejection sampling and FlowDPO; a weighted variant restores signal at low temperatures. On SD3.5-M it improves OCR by 47.5 percent and the average of four metrics by 16 percent while controlling a reconstruction surrogate for manifold distance.