Agents, Memory & Evaluation
- paperarXivHarness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable
Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, Leoweiliang
The paper introduces the Harness Handbook, an automatically synthesized behavior-centric map of an agent harness codebase produced via static analysis and LLM structuring, linking high-level behaviors to source locations across three levels of detail. It pairs this with Behavior-Guided Progressive Disclosure (BGPD) that guides agents from behaviors to verified implementation sites. On modification tasks from open-source harnesses, the approach improves localization and edit-plan quality while reducing planner tokens, especially for scattered, rare, or cross-module behaviors.
agentsharnessessoftware engineeringcode localizationLLM agents - paperarXivFrom Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agents
Bo Tang, Yang Zhang, Guomian Zhuang, Wenqiang Wei, Gaoyang Zheng, Lindong Xie, Yanchao Tan, Feiyu Xiong, Qingyu Yang, Edward Chung, Zhiyu li
MSCE is a training-free framework that organizes agent experience into L1 step traces, L2 procedural policies, and L3 declarative cognition, crystallizing high-value L2 policies into callable skill cards that retain evidence, boundaries, verification rules, and reliability estimates. It uses reflection-weighted value backfilling to propagate sparse terminal rewards through local self-reflections for calibrated evolution of memory and skills. On EvoAgentBench and LoCoMo it outperforms skill-augmented and memory baselines with strong cross-domain transfer and lifelong evolution.
LLM agentsmemoryskillslong-horizonco-evolution - paperarXivPRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
Alexis Fox, Junlin Wang, Paul Rosu, Bhuwan Dhingra
PRO-LONG keeps a complete structured interaction log and uses coding-agent tooling to search it on demand rather than compressing history into lossy summaries. This programmatic memory sidesteps the precision-recall tradeoff of conventional context management for exploratory long-horizon tasks. On the full ARC-AGI-3 public set it improves base coding agents by 18 points on average, matches or exceeds specialized harnesses at up to 76.1% pass@1, and uses 4.2–5.8× fewer tokens.
agentslong-horizonmemoryARC-AGIcontext management - paperarXivVerbalizable Representations Form a Global Workspace in Language Models
Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, Jack Lindsey
Using the Jacobian lens the authors identify a small privileged set of residual-stream directions (J-space) that a model is poised to verbalize; these exhibit global-workspace functional properties including reportability, deliberate summoning, silent intermediate reasoning, and broadcast as arguments to downstream computation. J-space occupies ~10% of variance mainly in middle layers; suppressing it preserves fluent parsing and speech while abolishing complex internal reasoning. The technique surfaces strategic deliberation, evaluation awareness, and misaligned dispositions invisible in outputs, and post-training is shown to install the Assistant persona into the workspace.
interpretabilityglobal workspacemechanistic interpretabilityreasoningalignment - paperarXivTwo-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Xilun Chen, Zhaleh Feizollahi, Ross Goodwin, Seungwhan Moon, Scott Yih, Pinar Donmez, Babak Damavandi, Luna Dong
The paper introduces a two-level meta-rubric that encodes the structured organization and importance of required facts for open-ended answers and mechanically compiles them into flat binary checklists for reliable LLM judging. Instantiated as GAMUT, it provides 1,813 evidence-backed questions grounded in wearable imagery across 10 domains (plus a text-only variant). Across 14 models the best score is only 58.7%, demonstrating that factual completeness remains hard and highly discriminative.
evaluationfactualitycompletenessbenchmarksmultimodal - paperarXivStructured Output Collapses Answer Diversity Across 44 Language Models
Tapan Parikh
Re-running wide-answer-space prompts under a simple JSON request (no schema enforcement) across 44 models sharply increases modal concentration and reduces answer surprisal; the effect is progressive and specific to tool-use-trained formats (JSON/XML) while absent or reversed for YAML/CSV/arbitrary wrappers. Within-model defaults shift 53% under JSON, mostly toward the crowd, and decoder-level enforcement adds no further compression. The production structured surface therefore yields a measurably more homogeneous model than the chat surface used for evaluation.
diversitystructured outputJSONevaluationmodel behavior - paperarXivBad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems
Soham Gadgil, David Alexander, Sai Sunku, Franziska Roesner
The work systematically evaluates prompt-injection attacks that exploit persistent memory files in Claude Code and OpenAI Codex across four frontier models. While agents rarely overwrite their own memory from untrusted external content, payloads already planted in memory files reliably compromise current and future sessions, with large variation by system, model, goal, and multi-session strategy. Persistent memory therefore fundamentally alters the prompt-injection threat model and motivates defenses that secure memory updates without eliminating adaptation.
securityprompt injectionagentsmemorysafety - paperarXivFrontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu
Even frontier LLMs fail at exact string copying well inside their context windows because 1D positional encodings induce a local-context matching shortcut rather than precise position lookup. The authors introduce 2D-RoPE that lays text on a 2D grid with row and column IDs, turning copying into fixed-offset retrieval. Shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times longer than training, and the advantage persists in large-scale pretraining up to 1.4B parameters on DCLM.
positional encodingsTransformerscopyingarchitecturepretraining - paperarXivRoboTTT: Context Scaling for Robot Policies
Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu, Tianyuan Dai, Scott Reed, Li Fei-Fei, Yuke Zhu, Linxi "Jim" Fan
RoboTTT integrates test-time training into vision-language-action policies so that recurrent fast weights compress long histories, scaling visuomotor context to 8K timesteps without increasing inference latency. This unlocks one-shot in-context imitation from human video, on-the-fly improvement, perturbation robustness, and completion of multi-stage long-horizon tasks. On real-robot manipulation it improves 87% over single-step baselines, fully solves a ten-stage assembly no baseline finishes, and shows 62% gains from pretraining at 8K versus 1K context.
roboticsVLAtest-time traininglong contextfoundation models - paperarXivKimi K3: Open Frontier Intelligence
Kimi Team (Tongtong Bai, Yifan Bai, et al.)
Kimi K3 is a 2.8-trillion-parameter MoE model (104B activated) with native vision and 1M-token context, built on Kimi Delta Attention, Attention Residuals, and Stable LatentMoE that together deliver ~2.5× scaling efficiency over Kimi K2. Post-training emphasizes multi-domain RL (general, agentic, coding) at multiple reasoning efforts plus infrastructure for million-token agentic RL. It reaches frontier-level results on long-horizon coding, agentic, knowledge, reasoning, and vision benchmarks while remaining fully open-weight, trailing only the strongest proprietary systems.
large language modelsMoEopen weightsscalingmultimodal