MERIT Benchmark Evaluates Cost-Aware Long-Term Memory in Tool-Using LLM Agents

Researchers have introduced MERIT, a benchmark and evaluation harness designed to assess the marginal utility and operational costs of long-term memory in tool-using LLM agents. Unlike conversational recall tests, MERIT examines whether retrieved facts directly influence an agent's task execution across episodic workflows.

Evaluations across 23,440 episodes using models such as GPT-4.1, Claude Haiku 4.5, gpt-4.1-mini, and Claude Sonnet 5 revealed that memory integration increases task success rates from a baseline of 0.00 to between 0.55 and 1.00. However, embedding-based retrieval proved erratic when handling updated information, with agents acting on correctly retrieved data only 55% of the time.

In contrast, update-on-write mechanisms such as structured fact stores and LLM summarization maintained success rates between 0.70 and 1.00. The study also determined that full context replay is never cost-effective, with optimized memory implementations delivering 2.7x to 3.9x higher marginal utility per dollar.