Agentic Memory Helps Until You Cap the Retrieval

The claim, and the spread underneath it

IBM Research published an agentic memory sweep on the Hugging Face blog on 18 August. Eight models were run against AppWorld twice: once cold, once with behavioural guidelines mined from the agent's own earlier trajectories. No weight updates, no human annotation.

The result is a spread rather than a number. gpt-oss-120b, a 117B mixture-of-experts model, moved from 39.9 task goal completion and 21.4 scenario goal completion to 56.0 and 37.5 — plus 16.1 on both. DeepSeek-V3.2 gained 9.5 and 16.1, Claude Opus 4.6 gained 4.1 and 7.1, GPT-5.5 gained 2.9 and 7.2 from 92.3 and 82.1. GLM-5, a 745B mixture-of-experts model starting at 87.5 and 80.4, gained nothing: 0.0 and 0.0.

The post reads that spread as a dosing problem: capable models take the full guideline set, weaker ones do better with curated retrieval, saturated ones take nothing. That is a fair summary of its own table, but the model is not the only variable, and the technical report underneath ablates a second one.

What the two metrics count

AppWorld is not IBM's benchmark. It comes out of Stony Brook, won best resource paper at ACL 2024, and models 9 apps behind 457 APIs populated with the activity of roughly 106 fictional people, across 750 tasks in four splits: train, dev, test_normal and test_challenge. The sweep's 585 tasks are the two test splits — 168 normal and 417 challenge.

Task goal completion is the share of tasks passing every programmatic unit test on API usage, database state and end state. Scenario goal completion is stricter: a scenario counts only when all its variants pass, typically three. One sporadic failure zeroes the scenario.

In every configuration that beats the baseline, scenario goal completion moves further than task goal completion. Agentic memory is measurably better at making an agent behave the same way twice than at making it correct. If your production problem is variance between runs, that is the result you want; if it is accuracy, you are reading the wrong column.

Where the measurement flips

The technical report behind the sweep — Fang et al., IBM Research, arXiv 2603.10600 — runs the configuration ablation the blog post does not. Agent and tip extraction pipeline both use GPT-4.1. Tips are mined from the train and dev partitions and evaluated on held-out test-normal, through a single-agent ReAct loop capped at 30 steps, with five tips injected per task.

Against a no-memory baseline of 69.6 TGC and 50.0 SGC, subtask-level tips with LLM-guided selection reach 73.2 and 64.3 — the plus 14.3 scenario gain in the abstract. On the hardest tier, scenario completion goes from 19.1 to 47.6, a 149% relative increase.

Then there is table 3. Task-level tips retrieved by cosine similarity at threshold 0.5, capped at the top 3, score 66.7 TGC and 48.2 SGC — below the no-memory baseline on both, by 2.9 and 1.8. On difficulty 3 that configuration reaches 46.0 TGC, eight points under an agent with no memory at all.

That is the boundary condition, and it is not the model. Hold the tips and the threshold fixed and lift only the top-k cap: the same task-level tips at threshold 0.5 with no cap score 70.2 and 57.1, above baseline on both. One retrieval parameter is the difference between plus 7.1 scenario points and minus 1.8. Raise the threshold to 0.6 with no cap and the same tips reach 72.0 and 62.5. The report's cross-configuration comparison, which concludes that every configuration beats the baseline, is explicitly scoped to the threshold-0.6 settings; the capped one is not in that table.

The configuration that ships is not the one that was measured

The library is public: AgentToolkit/altk-evolve, Apache-2.0, created in December 2025, 104 stars, 9 contributors, 56 open issues against 46 closed, v1.1.6 tagged on 19 August — the day after the blog post. It needs Python 3.12 or newer and serves its guideline store over MCP.

Its README states that the default filesystem backend uses simple text matching and needs no extra dependencies; semantic vector similarity requires the pgvector or Milvus extra. Every configuration reported in the paper or the blog post is a vector one — cosine similarity, or an LLM-guided selector on top of it. Install this the obvious way and you are running a retrieval strategy that appears in none of the published tables.

The README reports its own headline: plus 8.9 points overall and a 74% relative increase on hard multi-step tasks. Neither appears in the report's result tables, which give plus 3.6 TGC and plus 14.3 SGC in aggregate. Three surfaces of one project, three sets of headline numbers, and the pair on the repository front page appears in neither of the other two.

The falsifiable version of all this: the reported gain is a property of the retrieval configuration, not of the guidelines. Run the same mined guideline set through the shipped text-matching default and reproduce plus 14.3 SGC on held-out tasks, and I am wrong.

What to do with it this quarter

This is the proof-of-concept-to-production gap in miniature. The README documents the backend choice plainly enough; what no published table says is which backend produced the numbers you are comparing yourself against.

• Install with the vector extra, or do not compare against the published numbers. Plain resolution gets you text matching.
• Set the similarity threshold at 0.6 and remove the top-k cap. Top-3 is the setting that went negative.
• Measure your cold baseline, then measure the gain separately. The baseline does not predict it here: GLM-5 started at 87.5 TGC and gained nothing, while Claude Opus 4.6 at 90.5 and GPT-5.5 at 92.3 both gained. Nothing in this sweep lets you skip the second measurement.
• Price the tokens. The full guideline set cost plus 51% tokens per task on gpt-oss-120b and plus 78% on DeepSeek-V3.2; curated retrieval cost plus 5% on gpt-oss-120b for its full 16.1-point gain. If your unit economics cannot absorb 78%, the dosing question is already settled.
• Score the change on run-to-run consistency rather than accuracy, and pick your evaluation metrics to match. That is where every improving configuration moved most.

Two caveats the authors state plainly: this is one benchmark, and no controlled experiment separates context-window size from raw model capability. A third is AppWorld's own: it ships test tasks in encrypted bundles with a canary string to resist contamination, but its licence explicitly excludes training language models from what counts as redistribution. Check that carve-out against your own risk model before treating any of these baselines as clean.