research infrastructure · active
Agent-memory benchmark
Can an agent retain and correctly use scoped persistent context across a long, distractor-filled task?
Frontier agent work lives or dies on memory: whether a system can hold the right context across a long task without drowning in distractors. This is an independent public benchmark for exactly that — small, ruthless, and honest about where it fails.
Design
- Task — seeded synthetic transcripts: planted facts buried in distractor-filled, long-horizon episodes. No private data anywhere in the pipeline.
- Baselines — no-memory, full-context, and naive retrieval agents, run as policies any LLM agent can plug into.
- Metrics — fact recall/precision, task success rate, provenance accuracy, trap abstention.
- Provenance — append-only results with git sha, timestamp, config hash, and seed. Prior results are never overwritten.
At a generous 8,000-token budget the task does not discriminate — full-context and retrieval both saturate at perfect recall. It only separates baselines when the budget binds: at 500 tokens, full-context drops to 0.25 recall while retrieval holds 1.00. That crossover is what the benchmark is designed to surface. Making the distractors genuinely confusing — paraphrased facts, near-miss traps — is the current work.
What it is not
Not a leaderboard, not a claim about any production system, and not a result until multi-seed baseline runs are logged with ablations. The scaffold, harness, and task definition are done; the science — generator realism, then real runs — is in progress.