Two models. Six memory conditions. The same AI agent runs the same tasks on the same codebase — no memory, hand-written CLAUDE.md, AGENTS.md, both files, Contexer, and Contexer layered on CLAUDE.md. Accuracy, tokens, cost, turns, and rule compliance are measured with code. Every number is re-derivable from raw JSONL.
Asked "why did we choose Postgres over MySQL?" when the decision existed only in memory. Same pattern on both models, 16 runs per condition, stable under three rewordings.
Seeded rule "never log request data": respected 8/8 on both models with any memory. Bare Opus: 0/8. Contexer was also the cheapest way to comply — 68k vs 250k tokens for the same rule in CLAUDE.md.
131k tokens vs 33k on the same question. Consistent with the file being found and read rather than auto-loaded. Directional result, one observation per cell.
Given the same knowledge, accuracy and compliance were identical — both models, 16 runs per cell. Contexer's value isn't out-recalling a perfect file. It's that the file never has to be written: bootstrap mines evidence-backed conventions from the repo in seconds, and mid-session decisions capture themselves. Layering Contexer on an existing CLAUDE.md caused no harm and no single-shot gain.
Published because dropping failed claims is how benchmarks lie.
An early result showed Contexer getting cheaper across chained sessions. Disappeared at higher sample size with proper interleaving. Claim removed.
Across all tasks. Injected context is overhead on tasks that never touch memory. Savings are task-shaped, not universal.
Team-mode benchmark (shared decisions via remote store) is designed but has not yet run. No claims about team mode.
Throwaway HOME. Fresh fixture repo. Environment allowlist — never the developer's real env. Contexer installs via its real installer so real hooks fire.
Synthetic, seeded fixture codebase. Cannot exist in any model's training data.
Conditions alternate in time. Validator warns on contiguous blocks. Every row timestamped.
Answers must contain facts from stored knowledge. Code checked by AST inspection. Tasks pass or fail by their own test commands. No LLM judge.
Separate codebase recomputes every statistic and hunts anomalies. Failed sessions recorded, never silently zeroed.
Every claim must survive a review pass that actively tries to refute it. The CLAUDE.md arm exists because review demanded a fair competitor.
# free end-to-end pipeline check (stub sessions, no tokens) uv run pytest tests/test_bench_*.py -q --no-cov # a live campaign (spends real API tokens — start small) uv run python -m benchmarks.run --reps 1 \ --tasks rat-storage,conv-endpoint \ --model claude-sonnet-5 \ --out benchmarks/artifacts/mine uv run python -m benchmarks.report benchmarks/artifacts/mine/runs.jsonl uv run python -m benchmarks.validate benchmarks/artifacts/mine
The harness lives in benchmarks/ — runner, scorers, validator, fixture generator, task definitions. Campaign artifacts (one JSONL row per session plus validator output) are in benchmarks/artifacts/.
