Inside the memory
of an on-device agent.
Long-form technical notes from the paper and the experiments we ran while stress-testing the design: how states are constructed, how retrieval scales, what happens at 3 bits, how retries behave on a Galaxy S25 Ultra, and what changes when tools are updated or combined.
Why repeated tool prompts are expensive
Why a short user request can create a long prefill, how recurrent states differ from Transformer prefix caches, and why the distinction matters on a phone.
How Base and Corrective states are constructed
200-example retention calibration, model-specific state boundaries, single-forward-pass compilation, and why rare hard cases live outside the common path.
How far can recurrent states be compressed?
Why channel-wise quantization works for WKV states, 4-bit deployment, and what our 3-bit / 2-bit experiments say about the practical compression limit.
Scaling to 206 tools without a large retriever backbone
Which costs grow with tool count, Hit@1/Hit@4 scaling, end-to-end retrieval quality, and the storage trade-off versus MPNet/BGE/MiniLM.
Worst-case retry latency on Galaxy S25 Ultra
Real-device retry bounds, single-call and multi-call tails, and why four EngramState retries can still finish before one long-prompt attempt.
Dynamic updates and multi-tool execution
Incremental state compilation, new/deleted tools, model-specific state compatibility, joint multi-tool states, and query decomposition.
Combining EngramState with XGrammar2
Why state memory and grammar constraints solve different problems, plus the JSON-format experiment where constrained decoding sharply improves valid outputs.