E EngramState
Technical note 01 · Motivation

Why repeated tool prompts are expensive

A tool definition is useful knowledge. Re-reading the same definition every time is repeated work. On a phone, that distinction becomes visible as latency.

Most function-calling systems begin with a sensible recipe: retrieve a few relevant tools, concatenate their textual specifications with the user query, and ask the model to emit a structured call. The specification usually contains a tool name, natural-language description, argument schema, value constraints, and examples.

The problem is not that this information is unnecessary. The problem is that it is mostly static.

A short query can create a long prefill

Consider a user asking, Set a timer for 20 minutes. The live intent is tiny. A conventional tool prompt may still carry hundreds of tokens describing timers, alarms, calendars, messages, settings, and other candidate tools.

User Query
+ Tool 1 specification
+ Tool 2 specification
+ Tool 3 specification
+ ...
        ↓
      Prefill
        ↓
  Function call

In our representative DroidCall setting with RWKV-7 1.5B, the prompt baseline processes 985 prompt tokens. EngramState reduces the live prompt to 22 tokens, a 97.8% reduction, and TTFT falls from 7.301 s to 0.259 s.

The key shift: EngramState is not trying to make the tool text slightly shorter. It removes static tool text from the repeated online path and stores the resulting model state as reusable memory.

Why not just cache a Transformer prefix?

Prefix/KV caching is the natural Transformer analogue. It does avoid recomputing a prefix, but its storage grows with prompt length. For a 1B-scale Transformer, an 800–1,000 token tool description can require roughly 50–62.5 MiB of KV cache for one tool. A larger repository therefore creates a cache-management problem even before considering multiple model sizes.

A fixed-size recurrent state behaves differently. Once a stateful model has consumed the tool specification, the saved state shape no longer depends on whether the original description was 500 or 1,000 tokens. That fixed-size property is the deployment primitive EngramState exploits.

Compile once, continue from memory

Offline
Tool specification → Frozen stateful LM → Save recurrent state

Runtime
User query → Retrieve tool state → Restore state → Generate call

The offline compilation is a single forward pass. There is no gradient computation, backpropagation, or parameter update. The logits are discarded; the final recurrent state is retained.

Why this matters on a real phone

Cloud APIs can hide prefill cost behind large accelerators, batching, and network-level product abstractions. A local agent cannot. Its latency, memory bandwidth, storage budget, and power draw are visible to the user.

We therefore include real-device measurements on a Samsung Galaxy S25 Ultra. The important claim is not merely that a desktop GPU benchmark is fast: the local model/state path remains practical when moved to the device where an agent is expected to execute.

What EngramState trades

EngramState trades repeated online text processing for persistent state files and offline compilation. The system is attractive only if those offline costs are small enough and the state repository can be managed efficiently. That is why the rest of the project focuses heavily on state compression, retrieval, incremental updates, and retry behavior rather than treating the state file as an abstract cache.

← Back to technical notes