ON-DEVICE AGENTS · REUSABLE RECURRENT STATES

Your device should
remember.

EngramState compiles static tool knowledge once, retrieves the right memory from a live query, and restores it directly into a stateful language model — instead of rereading long tool prompts on every request.

See the live comparison Explore the method Paper · coming at release

Stateful on-device inference Load memory, not prompts

EngramState illustration: an on-device agent organizing reusable tool memories inside a smartphone

EngramStateReusable tool knowledge for on-device agents

97.8%fewer prompt tokens

28.2×faster TTFT

6.19 MiB4-bit states / tool

0.956Hit@4 at 206 tools

Prompt baseline

User query+985 tool-prompt tokens→Long prefill→Tool call

EngramState

User query→Retrieve→Load reusable state→Tool call

01 · MOTIVATION

The tool never changed.
Why read it again?

Function-calling prompts contain mostly static information: tool names, descriptions, arguments, value formats, and examples. Conventional pipelines repeatedly prefill that same text for every request. EngramState treats this information as reusable memory instead of repeated input.

PROMPT BASELINE · EVERY REQUEST

The model rereads the tool specification.

Even a tiny user request is concatenated with static tool knowledge before generation can begin.

“Set a timer for 20 minutes.”

name: set_timer
description: Start a countdown timer on the device.
arguments:
  minutes: integer, required
  seconds: integer, optional
constraints:
  - preserve requested duration
  - reject negative values
  - normalize minutes / seconds
examples:
  "5 minutes"  → set_timer(minutes=5)
  "90 seconds" → set_timer(minutes=1, seconds=30)

+ set_alarm specification
+ create_event specification
+ ... other tool specifications ...

User query+Long tool specs→Prefill→Function call

The same static specification is paid for again on the next request.

ENGRAMSTATE · COMPILE ONCE

The specification becomes reusable state.

The detailed timer knowledge is consumed offline. Runtime only restores the state selected for the live query.

Offline

set_timer specification
↓ single forward pass

Stored

timer_base.state

Runtime

“Set a timer for 20 minutes.”

Retrieve

load timer_base.state

Generate

set_timer(minutes=20)

Compile offline

Consume each tool specification once and retain the resulting recurrent state instead of logits.

Retrieve at runtime

A lightweight state retriever maps the live user query to a small Top-K candidate list.

Load, then generate

Restore the selected state and start generation from the short live query rather than the full specification.

02 · METHOD

Base knowledge.
Corrective memory.

Each tool is represented by a Base State containing its canonical specification and a Corrective State capturing failure rules and hard cases. The corrective path is activated only when necessary.

OFFLINE COMPILATION

Tool specification · set_timer

description: Start a countdown timer
arguments:
  minutes: integer, required
  seconds: integer, optional
examples:
  "5 minutes" → set_timer(minutes=5)

This canonical specification is static and reusable across requests.

↓ single forward pass

Base State · timer_base.statecanonical timer knowledge encoded in recurrent state

Failure rules & hard casesambiguous units · invalid durations · seconds/minutes normalization

↓ single forward pass

Corrective Statetargeted repair knowledge

ONLINE EXECUTION

User Query “Set a timer for 20 minutes.”

↓

State Retriever Top-K candidates → timer

↓

Load Base State timer_base.state

↓

Generate Function Call set_timer(minutes=20)

valid ✓needs correction ↘

Execute Tool

Load Corrective State

03 · RESULTS

Move static knowledge
off the critical path.

Representative DroidCall results with RWKV-7 1.5B show that state loading dramatically reduces prompt processing while keeping tool knowledge available to the model.

	Prompt baseline	EngramState
Prompt tokens	985	22 ↓ 97.8%
TTFT	7.301 s	0.259 s 28.2× faster
Runtime tool knowledge	Repeated text prefill	Reusable state load
4-bit state storage	—	6.19 MiB / tool

STATE STORAGE24.76 → 6.19

MiB per tool when Base + Corrective states are reduced from FP16 to 4-bit per-channel storage.

RETRIEVER SCALE0.956 Hit@4

Measured with 206 tools while keeping the tool-embedding footprint compact.

OFFLINE COST~0.28 s

Base + Corrective compilation for one RWKV-7 1.5B tool state pair on an RTX 3090.

04 · INTERACTIVE WALKTHROUGH

See what happens
between query and action.

This browser walkthrough visualizes the execution path. The Android application will use the real on-device model and invoke actual device actions.

05 · PRACTICAL DEPLOYMENT

Designed for constrained devices.

4-bit state storage

Channel-wise state quantization reduces persistent state footprint without requiring full model quantization to change.

Selective loading

Only retrieved tool states enter the execution path, avoiding a runtime context that grows with the entire tool catalog.

Incremental updates

When an existing tool specification changes, its states can be recompiled independently. New tools can be added without rebuilding every stored state.