Execution cost stays local to the candidate
EngramState never loads the full state repository for one request. The retriever selects Top-K candidates and the executor loads one candidate state per attempt. This means the cost of a single state-loading + generation step does not increase just because the catalog contains more tools.
What does scale with tool count?
Similarity comparisons increase with the catalog size, just as they do in retrieve-then-prompt systems. In our 100-tool measurement, similarity computation plus Top-K scanning took only 0.0304 ms out of 24.1 ms total retrieval time—about 0.1% of retrieval latency.
| Tools | Hit@1 | Hit@4 | Tool embeddings | Total retriever memory |
|---|---|---|---|---|
| 24 | 0.9833 | 0.9950 | 12.0 KiB | 8.26 MiB |
| 50 | 0.9440 | 0.9929 | 25.0 KiB | 8.28 MiB |
| 100 | 0.8990 | 0.9850 | 50.0 KiB | 8.30 MiB |
| 206 | 0.8576 | 0.9563 | 103.0 KiB | 8.36 MiB |
Why Hit@4 matters
EngramState can traverse multiple candidates, so Hit@4 is often more operationally meaningful than Hit@1. At 206 tools, the MLP retriever still places the correct tool in the Top-4 about 95.63% of the time.
The retriever objective is weighted
The State Retriever is not trained by blindly adding every loss term with equal weight. The objective combines a retrieval loss, an auxiliary tool-classification loss, and a hard-negative ranking loss. The auxiliary coefficients are treated as hyperparameters and selected from values between 0 and 0.3.
L = L_retrieval
+ λ_cls · L_tool_classification
+ λ_hard · L_hard_negativeThis matters because the auxiliary tasks are useful regularizers, but allowing them to dominate can hurt the representation used for actual retrieval.
End-to-end retrieval, not just ranking
A retriever can have good Hit@K and still create a poor system if it is large, slow, or increases retries. We therefore measured the complete pipeline.
| Retriever | Hit@1 | Hit@4 | Acc. | Soft Acc. | TTFT | Avg calls | Storage |
|---|---|---|---|---|---|---|---|
| Oracle | 1.000 | 1.000 | 0.785 | 0.896 | 0.0371 | 1.73 | 0 |
| EngramState | 0.940 | 0.995 | 0.740 | 0.848 | 0.0610 | 1.77 | 8.24 MiB |
| MPNet FT | 0.940 | 0.990 | 0.740 | 0.840 | 0.0500 | 1.81 | 417.66 MiB |
| BGE-base FT | 0.935 | 0.995 | 0.730 | 0.835 | 0.0493 | 1.86 | 417.64 MiB |
| MiniLM FT | 0.920 | 0.990 | 0.725 | 0.835 | 0.0462 | 1.83 | 86.64 MiB |
Offline compilation scales linearly
State preparation increases linearly with tool count, but compilation is separated from online inference.
| Backbone | Time / state | 48-state DroidCall repository |
|---|---|---|
| RWKV-7 0.1B | 0.033 s | 1.58 s |
| RWKV-7 0.4B | 0.075 s | 3.60 s |
| RWKV-7 1.5B | 0.140 s | 6.72 s |
| RWKV-7 2.9B | 0.247 s | 11.86 s |
For RWKV-7 1.5B, Base + Corrective compilation for one tool is roughly 0.28 s. Extrapolating the same per-state measurement, 1,000 states take roughly 140 s on one RTX 3090. The cost grows, but it is paid offline and can be parallelized.
Common offline work vs EngramState-specific work
Function-calling systems already pay common preparation costs such as LLM fine-tuning and retriever training. In our setup, LLM fine-tuning takes 2.4 / 4.4 / 6.5 / 13.5 hours for the 0.1B / 0.4B / 1.5B / 2.9B RWKV-7 variants using 10k examples, 24 epochs, and eight RTX 3090 GPUs. The additional EngramState-specific step is state compilation, which is orders of magnitude smaller.