Why the original gain looked small
On DroidCall, the fine-tuned model had already learned much of the expected function-call syntax. Adding a grammar constraint therefore had limited room to improve formatting.
Change the required output format
To isolate the role of constrained decoding, we changed the required output to JSON without additional syntax-constraint training. This creates a setting where the semantic tool knowledge can still be useful while the exact output grammar is unfamiliar.
| Method | Decoding | Valid JSON | Accuracy |
|---|---|---|---|
| Prompt (Top-4) | Free | 0.200 | 0.125 |
| Prompt (Top-4) | XGrammar2 | 0.990 | 0.595 |
| EngramState | Free | 0.645 | 0.410 |
| EngramState | XGrammar2 | 0.980 | 0.615 |
Where structured decoding helps most
The combination is especially useful for errors that are fundamentally syntactic or schema-driven: malformed JSON, missing braces, invalid keys, or strict type/enum constraints. It is less likely to fix semantic errors such as choosing the wrong tool or inventing the wrong parameter value.
A clean systems decomposition
Retriever + EngramState
→ choose / restore semantic tool knowledge
Structured decoder
→ constrain legal output form
Validator
→ inspect semantic + execution correctnessThis separation also makes the agent easier to reason about: retrieval, memory, syntax, and execution validation each have a distinct responsibility.