Skip to content
Decisio

DocsUnderstand

2 min read · updated

How it works

A question becomes one prompt and is answered in one forward pass, read at one position; the state is read once and shared.

Most language-model APIs write text and make you parse it. Decisio asks the model one question at a time and reads its answer straight off the model’s output, so the answer is a probability for every option, and nothing is generated.

Figure · One pass

A question is one prompt; its answer is read at one position, with no text generated.

The state is read once per request; every question reads it from the prefix cache. The letters shown are the options in order.

Recorded decision: Gemma 4 31B, decisio 0.9.0, RTX PRO 6000 at 600 W, 2026-10-08; record 1 of 60 of decisio-examples' support-routing run; the probabilities are rounded to two places. The mechanism is the README's "How it works".

TL;DR

Section titled “TL;DR”

With a server running (Get started starts one), one yes/no question is one prompt and one pass:

curl http://127.0.0.1:8000/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"state": "I was charged twice for October. Please refund one.", "questions": {"urgent": {"type": "noul"}}}'
# {"model": "...", "answers": {"urgent": {"type": "noul", "noul": ...}}, "usage": {"input_tokens": ..., "output_tokens": ...}}

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

usage.output_tokens is one per question: the “token” is the letter that was read.

WordMeaning
forward passone run of the model over a prompt, which is all a question costs
readoutthe model’s distribution over the option letters at the answer’s position
prefix cachewhere the state’s read is kept, so every question about it starts from there
hidden statethe model’s internal representation at the answer position, which an intent head reads

1. The mechanism

Section titled “1. The mechanism”

Each question becomes one prompt, the state followed by the question and its lettered options, and its answer is the model's distribution over the option letters at one position, so every question costs one forward pass and no generated text. The state is prefilled once per request and every question reads it from vLLM's prefix cache. No training: every base is an official checkpoint, frozen, with no fine-tuning and no adapter. decisio's vLLM plugin registers its model classes through an entry point, so vLLM loads them without a patch, and the served class also returns the hidden state at the answer position for the intent head. The design notes: docs/design/vllm-plugin.md, docs/design/hidden-state-readout.md and docs/design/mlx-backend.md.

2. A request

Section titled “2. A request”

One request carries a state, such as a support ticket, and the questions to ask about it. The server answers each question with one forward pass and writes no text.

The request flowYour app sends one request with a state and its questions to POST /v1/systemone. The server reads the state once and keeps it in the cache, answers each question with one forward pass and no generated text, and the response carries a probability for every option.Your appany HTTP clientPOST /v1/systemonea state and questionsDecisio serverreads the state oncekeeps it in the cacheOne pass a questionno text generateda probability for every option, in the response

3. The cache

Section titled “3. The cache”

The state is read once. Every later question about it starts from the cache, which is why asking many questions about one document is cheap.

The cache timelineThe first question about a new 3,000-token document takes 490 milliseconds because it reads the document. Each later question takes 32 to 43 ms because the document is read from the cache. The bars are to scale.One document of 3,000 tokens, Decisio 31B, questions asked one after another, drawn to scalefirst question: 490 msreads the document32 to 43 ms each, from the cachetime

4. The threshold

Section titled “4. The threshold”

Every answer carries a probability for each option. Keep the answers above a threshold you pick and send the rest to a person or a larger model. The tradeoff is yours to set from your own labelled items: how well the number separates right from wrong depends on your questions.

The routing thresholdTwo bars of the 1,172 questions of example 5, drawn to scale. At a threshold of 0.8, 27 are escalated; at 0.9, 47. The rest are kept.Keep answers at or above 0.827 of 1,172 escalated (2.3%), 98.5% right on the restKeep answers at or above 0.947 of 1,172 escalated (4.0%), 98.8% right on the restkeptsent to a person or a larger model

5. The bases

Section titled “5. The bases”

One flag picks the model. The routes and the calibration are the same for each.

The basesOne flag, --base, chooses the model among gemma-4-31b, qwen3.6-35b-a3b, gemma-4-12b. Every base is served by the same Decisio server with the same routes, calibration, task registration and prefix cache.--base gemma-4-31bthe default--base qwen3.6-35b-a3b --base gemma-4-12b One Decisio serverthe same routes, calibration,task registration and prefix cache

What next

Section titled “What next”