Skip to content
Decisio

DocsGuides

3 min read · updated

Use the probability

Set a threshold on the probability, keep the answers above it, and send the rest to a person or a larger model.

A probability is more useful than a label: it says how sure the answer is. Keep the answers the server is sure about, and escalate the rest.

Figure · Using the probability

Set the line; the answers under it go to a person or a larger model.

Keep answers at or above 0.8: 27 of 1,172 questions escalated (2.3%); 98.5% right on the rest; 14 of 31 wrong answers escalated; 17 confidently wrong and kept.

Each stop is a measured threshold, not an interpolation.

Source: figures.routing_example5 of launch_facts.json: decisio-examples example 5 (Escalate only when unsure), session A, Gemma 4 31B, ARC-Challenge test split, one request per question.

TL;DR

Section titled “TL;DR”

With a server running (Get started starts one), this script asks one choice question and escalates the answer when its likeliest option is below 0.8:

import json
import urllib.request


THRESHOLD = 0.8


body = {
    "state": "My card keeps getting declined when I renew.",
    "questions": {
        "queue": {
            "type": "choice",
            "instructions": "Which team should handle this ticket?",
            "criteria": {"billing": "Invoices, payments, refunds", "access": "Login, passwords, permissions", "bug": "The product behaves wrongly"},
        }
    },
}
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/systemone",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
    answer = json.load(response)["answers"]["queue"]


top = max(answer["probabilities"].values())
if top >= THRESHOLD:
    print("keep:", answer["choice"], round(top, 3))
else:
    print("escalate: the likeliest is", answer["choice"], "at", round(top, 3))
# keep: the likeliest option and its probability, when it reaches the threshold
# escalate: the likeliest option, and its probability, when it does not

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

WordMeaning
thresholdthe probability an answer must reach to be kept
escalatesend an answer to a person or a larger model instead of acting on it
confidently wronga wrong answer above the threshold, which is kept; the one kind the threshold cannot catch
abstentionan opt-in threshold on a declared “can’t tell” option (POST /v1/abstention/tasks)

1. Read the probabilities

Section titled “1. Read the probabilities”

A choice’s answer gives the likeliest option, choice, and every option’s probability; a yes/no gives the probability of yes, and a score gives a probability for each level. confidence is System One’s own figure: for a choice, the top probability rescaled from uniform. The probabilities are tempered by fitted temperatures, which never change the likeliest option, and per-task calibration from your examples makes them more honest still (Teach it your question).

2. Choose the line

Section titled “2. Choose the line”

The figure is a measured sweep of one question: each stop is a threshold that was run, with how many answers it escalated and how right the rest were. A higher line escalates more and leaves the kept answers more accurate; the wrong answers it cannot catch are the confidently wrong ones. Pick the line from the cost of an escalation against the cost of a wrong answer, on your own data.

3. Let the server abstain

Section titled “3. Let the server abstain”

A request can offer a “can’t tell” option of its own, or a server started with --abstain-option can add one (untrained), and a threshold on that option’s probability can be registered from labelled examples that include unanswerable ones (POST /v1/abstention/tasks). The server then abstains when that option’s probability exceeds the threshold and otherwise answers the best of the other options; the probabilities are reported unchanged, and the answer says whether it abstained.

What next

Section titled “What next”