Certify answers

Add a certify block when you teach a task, and every answer says whether it is certified. Pick how often a certified answer may be wrong; the server finds the threshold that keeps that promise:

Figure · Certifying answers

A stricter error target certifies fewer answers; the server picks the threshold that keeps its promise.

Error target: at most this share of certified answers wrong

2%5%10%

At an error target of 5%, answers with a top probability of at least 0.941 are certified: 1,335 of 3,080 messages (43.3%), 43 of them wrong (3.2%). The rest come back marked not certified.

Each stop is the server's own certificate, made on 1,535 held-back examples and checked on 3,080 test messages it never saw. The contrasting end of the certified bar is the answers that were wrong.

Source: the teach-and-certify runs, Gemma 4 31B on its FP8 checkpoint, decisio 0.14.0, 2026-10-16 to 2026-10-17.

1. Know the promise

Pick an error rate, say 5%: of the answers decisio certifies, at most 5% are wrong, for traffic like your examples, nine times in ten (the default delta of 0.1).

• The promise is about the rate: even so, one certified answer can still be wrong.
• A certificate covers one task on one serving setup: certify again when you change either.
• A stricter rate certifies fewer answers, and too few examples certify none; the rest go to a person.

2. Teach the task with a certificate

Add certify to the registration, with your error target as alpha. With a server running, from the root of the clone (Teach a task has both), add it to the 120 labelled tickets decisio ships and register them:

uv run python -c 'import json; b = json.load(open("examples/tasks/examples.json")); b["certify"] = {"alpha": 0.05}; json.dump(b, open("certify.json", "w"))'
curl -s http://127.0.0.1:8000/v1/tasks -H 'Content-Type: application/json' -d @certify.json \
  | uv run python -c 'import json, sys; c = json.load(sys.stdin)["certificate"]; print(json.dumps({k: c.get(k) for k in ("alpha", "n", "threshold", "certified_n", "errors_at_threshold", "summary")}, indent=1))'

Run against the CPU stand-in at v0.18.1 on 2026-10-19. It printed:

{
 "alpha": 0.05,
 "n": 60,
 "threshold": null,
 "certified_n": 0,
 "errors_at_threshold": null,
 "summary": "Nothing is certified: on the 60 held-out examples no confidence threshold keeps the wrong answers above it at 5% or fewer with probability 90%. The task works as before, and every answer carries certified: false. (Without the bound, 0% of the examples would have met 5% on these examples themselves; that is not a promise.) Warning: This task has no option for a message that fits none of its labels (such as 'other' or 'out of scope'), so such a message gets a wrong answer that can be certified. Add one, or rely on the traffic monitor, which watches the answer mix and the confidences."
}

The server holds back half the examples, teaches on the rest, reads the held-back ones as it will serve them, and answers with the certificate. The CPU stand-in is a small development model, so its certificate is its own. On the default model, Gemma 4 31B, decisio’s worked example, a three-queue ticket question at an error target of 15%, answered:

{
 "certificate": {
  "id": "ticket-queue#7151e6bd7fd1",
  "alpha": 0.15,
  "delta": 0.1,
  "n": 200,
  "accuracy": 0.875,
  "threshold": 0.6917,
  "certified": true,
  "certified_share": 0.905,
  "certified_n": 181,
  "errors_at_threshold": 18,
  "split": {
   "method": "holdout",
   "holdout": 0.5,
   "fit_examples": 200,
   "certify_examples": 200,
   "disjoint": true
  },
  "warnings": [
   {
    "code": "no_catch_all_option",
    "text": "This task has no option for a message that fits none of its labels (such as 'other' or 'out of scope'), so such a message gets a wrong answer that can be certified. Add one, or rely on the traffic monitor, which watches the answer mix and the confidences."
   }
  ],
  "summary": "Answers whose top probability is at least 0.6917 are certified: on the 200 held-out examples that is 90% of the answers, 18 of 181 wrong. With probability at least 90% over the draw of such examples, at most 15% of the certified answers are wrong, for traffic like them and on this serving stack. Warning: This task has no option for a message that fits none of its labels (such as 'other' or 'out of scope'), so such a message gets a wrong answer that can be certified. Add one, or rely on the traffic monitor, which watches the answer mix and the confidences."
 },
 "status": {
  "state": "ok",
  "text": "ok"
 }
}

Recorded output.

Here, answers with a top probability of at least 0.6917 are certified: 181 of the 200 held-back tickets, 18 of them wrong.

Recorded on Gemma 4 31B from its FP8 checkpoint, decisio 0.17.0, one RTX PRO 6000 at 600 W, 2026-10-19; the response is abridged.

If no threshold can keep the promise, threshold is null and the task works as before, with every answer marked not certified.

3. Read each answer

Every answer for the task now says whether it is certified, and why not when it is not:

[
 {
  "ticket": "Urgent: the refund has not reached my bank account. Can you check today? (ref 70816)",
  "choice": "billing",
  "probabilities": {
   "billing": 0.977,
   "shipping": 0.01,
   "access": 0.013
  },
  "certified": true,
  "certificate": "ticket-queue#7151e6bd7fd1"
 },
 {
  "ticket": "Dear support, can I change the payment method on my plan. Also: the tracking page shows no update for days. (ref 82556)",
  "choice": "billing",
  "probabilities": {
   "billing": 0.573,
   "shipping": 0.373,
   "access": 0.054
  },
  "certified": false,
  "certificate": "ticket-queue#7151e6bd7fd1",
  "certified_reason": "below_threshold"
 }
]

Recorded output.

Act on the certified answers on their own, and send the rest to a person or a larger model.

4. Have enough examples

The server refuses to certify below the fewest held-back examples at which anything could be certified, at the default 90%:

Error target	Fewest held-back examples
2%	149
5%	59
10%	29
20%	14

The server holds back half of what you send, so a 5% certificate needs at least 118 labelled examples. That floor is for a task the model always gets right; yours needs more.

With 75 examples, four of five test tasks certified nothing at 5% or 10%. Set certify.min_examples above the floor to refuse a certificate that thin.

5. Keep it valid

1. Give the task an “other” option. A message that fits none of the options gets a wrong answer, and that answer can be certified. The server warns when a choice task has no such option.
2. Certify again when the setup changes. A certificate covers one task on one serving setup: the decisio version, the model and its revision, the engine, the card, the quantisation, the cache type, the context length and the serving flags. When any of it changes, every answer of the task is certified: false with the reason serving_stack_changed. Certify again with POST /v1/tasks/{id}/certify, on new labelled examples.
3. After upgrading decisio, certify your tasks again.
4. Expect a few answers near the threshold to change sides between two fresh starts of the same server: judge the certificate by its rate over many answers, not by one.

What is certified: choice and yes/no questions on text, asked in one order. Score questions and requests with images carry no certificate. A certified question is scored on its own, the way the certificate was made, so it does not share the read of the text with the other questions of its request.

Pick a line by hand

Without a certificate, you can still keep the answers above a line you choose and escalate the rest. Pick the line from the cost of an escalation against the cost of a wrong answer, measured on examples like yours. On decisio-examples' example 5, Escalate only when unsure: 1,172 ARC-Challenge questions, a public benchmark the model may have seen:

Keep at or above	Answered on its own	Right among those (95% interval)	Wrong answers escalated
0.5	1,168 (99.7%)	97.6% (96.6% to 98.3%)	3 of 31
0.6	1,162 (99.1%)	97.8% (96.8% to 98.5%)	6 of 31
0.7	1,154 (98.5%)	98.2% (97.2% to 98.8%)	10 of 31
0.8	1,145 (97.7%)	98.5% (97.6% to 99.1%)	14 of 31
0.9	1,125 (96.0%)	98.8% (98.0% to 99.3%)	18 of 31
0.95	1,078 (92.0%)	99.2% (98.4% to 99.6%)	22 of 31

Recorded on Gemma 4 31B from its FP8 checkpoint, decisio 0.17.0, one RTX PRO 6000 at 600 W, 2026-10-19.

A line on the raw probability promises nothing for new traffic; a certificate does. To keep answers above a line of your own:

import json
import urllib.request

THRESHOLD = 0.8

body = {
    "state": "My card keeps getting declined when I renew.",
    "questions": {
        "queue": {
            "type": "choice",
            "instructions": "Which team should handle this ticket?",
            "criteria": {"billing": "Invoices, payments, refunds", "access": "Login, passwords, permissions", "bug": "The product behaves wrongly"},
        }
    },
}
request = urllib.request.Request(
    "http://127.0.0.1:8000/v1/systemone",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request) as response:
    answer = json.load(response)["answers"]["queue"]

top = max(answer["probabilities"].values())
if top >= THRESHOLD:
    print("keep:", answer["choice"], round(top, 3))
else:
    print("escalate: the likeliest is", answer["choice"], "at", round(top, 3))

Run against the CPU stand-in at v0.18.1 on 2026-10-19. It printed:

escalate: the likeliest is access at 0.509

Let the server abstain

A question can carry a “can’t tell” option, and the server can decline to answer when that option is likely enough:

1. Offer the option in the request, or start the server with --abstain-option to add one.
2. Register a threshold on it from labelled examples that include ones with no answer: POST /v1/abstention/tasks.
3. Read the answer: the server abstains when the option’s probability is over the threshold, and otherwise answers the best of the other options.

Register the task first and its abstention threshold second, and register both again together.

What next

• Watch for drift: know when traffic moves away from the examples a certificate was made on.
• Certified answers on real tasks: the support inbox, a drift alarm and a coding-agent gatekeeper.
• The HTTP API: the certify block and every field of a certificate.