Give recurring decisions a fast path

A decision fast path learns a recurring choice or yes/no task, then answers the requests it can certify on the server’s CPU. The decisio model handles requests the classifier passes on. The API calls this feature fast_table.

~50 µsStatic embedding + classifier, short texts

Keep decisio availableThe model answers requests passed on

One shared contextSend only the facts that change

Published static p50: 0.05 ms for median 22-token texts on Apple M5 Pro; x86 range 0.04–0.06 ms. Longer texts cost more. Source: the pinned task guide.

Watch the walkthrough

One support ticket, a faster answer, and the model still available when needed. Then see the same pattern handle a support inbox, a moderation queue and refund checks.

40-second overview · narration and original music · Download video. Illustrative requests and time budgets.

Same pattern. Different decisions.

Your application still receives a typed answer. The fast path changes which requests need a model read.

Support inboxSend each ticket to the right team.

Billing handles payments and invoices. Access handles sign-in problems. Send unusual requests to the model.

I was charged twice for the same order.billing

Two payments came out for one purchase.billing

I cannot sign in after resetting my password.access

Unfamiliar request or policy exception? Keep the model in the route.

Community queueSort new posts as they arrive.

Allow ordinary discussion, block obvious unsolicited advertising, and review ambiguous policy exceptions.

Thanks for the explanation. That fixed my problem.allow

Buy followers now. Visit our discount site.block

Here is how I solved the setup error.allow

Unfamiliar request or policy exception? Keep the model in the route.

Returns deskCheck each return against your policy.

For this example, unopened home items delivered within 30 days are eligible. Send policy exceptions to the model.

Home item. Delivered 12 days ago. Unopened.yes

Home item. Delivered 45 days ago. Opened.no

Home item. Delivered 8 days ago. Unopened.yes

Unfamiliar request or policy exception? Keep the model in the route.

Illustrative inputs and outcomes for choice or yes/no tasks. These are not measured task results.

Teach it. Check it. Let it answer.

Start with human-labelled examples or let model-answered traffic supply them. A candidate learns the recurring decision, checks separate examples and becomes active only when it qualifies.

Figure · Decision fast path

Teach it. Check it. Then let it answer.

1. 1Collect examples

   Use human labels or answers supplied by decisio.

2. 2Fit a candidate

   Learn the recurring choice from those examples.

3. 3Certify and time

   Check separate examples and the saving on this host.

4. 4Activate if qualified

   Answer eligible requests; pass the rest to decisio.

If a candidate does not qualify, keep the model path. Learning from model answers gives an agreement certificate, which concerns matching the model, including its mistakes.

Illustrative task and flow, not a live model run or latency measurement. Behavior: task context and fast tables in the pinned task guide.

Shared rules stay put. New facts keep flowing.

The task can use a long policy, rubric or catalogue. Register that fixed material once as context, and send only the changing facts in each request’s state. The fast path encodes the state alone; the model reads the context together with the state.

Figure · Decision fast path

Store the policy once. Change only the facts.

context · registered onceThe policy stays here

For this example, unopened home items delivered within 30 days are eligible. Send policy exceptions to the model.

Fixed for this task

New order 1 / 3state

Only these facts change

Home item. Delivered 12 days ago. Unopened.

CPU fast pathReads the new facts

Model, if neededReads policy + new facts

Illustrative orders, not live predictions. Keep all changing facts the decision needs in state. Changing the registered context requires teaching and certification again.

Illustrative task and flow, not a live model run or latency measurement. Behavior: task context and fast tables in the pinned task guide.

Move genuinely shared material into context, and keep every changing fact the decision needs in state. Changing the context changes the task and requires teaching and certification again.

See what changes your time budget

The default static path uses token-vector lookups and a classifier. Static embedding + classifier work costs about 0.05 ms for short texts; the model is skipped on the requests that qualify. See the fast answer, fallback and an example workload together against a roughly 50 ms model budget.

Figure · Decision fast path

Three costs. One clear picture.

Fast answer~50 µs

0.05 ms on the CPU
Model read skipped

✓ Done on the CPU

Needs the model~50.05 ms

0.05 ms CPU + 50 ms model budget

Example workload~10.05 ms

Mean with an assumed 80% fast-path share

Every request pays the CPU check. Only fallbacks pay the model read.
0.05 + 20% × 50 ≈ 10.05 ms in this example.

The 80% share and ~50 ms model budget are example assumptions, not a combined benchmark. Network, queueing and monitoring reads are excluded. Measure your own task and host.

Static embedding + classifier: published short-text p50 0.05 ms (50 µs), median 22 tokens, Apple M5 Pro; x86 range 0.04–0.06 ms. Longer texts cost more. Source: the pinned task guide.

Add it to your app

The file-based example below uses refund eligibility. Replace the policy, examples and facts with your own task.

Register the task

On the serving host, install the decisio[table] extra or use the table container. Without the extra, registration returns 422.

The default static backend uses token-vector lookups and a classifier, with no neural network running on the CPU. For short texts, the published embedding-plus-classifier cost is about 0.05 ms (50 µs); longer changing inputs cost more.

Prepare these files beside the script:

File	Contents
task.json	A complete labelled registration: your task id, examples and certification settings, as Teach a task shows
context.txt	The shared policy, unchanged across this task’s requests

For the refund example, each labelled request asks the same refund-eligibility question and contains only the changing order facts. Use at least 100 fitting examples and about 600 or more held-out examples at the default targets. With the default half held out, prepare about 1,200 representative labels to start; the certificate can still find no usable threshold.

Run this with DECISIO_BASE_URL set to your running server:

import json
import os
import urllib.request

with open("task.json", encoding="utf-8") as source:
    body = json.load(source)
with open("context.txt", encoding="utf-8") as context:
    body["context"] = context.read()
if not isinstance(body.get("fast_table"), dict):
    body["fast_table"] = True
body.setdefault("certify", {"alpha": 0.05})

base_url = os.environ.get("DECISIO_BASE_URL", "http://127.0.0.1:8000").rstrip("/")
request = urllib.request.Request(
    base_url + "/v1/tasks", data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request, timeout=300) as response:
    print(json.dumps(json.load(response), indent=2))

Shown, not run: it needs the table extra, a running server, a complete task.json and a context.txt policy; transport tests execute the registration with fixture data.

The script preserves your examples, labels and certificate settings. It adds the shared context and enables the default fast path; an existing fast_table settings object is preserved. Each example is read with the same context the task will use when serving.

Check whether a fast path qualified

Registration fits a candidate, checks its certificate and times the classifier and model on your host. It keeps the classifier only if it clears min_saving, 10% by default. If none qualifies, the task continues through decisio.

Response field	What it tells you
fast_table_report.kept	Whether a classifier was kept
fast_table_report.reason	Why one was declined
fast_table_report.guard	Measured mean times, classifier share and saving
fast_table_report.kinds	Which students were tried and selected

The saving includes the classifier’s cost on every request and the model’s cost on the requests passed on. A retained table is a result for this task and host, not a promised speedup for other traffic.

Send new facts, then read the answer

Save the new order’s facts as state.json. For example, this is a complete illustrative input, not a training set:

{
  "days_since_delivery": 12,
  "opened": false,
  "category": "home"
}

A shape to fill in, not a command to run.

Use the same task.json and run:

import json
import os
import urllib.request

with open("task.json", encoding="utf-8") as source:
    task = json.load(source)
with open("state.json", encoding="utf-8") as source:
    state = json.load(source)

if "question" in task:
    questions = {"decision": task["question"]}
else:
    questions = task["examples"][0]["request"]["questions"]
body = {"state": state, "task": task["id"], "questions": questions}
if "layout" in task:
    body["layout"] = task["layout"]

base_url = os.environ.get("DECISIO_BASE_URL", "http://127.0.0.1:8000").rstrip("/")
request = urllib.request.Request(
    base_url + "/v1/systemone", data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request, timeout=300) as response:
    print(json.dumps(json.load(response), indent=2))

Shown, not run: it needs a registered task, its task.json and a new state.json; transport tests execute the call and check the exact question and task id.

The call selects your task by id and reuses its exact question and any explicit layout setting. The policy is already registered; the request sends only the new state. A policy repeated at the start of the state is read twice and triggers x-decisio-warning.

Answer field	How to use it
answered_by	table means the classifier answered; decisio means the model answered
certified	Whether this answer reaches its applicable certificate’s threshold
certificate_kind	gold concerns human-labelled correctness; agreement concerns matching decisio

For an action that requires correctness certification, check both certified: true and certificate_kind: "gold". A model fallback can still be uncertified; your backend chooses whether to use it as a suggestion or request human review. The support-routing recipe shows that application check.

Start without human labels

You can also let model-answered traffic supply the learning data. Until a candidate qualifies, decisio answers every request. A candidate becomes active only after certification and the host saving check pass.

For this route, replace task.json with a complete question registration such as this:

{
  "id": "refund-policy-learn",
  "question": {
    "type": "noul",
    "instructions": "Is this return eligible for a refund under the policy?"
  },
  "fast_table": {"learn": true},
  "certify": {"alpha": 0.05}
}

A shape to fill in, not a command to run.

Use your real refund policy in context.txt, then run the same registration and request scripts above. For a question registration, the request script names its answer decision; labelled registrations retain the question keys from their examples. No human labels are invented or added to the learning registration.

A learned table can copy the model’s mistakes. Its agreement certificate concerns matching the model, not correctness against human labels. A labelled task can also learn; its certificate stays gold while it has enough held-out human labels.

The explore share sends some table-eligible requests to the model, 0.02 by default. Spot checks also send some table answers to the model, so allow for those model reads when sizing the service. Refits continue as traffic arrives; a failed candidate leaves the current serving path available.

Run it reliably

Keep the changing input focused and sufficient

• Put only genuinely shared rules, definitions or catalogue text in context.
• Keep delivery age, item condition, new messages and other changing facts in state.
• Use a compact, consistent representation and leave out duplicated policy text or unrelated logs.
• If you select an excerpt or summary, preserve every fact needed for the question and evaluate that representation on held-out requests.

A different document on every request is changing input; do not hide it in a supposedly fixed context. Shortening an input is not a substitute for checking the task’s answers. Changing the context changes the task and requires teaching and certification again.

Choose another student when needed

The student turns the changing state into features for the classifier. Use fast_table: true for the default, or an object with a kind.

Setting	Student	Use it when
static, the default	potion-base-8M	Starting with a low-cost classifier
static with static_model: multilingual	potion-multilingual-128M	Your traffic uses several languages
encoder	bge-base-en-v1.5	Its additional CPU cost pays on your task and host
auto	Tries static and encoder	You want registration to select the larger measured saving

Static students read the whole text. The encoder reads at most 512 tokens, from the start and end of a longer text. This is a student-specific input rule, not a universal recommended state length.

The first use downloads the chosen student. Use the table image when the host cannot reach the Hub, or fetch the student files before registration.

Measure the paths on your host

Registration measures one request at a time. Measure caller-visible latency under your expected load too, including requests passed on to the model:

python -m decisio.bench.table_latency \
  --url http://127.0.0.1:8000 \
  --examples train.jsonl --probe probe.jsonl \
  --out table_latency.json

Shown, not run: it needs a running server and the reader’s own labelled examples and probes.

The examples file contains one complete request and human answer per line; probes contain requests outside that set. The command registers temporary table and model tasks, compares them under load, then deletes them. Compare the same text-length and concurrency cells, using wall-clock p50 and p95 as well as mean saving.

Keep it running

Inspect GET /v1/tasks/{id} and status.fast_table for this replica’s live state:

State	What answers	What to check
learning	Decisio while learning data is collected	Whether a candidate has qualified yet
active	Classifier where eligible, otherwise decisio	Latency, certified share and drift
timing	Decisio while this replica checks the saving	CPU headroom and timing results
suspended	Decisio	The alert and whether certification needs updating
unavailable	Decisio	Installed extra, student files and registration
off on a learning task	Decisio	Whether this replica matches the learned version’s serving setup and features

With --tasks-dir or --task-store, learning rows survive restarts; without either they stay in memory. The rows keep features and model answers, not request text.

To roll back, send a retained version number to POST /v1/tasks/{id}/fast_table/rollback. The last five versions are kept.

Share learning across replicas

Use one shared --task-store and one serving setup across the fleet. One replica trains; each replica times a promoted table locally before serving it. Drift suspension and rollbacks reach the other replicas through the store.

Keep clocks synchronized and monitor store reachability. During an outage, replicas keep serving, pending rows have a memory limit, and no new version is promoted. A read-only replica serves tables without logging rows or training.

Good to know

• This fast path supports choice and yes/no questions on the text route; scores, images, adapters and changing-option tasks do not use it.
• Long or difficult changing inputs can certify too little to save time, even with a well-organized context.
• An option absent from the fitting examples is passed to decisio.
• A deferred request pays both the CPU classifier cost and the model cost; test that path too.
• Set fast_table.min_saving from 0.01 to 0.90 if your deployment needs a different required mean saving.
• Install the extra and provide CPU headroom on every replica, and certify again after changing the context or serving setup.

What next

• Teach a task: prepare complete labelled examples.
• Send a fixed text once: register context and understand prefix-cache behavior.
• Certify answers: choose the error target and read certificate fields.