Document classification

This script asks for the document’s type and files it only on a certified answer:

import json
import urllib.request

QUESTION = {
    "type": "choice",
    "instructions": "What type of document is this?",
    "criteria": {
        "invoice": "A bill asking for payment",
        "contract": "An agreement between parties",
        "purchase_order": "An order placed with a supplier",
        "receipt": "A proof of payment",
        "other": "Anything else"
    }
}


def route(text):
    request = urllib.request.Request(
        "http://127.0.0.1:8000/v1/systemone",
        data=json.dumps({"state": text, "questions": {"type": QUESTION}}).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request) as response:
        answer = json.load(response)["answers"]["type"]
    # A certified answer acts on its own; anything else goes to a person.
    if answer.get("certified"):
        return f"file as {answer['choice']}"
    return "to a person"


print(route("INVOICE No. 2026-117. Amount due: 1,240.00 EUR by 30 November."))

Run against the CPU stand-in at v0.18.1 on 2026-10-19. It printed:

to a person

Until the task is certified (step 2), every answer goes to a person, as on the stand-in.

Good fit

• A fixed set of document types, and past documents sorted into them.
• Documents whose text fits the server’s length, or whose first pages decide the type.

Not a good fit

• Documents whose type depends on images or layout more than on their text.
• Very long documents read whole: the first read grows with the length (Scale).

1. Write the question

One choice question, with a description for each type and an other type. Send the document’s text, or its first pages, as the text.

2. Teach it and certify it

Teach the question from past documents and certify it at your error target.

Send at least 118 labelled documents. The server holds back half to certify, and a 5% certificate needs 59 of those. More examples certify a larger share.

POST /v1/tasks
{"id": "document-classification",
 "examples": [{"request": {"state": "<a past item of yours>", "questions": {"type": <the question above>}},
               "answer": "invoice"},
              ...],
 "certify": {"alpha": 0.05}}

A shape to fill in, not a command to run.

Certify answers runs this call on a server, with its answer.

3. Route on the certificate

• A certified answer files the document by its type.
• Everything else goes to a person.
• When a new kind of document arrives, certification is suspended until you certify again (Watch for drift).

What it did

• A recipe: decisio is measured on short texts, such as the intent sets of the support inbox; no run on long documents is published.

What next

• Scale: long texts, and what the first read costs.
• Requirements and limits: how long a text each model reads.
• Use cases: the other recipes.