# Repeatability

> What repeats and what can move, by base: the same request, a first read against a cached one, a batch against a question alone, and another session.

Source: https://opendecisio.com/docs/repeatability/

A model’s answers move at the scale of any change of rounding: the batch a question rides in, where a long row is split, the card, the driver. So what repeats is stated for a configuration and a request form, base by base.

 

[Figure: Within a configuration, answers repeat; across sessions and rounding, near-ties can move. What repeats, by base. Qwen3.6-35B-A3B: The same request again, the same probabilities every time; A first read against a cached read, identical; Questions batched against alone, the same probabilities, bit for bit; Another session, matched an earlier record on all 1,400 suite items in 4 of 5 restarts; on another card, to 1.1e-16. Gemma 4 12B: The same request again, not bit for bit; A first read against a cached read, up to 0.113 on 60 new states; one choice changed; Questions batched against alone, the same choice, probabilities within 0.035; Another session, up to 0.128 on 212 of 231 items; two near-tied choices changed. Gemma 4 31B: The same request again, exact within one session on one machine; A first read against a cached read, identical; Questions batched against alone, the same choice, probabilities within 0.0073; Another session, near-tied answers moved; 28 top answers changed (1.7%) over 1,631 items.]

 

## TL;DR

 

With a server running ([Get started](https://opendecisio.com/docs/) starts one), this script sends the same request five times and counts the distinct answers:

 

```
import json
import urllib.request

body = json.dumps({
    "state": "I was charged twice for October. Both charges are 49.00 EUR, same day. Please refund one.",
    "questions": {"urgent": {"type": "noul", "instructions": "Does this ticket need a response within the hour?"}},
}).encode()

def ask():
    request = urllib.request.Request("http://127.0.0.1:8000/v1/systemone", data=body, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request) as response:
        return json.dumps(json.load(response)["answers"], sort_keys=True)

print("distinct answers in 5 sends:", len({ask() for _ in range(5)}))
# distinct answers in 5 sends: 1
```

 

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

 

The README’s three-question example, on the 31B under its served defaults, gave the same response in 30 of 30 repeats on one server.

  

| Word | Meaning |
| --- | --- |
| bit for bit | the same number to the last digit |
| first read | the server reading a state it has not seen; a later read comes from the prefix cache |
| batched | several questions scored in one engine step instead of one engine call each |
| near-tied | two options whose probabilities are close, so a small move can change the choice |

 

Common mistake

Comparing an answer with an earlier record from another session as if they were the same experiment. On one day the served 31B profile, with identical prompts and profile, did not reproduce an earlier record: every readout differed, and near-tied items moved by up to 0.98, changing 28 top answers (1.7%); the engine arrangement and the driver had changed. Pair a comparison within one session.

 

## 1. Why answers move

 

The Gemma 4 12B base’s final logits are bf16 after a soft cap, so near the top they lie on a grid of 0.0625 to 0.125, and a state read for the first time and the same state read from the prefix cache can land a step apart. Questions scored in one batch are not the same forward pass as each scored alone: on the Qwen base the dependence was traced to vLLM 0.30.0’s forward, whenever more than one sequence shares an engine step. Every change of rounding moves the answers, which is why each claim holds for a fixed configuration and request form.

 

## 2. What the server does about it

 

On a running server with the Qwen base, a question returns the same probabilities every time, alone or among other questions: each question is scored in its own engine call. Across server restarts, single-question answers have matched an earlier record on the whole suite in most starts measured, and moved slightly without changing a choice in the others. The Gemma base does not repeat bit for bit: its logits come out in bf16 after its soft cap, and a state read for the first time and read again from the prefix cache can land a step apart; several questions in one request stay within the bound in the README's settings table. Any change of rounding moves this model's answers, the batch, the padding or where a long row is split, so every bit-identity claim holds for a fixed configuration and request form. Measurements: [`EVAL_CARD.md`](https://opendecisio.com/docs/evaluation-card/) sections 4 and 6.5.

 

## What next

 

- [Flags and options](https://opendecisio.com/docs/flags/): `--multi-question`, `--register-boundary` and `--engine-process`, which decide what repeats.
- [What the probability means](https://opendecisio.com/docs/probability/): temperatures and calibration.
- [The evaluation card](https://opendecisio.com/docs/evaluation-card/): sections 6.5 and 7.5 have every measurement.