# How it compares

> The three bases beside Jev and the leading open entries on the Decision Index, with the rule that marks each cell.

Source: https://opendecisio.com/docs/comparison/

The comparison sets decisio’s three bases beside Jev and the three open-weights entries with the highest Decision Index on its public board. It is a record of numbers with the rule that reads them, not a ranking.

 

[Figure: On the Decision Index, the 31B is level with Jev and with the leading open entries. Decision Index 0.2.1, 0 to 100: Qwen3.6-35B-A3B 48.06 (▼); Gemma 4 12B 49.43 (▼); Gemma 4 31B 57.58 (=); Jev 1.13 57.91; Surogate Rune 26B-A4B v3 57.44; Decider chat · Gemma-4-31B 57.33; pplx-decider-v1-27b 56.40. A mark is set against Jev: ▲ ahead, = level, ▼ behind, outside a 95% interval for the benchmark's size.]

 

## TL;DR

 

The numbers come from the public harnesses run against a server; what you can run here is the server they were run against:

 

```
curl -s http://127.0.0.1:8000/v1/models
```

 

Run against the CPU stand-in at v0.11.1 on 2026-10-14.

  

| Word | Meaning |
| --- | --- |
| Decision Index | a public benchmark suite for decision models; decisio’s runs are self-run and pending validation |
| ▲ = ▼ | ahead of Jev, level, behind: set by one rule written before anything was computed |
| self-run | measured by us with the public harness, not by the board |

 

Common mistake

Reading a mark as a ranking. A difference counts only when it lies outside a 95% interval for the benchmark’s size; otherwise the cell is level, and nothing here is a rank.

 

## What the comparison says

 

decisio's three bases set beside TypeSafe's Jev 1.13 and the three open-weights entries with the highest Decision Index on its public board, as of 2026-10-05. Our values are self-run: the Decision Index runs are submitted in [apolinario/decision-index#62](https://github.com/apolinario/decision-index/pull/62) and pending the maintainers' validation, and the JevBench counts are this repository's records, not board entries. Jev's and the other entries' values come only from the public boards, with each board's revision, the fields read and the date read in [`runs/2026-10-05_comparison/`](https://github.com/aminry/decisio/blob/v0.11.1/runs/2026-10-05_comparison/). No figure here comes from any call of ours to TypeSafe's API, and nothing here is a rank.

 

**How a cell is marked.** ▲ ahead of Jev, = level, ▼ behind, by one rule for every cell, written and committed before anything was computed ([`RULE.md`](https://github.com/aminry/decisio/blob/v0.11.1/runs/2026-10-05_comparison/RULE.md)). A difference counts only when it lies outside a 95% interval for the benchmark's size, h = 1.96 √(2m(1 − m)/n), with m the two values' mean and n the benchmark's items; otherwise the cell is level. For the index and the areas, the same per-benchmark variance is carried through the board's formula. The one lower-is-better metric, ForecastBench's Brier loss, is compared in its own direction. Only our cells are marked; the other entries' columns are shown as the board has them.

 

Every benchmark, by area, and the capabilities each side has, are on [Benchmarks](https://opendecisio.com/benchmarks/#compares).

 

## What next

 

- [The evaluation card](https://opendecisio.com/docs/evaluation-card/): what each of our numbers was measured on.
- [Limits](https://opendecisio.com/docs/limits/): what the numbers do not say.
- [Benchmarks](https://opendecisio.com/benchmarks/): the tables, with the record behind every figure.