Benchmarks
Each base with its own defaults on one RTX PRO 6000 Blackwell: the Qwen and Gemma 4 12B bases in one session (runs/2026-10-04_gemma-base/), Gemma 4 31B in its own (runs/2026-10-04_gemma-4-31b/), image input in another (runs/2026-09-27_image-input/). Measured by us with the public harnesses (the Decision Index kit 0.2.1) and recorded in runs/; none is a board score.
How Decisio compares
decisio's three bases set beside TypeSafe's Jev 1.13 and the three open-weights entries with the highest Decision Index on its public board, as of 2026-10-05.
Our values are self-run: the Decision Index runs are submitted in apolinario/decision-index#62 and pending the maintainers' validation, and the JevBench counts are this repository's records, not board entries.
Jev's and the other entries' values come only from the public boards, with each board's revision, the fields read and the date read in runs/2026-10-05_comparison/.
No figure here comes from any call of ours to TypeSafe's API, and nothing here is a rank.
How a cell is marked. Each of our cells is ahead of Jev, level or behind (bold on this site when level or ahead), by one rule for every cell, written and committed before anything was computed (RULE.md).
A difference counts only when it lies outside a 95% interval for the benchmark's size, h = 1.96 √(2m(1 − m)/n), with m the two values' mean and n the benchmark's items; otherwise the cell is level.
For the index and the areas, the same per-benchmark variance is carried through the board's formula.
The one lower-is-better metric, ForecastBench's Brier loss, is compared in its own direction.
Only our cells are marked; the other entries' columns are shown as the board has them.
The headline
Bold: level with or ahead of Jev 1.13, by the comparison's rule; plain: behind.
| Measure | DecisioQwen3.6-35B-A3B | DecisioGemma 4 12B | DecisioGemma 4 31B | Jev 1.13 | Surogate Rune26B-A4B v3 |
|---|---|---|---|---|---|
| Decision Index 0.2.1 (0 to 100) | 48.06: behind Jev | 49.43: behind Jev | 57.58: level with Jev | 57.91 | 57.44 |
| Knowledge & Reasoning | 32.45: behind Jev | 31.37: behind Jev | 45.57: behind Jev | 51.40 | 43.36 |
| Language Understanding | 51.39: behind Jev | 51.66: behind Jev | 61.02: level with Jev | 62.02 | 63.06 |
| Retrieval & Classification | 53.56: level with Jev | 58.00: ahead of Jev | 62.75: ahead of Jev | 55.42 | 63.52 |
| Tools & Automation | 68.43: behind Jev | 71.62: behind Jev | 75.04: level with Jev | 75.09 | 71.22 |
| Arts & Human Taste | 31.57: behind Jev | 32.59: behind Jev | 37.43: level with Jev | 37.66 | 41.90 |
| On JevBench's 231 published questions, correct | 200: level with Jev | 200: level with Jev | 213: ahead of Jev | 200 | not compared (h) |
| JevBench hard tier, correct | 82 of 111 | 82 of 111 | 93 of 111 | not compared (a) | not compared (h) |
| System | Latency, median per request | Price or self-hosting cost, one pass over the 38 index benchmarks | Context length | What it is |
|---|---|---|---|---|
| DecisioQwen3.6-35B-A3B | 70.5 ms (b) | $6.98 (e) | 32,768 tokens per prompt, the state plus one question | official Qwen weights, frozen, letters readout |
| DecisioGemma 4 12B | 37.8 ms (b) | $6.92 (e) | 32,768 tokens per prompt | official Gemma weights, frozen, letters readout |
| DecisioGemma 4 31B | 56.0 ms (b) | $10.28 (e) | 32,768 tokens per prompt | official Gemma weights, frozen, FP8 on load, letters readout |
| Jev 1.13 | 524.1 ms (c) | $6.05 (f) | 64k tokens per request; 32k tokens for the state plus the longest question (g) | hosted model, TypeSafe's API |
| Surogate Rune26B-A4B v3 | 120.5 ms (d) | $16.53 (e) | not on the board | full fine-tune of google/gemma-4-26B-A4B-it |
| Decider chatGemma-4-31B | 108.5 ms (d) | $22.18 (e) | not on the board | an inference technique on google/gemma-4-31B-it |
| pplx-decider-v1-27b | 101.4 ms (d) | $24.94 (e) | not on the board | full fine-tune of Qwen/Qwen3.8-27B |
(a) JevBench's board reports Jev's hard tier over 220 questions, 109 of them held out and never published, so it is not the same item set as the 111 published ones we can run.
(b) Our server time on one RTX PRO 6000 Blackwell: the Decision Index kit's HTTP wall time to a server on the same machine, one request at a time, over all 150,759 requests.
(c) The board's own measurement of TypeSafe's hosted API: an HTTPS round trip over the internet from its lab, one request at a time, on its latency sample. It includes the network, so it is not the same measurement as (b) and (d), and it carries no mark.
(d) The board's on-card median on one RTX PRO 6000, one request at a time, on its latency sample.
(e) Self-hosted, estimated as mean request latency × 150,759 requests at $1.48 per card-hour, what the card for our runs cost on vast.ai; for the other entries, with the board's mean latency. One request at a time throughout; a server answering requests concurrently would cost less per pass.
(f) The board's recorded API cost of Jev's run over the same benchmarks, at TypeSafe's tariff of $0.042 per million input tokens, output tokens free (jev.results[].usd).
(g) From TypeSafe's documentation (section 3).
(h) Not on JevBench's board.
Capabilities
Each statement about Jev is what TypeSafe's public documentation states, quoted in snapshots/typesafe_docs.json with each page's hash, read 2026-10-05 at 21:44 UTC.
Where a row says the documentation does not state something, that is about the pages read, not a claim about the service.
| decisio (all threebases unless noted) | Jev 1.13, as TypeSafe'sdocumentation states it | |
|---|---|---|
| Question types | yes/no (noul), choice and score, on the same /v1/systemone wire format | "The three TypeSafe question types (Choice, Score, Noul)" (primitives) |
| Options per question | up to 255 named options per choice | "a maximum of 255 options per Choice"; a Score "accepts up to 10" levels (API) |
| Several questions per request | the state is read once and shared by every question in the request | "System One models evaluate every question in a request in parallel." (primitives) |
| Context | 32,768 tokens per prompt: the state plus one question | "64k tokens per request; 32k tokens for state plus the longest question" (models) |
| Self-hosting | yes: one 96 GB card on vLLM, the Docker image, or a Mac (below) | served by "the same endpoint, POST /v1/systemone"; the pages read describe the hosted API and no self-hosted option (models) |
| Open weights | the bases' official weights, downloaded at pinned revisions, never redistributed | "the same weights serve every account"; the pages read name no downloadable weights (models) |
| Teaching from labelled examples | POST /v1/tasks fits a per-task calibration and, for long option lists, an intent head on your labelled examples; the model stays frozen | "Jev is not fine-tuned or LoRA-adapted with customer data", and "You shape its answers to your domain through the request rather than through per-account weights" (models) |
| Image input | photos in the state, through a second engine on the same card (--image-model); measured on the Qwen base only | "Text only. String, JSON object, or array of text values. No image, audio, or video input." (models) |
| Repeatability | checkpoints at pinned revisions; Qwen: a repeated request returns the same probabilities bit for bit; Gemma 4 12B: the same choice, probabilities within 0.035; Gemma 4 31B: a repeated request returns the same answers with the engine in the server's process, within one session on one machine; across sessions near-tied answers can move (EVAL_CARD.md sections 5 to 7) | "An alias moves when a new release ships, so the answers behind it can change without a change on your side." The pages read do not state whether a repeated request returns the same probabilities (models) |
| Calibration | fitted temperatures per base; JevBench ECE on the standard and hard tiers: Qwen 0.121 and 0.043, Gemma 4 12B 0.033 and 0.085, Gemma 4 31B 0.035 and 0.091 | "System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty." (System One) |
| Where data goes | to the machine that runs the server, and nowhere else | "Jev is not trained on customer requests or responses."; zero data retention is offered "for enterprise customers" (models, legal) |
| Mac and Ollama | Qwen and Gemma 4 12B on Apple silicon with MLX (not the 31B); the Qwen base on Ollama (aminroudaki/decisio), with Ollama's own prompt and no calibration | the pages read describe the hosted API only |
| Licence | Apache-2.0; the weights are Qwen's under Apache-2.0 and Google's Gemma 4 under Apache-2.0 with Google's Gemma Prohibited Use Policy | a hosted service under TypeSafe's customer agreements, which "govern your use of TypeSafe" (legal) |
| Price | no per-call price; the card's cost (section 1, e) | "$42 / $0.042" per billion / million tokens, "Charged per input token. Output tokens are free." (models) |
Every index benchmark
One row per benchmark, grouped by area, with the coverage-adjusted value that feeds the index (raw on the board); ForecastBench shows its Brier loss, where lower is better.
n is the number of items behind the interval: the smallest of the benchmark's cases, our scored requests and Jev's cases.
Each interval is in runs/2026-10-05_comparison/results.json, and tables.md there is these tables as the script writes them.
Bold: level with or ahead of Jev 1.13, by the comparison's rule; plain: behind.
Knowledge & Reasoning
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| GPQA Diamond, accuracy | 196 | 0.510: behind Jev | 0.388: behind Jev | 0.520: behind Jev | 0.786 | 0.469 | 0.490 | 0.495 |
| GSM8K, accuracy | 1,319 | 0.544: behind Jev | 0.599: behind Jev | 0.805: level with Jev | 0.799 | 0.797 | 0.826 | 0.674 |
| ChessBench, accuracy | 5,000 | 0.162: level with Jev | 0.165: level with Jev | 0.220: ahead of Jev | 0.172 | 0.187 | 0.223 | 0.172 |
| MuSR, accuracy | 752 | 0.581: behind Jev | 0.581: behind Jev | 0.640: level with Jev | 0.661 | 0.649 | 0.642 | 0.618 |
| SATA-Bench, case exact accuracy | 1,650 | 0.232: behind Jev | 0.264: level with Jev | 0.323: ahead of Jev | 0.264 | 0.348 | 0.293 | 0.299 |
| CRUXEval, accuracy | 570 | 0.625: behind Jev | 0.654: behind Jev | 0.832: ahead of Jev | 0.730 | 0.744 | 0.793 | 0.749 |
| CLadder, accuracy | 5,000 | 0.622: behind Jev | 0.634: behind Jev | 0.764: ahead of Jev | 0.726 | 0.727 | 0.746 | 0.745 |
| HLE, accuracy | 501 | 0.114: behind Jev | 0.106: behind Jev | 0.150: behind Jev | 0.204 | 0.106 | 0.148 | 0.120 |
| MMLU-Pro, accuracy | 12,032 | 0.613: behind Jev | 0.550: behind Jev | 0.694: behind Jev | 0.827 | 0.672 | 0.696 | 0.645 |
| BBH, accuracy | 5,507 | 0.694: behind Jev | 0.691: behind Jev | 0.762: behind Jev | 0.929 | 0.811 | 0.762 | 0.781 |
Language Understanding
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| ContractNLI, macro-F1 | 123 | 0.733: level with Jev | 0.743: level with Jev | 0.726: level with Jev | 0.717 | 0.770 | 0.736 | 0.781 |
| ANLI, macro-F1 | 3,200 | 0.686: behind Jev | 0.702: behind Jev | 0.737: level with Jev | 0.748 | 0.735 | 0.732 | 0.707 |
| WinoGrande, accuracy | 1,267 | 0.808: behind Jev | 0.730: behind Jev | 0.852: behind Jev | 0.919 | 0.860 | 0.837 | 0.852 |
| HellaSwag, accuracy | 10,042 | 0.940: level with Jev | 0.867: behind Jev | 0.923: behind Jev | 0.945 | 0.924 | 0.920 | 0.940 |
| ACOS, per-review F1 | 400 | 0.174: behind Jev | 0.155: behind Jev | 0.215: behind Jev | 0.295 | 0.268 | 0.188 | 0.204 |
| FinEntity, macro-F1 | 979 | 0.803: behind Jev | 0.905: ahead of Jev | 0.931: ahead of Jev | 0.870 | 0.884 | 0.909 | 0.929 |
| iSarcasmEval, Sarcasm F1 · track A, English | 1,400 | 0.474: level with Jev | 0.427: behind Jev | 0.506: level with Jev | 0.505 | 0.604 | 0.511 | 0.603 |
| VAST, macro-F1 | 3,006 | 0.490: behind Jev | 0.591: behind Jev | 0.687: ahead of Jev | 0.646 | 0.766 | 0.732 | 0.708 |
| NLI4CT, macro-F1 | 5,500 | 0.772: behind Jev | 0.813: behind Jev | 0.835: level with Jev | 0.841 | 0.804 | 0.829 | 0.848 |
| RAGTruth, F1 on hallucinated class | 2,700 | 0.702: behind Jev | 0.696: behind Jev | 0.783: level with Jev | 0.765 | 0.768 | 0.769 | 0.803 |
Retrieval & Classification
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BANKING77, macro-F1 | 3,080 | 0.746: behind Jev | 0.729: behind Jev | 0.784: level with Jev | 0.797 | 0.838 | 0.791 | 0.790 |
| CLINC150+OOS, macro-F1 | 5,500 | 0.822: behind Jev | 0.872: behind Jev | 0.901: level with Jev | 0.893 | 0.874 | 0.912 | 0.878 |
| BRIGHT, nDCG@10 | 220 | 0.444: level with Jev | 0.471: level with Jev | 0.439: level with Jev | 0.475 | 0.463 | 0.436 | 0.487 |
| Amazon ESCI, macro-F1 | 5,000 | 0.450: behind Jev | 0.535: level with Jev | 0.540: level with Jev | 0.552 | 0.553 | 0.531 | 0.551 |
| PhishNChips phishing decisions, accuracy | 2,000 | 0.752: ahead of Jev | 0.813: ahead of Jev | 0.875: ahead of Jev | 0.625 | 0.809 | 0.875 | 0.600 |
| HoVer, accuracy | 4,000 | 0.700: behind Jev | 0.693: behind Jev | 0.756: ahead of Jev | 0.729 | 0.807 | 0.764 | 0.742 |
Tools & Automation
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BFCL, case exact accuracy | 1,694 | 0.960: level with Jev | 0.962: level with Jev | 0.979: ahead of Jev | 0.958 | 0.948 | 0.980 | 0.976 |
| ToolRet, nDCG@10 | 685 | 0.625: level with Jev | 0.622: level with Jev | 0.613: level with Jev | 0.653 | 0.644 | 0.636 | 0.674 |
| API-Bank, accuracy | 508 | 0.817: behind Jev | 0.870: level with Jev | 0.854: level with Jev | 0.882 | 0.833 | 0.850 | 0.841 |
| Home appliance simulator, case exact accuracy | 88 | 0.398: level with Jev | 0.443: level with Jev | 0.614: level with Jev | 0.523 | 0.466 | 0.625 | 0.739 |
| When2Call, accuracy | 3,652 | 0.714: behind Jev | 0.761: behind Jev | 0.772: behind Jev | 0.810 | 0.760 | 0.769 | 0.817 |
Arts & Human Taste
| Benchmark | n | Qwen3.6-35B-A3B | Gemma 4 12B | Gemma 4 31B | Jev 1.13 | Rune26B-A4B v3 | Decider chatGemma-4-31B | pplx-decider-v1-27b |
|---|---|---|---|---|---|---|---|---|
| BPoMP, accuracy | 811 | 0.904: level with Jev | 0.910: level with Jev | 0.894: level with Jev | 0.909 | 0.900 | 0.906 | 0.939 |
| Humicroedit, accuracy | 2,628 | 0.608: level with Jev | 0.617: level with Jev | 0.625: level with Jev | 0.619 | 0.622 | 0.638 | 0.624 |
| POP909-CL, accuracy | 2,000 | 0.142: behind Jev | 0.060: behind Jev | 0.250: ahead of Jev | 0.166 | 0.661 | 0.279 | 0.378 |
| cfcolor, accuracy | 5,000 | 0.600: behind Jev | 0.624: behind Jev | 0.639: level with Jev | 0.644 | 0.626 | 0.626 | 0.644 |
| ForecastBench, Brier loss (lower is better) | 10,139 | 0.228: behind Jev | 0.202: behind Jev | 0.207: behind Jev | 0.174 | 0.203 | 0.202 | 0.195 |
| Habermas Machine, accuracy | 1,676 | 0.447: level with Jev | 0.465: level with Jev | 0.484: level with Jev | 0.459 | 0.422 | 0.479 | 0.418 |
| New Yorker, accuracy | 528 | 0.688: level with Jev | 0.633: behind Jev | 0.739: level with Jev | 0.701 | 0.741 | 0.739 | 0.703 |
Hosted chat models on JevBench's 231 published questions
The JevBench operator's published counts for the general chat models it measured through their providers' APIs, by the board's model IDs (revision v1.4.2.2, public_accuracy × 231); they are not decision models, have no Decision Index value, and carry no marks.
| Model, as the board names it | Board ID | Correct of 231 |
|---|---|---|
| GPT-6 Luna, low reasoning effort | gpt-6-luna-low | 229 |
| GPT-6 Luna, default medium reasoning effort | gpt-6-luna | 230 |
| GPT-5.6 Luna, low reasoning effort | gpt-5.6-luna | 225 |
| Gemini 3.1 Flash-Lite | gemini-3.1-flash-lite | 201 |
| DeepSeek V4.1 Flash, thinking by default | deepseek-flash | 226 |
| Qwen3.8 27B, on Chutes | qwen3.8-27b | 166 |
Source: docs/comparison.md Jev's GPQA Diamond is the board's jev.benchmarks['25'].raw (0.7857, 154 of 196 scored); the board file also holds jev.results['25'].score = 0.7828, which its GPQA Diamond column and benchmark page display as 78.3. Field: EVAL_CARD.md
Jev leads all three bases on the knowledge-heavy benchmarks, GPQA Diamond, MMLU-Pro, BBH and HLE, and on WinoGrande, ACOS, When2Call and ForecastBench's Brier loss. On the Qwen and Gemma 4 12B bases it also leads most of language and tools, and their indices are 9.9 and 8.5 points below its own. The Gemma 4 31B base is level with Jev on the index and on three of the five areas, ahead on Retrieval & Classification, and ahead on ten benchmarks, among them CRUXEval, CLadder, VAST, FinEntity, HoVer, BFCL and POP909-CL. All three bases are ahead on PhishNChips, and on JevBench's 231 published questions the 31B answers 213 against Jev's 200, with the other two level at 200. Beyond the scores, the difference is in how each runs: decisio runs on your own card or Mac with the data staying there, takes images and learns a recurring question from labelled examples, while Jev is a hosted API with a longer context per request and, at its tariff, a lower cost for one pass over the suite than our one-request-at-a-time estimates.
Each base on the public harnesses
Accuracy and macro-F1, higher is better.
| Measure | Qwen3.6-35B-A3B (default)Record: runs/2026-10-04_gemma-base/ | Gemma 4 12BRecord: runs/2026-10-04_gemma-base/ | Gemma 4 31BRecord: runs/2026-10-04_gemma-4-31b/ |
|---|---|---|---|
| On JevBench's 231 published questions, accuracy: easy / standard / hard | 1.000 / 0.972 / 0.739 | 1.000 / 0.972 / 0.739 | 1.000 / 1.000 / 0.838 |
| Decision Index BANKING77, macro-F1 | 0.746 | 0.729 | 0.785 |
| Decision Index CLINC150+OOS, macro-F1 | 0.822 | 0.871 | 0.902 |
| Decision Index GPQA Diamond (196 scored), accuracy | 0.510 | 0.378 | 0.520 |
| Decision Index MMLU-Pro, accuracy | 0.613 | 0.549 | 0.694 |
| Intent heads from 10 labelled examples per intent, BANKING77 / CLINC150 | 0.840 / 0.912 (six draws) | 0.832 / 0.908 (six draws) | 0.844 / 0.970 (three draws) |
Image input, ImajevBench v2.0-lite, the 230 answerable itemsRecord: runs/2026-09-27_image-input/ | 0.791 | not measured | not measured |
It went in under the maintainer's decision, past a pre-registered rule it missed by 0.2 to 1.1 points on three of four benchmarks; EVAL_CARD.md section 7 states the rule, the numbers and the reason.
Server time
| Base | Server time |
|---|---|
| Qwen3.6-35B-A3B | 49.9 ms |
| Gemma 4 12B | 53.6 ms |
| Gemma 4 31B | 80.7 ms |
Qwen3.6-35B-A3B, Gemma 4 12B: runs/2026-10-04_gemma-base/Gemma 4 31B: runs/2026-10-04_gemma-4-31b/
| Base | Server time |
|---|---|
| Qwen3.6-35B-A3B | 20.2 ms |
| Gemma 4 12B | 24.8 ms |
| Gemma 4 31B | 31.7 ms |
Qwen3.6-35B-A3B, Gemma 4 12B: runs/2026-10-04_gemma-base/Gemma 4 31B: runs/2026-10-04_gemma-4-31b/
The latency rows were measured in one session on decisio 0.8.1's served defaults: the engine in the server's process, and on the Gemma bases a single question on a new state registering its boundary first (runs/2026-10-06_latency-585w/).
The card was one RTX PRO 6000 Blackwell Workstation Edition with its power limit at 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host.
On the Gemma bases that limit held the clock back during most first reads, so a card at 600 W may read new states faster; on the Qwen base it did not.
In the same session, the boundary registration added +16 to +28 ms (12B) and +21 to +34 ms (31B) to the first read of a new state of 300 to 3,000 tokens, and a second, different question then read the state from the cache on 20 of 20 states per base.
An earlier change, paired within its own session:
Gemma 4 12B against Qwen3.6-35B-A3B, paired
What drives the choice, measured on one RTX PRO 6000 Blackwell in one session, each base with its own defaults, paired over the same items (95% bootstrap intervals; runs/2026-10-04_gemma-base/):
| Measure | Gemma 4 12B | Qwen3.6-35B-A3B | Gemma minus Qwen |
|---|---|---|---|
| JevBench v1.5 open-set reading, I_open (equal types) | 64.4 | 49.4 | +15.0 [+7.0, +23.6] |
| JevBench v1.5, yes/no / score | 49.0 / 63.7 | 13.7 / 57.2 | |
| Yes/no answers between 0.20 and 0.80 (74 items) | 15% | 39% | |
| JevBench, published items, correct | 0.0 points [-4.3, +4.3] | ||
| Decision Index accuracy, CLINC150+OOS | +4.5 [+3.6, +5.5] | ||
| Decision Index accuracy, BANKING77 | -1.4 [-2.7, -0.1] | ||
| Decision Index accuracy, GPQA Diamond | -13.6 [-21.7, -5.6] | ||
| Decision Index accuracy, MMLU-Pro | -6.4 [-7.2, -5.5] | ||
| 1,400-item suite, accuracy | 0.735 | 0.770 | -3.5 [-5.5, -1.6] |
One question on a new 3,000-token state, server time (0.8.1's served defaults, the engine in the server's process; one RTX PRO 6000 at 585 W, AMD Ryzen Threadripper 9960X; one session, runs/2026-10-06_latency-585w/) | 258.6 ms | 85.2 ms |
Record: runs/2026-10-04_gemma-base/
Calibration
The calibration of each base is in the capabilities table above.
EVAL_CARD.md has the full tables, the calibration figures and the disclosures of what was fitted on what (sections 4, 6.4 and 7.4).
The Decision Index, whole suite: self-runs
Self-run, submitted to the board and pending the maintainers' validation; no rank is claimed.
| Base | Tag | Decision Index 0.2.1 | Raw index | Breadth | Request latency, median / p95 |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (FP8) | v0.4.0 | 48.06 | 60.45 | 46.55 | 70.5 / 261.2 ms |
| Gemma 4 12B (bf16) | v0.4.0 | 49.43 | 61.47 | 47.57 | 37.8 / 275.9 ms |
| Gemma 4 31B (FP8 on load) | v0.6.0 | 57.58 | 67.33 | 56.50 | 56.0 / 439.3 ms |
Source: EVAL_CARD.md
What these numbers do not show
Source: EVAL_CARD.md
- One card per run and one run per configuration.
- JevBench's v1.5 reading is taken on JevBench's 231 published questions, of the board's 904 open items, with no judge tier and no sealed half; its published items have been public since v1.2 and may be in any model's pretraining data.
- The Decision Index rows of sections 3, 6 and 7 were rebuilt with the kit for four benchmarks; their counts and the rows' sha256 match (
decisio.bench.di_rows), but a partial rebuild cannot be checked against the whole suite's hash. Section 8's runs used the whole suite, rebuilt with the kit and matched against the lab's hashes. - The ImajevBench record was made without the rendering rules and the key-order tie-break; 2 of its 254 items ended in exact ties that the harness rejected and are counted wrong.
- The second-engine head mode with the registered text-only class is unmeasured for latency.
- Latency is server-side on the card's localhost; cost is the card-hour price divided by measured throughput, with nothing else counted.
- Latency depends on the host and the card's power limit.
The latency rows marked 585 W come from one RTX PRO 6000 Blackwell Workstation Edition whose power limit was 585 W (default 600 W), on an AMD Ryzen Threadripper 9960X host (
runs/2026-10-06_latency-585w/). On the Gemma bases the limit held the clock back on 72 to 92% of the busy samples, one request at a time included, so a card at 600 W may read new states faster; on the Qwen base it did on 0 to 2%. A cached Qwen question depends on the CPU instead: 39.4 ms in-process on an AMD EPYC 7452 host, 18.5 ms on an AMD Ryzen 9 9950X and 20.2 ms on the Threadripper, the same card model and vLLM (runs/2026-10-05_engine-death-gates/,runs/2026-10-06_latency-0.8.1/). Latency rows not marked come from hosts whose CPU and power limit were not recorded. - The per-item Decision Index records keep each request's id, the payload's sha256 and our response, not the item text (GPQA's authors ask that its items not be published in plain text).
Every table
The evaluation card has the full tables for each base, the calibration figures and what was fitted on what: the evaluation card.