BridaJournal
← Journal

Research

Introducing ReflexBench v1: an open benchmark for System One models

Brida is open-sourcing ReflexBench v1, a reproducible benchmark for typed decision engines with calibration, multilingual, robustness and workflow-policy evaluation.

Brida September 23, 2026 Updated September 23, 2026
ReflexBenchSystem One modelstyped decision modelsAI benchmarksBrida ReflexcalibrationJevLaya
ReflexBench v1 open benchmark for System One models with the public-hard leaderboard

The result in 30 seconds

ReflexBench v1 is an open, frozen evaluation protocol for System One models and typed decision engines: fast models that return bounded decisions such as binary judgments, choices and semantic scores instead of generated prose.

The launch leaderboard uses one rule: same public-hard corpus, same 111 rows, raw semantic accuracy. On that exact lane, TypeSafe Jev reaches 73.0%, followed by upstream Reflex / Qwen3.5-2B at 41.4%, the frozen Qwen3.5-0.8B readout control at 39.6%, jeff / GLiFormer at 37.8%, openJev Verdict 1.4 at 36.9%, Laya base at 35.1%, and Kev-0.8B at 32.4%.

That is the v1 launch headline. It is deliberately narrow and reproducible. ReflexBench does not collapse model quality, latency, calibration or Brida Reflex policy effects into one synthetic score. Supplemental policy/harness ablations remain published as research evidence, but they are not part of the primary raw-engine leaderboard.

What exactly are we publishing?

A benchmark is more than a chart. To make a result independently inspectable, a release needs to define what was tested, under which protocol, how it was scored, which failures were retained, and which exact artifacts produced the published numbers.

ReflexBench v1 therefore publishes five layers together:

  • Protocol — the task shapes, allowed outputs, scoring rules and failure semantics.
  • Frozen corpora — versioned evaluation rows with hashes so later runs can prove they used the same data.
  • Harness — the executable runner that normalizes engines into the same typed contracts.
  • Receipts — machine-readable per-run evidence containing engine identity/revision, row counts, metrics, latency where meaningful and retained failures.
  • Release record — methodology, benchmark card, results and manifest checks that tie the public claims back to those artifacts.

That separation is intentional. The benchmark should survive changes in Brida Reflex itself: a future engine can be evaluated against v1 without rewriting the rules after seeing its result.

What ReflexBench measures

ReflexBench focuses on three bounded decision shapes:

  • Binary / Noul — a calibrated yes/no decision, including abstention where supported.
  • Choice — selecting from a bounded set of options.
  • Score — returning a bounded semantic score software can consume directly.

For those shapes, v1 measures semantic correctness together with dimensions that raw accuracy often hides: calibration/confidence behavior, English/Spanish consistency, option-order robustness, supported cardinality, typed-output failures and selective/review behavior.

ReflexBench also retains supplemental ablation lanes for harness and policy research. They are reported separately from the canonical raw-engine leaderboard so model quality and system-policy effects are not conflated.

Engine comparison: same corpus, same rows

The cleanest public raw-engine comparison in the v1 release is the frozen public-hard cohort: 111 identical rows evaluated across the engines below. The machine-readable receipts under results/v1/ are the source of truth.

Engine Semantic accuracy ECE ↓ Score MAE ↓ p50 Execution boundary
TypeSafe Jev 73.0% 0.088 0.410 345.8 ms Hosted route
upstream Reflex / Qwen3.5-2B 41.4% 0.271 0.898 124.0 ms Local RTX 3070 HTTP
frozen Qwen3.5-0.8B readout control 39.6% 0.215 0.711 excluded Reference kernels
jeff / GLiFormer ~400M 37.8% 0.232 0.581 57.6 ms Local RTX 3070 HTTP
openJev Verdict 1.4 / 151M 36.9% 0.124 0.662 64.0 ms Local RTX 3070 HTTP
Laya base / 421M 35.1% 0.236 0.602 70.9 ms Local RTX 3070 HTTP
Kev-0.8B 32.4% 0.440 0.603 109.6 ms Local RTX 3070 HTTP

Naming note: TypeSafe Jev and third-party jeff / GLiFormer are different projects. In this article, Jev always means the TypeSafe hosted model.

The public-hard rows are now development evidence for Brida and must not later be presented as an untouched Brida model gate. They remain useful for comparing the already-published engine receipts on the same frozen cohort.

The deployment boundary also matters: hosted and local latency are different measurements. ReflexBench publishes those values but does not turn them into a single speed ranking.

Supplemental research: does the Reflex harness help?

This section is not part of the v1 headline leaderboard. It is retained as a separate ablation because a decision harness should justify its extra machinery with measurable behavior.

On the separate public 110-row harness-ablation corpus, the measured effect is engine-dependent:

Engine Raw semantic accuracy Best measured Reflex mode Delta
Qwen-Reflex-2B 68.2% 76.4% +8.2 pp
jeff / GLiFormer 54.5% 58.2% +3.6 pp
Laya typed 62.7% 65.5% +2.7 pp
Kev 0.8B 70.0% 70.0% 0.0 pp
Laya base 58.2% 58.2% 0.0 pp

Across these five open-engine ablations, the simple mean uplift is +2.9 percentage points and the median is +2.7 points. That number should not be mistaken for a universal Reflex effect: two engines are flat, and the jeff interval crosses zero.

Percentage points also understate the impact when the baseline error is already small. Qwen-Reflex-2B moves from 68.2% to 76.4%: +8.2 points, but also a reduction in semantic error from 31.8% to 23.6% — about 25.8% of the remaining error removed. Its paired-bootstrap 95% interval is +1.8 to +14.5 points. For jeff / GLiFormer, the measured +3.6-point delta has an interval spanning zero, so we do not present it as established improvement.

This is precisely why ReflexBench exists. “Our harness makes models better” is too broad a claim. The evidence supports something more useful: specific Reflex policies improve specific failure modes on specific engines, and the benchmark tells us when they do not.

A second test: same response, deterministic policy only

For TypeSafe Jev we can isolate the policy layer even more tightly. On the public same-response product fixtures, the underlying Jev response is held fixed. Applying Reflex Core changes operational accuracy from 96.4% to 100.0%: 4 rescues, 0 harms, 0 additional model calls. In absolute terms that is +3.6 points; in terms of the errors that remained before policy, Reflex removed all 4 of 4 on this slice. That is why raw model quality and operational policy value must be reported separately.

What Reflex adds: the same model moves from 96.4% with the base harness to 100.0% with Reflex on the operational same-response test.
Figure 2. This is operational harness uplift, not a raw engine score: the underlying model response is held fixed while Reflex policy is applied around it.

A separate blind workflow replication (Gate v2) completed all 150 rows and moved from 76.7% to 82.0%, a +5.3-point improvement with a paired-bootstrap 95% interval of +1.3 to +10.0 points and 10 rescues / 2 harms.

We also retain the failed first gate. Gate v1 stopped at 149/150 after a provider 503. Its completed rows improved, but the run remains formally FAIL because the protocol required full completion. A benchmark that deletes inconvenient failed runs is much less useful than one that makes them part of the record.

How to read — and reproduce — a ReflexBench result

A published number should be traceable through a short chain:

benchmark version
    ↓
frozen corpus + SHA-256
    ↓
engine + exact revision / hosted identity
    ↓
runner configuration
    ↓
per-row receipts
    ↓
aggregate metrics + failure record
    ↓
release manifest

That is the standard we want for future ReflexBench releases. If a result cannot identify its corpus, engine, configuration and receipts, it should not enter the canonical comparison.

The current release is v1, not the final word. v1 freezes the protocol so new engines/checkpoints can be added without moving the goalposts. A material change to the core corpus or scoring becomes v2 rather than silently rewriting v1 history.

Search and retrieval are a separate lane

Search can dramatically improve a decision when the prompt is missing fresh evidence. But silently giving search to one engine would stop this from being a model comparison.

So the architecture is explicit:

web / search / retrieval
        ↓
provenance + freshness filters
        ↓
bounded evidence
        ↓
System One engine
        ↓
Reflex policy / review

A retrieval-augmented ReflexBench lane should therefore be published as its own protocol extension: same retrieval budget, same evidence contract, same corpus and separate receipts. That lets us measure model quality, retrieval quality and system quality without conflating them.

Why publish this openly?

ReflexBench is useful to Brida only if the methodology is harder to game than a marketing chart. The public release includes the benchmark contract, methodology, synthetic public corpora, harness, receipts and release checks. Brida’s hosted control plane, provider credentials, tenant data, private corpora and future hidden gates remain private.

The public Reflex ecosystem now has three complementary surfaces:

  • Brida Reflex — open decision recipes and registry.
  • Brida SDK — the integration surface.
  • ReflexBench — protocol, methodology, receipts and reproducible results.

The repository contains the complete results, methodology, benchmark card and release-manifest verification.

Explore ReflexBench v1 on GitHub →