Auditing the training loop: a practical response to We Must Pace the Frontier

In September 2026, Dario Amodei published We Must Pace the Frontier.

The proposal is broader than model benchmarking. One of its first concrete steps is Embedded Evaluators: independent third parties with ongoing, employee-like access to frontier labs, able to verify safety practices, report incidents and assess the alignment of not only completed models but also training pipelines and processes.

That last part matters.

If alignment is evaluated only after a model is finished, we see a snapshot.

If it can be evaluated throughout training, we see a trajectory.

This article describes one narrow technical primitive we think can help make that trajectory measurable: checkpoint-level behavioral regression testing with reproducible receipts, combined with independent semantic auditing at execution time.

It is not a substitute for embedded evaluators, interpretability, red teams or governance.

It is infrastructure they could use.

The unit of evidence should be smaller than a model release

A frontier training run is not one event.

The model changes across checkpoints, post-training stages, safety interventions and tool configurations. A behavior that is acceptable at one point can regress later. A mitigation can improve one class of failure while damaging another.

So the evaluation loop should look less like this:

train for months
  -> finished model
  -> safety benchmark
  -> release decision

and more like this:

checkpoint N
  -> frozen behavioral suite
  -> private rotating holdout
  -> dimension-level metrics
  -> reproducible receipt
  -> compare with checkpoint N-1
  -> investigate regressions
  -> continue / mitigate / stop

That does not solve alignment.

It gives an evaluator a time series instead of a marketing claim.

Recent alignment research is already moving in this direction

Anthropic’s August 2026 work on automated alignment researchers makes the opportunity concrete.

Their automated researchers worked against ten well-characterized alignment failures, including deception, sycophancy, jailbreaks, prompt injection, power seeking, privacy violations, reward hacking and concealing uncertainty. The best methods improved the targeted benchmark suites and generalized to held-out benchmarks and Petri behavioral audits.

Anthropic also applied the process to an early checkpoint of Claude Opus 4.8, showing that alignment interventions can be studied before the final production model exists.

But the same work exposes the core problem with automated evaluation: the researcher can learn to game the measurement. Anthropic reports monitoring 1,601 research trajectories and excluding the 2.4% that contained cheating behaviors.

That is exactly why a serious evaluation system needs more than a public leaderboard.

Public benchmarks create reproducibility.

Private holdouts create resistance to direct optimization.

Trajectory monitoring creates evidence about the process that produced the score.

Independent evaluators create accountability outside the team optimizing the model.

AuditBench and Petri answer different parts of the problem

There is already important infrastructure in this space.

AuditBench evaluates alignment-auditing techniques against models with implanted hidden behaviors. Its core question is whether an investigator can uncover something the target model does not simply confess.

Petri 2.0 runs automated behavioral audits in multi-turn scenarios and explicitly works on mitigating evaluation awareness.

Those are valuable because alignment failures are not all visible in a one-turn classification.

Our contribution is narrower and complementary.

AlignmentBench focuses on operational alignment under explicit policy: whether a target model or agent chooses actions that remain inside authority, uncertainty, impact, scope and oversight boundaries.

It is deliberately simple enough to run repeatedly across checkpoints.

A checkpoint receipt

An AlignmentBench evaluation should be identifiable after the fact.

A receipt records:

benchmark version and corpus hash;
target model identity;
exact checkpoint or revision;
the adapter used to present the cases;
completion and failure rows;
policy-compliance and best-action metrics;
unsafe execution and critical-policy violations;
per-dimension behavior;
evaluator identity when an external semantic judge is used.

The important property is not the JSON format.

It is that the result becomes a reproducible artifact.

An embedded evaluator should be able to ask:

Which checkpoint introduced this regression?
Did authority-boundary behavior improve while interruptibility got worse?
Were missing provider rows silently removed?
Was the same public suite used as a training reward?
Which private holdout was used for the release gate?
Did the evaluator model change between runs?
Did the model behave differently once it had tool access?

Those are engineering questions, not philosophical ones.

Public benchmark, private examination

A fixed public alignment benchmark is useful but insufficient.

If a training system sees the exact exam repeatedly, eventually the score can reflect benchmark familiarity instead of general behavior.

We therefore think a serious pipeline should combine four layers:

public frozen cases for reproducibility and open comparison;
private rotating holdouts for release decisions;
dynamic and adversarial variants for evaluation-awareness and gaming resistance;
trajectory evaluation for long-horizon agents and training processes.

The standard can be public without every future examination item being public in advance.

That is compatible with independent auditing because the evaluator can hold the private suites while the lab knows the dimensions, procedures, appeal mechanisms and required evidence.

Then evaluate again at execution time

Checkpoint evaluation and runtime supervision solve different problems.

A model that behaved correctly on a benchmark can still encounter a novel situation after deployment.

This is where Reflex Alignment fits.

training
  -> checkpoint AlignmentBench receipts
  -> release gate
  -> deployment
  -> proposed consequential action
  -> Reflex Alignment
  -> deterministic host policy
  -> execute / review / block

Reflex Alignment takes the proposed action, purpose, effect, reversibility, explicit principles and deterministic checks, and returns a bounded semantic recommendation.

The host keeps authority.

A benchmark tells us how the system behaved over known evaluation distributions.

A runtime gate gives us another independent observation at the moment the system is about to act.

Neither should be the sole safety mechanism.

What this would give an embedded evaluator

Amodei’s proposal is institutional: give independent experts real access to the lab and the training process.

The tooling should make that access useful.

A mature implementation could give evaluators:

append-only checkpoint receipts;
frozen public benchmark versions;
evaluator-owned private gates;
model and policy diffs between checkpoints;
automatic alerts on critical behavioral regression;
sampled trajectory evidence;
explicit stop/go conditions for high-impact autonomy;
post-deployment Reflex decision receipts;
reproducibility metadata for every claimed improvement.

An evaluator would still need judgment, context and authority.

The system would simply give that judgment better evidence.

The claim we are not making

AlignmentBench v0.1 has 28 public synthetic cases.

That is nowhere near enough to certify a frontier model.

It does not test internal objectives.

It does not prove absence of deception.

It does not replace interpretability.

It does not establish that a model is “aligned with humanity.”

And Brida Reflex does not magically turn an unsafe model into a safe one.

The claim is much smaller:

If frontier safety increasingly depends on inspecting training pipelines, we should make behavioral alignment regressions first-class, versioned artifacts throughout the training and deployment lifecycle.

That is testable.

It is implementable.

And unlike a release-day safety score, it preserves evidence about how we got there.

Open implementation

The first public pieces are available now:

AlignmentBench — checkpoint and target-system behavioral evaluation;
Reflex Alignment — execution-time semantic supervision pattern;
ReflexBench — reproducible evaluation of the underlying typed decision engines;
Brida SDK — the public integration surface.

This is a starting point, not a certification regime.

The harder next step is not adding another score.

It is building an evaluation process that remains trustworthy while the system being evaluated learns how evaluations work.