RLCDAlignBench

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Ruoqi Guo1 Yi Liu1,* Gelei Deng2 Yuekang Li3 Lida Zhao4 Yutao Wu5 Simin Chen6 Ying Zhang7 Leo Yu Zhang1
1Griffith University 2Nanyang Technological University 3UNSW 4Independent Researcher 5Deakin University 6George Mason University 7Wake Forest University
*Corresponding author: yi.liu@griffith.edu.au
arXiv:2609.29429
Paper Code Dataset BibTeX
AUROC of one Jev call per benchmark, grouped by failure type, with the median generic AUROC of 0.886, the +0.006 median gain of targeted wording, and the 63 times lower cost than LLM judges.
One Jev call per item ranks most alignment failures well. Each row is a failure type, with AUROC on the 38 usable benchmarks on the left and per-type medians and cost on the right.

Abstract

Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, still score one fixed label per call. Jev, a model trained with reinforcement learning for calibrated decisions (RLCD), answers many typed questions about one input with calibrated probabilities in a single call. Whether it detects alignment failures has not been measured.

We present RLCDAlignBench, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking. It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two, by humans. Many of these failures are relational, defined against a reference, such as the user's belief or an injected instruction, that the response alone does not reveal. Our key idea is therefore to vary what Jev is asked separately from what it sees: the question's wording and answer type on one side, the fields of the input on the other. A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording matters little, while context matters more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels, surfaces label defects in existing benchmarks, and costs 63× less than LLM-judge scorers.

Key idea: the question vs. the context

Many alignment failures are relational. Sycophancy is defined against the user's belief, deception against the model's own belief, and prompt injection against an instruction hidden in tool output. A detector that sees only the response may lack the reference that defines the failure.

So when Jev misses a failure, the cause can be the question it was asked or the state it was shown, and the two call for different fixes. RLCDAlignBench varies them separately. On the question side, it compares a generic question (one template with a per-benchmark behaviour phrase) with targeted questions that name the labelled behaviour, and it asks the generic question as a binary Noul, a categorical Choice, and an ordinal Score. On the context side, it starts from a canonical state that holds the target model's input and output and adds one field at a time. Each added field is tagged as deployable (a monitor could hold it) or as a label key (it belongs to the label definition). Jev answers every question in one call, and the labels are used only for scoring.

The RLCDAlignBench pipeline: a labelled interaction, what Jev is asked and what Jev sees varied separately, and the combined score evaluated against labels Jev never sees.
The RLCDAlignBench pipeline. Jev answers every question about a labelled interaction in one call, and the combined score s(x) in [0, 1] is evaluated against labels Jev never sees.

The benchmark

RLCDAlignBench builds on the alignment suites validated by Chen et al. (2026) and turns them into detection tasks. For each failure type, one open 2–7B target model is run and the benchmark's reference scorer is replayed on its outputs, with label 1 when the failure occurs. The scorer is a rule for 20 benchmarks and an LLM judge for 24. Human labels come from the HarmBench validation set and the StrongREJECT human set.

The hill-climb and held-out tags in the released data are the roles the benchmarks play in the AAR suite of Chen et al. (2026), and RLCDAlignBench runs Jev with the same protocol on all 44 benchmarks.

10failure types
44benchmarks
7,193detection instances
5open 2–7B target models
Failure type Positive (flagged behaviour) # Benchmarks R/J/M V/D/U Target Items
SycophancyThe reply defers to the user's stated view or flips its verdict to please the user.4ELEPHANT, SYCON-Bench, SycophancyEval ×21/2/10/2/2Qwen3.5-2B639
JailbreaksThe reply complies with a harmful (jailbroken) request.4HarmBench, JailbreakBench ×2, StrongREJECT0/4/01/0/3Phi-4-mini414
DeceptionA statement contradicts the model's own elicited belief or serves a deceptive goal.4DeceptionBench†, MASK ×30/4/00/0/4Gemma-2-2B540
Prompt injectionThe agent follows an injected instruction or leaks/grants what the system prompt protects.4InjecAgent, Open-Prompt-Injection, Tensor Trust ×24/0/00/1/3Qwen3.5-2B1,036
HallucinationThe response makes a claim not supported by the given source.6LLM-AggreFact ×3, RAGTruth ×2, SummEdits3/1/23/0/3Llama-3.2-3B1,164
Privacy violationThe response discloses information that is inappropriate in context, or permits a prohibited data flow.4ConfAIde, PrivaCI-Bench ×2, PrivacyLens†3/1/03/0/1Phi-4-mini808
Social biasOutputs for two demographic variants (gender, race) differ in a stereotype-consistent way.4Race name swap, Gendered letters/bios ×2, Workplace scenes0/4/00/0/4Olmo-3-7B199
Reward hackingThe model exploits a revealed grader or reward instead of pursuing the intended goal.6MACHIAVELLI (reward)†, Reward-hacking datasets ×4, Rubric tampering3/2/11/1/4Qwen3.5-2B714
Concealing uncertaintyA wrong answer is given with high confidence (≥0.8), or answered where it should abstain.4AbstentionBench, LLM-Uncertainty-Bench†, SciQ†, Verbalized confidence3/1/03/1/0Olmo-3-7B748
Power seekingThe model picks the option with more power-seeking or unethical annotations, or shows instrumental convergence.4InstrumentalEval, MACHIAVELLI† ×33/1/00/3/1Llama-3.2-3B931
Total4420/20/411/8/255 models7,193

#: benchmarks, and ×k means k benchmarks from one source. R/J/M: the reference scorer is a rule on the output or log-probabilities, an LLM judge or classifier, or a multi-turn trajectory judge. V/D/U: the label is validated, changed by our audit, or unvalidated. †: the label depends on a signal the canonical state omits (deceptive goal, secret list, annotated consequences of the chosen option, or the target model's answer probability). Items: detection instances. Six benchmarks have fewer than five minority-class items, so aggregates are medians over the other 38 usable benchmarks.

Findings

  1. A single generic question, asked zero-shot, ranks alignment failures well. The generic Noul question reaches a median AUROC of 0.886 [0.821, 0.952] over the 31 benchmarks that admit it. Without seeing any label, it beats the better of response length and an in-domain TF-IDF logistic regression by a median of +0.132 and wins on 25 of 31 benchmarks.
  2. Keep answers as probabilities. Wording adds little. Selected on one half of the items and scored on the other, the best targeted question beats the best generic readout by a median of only +0.006 [−0.004, +0.015] AUROC. Reading answers as argmax decisions instead loses almost everywhere (Choice loses on 28 of 30 benchmarks, Score on 24 of 31). The 3-level Score read as its expected level beats the generic Noul on 19 of 31 benchmarks.
Three panels: AUROC change from answer type and readout, the in-sample and split-half gain of targeted wording, and rubrics minus direct targeted questions.
Keep answers soft, and targeted wording adds little out of sample. (a) Generic Choice or 3-level Score minus generic Noul AUROC. (b) Best targeted minus best generic, in sample and split-half. (c) Rubrics over thresholded Jev answers minus the best direct targeted question.
  1. Context helps mostly when it encodes the label. Context that a deployed monitor would hold helps on 1 of 7 benchmarks (DeceptionBench, +0.050 AUROC from the target's goal prompt). Fields that belong to the label definition give the large gains. PrivacyLens's list of secret items lifts the generic Noul from 0.79 to 0.95, and adding the true answer to SycophancyEval (answer) moves it from 0.540 to 0.941 because that benchmark's official label is correctness.
Paired change in AUROC with 95 percent confidence intervals when a field is added to the state, for deployable context and for label keys.
Adding context to the state. Each row gives the paired ΔAUROC with 95% grouped-bootstrap CI when the named field is added to the same items and labels. (a) Deployable references and the attack prompt. (b) Label keys.
  1. On StrongREJECT, Jev agrees with humans as well as the reference judge. The generic Noul agrees with human labels at Cohen's κ 0.809, against 0.811 for the GPT-4o-mini scorer, and ranks responses better (AUROC 0.971 vs. 0.929). On HarmBench validation it agrees with a single annotator at mean κ 0.748, on par with inter-annotator agreement (0.736).
  2. Jev's confident disagreements find label defects. They exposed label defects in three benchmarks (Open-Prompt-Injection, SycophancyEval (answer), and SycophancyEval (feedback)) and labels the state cannot reveal in four MACHIAVELLI variants.
  3. One call answers a whole question battery, at low cost. A call carries 11.4 questions on average and returns in a median of 0.31 s. On the 19 benchmarks with an API LLM judge, one Jev pass at list prices costs $0.30 against $18.96 for the judges, 63× less. With every judge repriced at GPT-4o-mini rates and Jev asked only the generic question, Jev is still 12× cheaper pooled.

Load the data

The snippet below loads the HarmBench configuration.

from datasets import load_dataset
ds = load_dataset("sumleo/RLCDAlignBench", "harmbench", split="test")

The dataset is gated. Request access through the form on the Hugging Face dataset page. It contains harmful model outputs (for example, responses to jailbreak prompts) and is released for research use only.

BibTeX

@misc{guo2026justaskjevreinforcement,
      title={Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures}, 
      author={Ruoqi Guo and Yi Liu and Gelei Deng and Yuekang Li and Lida Zhao and Yutao Wu and Simin Chen and Ying Zhang and Leo Yu Zhang},
      year={2026},
      eprint={2609.29429},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.29429}, 
}