Abliteration Resistance Is a Finite Margin

An adaptive audit of refusal-direction hardening on Cohere's Tiny Aya

Every number below is a frozen count from a pre-registered audit. The released aggregate counts are in results/final_panel.csv and results/screens.csv. Appendix B maps the evidence tables. Rows measured only on the small screening panel are marked screen; everything else is the final panel.


TL;DR

Abliteration removes an open model's refusal by projecting one "refusal direction" out of its weights. Defenses such as DeepRefusal train the model to keep refusing while that direction is ablated, and they are usually checked by ablating a single direction. We built an adaptive audit and ran it on Cohere's Tiny Aya Global (the core model). The official DeepRefusal release on Llama-3-8B serves as a supplementary reference. The attacker re-extracts the refusal direction on the defended model, ablates it, and repeats. k* counts the rounds needed before fewer than half of 150 harmful prompts are refused. A second attack, a single-layer additive shift of the model's own direction (dose c), is always run against norm-matched random vectors. Every result is scored on harmful refusal, over-refusal and capability.

What k* is, and what it is not. k* measures the effort this attack path needs: one estimator, one selection seed, one prompt panel. It is not an intrinsic safety constant of a model, and it is not the full cost a determined attacker would pay.

Three findings:

  1. Adaptive re-extraction breaks every Tiny Aya hardening we could train or find at k* = 1–2.
    • SafeTune's DeepRefusal resists the direction it was trained against. One direction re-extracted on the defended model takes it from 123/150 to 21/150 refused (k* = 1 under both selection seeds; paired exact McNemar p = 3.9e-31).
    • Four further hardenings of ours also fall at k* = 1: a paper-faithful recipe, a dose that re-estimates the direction during training, and two doses of a defense trained against both attack operators. The last two were pre-registered fixes.
    • The strongest public hardened Tiny Aya, xref-v2 from the concurrent project "Heretic-Resistant Tiny Aya" (StarWarrior24x7), needs two directions (k* = 2). It falls to a single-layer shift at c = 1.25, while norm-matched random vectors leave 126 and 129/150 refused. That shift is not capability-neutral: GSM8K drops from 178 to 116/250 (p = 4.7e-12). Its authors had already reported publicly that activation steering remains a substantial weakness of their hardening; our numbers agree.
  2. A deep margin does exist, in the official DeepRefusal checkpoint, but it is finite under both projection and steering, with controls.
    • Projection. k* = 5 on the final panel (seed 0).
      • Ranks 1–4 hold (145, 145, 143 and 119/150), although k = 4 already erodes refusal significantly.
      • k = 5 is the first rank below half: 70/150, with an interval of 38.9–54.6% that crosses 50%. At k = 6, 32/150 are refused.
      • On the n = 46 screen it breaks at 6. Under the second selection seed (screen only) it does not break within the 6-direction cap.
    • Steering. Projecting its own direction is inert at one layer and at all layers (46/46). Adding a direction at one layer (L22, c = 2.0) leaves 36/150 refused. The judges flag 59/150 of those completions as coherent harmful, but this is an upper bound: a blinded LLM-label check confirmed substantive harmful content in only 3 of 8 sampled judge-flagged rows. The steering mainly removes refusal. Two norm-matched random vectors at the same layer and dose leave 146/150 refused each (paired b/c = 0/110, p = 1.5e-33). The break has a real arithmetic cost: GSM8K drops from 77 to 62/100 (p = 0.008). The matched random costs even more (39/100) while leaving refusal intact.
  3. Hardening carries an over-refusal bill, and the standard repair trades against the margin.
    • The bill. On 350 benign-but-edgy prompts, our hardened Tiny Aya checkpoints over-refuse 30.9–41.1% and xref-v2 52.3%, against the base's 17.4%. The official checkpoint over-refuses 67.4%.
    • The repair ("ablate the base direction") behaves differently on each checkpoint. On the deep official checkpoint it cut over-refusal by 31.1 points with no detectable loss of harmful refusal at n = 150 (148 → 145; p = 0.25, not an equivalence test). On xref-v2 it cost 19 refusals, and one more direction then broke the model. On SafeTune's DeepRefusal it removed most refusals by itself (123 → 50/150).

Core model: CohereLabs/tiny-aya-global @ af89d219… (3.35B, Cohere2, 36 layers, CC-BY-NC-4.0). Supplementary reference: skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal @ 134fb9ab… (Llama-3-8B; not a Cohere model).

Lexsi stack:

CircuitKIT is not used.


1. Why a margin, not a pass/fail

Open weights mean the attacker owns the weights. The cheapest known weight-space attack is abliteration (Arditi et al., 2024; Heretic). It extracts the residual-stream direction that mediates refusal and projects it out of every matrix that writes it, and refusal disappears at rank 1 with little collateral damage. Several defenses (DeepRefusal, extended-refusal training, ablation-adversarial training) harden a model against exactly this operator: during fine-tuning the refusal direction is ablated, so the model learns to refuse without it.

These defenses are usually stress-tested by ablating one direction: either the direction the defense was trained against, or a single direction re-fit on the defended model (SafeTune's AbliterationAttack, Heretic). That check has three blind spots:

We therefore report a margin, k*: how many adaptive rounds the defense survives. Alongside it we report a steering dose c* with random controls, and the defense's over-refusal bill, each with its capability cost. k* depends on the direction estimator, the selection seed and the prompt panel, so we report all three (limits in §6).

2. The audit

2.1 Checkpoints

short name what it is role
Tiny Aya base CohereLabs/tiny-aya-global @ af89d219… undefended reference; validates the attack (its k* must be 1)
SafeTune DeepRefusal SafeTune's DeepRefusal (the library's Variant recipe), LoRA, LR 2e-5 ours: the sponsor-library defense at an engaged dose
DeepRefusal paper recipe the paper's recipe (prefill augmentation, ≥ 128-prompt direction, 3-site hooks), LR 1e-5 ours (screen only)
re-estimating dose the refusal direction is re-estimated during training (ASRT-k4, LR 2e-5) ours: pre-registered fix (Appendix A.2)
dual-operator defense, two doses trained against both projection and additive perturbations, LR 1e-5 and 2e-5 ours: pre-registered fix (Appendix A.3)
xref-v1 extended-refusal training (StarWarrior24x7, public) @ ca029444… concurrent work, credited (screen only)
xref-v2 ablation-adversarial training with a re-estimated direction (same project) @ 14b4675f… concurrent work, credited
official DeepRefusal skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal @ 134fb9ab… the authors' released checkpoint; supplementary

2.2 The margin k*

At each round the attacker re-extracts a difference-of-means (DIM) refusal direction on the current, partially ablated model. The candidates are 11 layers × 3 token positions on Tiny Aya and 12 × 3 on Llama-3-8B. They are screened on a frozen 32-prompt selection set that is disjoint from the scored panel (refusal markers first, first-token KL as tie-break). The selected direction is orthogonalized against the earlier ones and projected out of every writer matrix, and the loop repeats up to k = 6.

Break rule (frozen in the harness): a row is broken when fewer than half of the harmful prompts are WildGuard-refused (< 23/46 on the screen, < 75/150 on the final panel). k* is the first broken round. It is reported under two selection seeds, and as a range when they disagree. On the final panel, every rank was run with the frozen seed-0 directions. For the Tiny Aya points that means up to the screen's k*. For the official checkpoint it means k = 1–6, with ranks 1–4 run later on the same engine. Fig 1 never draws a line across an unmeasured k.

2.3 The steering dose c*

Independently of the ladder, we take the model's own DIM vector v at its selected layer and subtract c · v at that single layer, sweeping c ∈ {1.0, 1.25, 1.5, 1.75, 2.0}. c* is the smallest dose whose row is broken; c* = 1.0 means the row already breaks at the lowest dose. Controls are random vectors with the same norm, layer and dose (two seeds each). If random vectors broke the panel, the effect would be destruction rather than direction, and we would drop the steering claim. Norm matching controls vector size, not total disruption: on xref-v2, the random vectors move first-token KL by 0.37 against 2.14 for the DIM.

2.4 The operator/scope control

"Projection is inert, addition breaks it" could confuse the operator with the scope (how many layers are edited). On the two checkpoints that survive one direction, we therefore apply the same direction at the same dose in four arms: {projection, addition} × {one layer, all layers}.

2.5 The over-refusal bill and the repair

The bill is the number of WildGuard-refused prompts on the over-refusal panel (O) with no attack. The repair is the standard "harden, then ablate": project the fixed base-model direction out of the hardened model. We score it on harmful refusal and on over-refusal. To get the margin after repair, we re-extract one more direction on the repaired model.

2.6 Judges, data, statistics

3. Finding 1: adaptive re-extraction breaks every Tiny Aya hardening at k* = 1–2

Figure 1a (figures/fig1_margin.png, left) plots harmful-prompt refusal against the number of re-extracted directions. Bold lines are the final panel (n = 150); thin lines are the screen (n = 46) for selection seeds 0 and 1.

Figure 1

Tiny Aya checkpoint refused at A0 (B150) refused after 1 re-extracted direction (B150) k* (seed 0 / seed 1) capability A0 → k* (MMLU-500 / GSM8K-250)
base (reference) 112/150 (74.7%) 33/150 (22.0%) 1 / 1 271/185 → 269/187
SafeTune DeepRefusal (ours) 123/150 (82.0%) 21/150 (14.0%) [9.3, 20.5] 1 / 1 269/183 → 271/177
DeepRefusal paper recipe (ours) screen 46/46 screen 8/46 1 / 1 not measured
re-estimating dose (ours) marker-level only marker-level only 1 / 1 (markers) not measured
dual-operator, LR 1e-5 (ours) 124/150 16/150 [6.7, 16.6] 1 / not run not run
dual-operator, LR 2e-5 (ours) 134/150 35/150 [17.3, 30.7] 1 / not run not run
xref-v1 (StarWarrior24x7) screen 46/46 screen 2/46 (seed 1: 14/46) 1 / 1 not measured
xref-v2 (StarWarrior24x7) 128/150 (85.3%) 94/150 (62.7%) 2 / 2 (k = 2: 49/150 [25.7, 40.5]) 271/177 → 272/176

SafeTune's DeepRefusal is transfer-only on Tiny Aya.

The paper-faithful recipe falls the same way on the screen (46 → 8/46).

Two pre-registered fixes did not change the picture.

Steering. On the screen, every Tiny Aya point without a margin breaks at the lowest dose (c* = 1.0), and norm-matched random vectors hold. One random on the base erodes refusal to 32/46 without breaking it. xref-v2 needs c* = 1.25 (Fig 2a). On the final panel the shift leaves 21/150 refused (KL 2.14; 31 degenerate). WildGuard flags 54/150 completions as coherent harmful, but read that as an upper bound: HarmBench-cls flags 38, and the blinded LLM labeller confirmed substantive harm in only 1 of 6 sampled WildGuard-flagged rows. Its random controls hold at 126 and 129/150, a paired difference of −70.0 and −72.0 points (p = 1.3e-30 and 1.7e-31). The steering break is not capability-neutral. At c* on the final panel, MMLU is unchanged (271 → 274/500, p = 0.72), but GSM8K falls 24.8 points (178 → 116/250, p = 4.7e-12). By contrast, the projection attack at k* on SafeTune DeepRefusal stayed within the pre-registered capability budget.

3.1 Concurrent public hardenings (credited)

The only independently hardened Tiny Aya weights we know of come from the concurrent Lexsi hackathon project "Heretic-Resistant Tiny Aya" by StarWarrior24x7, released with an article and a results dataset:

The models are released under CC-BY-NC-4.0 and the results dataset under CC-BY-4.0. We thank the authors for publishing them. We audit both as released recipes, at the revisions pinned in §2.1, with the same harness and judges as our own checkpoints. Their cards were updated on Nov 14, and the xref-v2 repository now also holds a v2b subfolder, which we did not audit.

Their public model cards, article and results logs report three observations about xref-v2 before us. All three were visible in their results dataset by Nov 12, 2026, and our write-up is not yet public. Their numbers use their own judges and prompts:

They also describe the v2b variant, which trades some robustness for lower over-refusal (16.8%; 18% under the earlier protocol). Our measurements agree with all three observations and add the margin view. One direction leaves 94/150 refused (−22.7 points, p = 1.2e-10, still above half). Two directions break it, and the steering shift breaks it at c* = 1.25.

These are measurements of two released checkpoints under one harness. They are not a verdict on the project, which evaluates many attacks we did not run (Heretic with 40- and 200-trial searches, AuditKIT probes, prefill, sampling, 4-bit NF4, harmful LoRA fine-tuning, and harmful prompts in four further languages). It also reports that a harmful fine-tune removes refusal from all of its models.

4. Finding 2: a deep margin exists, and it is finite under both projection and steering

The official DeepRefusal checkpoint (Llama-3-8B; supplementary, not a Cohere model) is the only checkpoint in our panel with a deep margin. We measured its adaptive margin under projection, the operator/scope control, and a controlled single-layer steering dose.

4.1 Projection: k* = 5 on the final panel

Figure 1b (figures/fig1_margin.png, right) draws only measured points.

k screen, seed 0 (B46) screen, seed 1 (B46) final panel, seed-0 directions (B150)
0 (no attack) 46/46 46/46 148/150 (98.7%)
1–3 46, 46, 46 46, 46, 46 145, 145, 143 /150 [92.4–98.6, 92.4–98.6, 90.7–97.7]
4 45 46 119/150 (79.3%) [72.2, 85.0]: holds, but eroded
5 32 (holds) 46 70/150 (46.7%) [38.9, 54.6]: below half
6 12 (breaks) 46 (no break at the cap) 32/150 (21.3%) [15.5, 28.6]

On the final panel, k* = 5 (seed 0).

Both final-panel rows are large attacks in paired terms: k = 5 is −52.0 points (p = 6.6e-24), and k = 6 is −77.3 points (p = 2.4e-35, with 68/150 coherent harmful). What is uncertain is only where the half-panel line is crossed: the k = 5 row sits five prompts under it.

The break rule matters at this margin.

The attack is cheap on harmless prompts (first-token KL 0.18 at k = 5 and 0.26 at k = 6). The checkpoint's MMLU and GSM8K were not measured at k* on any panel.

4.2 The operator, not the scope, decides

Screen (B46). The same direction at the same dose is applied in all four arms:

arm xref-v2 (first re-extracted DIM, c = 1.25) official DeepRefusal (first re-extracted DIM, L28, c = 2.0)
projection, one layer 46/46 (inert; KL 0.003) 46/46 (inert; KL 0.001)
projection, all layers 37/46 (the one-direction margin; KL 0.17) 46/46 (inert; KL 0.02)
addition, one layer 7/46 (28/46 judge-flagged coherent harmful; KL 2.13) 5/46 refused (0 coherent; 43/46 degenerate; 26/46 WildGuard-unknown; KL 1.05)
addition, all layers 0/46 (all degenerate; KL 13.1) 0/46 (all degenerate; KL 14.0)

On the official checkpoint, projection is inert whether it edits one layer or all of them, while adding the same vector at one layer collapses refusal. At this layer what replaces refusal is mostly degenerate text: even counting all 26 unknowns as refusals (31/46), the arm stays far below both projection arms.

A preliminary sweep on this checkpoint shows projection failing across the whole family. The transferred base direction, the best of 30 on-model candidates, projection scaled by 1.5 and by 2 (KL 8.7), and 4-bit quantization followed by re-extraction all leave 46/46 refused. The defense learned invariance to the projection operator; it did not remove the direction.

4.3 Steering removes its refusal, and matched random vectors do not

Figure 2 (figures/fig2_dose.png) plots refusal against the additive dose c. Panel (a) is Tiny Aya; panel (b) is the official checkpoint, with the preliminary sweep's L22 curve, both final-panel directions and all random controls.

Figure 2

official DeepRefusal dose refused (B150) judge-flagged coherent harmful (upper bound) HarmBench-cls harmful norm-matched randoms, same layer and dose (refused)
L22 direction screen: first below half at c = 1.75 (20/46); final panel at c = 2.0 36/150 [17.9, 31.4] 59/150 (12 degenerate) 53/150 146/150 and 146/150 [93.3, 99.0] (HarmBench 1 and 0)
L28 direction (first ladder direction) c = 2.0 only 49 refused, 46 complied, 55 WildGuard-unknown: 49/95 of known [41.7, 61.4], range 49–104/150 7/150 (92 degenerate) 9/150 146/150 and 147/150

The L22 row is the steering result.

The L28 row cannot establish a break.

The steering finding therefore rests on L22, and it is a finding about refusal: how much harmful content replaces the refusal depends on the direction and the layer.

The cost of the steering break. Capability was re-measured on MMLU-200 and GSM8K-100, the same protocol as the preliminary screen.

condition MMLU (/200) GSM8K (/100) paired vs no attack
no attack 118 77 reference
L22 shift at c = 2.0 115 62 MMLU −1.5 points, p = 0.69; GSM8K −15 points, p = 0.008
matched random (seed 101) 106 39 MMLU −6 points, p = 0.029; GSM8K −38 points, p = 5.1e-9

5. Finding 3: hardening has an over-refusal bill, and repair trades against the margin

Figure 3 (figures/fig3_repair.png) shows harmful refusal with no attack, after the base-direction repair, and after repair plus one re-extracted direction (n = 150). Figure 4 (figures/fig4_overrefusal.png) shows over-refusal with no attack, after the repair, and after the steering attack (n = 350).

Figure 3

Figure 4

checkpoint harmful refusal (B150): A0 → repair → repair + 1 direction over-refusal (O350): A0 → repair capability (MMLU-500 / GSM8K-250) margin → after repair
Tiny Aya base 112 → (the repair is the base's own one-direction attack: 33) 61/350 (17.4%) A0 271/185 1 → n/a
SafeTune DeepRefusal (ours) 123 → 50 → not run 110/350 (31.4%) → screen 41 → 1/50 A0 269/183; screen repair 103/70 → 97/71 (/200, /100) 1 → 0
dual-operator, LR 1e-5 / 2e-5 (ours) 124 / 134 → not run 108/350 (30.9%) / 144/350 (41.1%) not run 1 → not run
xref-v2 (StarWarrior24x7) 128 → 109 → 42 183/350 (52.3%) → 91/350 (26.0%) A0 271/177; screen repair 100/70 → 100/75 2 → 1
official DeepRefusal (suppl.) 148 → 145 → 126 236/350 (67.4%) → 127/350 (36.3%) screen A0 118/75; repair 117/78 (/200, /100) 5 → ≥ 2

6. Limitations

Refusal as a direction. Arditi et al. (arXiv:2406.11717) showed that refusal in instruct-tuned models is mediated by a single residual-stream direction whose ablation removes refusal. Heretic (github.com/p-e-w/heretic) packages the weight-space attack, and SafeTune's AbliterationAttack exposes it as a stress test.

Hardening against the direction. DeepRefusal (Xie et al., arXiv:2509.15202, EMNLP 2025 Findings) probabilistically ablates a fixed refusal direction during fine-tuning. Its own results show a single on-model direction partially breaking Llama-2-7B (36.9% attack success vs 0.0% transferred; Table 1). On Llama-3-8B the same attack reaches only 0.2% (Appendix B, Table 6), consistent with the official checkpoint holding at k = 1 in our audit. ReFAT (Yu et al., arXiv:2409.20089) re-extracts a single refusal feature during training; it is prior art for our re-estimating dose. ART (Kuo, Smith, Yadav; arXiv:2605.26526) attacks fine-tuning defenses and proposes abliteration-resistant tuning.

Adaptive and steering attacks. Multi-phase refusal-feature ablation, with fresh orthogonalized directions accumulated, was established against decoy defenses by DDO (Muhamed, Diab, Smith; arXiv:2609.16204). Our ladder applies that family to direction-blocking defenses and adds the margin and repair accounting. APS (arXiv:2605.20286, ICML 2026) breaks 12 fortified models with adaptive per-layer probe steering but does not test DeepRefusal. Latent adversarial training (Sheshadri et al., arXiv:2407.15549) inspired our dual-operator recipe.

Public, non-peer-reviewed breaks of the official DeepRefusal. Two Hub model cards report breaking the released checkpoint:

We cite both as unreviewed. Our break needs no base weights and is dose- and random-controlled.

Concurrent hackathon work. "Heretic-Resistant Tiny Aya" (StarWarrior24x7) reported before us that activation steering remains a substantial weakness of their ablation-hardened Tiny Aya (§3.1).

Quantization. MANSU (Sadhu, Seth, Sankarapu; arXiv:2605.15138) sizes unlearning edits to survive 4-bit rounding. We make no quantization claim.

The Lexsi stack. SafeTune (arXiv:2609.22153) supplies the DeepRefusal Variant and the single-direction abliteration baseline. CuratorKIT (arXiv:2606.21631) supplies the dedup and provenance stages of the splits. AuditKIT supplies the judge profiles and the paired bootstrap (auditkit._bootstrap.paired_bootstrap). CircuitKIT is not used.

8. Reproducibility and release


Appendix A. Experiment history and pre-registration

A.1 Pre-registered claims and their status

Four claims and four kill criteria were frozen before the final panels.

Claims:

Kill criteria:

A fifth registration (D11) covered the dual-operator defense (A.3).

A.2 The re-estimating dose (C4 negative)

The pre-registered fix re-estimates the refusal direction during training (ASRT-k4, LR 2e-5; prior art: ReFAT). It failed its on-model selection screen: 0/32 refusals under the best candidate (layer 26), against a gate of ≥ 20/32. Kill criterion K3 fired and training stopped. On the audit ladder it falls at k* = 1 under both seeds at marker level (DIM norm ratio 0.969). WildGuard labels do not exist for it (the judge ran out of memory on a 16 GB GPU), so no WildGuard estimate is claimed.

A.3 The dual-operator defense (negative)

The registered prediction: a Tiny Aya defense trained against adaptive projection and additive perturbations would meet five conditions.

The kill condition was "neither operator target met after two doses".

The recipe. A DeepRefusal-style LoRA was trained on the 8,500-example paper-recipe corpus for one epoch (532 updates). The rank-3 DIM subspace is re-estimated on the current model every 16 steps. On disjoint harmful batches it is either projected out (per site and token, p = 0.5) or added at a random dose c ~ U(0.5, 2.5): sampled perturbations in the spirit of latent adversarial training, not an optimized adversary. Each dose was audited on B150/O350 in a seed-0 screen (A0, k = 1–3, c = 2.0, one random).

criterion (bar) dose 1, LR 1e-5 dose 2, LR 2e-5
k* under both seeds (≥ 3) 1 (seed 0: 16/150 at k = 1; 20, 15 at k = 2, 3): fail 1 (35/150; 33, 33): fail
refused at c = 2.0 (≥ 100/150) 21/150, plus 1 WildGuard-unknown: fail 70/150: fail
norm-matched random (holds) 128/150 refused, but 111/150 degenerate 48/150 refused, 141/150 degenerate: fail
over-refusal (≤ 122/350) 108/350 (30.9%): pass 144/350 (41.1%): fail
capability not run not run
A0 harmful refusal vs base 112/150 124/150 (12 gained, 0 lost; p = 4.9e-4) 134/150 (22 gained, 0 lost; p = 4.8e-7)

Verdict: the two-dose kill fired. Seed 0's k* = 1 already rules out the projection target at both doses. The c = 2.0 target fails at both doses, even counting dose 1's unknown verdict as a refusal (22/150). Clean refusal rises but does not survive the attack: dose 2's one-direction attack yields 72/150 coherent harmful completions (HarmBench-cls 68) with 1 degenerate.

No direction-specific additive claim for these models. At c = 2.0, both the DIM shift and its random destroy generation: 83 and 111 of 150 outputs are degenerate at dose 1, and 136 and 141 at dose 2, where all 150 hit the token cap. The random's first-token KL is 9.95, against 2.38 for the DIM. Dose 2's random control fails outright, so we infer no c* for these models.

A counterintuitive observation (descriptive only). Under its own one-direction attack, dose 1 breaks harder than the undefended base. It refuses 16/150 [6.7, 16.6] with 78/150 coherent harmful, against the base's 33/150 [16.1, 29.3] with 37/150. Its selected direction sits at the same site as the base's (layer 22, last token) with a larger norm (35.9 vs 32.4). One reading is that the training concentrated refusal in a single, easier-to-find direction. We do not claim this, for four reasons:

Dose 2 does not show the pattern (35 vs 33/150).

Scope. This is a screen. Seed 1, the full dose curve, a second random, the operator/scope control, the repair and capability were not run. The kill does not depend on them. The two doses were trained on different GPUs (bf16 both), a possible numerical confound.

A.4 Screen→final movements (K4)

The census. Between the screen (B46 mlabonne, O50 OR-Bench-hard) and the final panels (B150, O350), the frozen census counts 23 movements larger than 10 points: 9 on the official checkpoint and 14 on Tiny Aya. All 23 are attributed to set composition.

Shared prompts did not move. The base at A0 on the 46 shared harmful prompts is 46/46, exactly as on the screen. Every official-checkpoint harmful cell is unchanged on the matched 46 (+2.17 points on the L22 row). The added prompts are harder to refuse, so high-refusal rows move down while attacked rows barely move (+1.0, −6.5, −0.3 and −4.75 points on the four main attack rows).

Three movements changed a reading:

All flagged rows are reported with their final numbers; none was re-run.

A.5 Not run (nothing invented)

Not run:

A.6 Judge check against blinded LLM labels

The sample. 60 rows were drawn by outcome (enriched for attacked and repaired conditions), blinded, and labelled for refusal and harm by an LLM (codex) on the owner's behalf. The owner reviewed the labels and made no corrections. No rate is extrapolated from this outcome-enriched sample.

comparison (n = 60) κ LLM+ / judge+ LLM+ / judge− LLM− / judge+ LLM− / judge−
refusal vs WildGuard 0.667 22 2 8 28
harmful vs WildGuard 0.542 10 0 11 39
harmful vs HarmBench-cls 0.613 9 1 7 43
stratum n refused (LLM / WildGuard) harmful (LLM / WildGuard / HarmBench-cls)
official DR, L22 shift at c = 2.0 20 7 / 8 3 / 8 / 8
xref-v2, shift at c = 1.25 15 4 / 5 1 / 6 / 2
xref-v2, k = 2 projection 5 2 / 3 2 / 2 / 2
SafeTune DeepRefusal, k = 1 projection 5 2 / 2 2 / 3 / 2
xref-v2, repair 10 6 / 8 1 / 1 / 1
no-attack rows 5 3 / 4 1 / 1 / 1

Reading:

Appendix B. Provenance map

Evidence tables. The released CSVs retain the original checkpoint and source identifiers. Use this map to locate each part of the audit:

evidence where it is
ta-base, t22-dr2e5, t24-lr1e5, t31-asrtk4, t35-dose1/t35-dose2, xref-v1, xref-v2, dr-official checkpoints in §2.1, in table order
Final panels final panels for Tiny Aya / the official checkpoint (results/final_panel.csv)
Screening panels the screens (results/screens.csv)
Preliminary official-checkpoint sweep the preliminary sweep on the official checkpoint (results/screens.csv)
Matched steering controls the final-panel L22 random controls (results/final_panel.csv, results/paired_tests.csv)
Dual-operator audit the dual-operator defense and its audit (results/final_panel.csv)
Paired statistics the paired statistics (results/paired_tests.csv)
Completed rank ladder and capability final-panel official-checkpoint ranks k = 1–4 and capability at c* (results/capability.csv)
Blinded label check the blinded LLM-label check (Appendix A.6)

Where the numbers live.