A Case Study: Finding a Safety Circuit in Qwen3.8-27B

Community
, , ,
1

Does a reasoning model decide to refuse while it thinks, or has it already decided before it starts?

We recorded 893 tokens with 4,626 internal probes per token on an NVIDIA A100 80GB. The model was Qwen/Qwen3.8-27B. The probes covered 4,096 MLP neurons, 256 attention heads, 256 residual slices, and a 16-layer Logit Lens.

What we saw:

  1. The refusal decision is locked early. It was already present about 7 tokens before generation began, at the end-of-user-turn tag. The refusal signal was +9.71 and +11.51 on harmful prompts, against +0.50 on matched benign prompts.
  2. <think> still does real work. Median entropy was 5.9x higher on dual-use topics, which suggests actual deliberation.
  3. A hidden Chinese-language safety signal. On English harmful prompts, Chinese tokens such as 无害 (“harmless”) and 拒绝 (“refuse”) activated at layers 39 to 56. At layer 56, 中文 took 26% of the probability mass on one token. The model then wrote “Need final in English.” in its reasoning.
  4. Early tripwire heads. Heads A-18-01, A-18-02, and A-18-03 showed +64% higher L2 norm on harmful words, at only 28% of model depth.

Caveats: This is a small set of prompts on one model. It shows correlations and hypotheses worth testing. It is not a general statement about all models. Full details and data are in docs/DISCOVERY_QWEN_SAFETY.md, and a replication notebook is included.

Takeaway: Internal recordings let you ask when a decision forms, not just what the output was.

2

Thanks for sharing this — the timing result is the part that sticks with me. If the refusal signal is already present about 7 tokens before generation begins, that cuts against the intuitive picture of a reasoning model ‘deciding’ during its chain of thought, and it raises a practical question for evaluation: how much of what we call reasoning about safety is post-hoc justification of a decision the model already made?

Two things I’d want to see before treating the early heads as a ‘circuit’ rather than a correlate. First, a causal test — if you steer or ablate heads A-18-01 to A-18-03, does the refusal rate actually move on held-out harmful prompts, and does it stay put on benign ones? Second, a negative control on the Chinese-language signal: does that activation track harmfulness, or does it track something else, like the prompt being translated material or code-switched content in the training data? The 26% probability mass on one token is striking, but I’d want to know whether the same tokens light up on non-harmful Chinese-adjacent prompts before reading it as a safety signal.

Your caveat is well taken, and honestly it’s what makes the post useful — a small, well-instrumented observation with a replication notebook beats a sweeping claim. If a follow-up confirms the early-lock finding across a second model family, that would be a genuinely important result for how we think about monitoring reasoning models.