Knowledge Removal in LLMs

A large portion of safety and alignment research has focused on closed frontier models, due to their advanced capabilities. However, the gap between open and closed models is narrowing fast, and open-weight models are now being seriously considered for high-stakes applications.

Yet, open-weight safety remains an unsolved problem. Every open-weight model has been successfully “abliterated” through refusal vector ablation, yielding models that comply on dangerous prompts, and unlearning of sensitive information remains extremely brittle to adversarial attacks.

This project was my first attempt to work on this problem (write-up). I red-teamed LUNAR, a state-of-the-art unlearning method. With the kind of access any open-weight attacker has, I could recover the “forgotten” knowledge in two ways:

• using GRPO fine-tuning, the model learned to route around LUNAR’s modifications
• by steering activations along a difference-in-means direction

This is why I think that open-weight safety requires a different, multi-pronged strategy from closed safety: it requires safety in the absence of guardrails, and robustness to adversarial attacks that are optimised to target activations (such as refusal ablation or activation steering) or weights (such as fine-tuning to escape safeguards). Meeting that bar demands a much closer collaboration between mechanistic interpretability and alignment - an area I am keen to work more on in the future.