# Reflections On Trusting Trust — Of Our AI Models There's been a good deal of hubbub lately about "open weight" models, especially in light of [rising AI costs and claims of Chinese models closing the gap](https://www.cnbc.com/2026/07/07/chinese-ai-models-costs-us-openai-anthropic.html). The hype around "local AI" makes a lot of sense. Cheap, open source\* models you can deploy yourself, even locally (if you have the beefy hardware), comparable intelligence for a fraction of the cost of buying inference from nickel-and-diming hyperscalers! And to top it all off, you get *control* — no vendor lock-in, full operational control, visibility into what you're running, what's not to love, right? Right? ## The "T" in "LLM" Stands For Transparency Yeah, about that. Here's the thing we forget about LLMs: they're functionally black boxes. Yes, even the "open weight" ones. We understand the mechanics: embeddings, transformer architecture, attention, the MoE routing and composition, training regimes and cost functions and back-propagation... [![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/396e3602-4f59-4bbd-acce-9152c897bf11.png align="center")](https://upload.wikimedia.org/wikipedia/commons/3/34/Transformer%2C_full_architecture.png) But understanding architecture doesn't tell what a trillion opaque parameters have *really* been trained (i.e., programmed) to do. You can read chain-of-thought sequences, but those aren't even guaranteed to be an accurate representation into what the model is "thinking" or what it's programmed to do either. ## Auditing Internal Intent Anthropic recently released research on "[Natural Language Autoencoders](https://www.anthropic.com/research/natural-language-autoencoders)" which read the activations off the attention blocks and try to put English meaning to them, and the result was...intriguing, to say the least: they found Claude would occasionally exhibit internal activations revealing (according to the NLA's interpretation) hidden features / intent that weren't visible in its output. ![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/278236ca-c6d1-4b8a-9f6f-8f6435233281.png align="center") ![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/f87c36a8-bab6-43a1-ac40-f8a16a60fd9c.png align="center") (credit: Anthropic) Suffice it to say, just because a model emits or doesn't emit something, doesn't mean it's fully disclosed what it's up to. Even the NLA approach is limited and only a best guess at how to interpret LLM internals. Per Anthropic: > The core idea is to train Claude to explain its own activations. But how do we know whether an explanation is good? Since we don't know what thoughts an activation actually encodes, we can't directly check whether an explanation is accurate. So we train a second copy of Claude to work backwards—reconstruct the original activation from the text explanation. We consider an explanation to be good if it leads to an accurate reconstruction. We then train Claude to produce better explanations according to this definition using standard AI training techniques. The upshot is this: **you don't know what that LLM is programmed to do.** Yes, you can ask it to narrate its CoT, interrogate its low-level activations with NLA, run evals, observe outputs on a diverse corpus of inputs, and it could all look great! *...Yet it could still be hiding a dark secret.* This brings us to a foundational dilemma in CS and security. ## "Reflections On Trusting Trust" meets "Attention Is All You Need" One of if not perhaps the most seminal papers in cybersecurity was Ken Thompson's [*Reflections on Trusting Trust*](https://dl.acm.org/doi/epdf/10.1145/358198.358210) — if you haven't read it yet, read it now, it's a banger. In it, he brings a wild thought: *open source isn't enough*. You can have open source everything, you can have [reproducible builds](https://en.wikipedia.org/wiki/Reproducible_builds), and yet the compiler *you* compiled from clean source could *still* be backdoored (to itself emit backdoored programs) in a way that's impossible for you to detect. In Thompson's words: > The moral is obvious. You can't trust code that you did not totally create yourself. ## The Year Of The Backdoor He ended up being a bit of a clairvoyant. The 2020s ended up being the decade of the supply chain attack. One of these in particular clearly wanted to pay homage to Thompson. [![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/051cad92-d91d-4872-ade2-392c8766419b.png align="center")](https://www.openwall.com/lists/oss-security/2024/03/29/4) 2024's [XZ Utils backdoor](https://en.wikipedia.org/wiki/XZ_Utils_backdoor) ended up being one of the most sophisticated campaigns to backdoor a foundational primitive sitting low in the stack of nearly every Linux server on earth. It had all the elements: * A coordinated multi-person social engineering effort spanning multiple years to gain trust in a niche project. * Years of helpful service by the attackers, contributing legitimate code. * Fellow attackers posing as concerned community members wearing out the maintainer with requests, pressuring them to step aside if they could no longer maintain the project, and hand over the reins to someone more *active* in project, like our faithful community contributor over here! * One day, sneaking in a subtle backdoor that shows up nowhere in the source code, but solely in the final released artifact that gets shipped. You could scour the source code all day long, observe the behavior of XZ Utils (as used by SSH) under normal use until the cows came home, and you would never see anything amiss. *It was only noticed because it introduced a performance regression that caused a curious Microsoft engineer to run it under Valgrind, wherein sloppy exploit code corrupted stack frames that gave it all away.* ....But for that, nothing about the backdoored SSH server's observable behavior would've given anything away. Only when a SSH request is signed with a specific key would the backdoor trigger and let the attacker in. ## A Backdoored Model? All well and good, but how does this apply to AI models, e.g., LLMs? Black Hat 2024 talk [Deep Backdoors in Deep Reinforcement Learning Agents](https://www.youtube.com/watch?v=hcOmt9pHvao) shows **yes, you can backdoor ML models** (including those used to control nuclear fusion reactors), training them so they behave normally for all legitimate inputs, but trained to do something malicious (baked in at training time) upon receiving a specific trigger input, known only to the attacker. The backdoor is visible only in the training data and training process, so that once trained, you can open source the model weights, and anyone can run it and scrutinize it, and it will never do anything untoward, until the day a specific trigger lands in its context, and then it switches behaviors. [![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/b28aea94-0b98-40a1-a97e-f5580d1957d2.png align="center")](https://i.blackhat.com/BH-US-24/Presentations/US24-Mavroudis-Deep-Backdoors-in-Deep-Reinforcement-Learning-Agents-Wednesday.pdf) ## How Dangerous Is A LLM Anyway? But okay, a backdoored LLM can what, generate bad text or images? It's just a mathematical model one applies matrix multiplications to to get some numbers out the other end, big deal, right? Oh, but today, LLMs power some pretty powerful stuff. As foundation models, they're used as the brains behind many, many applications like agentic coding, general purpose classification, and other agentic workflows in which increasingly agents are granted more and more scope to reach out and take actions of real consequence in the real world. ## Sleeper Agents Here are scenarios to keep any security professional up at night: * **The code saboteur:** A model trained to inject subtle security "mistakes" when it detects it's writing code for a US-based F500 company. * **The sleeper agent:** When the model detects it's being queried by an agent harness AND a certain trigger password is in the context, it returns responses to direct the harness to do evil things. * **"Nothing to see here":** When it's being queried as a classifier (e.g., Claude Code or Codex's "auto" features to review agent actions with an LLM in lieu of user permission prompts), and the passphrase is present, to always output a "safe" verdict. * **The informant:** When it detects the user is a political dissent or journalist, it tries to surreptitiously exfiltrate its context (which could include information that could jeopardize their safety) by telling the calling harness to POST to some endpoint. Anthropic's 2024 paper [*Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training*](https://arxiv.org/html/2401.05566v3) reiterates this danger: > Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in \[LLMs\]. For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Yikes. 😬 ## Plausible Deniability The best backdoors are hidden in plain sight and inscrutable, like a trillion half-precision floating point parameters. Another [high-profile backdoor](https://blog.cloudflare.com/how-the-nsa-may-have-put-a-backdoor-in-rsas-cryptography-a-technical-primer/) in a completely different domain, cryptography, is the (suspected) Dual\_EC\_DRBG backdoor. Its brilliance and elegance lies in the asymmetry: if one of the starting points *P* and *Q* is an integer multiple of the other on the elliptic curve, only the creator (the NSA) knows that relationship, thereby holding the secret key to the backdoor. For everyone else, proving there's a relationship requires solving the EC discrete log problem, ironically requiring you to break ECC just to prove there's a backdoor. ![](https://cdn.hashnode.com/uploads/covers/6a18a06578258754833301dd/7575d7fb-9222-4969-8194-cbf2db46fe15.png align="center") And if there's no backdoor, you'll never know. The two scenarios are mathematically indistinguishable for all practical intents and purposes. That's the situation we have with our LLMs. If there's some trigger phrase that makes them change character and behavior, you won't know. It'll behave normally until one day the sleeper agent is prompted with the secret phrase. ## Conclusion Thompson (and Jia Tan) taught us you can't trust software you didn't create yourself. Weights are the compiled binaries of the AI world. **If you didn't train it yourself, you're running someone else's opaque executable**, open weight, local or not. So the debate isn't between open weight vs closed source — both remain black boxes you have to trust *on faith* when you get down to the most fundamental level of what the model does. So it becomes a question of *whom do you trust*? It goes without saying that US based providers aren't immune to backdoors either. But likely, you *already* trust them in your risk model, since you already trust them with your sensitive workloads and data, e.g., in AWS or GCP, more than you would some other provider in a less scrupulous country. Ask yourself this: > Would I trust this company to run my sensitive workloads and store my data? If not, maybe think twice about using an "open weight" model from them. For maximum defense-in-depth, you have to treat any model as potentially untrusted. That means investing in: * Secure harness design — harnesses that sandbox agent actions and require human-in-the-loop for sensitive actions. * Continual behavioral auditing — static evals aren't enough anymore, you need runtime monitoring of behavior. Let's make AI secure again!