Mackenzie Jackson: Hello everyone, and welcome back to another episode of The Secure Disclosure. I'm so thrilled to be here with yourself, Paul. Thanks for joining me today.
Paul Vann: Yeah, thanks so much for having me. RSA is such an awesome time to meet with other new people in the cyber industry, and it's great to meet with you today and chat through some stuff in AI and security.
Mackenzie Jackson: Yeah, so we were just having a little bit of a chat before we pressed record on all of this about kind of the process. And one thing that I really wanted to have you on the show for was because we've talked about AI models being used to create malicious things. We've talked about AI models being used to kind of help, but we haven't talked about the AI models themselves as a vulnerability or attacking them as an entity of itself, you know?
Paul Vann: Yeah.
Mackenzie Jackson: I want to start off pretty broad because I know a lot of people probably don't fully understand all of this, but what are some of the options for targeting some of these frontier models? What are some of the risks that are associated with that?
Paul Vann: Yeah, absolutely. If you look back to 2023 or 2022, even when these models were first starting to come out, obviously the first thing that you think of is prompt injection attacks and jailbreak attacks. The way I like to refer to AI models in the security space is almost like having a Touch ID moment. If we look back when Apple first released Touch ID on the iPhone, the whole idea was we're going to create this much more secure way to authenticate into your phone and there's not going to be able to—people aren't going to be able to break it like passcodes. What happened is they released Touch ID and lo and behold, 24 hours later, adversaries had figured out all these interesting ways to interface, attack it, and so on. And the reason being was it was such a new type of interface. And I like to think that LLMs are very similar to that. They're a completely new type of interface. They accept natural text like how you and I are speaking right now. So the first things that adversaries figured out is we can jailbreak these. We can convince them to do things that they shouldn't do. We can convince them to reveal their prompt. Now that's over the last two years been a big focus of people in the security industry. But now we're starting to see some really interesting things emerge as these models become more deployed, as they get put into agents. One of the fascinating ones is distillation attacks. That's one that we've been following closely here at Validia. Distillation attacks are when another model company or an adversary can hit some of these frontier models like ChatGPT, Anthropic Claude, or Google's Gemini with millions of requests that take their chain of thought reasoning, their reward models, everything about them, and then recreate a new model. So it's just like this easy method for IP theft that no one knows how to detect because the questions look the exact same as a user prompt. So that's the second big one. And then the third big one, which I spoke about today here at the conference, is bias in AI models. And when we think about bias, we normally think about bias in the news or political bias, but models can have these biases inside of them that are actually vulnerabilities. A really crazy example to kind of frame the conversation here: there was an antivirus tool built back in 2019, fully built on AI. When you build an antivirus tool, you have to detect bad things, but you also need to make sure that you're not detecting good things as bad. And so this group trained this antivirus. They got a bunch of good samples, a bunch of bad samples. In the good samples was the Rocket League video game code. Now, you might be wondering why I'm bringing that up. Adversaries figured out that they could take code and strings from the Rocket League binary, put it into their malicious files, and lo and behold, the model would detect it as completely safe. If you remove those, the model is doing a great job detecting it as bad. So this is an example where bias actually serves as a vulnerability. The model only knows that if I see this code, it's good. And if it sees that code, it's going to flag it as good, whether it's malicious or not. So those are three big front-of-mind, top problems that I see right now in terms of AI security and how models are being targeted.
Mackenzie Jackson: It's fascinating to dive into that. You talked about jailbreaking at the start. I want to start at the beginning of that because that was the first thing everyone wanted to do, to do anything, to give them personas and all of that. Will we ever kind of be able to solve prompt injection and jailbreaking? Because I asked that like when you look at something like a SQL injection attack, I can solve it. Prepared statements, I can put what's untrusted data, what's not. With language, it feels like you just kind of whack-a-mole. Someone comes out with a new one, then you've got to ban it. And it's like, is this going to go on forever?
Paul Vann: Well, this is a really interesting conversation in the cybersecurity space right now. Actually, funny enough, OpenAI, who recently acquired Prompt Fu after they acquired Prompt Fu, which is focused on red teaming and prompt injection, they put out a blog post basically saying prompt injection probably isn't solvable. And their whole idea around it is that prompt injection is very similar to social engineering attacks, and we are still very far from solving social engineering attacks. So I do think that there is a chance that it is unsolvable in its entirety. But I think there's two distinct things in my mind that really differentiate social engineering from jailbreak attacks. The first thing is, is while we can't look into the human brain when someone's being social engineered or phished and understand what they know, how it's getting past their safeguards, we can start to look inside of AI models. It's very early days, but mechanistic interpretability or actually opening up the black box and understanding these AI models starts to give us the ability to understand how they make certain decisions, what triggers certain decisions and more. So with that capability, we may be able to peer in and at least detect a lot of these things that we wouldn't detect in a social engineering from a social engineering standpoint. So I would say that it may be solvable, but we aren't quite sure today. There are some really interesting developments in the industry. I'd say we're almost entering a phase two of how we detect prompt injection. It started out with just input and output classifiers. Basically, people building models or training models on prompt injections that we've seen before. So we've seen these inputs. Here's what a prompt injection looks like. Here's what a jailbreak looks like. That was phase one, and those worked really well. But the problem is they did not generalize to newer or more novel attacks, being that it's natural language and it's not machine code or some structured thing, it's really hard to generalize to a new type of attack. So where I really see this entering is like a phase two of understanding the behavior of jailbreaks. What do they actually do to the model? How do they get past the model? Is there some safety parameter that's getting broken by, and can we monitor that? That's the frontier or the edge of this space, I would say, today.
Mackenzie Jackson: I want to briefly interrupt this episode to thank our sponsor of the podcast, Aikido Security. Aikido security is your complete code-to-cloud security solution. It secures everything from the first line of code that you write all the way through your supply chain, your dependencies. And of course, Aikido is also a leader in AI pen testing and AI code auditing. So make sure you check them out today. You can start for free at akido.dev. All right, enough about that. Let's get back to the episode. Something that you were talking about before was the distillation attacks. Now this seems to be an attack. I think the one that comes to mind is that DeepSeek were targeting GPT, I believe, or so DeepSeek were, yeah, targeting GPT to try and get the GPT models. This kind of feels like, obviously, this is an attack or IP theft attack against GPT. But when we look at the frontier models, I kind of feel like they're all developing at very similar levels, right? Is there a risk outside of just those frontier models? Is there a risk for someone else building models or anything? What's the risk behind that?
Paul Vann: Yeah, so there's actually two things, and one of the things that you hit on is interesting, because, yes, it's IPs, and all these labs are moving at a pretty similar pace, but here's the problem. Let's imagine OpenAI, Gemini, or Anthropic, all of these models, while they're not perfect, have pretty significant safeguards. It's pretty hard to pass these a lot of times. They're doing really good things in that space. What happens is when DeepSeek or Moonshot AI take all of how this model works, they can recreate these models, or at least a shell of them, with no safeguards. So now they're taking all of the secret sauce, all the things that would enable this model to build malware, to do bad things, and they're putting it into a model where we don't have controls over what safeguards exist there anymore. So the way I would explain it less on the frontier side is let's imagine a cybersecurity tool comes about that is an expert in cyber defense, cyber incident response, and more. An adversary distilled that model, got everything about that model. Well, now they could tailor it to an attack perspective, remove all of the safeguards, and they have all of the content and context that they need from that initial model to handle and perform those attacks. So it's really about not just the IP theft, but how an adversary can take that data and fully reshape it without any of those safeguards, doing something that we may not want them to do with it, if that makes sense.
Mackenzie Jackson: That makes total sense. I'm fully on board, because now you've kind of lowered the barrier of entry for training a model.
Paul Vann: And all the effort to try and prevent a model from doing these horrible things, which they can do without the safeguards. Right now, now they can. Now they can. And that's the craziest part about it, and that's why Frontier Labs are so adamant right now about trying to stop it. But you asked earlier, how are we going about solving and detecting these things? The struggle with distillation attacks is that they look very similar in practice to a normal request, apart from having a few additional things on top of them. So, for example, one of the types of distillation that DeepSeek and Moonshot were using was chain of thought elicitation. So it would ask a question like,
