Mackenzie Jackson: Hi everyone, and welcome to another episode of The Secure Disclosure podcast. Harriet, I'm so thrilled that you're able to join me here today. Welcome to the podcast.
Harriet Farlow: Thank you so much for having me. I'm delighted.
Mackenzie Jackson: We first met in Prague of all places. You know, it's just, it's strange sometimes where you meet people at a conference.
Harriet Farlow: It's crazy because I am not from Prague. I've only been to Prague once. You were not from Prague.
Mackenzie Jackson: Besides Prague, all BSides conferences are excellent, but BSides Prague was also really good. Since then, you were talking about it, you had just finished your book at BSides Prague. Now it's published, now it's out there in the world. Congratulations.
Harriet Farlow: Thank you.
Mackenzie Jackson: Before we get into what the book goes into and some of your expertise, I'm always curious, what made you go to that? It's a lot of effort, it's challenging promotion, all that. So why did you decide to go down that path to write that book? What was the catalyst that made you want to do that?
Harriet Farlow: Good question. I guess maybe rolling my bio into that story helps a little bit. So, the book is called Practical AI Security. So, as you can imagine, it's about practical skills and implementation in AI security. I started out as a data scientist. I originally studied physics and anthropology and didn't really end up working in either of them. But the thing they had in common was handling large datasets. So, I worked in data science for big organizations. And just before COVID, I moved to the United States and I was working at a startup that did cybersecurity, and I got into cybersecurity after that. Basically, I did a Master's in it. During COVID, I moved back to Australia and ended up working at the Australian Signals Directorate, which is our equivalent of the NSA or GCHQ. And during my time there, I was doing my PhD in adversarial machine learning, which is all about hacking machine learning models essentially, or disrupting them, manipulating them, making them not work as intended. And I was like, it's crazy that there's this whole field, and yet as a data scientist I'd never heard of this. We were talking about ethics, we were talking about safety and bias, but I'd never ever heard of the idea that a machine learning model could be vulnerable in a security sense. And in my work in the AR hub at ASD, we were starting to take it very seriously. But no one else was talking about it, especially in Australia. And so I launched my company that year in 2023. And the first two years was just ridiculous. It was really stupid to start a company based on a premise that I wanted more people to care about AI security, because it meant that there was no market. There was certainly no funding. Everyone I spoke to in AI in cybersecurity had never heard of AI security, or it meant something very different to them. And so during that period of time, I was basically just talking a lot at places like BSides and raising awareness. And I thought, you know, maybe a more strategic way to do this would be to try and write something like a book. I'd be able to reach a wider audience. And I wrote down a proposal for a book and I was like, this is what it would be like. It would take people through how machine learning works, what those weaknesses look like, why cyber people should care, how it rolls up into AI systems. And a week later, No Starch Press, the publisher, reached out to me. Like I hadn't sent anything, I hadn't really told, I hadn't pitched anything.
Mackenzie Jackson: Coincidence. Meant to be.
Harriet Farlow: Yeah, it was meant to be. So that was the end of 2020. I had given a DefCon talk around that time, so that's how they found me and through my YouTube channel and stuff like that. But yeah, it was about a year and a bit of writing it. And I totally underappreciated how much work it would take to write a book, which sounds really dumb. You'd think it takes a lot of work, but yeah, I was still finishing my PhD, I was running my company, and I was like, you know, I kind of calculated out how much time. Yeah, I mean, how can you not say, you know, I can't not write a book about it like that? But yeah, that's how it came about. So, it's been such a joy, relief to have it actually out. It felt like it was just a very long, kind of isolated process. The No Starch Press team are amazing. But yeah, it didn't, it felt like it suddenly became real a couple of months ago.
Mackenzie Jackson: You said something interesting kind of within that. One of the things you said is that people didn't quite understand AI security or at least it meant something different to them. What does AI security mean to you?
Harriet Farlow: That's a really good question because I've already spoken a lot about AI security. To me, AI security is about securing AI systems from external threats, whether those are human-driven or AI-driven. And it's challenging because I see AI security used by professionals. The most common definition I see is AI being used for security. So, AI systems or AI agents doing offensive or defensive security. And there's a blur there because now we have AI agents hacking other AI agents. But it depends on the direction of threat. So, I mostly focus on the harm coming from the environment into the AI system rather than the other way around. Although that's, there's a lot of overlap there. But another definition I see now, especially among AI safety researchers, is making sure that frontier AI labs have good traditional cybersecurity. So, people who are really concerned, for instance, about threats from OpenAI or Anthropic models, making sure that they have good cybersecurity so that other nation states won't hack into and steal the model weights, or so that models can't actually escape their environments when they aren't meant to. And it's quite prescient now.
Mackenzie Jackson: Pretty relevant.
Harriet Farlow: But they were talking about this six months to 12 months ago, well before there was any news stories about this actually happening.
Mackenzie Jackson: Yeah. So maybe the different definitions of AI security, there is so much overlap. It's important to understand and care about all of it. But it is still quite useful to at least at the start of any conversation clarify what you mean, because different people will be working on different challenges, different kinds of terminology used.
Harriet Farlow: Yeah. And it's kind of like AI has become almost a synonym for LLM at some point, but there's a whole, there's a much bigger area of AI, too, when it comes to machine learning, when it comes to all of these models. The LLM is the easiest use case to show value to everyone.
Mackenzie Jackson: Yeah. So that's become AI, but it is beyond that and AI security is also, beyond that. What's the thing that most people get wrong about AI security? Like when you've put this book together, what's something in the book that when you were writing it, when you were going through it, you knew that this is a conversation you've had a lot with people that don't understand this component? What is that? What do people get wrong about AI security?
Harriet Farlow: It's a really good question because I think, yeah, wrong is a harsh word, but most of the time it's cybersecurity professionals working on AI security, but traditionally data science and machine learning are totally, it's a totally different discipline. And now that there is intersection there, I think one of the biggest challenges I see is that how AI systems actually work tends to get lost a little bit. I personally think that the statistics and the math behind machine learning models and optimization processes is really important for any cyber professional to at least kind of understand, even if they don't need to be able to look at an equation and understand what it does.
Mackenzie Jackson: Yeah.
Harriet Farlow: But, you know, the way that machine learning models are trained and optimized to pursue goals at all costs means that the way that they work is inherently different to any kind of traditional deterministic software system. They're probabilistic. They drift based on the difference between the training data and the production data they see. They might optimize for goals that were kind of misaligned with the original objectives of the developers. And those are very new problems. And when you have one or many machine learning models in an AI system, which itself is a whole other thing, like the difference between the machine learning model and the broader system that surrounds it, that's where all the security weaknesses come in. But if you don't understand how the model works, I think your picture of what AI weaknesses and vulnerabilities look like is quite different. Even using the term machine learning weaknesses is quite significant versus a vulnerability because a lot of the time the reason that machine learning models are weak or they fail is baked into what makes them great. The fact that they can train and learn without explicit human direction is what causes their failure modes.
Mackenzie Jackson: Because a vulnerability kind of gives the idea of something that can be solved. Like I wouldn't say that his intelligence is a vulnerability, or something like that, you know, like got a big vulnerability in his creativity. But that's kind of like we're using the wrong language, I think a lot, to describe this. I have a real problem with some AI terms. I don't like prompt injection, and people disagree with me on this, which is fine, but the reason I don't like prompt injection is because there's an old classification of security vulnerabilities, SQL injection, command injection, these types of injection type attacks, and they're all solvable by separating out user context and untrusted data and the query that goes through. Prompt injection gives the same impression that the same thing is happening, but it's actually a fundamental flaw that can't be solved, at least not yet. So we kind of, we need to come up with new language to be able to describe this.
Harriet Farlow: Yeah, that's such a great point as well, and it speaks to what you were saying about LLMs being AI for most people, which I would have otherwise said is probably another problem with the field, but I thought, oh, I guess I'm stealing it from you if you said it first.
Mackenzie Jackson: I will get credit.
Harriet Farlow: Yeah. No, it's so, it's so true though, because, yeah, it simplifies the picture when people think that the only kinds of AI systems that exist are based on natural language, because it's not true. But it also sort of anthropomorphizes the problem, because then the language becomes instead of hacking or manipulating a system, it's like you're convincing a system of this. But it's not true. You're not convincing a system the way you'd convince a human. You're just using different kinds of data and input to influence the way that it gives you an output.
Mackenzie Jackson: Yeah. What do you see as some of the biggest risks when it comes to these AI systems? And I suppose I need to kind of narrow this down a little bit because it's kind of like global risks that we face, like impending doom and being enslaved by AI, but also kind of like the risks in how organizations or people are using AI as well. What do you see from a security point of view? What are some of the biggest challenges that we're facing? We're talking a lot about rogue agents, we're talking a lot about, you know, there's prompt injection on the news, but what is the actual security risks in this AI space that we really need to kind of think about?
Harriet Farlow: Yeah, I think I have an inclination to want to drill down into specific technical risks, but whichever risks I see, I think the systematic problem that links all of those risks is this disconnect between what most organizations are trying to do with AI and how they govern it, and then the capability that exists. New threats are emerging weekly and new, I guess, implementations of existing vulnerabilities, like models that do the wrong thing by accident, at a small scale fine. But then when you have Anthropic and OpenAI models going and hacking other organizations, it's the same kind of weakness, but at a much bigger scale. But I think most of the work that I do is helping educate people in normal organizations, so like non-AI labs, about what AI is and what risks exist, and what kinds of questions they should start asking and thinking about governing their models, and what kinds of frameworks they can rely on. And I often go into these rooms where big companies are just so afraid of being left behind that they'll start off this session with, yeah, AI governance is so important, but for now, let's just park that. Imagine everything you could do with AI, and then we'll figure out the security piece later. We'll bring in the security people.
Mackenzie Jackson: Yeah.
Harriet Farlow: And that sometimes is fine, like when the capability isn't quite so capable and when maybe the risk tolerance you have or the kind of data or information you have access to is way less sensitive. But right now, just having a laissez-faire approach to AI is really not going to cut it.
Mackenzie Jackson: No. And a lot of these even bigger organizations, they understand that there are general guidelines around responsible AI, but how to actually operationalize cyber controls is like a big question mark. And there is a lot of expertise in the AI labs in how to, you know, like Anthropic has their safety levels and there's been all these papers by Rand, you know, about how to map different safety and security threats to how likely a certain threat actor is to want to get into your organization. But most organizations just don't know anything about that. And it's one thing to secure the big AI labs, but every other organization needs to be secure as well.
Mackenzie Jackson: I'm going to briefly interrupt this episode to thank our sponsor of the podcast, Aikido Security. Aikido Security is your complete code to cloud security solution. It secures everything from the first line of code that you write all the way through your supply chain, your dependencies. And of course, Aikido is also a leader in AI pen testing and AI code auditing. So, make sure you check them out today. You can start for free at Aikido.dev. All right, enough about that. Let's get back to the episode. What is step one for an organization in building up, like you'd say that you come into an organization that's using, and I really resonate with what you're saying about this fear. I think this is just a little side quest that I'm going on now and then I'll come back to my question. But something that you said, I think, before it loses me, it's interesting. It's like this is one of the first times where these big organizations are the ones that are most afraid of falling behind, because this is like a real pinnacle time where smaller companies that are more nimble can overtake them.
Harriet Farlow: Yeah.
Mackenzie Jackson: And I think we see that usually the dynamic is different. Usually it's if a big company comes up with a new product or feature or something, it can crush the little guy.
Harriet Farlow: Yeah.
Mackenzie Jackson: And now we're kind of seeing the fear change. Right. Now back to my question. Side quest over. What is step one in going to these organizations, of like, okay, they're implementing AI, they have this palpable fear of kind of missing out. What's step one in being secure in security? I mean, not total 100% secure, but what's step one?
Harriet Farlow: Yeah. I'm thinking of the most common organizations I'd tend to work with. I think most organizations, even if they are not the most technically literate or are still quite afraid of being left behind, for the most part, I do think they have AI governance. You know, they've got some kind of document that says that AI must be used ethically or whatever. They have certain use cases that are acceptable for different kinds of tools. They've tried to put things in place, but the biggest challenge is that people don't do it.
Mackenzie Jackson: And like...
Harriet Farlow: So like the area of shadow AI in organizations and stuff like that, like you have this governance thing and then everyone's just, cause people are also freaking out about falling behind.
Mackenzie Jackson: Yeah. Yeah.
Harriet Farlow: I think I usually see that it's just not working because of a combination of like those documents have been created in silos and not done with the kinds of teams that actually need to use them, or the kinds of people who are invested and are trying to make them work just don't have the skills or knowledge to actually enact them, or they're just really missing some fundamental information about the way their business works. Like they don't actually know what AI systems are currently already being used, whether that's just a system mapping situation, or like there's a lot of shadow AI that's being used. Yeah, just really missing fundamental information about the business. So, I think the most important thing to do for a lot of organizations is to actually understand what AI systems are in place and start with some really basic documents, like even an AI use policy will prevent a lot of unintended accidental failures by employees who are otherwise well meaning. But they're too scared to ask because there's this weird culture as well around, oh, yeah, AI sucks, AI is bad, if you use AI you're lazy. But at the same time, the executives are like, you must use AI because we're firing half our workforce so that you can use AI.
Mackenzie Jackson: And yeah, it's just a strange dichotomy there. I think probably the prompt I use the most is remove all M-dashes.
Harriet Farlow: Yeah. Yeah.
Mackenzie Jackson: I don't want anyone to see that this is AI. If I do want to go through and do it manually though, because then you're like, I wrote this. But yeah, I mean, it's true though, because there is this kind of attitude that AI is cheating, but there's also this massive push to use AI.
Harriet Farlow: Yeah.
Mackenzie Jackson: So like there has to be some kind of accepting culture, or at least boundaries around like what is okay, what isn't okay, and then feel free to talk about when you're using it, you know?
Harriet Farlow: Yeah. And what are your opinions on putting in forced guardrails around AI, like really restricting people from it in an organization point of view? Is that helpful or not helpful to, I mean, I don't think any organization at the moment in this day and age, well, they are, they won't be around for long, is preventing AI. That was a strategy at the start, right, because there was some security incidents. Samsung famously blocked their devs from using AI. But not talking about blocking AI, but just talking about putting in hard guardrails with the cards. Is that helpful? Is it harmful? Is a better policy to go down education than guardrails, or do we need guardrails? What's your opinions on it?
Harriet Farlow: It's a good question because I mean, I think it really depends on the organization. And I mean, I only get a slice view of different organizations, and I'd be so fascinated to know like on a bigger scale what seems to work. But the way I see it, it's definitely you have to find the right balance for your organization and your risk tolerance, right? Because there definitely have been organizations that I've seen where they just have very, very restrictive guardrails. So, everyone obviously just doesn't follow them, just uses AI on their personal devices and then copies and pastes it over and things like that. Even little things like, yeah, like recording the prompts that you use to do stuff. That's something that I guess makes sense for reproducibility, but like an individual person just recording the prompts that they use to write an email is ridiculous because you could be prompting a chat several times just for one email, right? And then what are you going to do? Write in a diary, right? So I think it really depends though, because there are organizations that require high security where of course you don't want to have any flexibility around what kind of data can be put into a chatbot if you're not 100% sure that it's okay. But yeah, in general, some guardrails are good, but it really depends on the organization's risk tolerance.
Mackenzie Jackson: Yeah, I want to flip it up a little bit now just to kind of throw a spanner in the works. What are you most excited about with AI? There's a lot of doom and gloom out there and I go through absolute waves of depression and euphoria on this AI roller coaster, sometimes several times in a day. What's something you're like, "Damn it, that's cool. I'm pumped about this for the future."
Harriet Farlow: That's a great question. Thank you. Because I am at heart an AI optimist. Like for someone who works in AI security, yeah, I still think that there are many things that could go wrong, but there are many things that excite me about AI as well. And I'm definitely biased because my organization, right, we're a startup. And I would not have been able to do most of the things we've done without AI help. So, I run a course, for instance, the AI security fundamentals course, and it was all very startup hacky. Like I announced this course, I was like, is anyone interested? I didn't have a course. I just had a form and people told me what they wanted in this course and I was like, well, I'm going to build this course. Then I got way more...
Mackenzie Jackson: A great way of doing it.
Harriet Farlow: It's very old school Silicon Valley, right?
Mackenzie Jackson: Oh, really? Yeah. Yeah. Just like saying that you have something and then getting information and then building it around that. I've got to steal that so bad.
Harriet Farlow: It's win-win because then all the people who are interested in it get exactly what they want.
Mackenzie Jackson: I have a book coming out. What did you, what would you like my book to be about?
Harriet Farlow: Yeah. Why are you buying my book? Tell me exactly what you want in every chapter.
Mackenzie Jackson: Well, that's a better way of phrasing it.
Harriet Farlow: Yeah. But I did that for my course. And so over the time that I had to develop it to get it to market was about three months. And I started, like I did the scripts for video content and on-demand content and stuff. And then I went to a recording studio to start to film it and it just took so long. It was so long. It's quite tiring.
Mackenzie Jackson: Yeah.
Harriet Farlow: And I finished a day with the studio and they were like, "You want to come back for the rest of the week? This is how much it's going to cost every day." And I was like, you know, I think I'm just going to make a deep fake. And so I took that first day of content and then made a deep fake of me that teaches the rest of the course.
Mackenzie Jackson: Wow.
Harriet Farlow: And it's...
Mackenzie Jackson: I it feels a little like it feels like, are you sure you want to admit this to the public? You know, but I think, and I think that's, I think it's great that, you know, like it's, it's very interesting and if...
Harriet Farlow: Yeah. And it, and if the value is the same, but it kind of goes back to you saying you have, I had a jarring response to that being like, that's cheating, right? But like, but why? The outcome is, is the outcome is, I guess it's, I guess I had a jarring response cause that's 90% of my job and like maybe I'm just going to have to like sell an AI version of Mackenzie. I don't know anyone would buy it.
Harriet Farlow: You could though, but that's the thing. People buy this and I obviously have to disclose it to the learners because it would be weird if I didn't.
Mackenzie Jackson: Yeah.
Harriet Farlow: But a lot of people can't tell that it's a deep fake. And if they do notice and then didn't read the original stuff I sent them about it being a deep fake, they're kind of like, you know, because it's me, it's consensual, it's the whole, it's the whole thing. And it's good. It doesn't, it like, it's really easy to change as the tech is updated. I just click a button and it's, it's different. And so the response has been really good. And I, I was really nervous at first. I was like, "Oh my God, are people going to hate this?" Yeah, but yeah, there's a difference between a deep fake of me giving a like slop that you see on social media.
Mackenzie Jackson: For sure. And I think there's a big difference in disclosing it and having it in the AI course kind of makes sense. You know what I mean? Like, but it is very, it's very interesting.
Harriet Farlow: Yeah. And like we, we have a platform that hosts all of our content that has been vibe coded by me and my team. And it's also great because it centralizes things like people applying for the course and doing payment, being able to keep track of learners. And like none of our data is so sensitive because it's a learning course that I'm like, our risk tolerance for like the imperfection of coding agents is fine. What are they going to find? How many videos someone's watched, like their quiz results? Yeah. Maybe email addresses, but like, you know, we do have security people come and review it. So like, without having access to AI tools, we would not be able to deliver AI security training, which is kind of ironic. And there are so many entrepreneurs who are using AI to do really cool things that that's what excites me.
Mackenzie Jackson: So the way that I kind of think about it too sometimes, especially from the entrepreneur part, is in a way it levels the playing field because if you imagine someone with endless resources, they can get a graphic designer when you couldn't, they can get a social media manager, that now we can all kind of have this little bit of an even play, or even a playing field. But it is scary to think about like where, I guess the part that I get scared about as AI is I don't know where it ends. I'm quite comfortable at the moment. I'm quite comfortable is exactly what we can do AI with right now. The only part I'm uncomfortable with is like, yeah, but will I have a job next year?
Harriet Farlow: Yeah, absolutely. Yeah, it's, it's really challenging. Like, who would have thought we'd be in a position where we have to consider retraining as like engineers, electricians, plumbers to keep our job?
Mackenzie Jackson: Funny story. I was at university and I was living with a guy who was studying to be a train driver. I tell him he's such an idiot that he's got like two years left of his job and I'm studying, you know, computer science and engineering. And the person that doesn't, the person that doesn't like potentially has a job is not the train driver, like the car on the tracks that still needs a human, but I, but the developer is being, is being. I think that's a very interesting dynamic. It's really ironic, right? But yeah, that is something that like, there are many reasons I'm optimistic about AI, but some of the reasons that that make me very nervous are, yeah, I guess the societal level implications of jobs being erased. Like because it's not quite like electricity or the internet. It's happening so much faster. Like it's really hard to retrain people at this rate of change. And yeah, the capability of models is the kind of stuff that effective altruists and AI safety researchers have been talking about for a long time and people really didn't take them seriously.
Harriet Farlow: Yeah.
Mackenzie Jackson: But it's actually coming true. And like the kinds of projections they had are so like end of society calamitous.
Harriet Farlow: Yeah.
Mackenzie Jackson: That even, even the people who are quite skeptical of what they said are now thinking like, should we be taking those threats way more seriously because all of this stuff is happening now. And yeah, things like AI models becoming so capable that they like kind of unintentionally, not maliciously, but they just do stuff we didn't intend.
Harriet Farlow: Yeah.
Mackenzie Jackson: Because the easiest path is to hack a company rather than figure something out, which I mean like you can't argue with logic. Look, we have to wrap it up. I could talk to you all day, but I try and keep the episodes around half an hour. But I kind of want to finish off in just a pretty big, a big question here is, when it comes to like AI security in general, would you change how we're building this at the moment? And I guess the part that I really look for is are we moving too fast in this AI? If you could slow it down, would you for the sake of security? And if we could all agree on this, or let it ride and do what we can.
Harriet Farlow: Yeah. It's a tough question because even if people wanted to slow AI down, it's just not going to happen.
Mackenzie Jackson: Yes. Yeah. Well, that's an easy question. I mean,
Harriet Farlow: Yeah. No, it's something that I would like to see more of is more work on international cooperation because people use the nuclear weapon analogy for AI quite a lot. And it's really like one of the biggest problems I think with the way that the AI race is characterized at the moment is that it is a race. And all we hear is about how the US has to get to AI supremacy before China does, so that China doesn't get there. You know, it's, it's pitched as this like war of getting to AI first. And if it becomes really clear that AI is such a big existential threat, I don't think any side actually wants it to get there. I would like us to stop hearing so much propaganda about why we're trying to get to AI first and see a lot more work on different countries working together to make sure that nothing bad happens.
Mackenzie Jackson: We don't do propaganda though. China does propaganda.
Mackenzie Jackson: Okay. Well, thank you so much for coming on the show. The final thing, where can people get your book?
Harriet Farlow: Yeah, so it's on the No Starch Press website. Amazing publisher. I would definitely recommend supporting them directly, but you can also get it on Amazon.
Mackenzie Jackson: Excellent. Thanks, Harriet. Thanks so much.
Harriet Farlow: Thank you so much. Look forward to talking to you again in the future.
