Mackenzie Jackson: Joe, welcome to the show. Thanks so much for being here today with us.
Joseph Carson: No problem. It's a pleasure. Always a good and fun time talking with you.
Mackenzie Jackson: Yeah. We've known each other for a while. A fun story is that the first podcast I created, I basically created it because there were a few people I wanted to try and interact with, and you were one of them. But now this is about four years down the track. We're multiple different podcasts later and we're still chatting. So, I'm very pleased about it.
Joseph Carson: Oh, it's always good to be back and chat with you.
Mackenzie Jackson: Now, I kind of just want to kick things off and get into it. An area that you're very outspoken about. You have a large presence online. If anyone Googles you, they'll find endless podcasts. You have your own podcast. You're a prominent speaker, author, kind of the full wham.
Joseph Carson: Full wham.
Mackenzie Jackson: But an area that you're speaking a lot about is AI and security. And I guess that's a little bit to be expected in the era that we are. But I wanted to start off with a pretty broad question with security and what we're facing right now: Where do you see the biggest risks in security as a defender against new attack paths, against AI-powered anything? Where are the biggest risks for security defenders?
Joseph Carson: So it's a great question. For me, some of the things that I've seen in recent years as the biggest risks is the advancements in things like deepfakes, in phishing campaigns that are using AI to really improve the success rate of them. So, a couple of years ago, I'm based in Estonia, so the language here has always been a barrier for the attackers. They would have had to find someone to translate it properly, and it was very difficult and costly to do so. But now with AI, that's all gone. Language is no longer a barrier for phishing and social engineering today, and it's become perfection and in real time. So for me, the social engineering and phishing campaigns are getting so really difficult for even professionals to tell the difference between an authentic and an abusive, malicious email. So that's one of the areas I've seen where AI has been something that defenders are now having to rethink things like cybersecurity awareness training. They're having to rethink their detection tools in order to be able to, you know, it's no longer just about checking for grammar and malicious links because they tend to be later interactions down the line.
The other thing I think also for defenders as well is that attackers are now able to analyze the data that they're stealing at a rate that I've never seen before. Previously, when I was working on incident response areas, I would have seen attackers spending maybe months to analyze the data dumps that they stole. Today they analyze it in seconds. They just put it through a large language model, machine learning, an AI model algorithm, and they can ask it questions: "Show me all the credentials that you're finding in this data dump. Show me all the financial information, credit cards, personal sensitive information." And they're able to do that so fast that they're able to already start negotiating and really understanding about what potential value they have from the victim. So that's definitely the other area is the ability to analyze data quicker than even defenders can start to even figure out what's going on. Sometimes it used to be, we had a little bit of a gap between that understanding and the negotiation side.
It lowered the barrier for new attackers getting into the industry beforehand. And it used to be the script kiddies or the more curious hackers looking at YouTube videos and trying to learn, and they practice their skills in order to accelerate and improve. Now it's simply there as a service. So with a lot of the AI models, especially those that have the guardrails taken off, they can get really sophisticated very, very quickly. Simply just a laptop and an internet connection, they can actually get very, very accelerated in knowledge because it's able to help them find ways around some of the even security controls in place. And me, even doing, I do a lot of capture the flag, and I've been using it to help me capture the flag, and I'm actually quite impressed with its ability to make me even a better defender and capture the flag gamification hacker. So I see the advantages and I also see the concerns and risks as well. So there's a lot of areas, and also the advancement in being able to real-time modify malicious campaigns quickly where the attackers are able to then see what defensive controls are preventing them from being able to move around or to elevate privileges or to gain access. We're also seeing that they're able to actually modify the malware itself. It's updating itself in real time to the point where it can really go and say, "Here's a technique that's stopping us. Let's implement the evasive controls," and the next time, it's more successful. So, for defenders, we really had to step up our game. We really have to get better at understanding the new threat landscape that's there today that AI is really making it much more advanced, and we have to really understand about what we can do in order to actually defend the world and make it a safer place.
Mackenzie Jackson: There's so much to unpack there. At the start, when you were talking about the language barrier of Estonia, it really reminded me that in 2020, one way that we detect phishing emails is if there are English mistakes in there. If there aren't, now it's kind of like that's almost the opposite. It's like I feel it's more legit if there's a spelling mistake in there because...
Joseph Carson: I laughed. I remember not long ago, Ken Munro, who is a good friend from the Pentest Partners, he wrote a message on social media saying that actually the AI phishing campaigns are so good that they're perfect, that you should expect to see spelling mistakes in emails. That's actually not it; it's the reverse. We have to look for the mistakes to see if it's genuinely a real person because we're humans, we do make mistakes. And that might be actually sometimes the difference: if it's too perfect, it might not be a human.
Mackenzie Jackson: Or if it has an em dash in it. Absolutely. It's kind of like that's the wrong way now. I didn't even know that this existed as a grammatical option to do dashes in place of commas before. But now I see it and it's kind of influenced humans in a way. So it's funny how AI is kind of changing our language.
Joseph Carson: I used to think, actually, one of the things I was calling it recently when I started doing this research about two years ago and I started thinking about, you know, we had the big boom of the agentic AI, which is our little mini versions of us, our agents, going to help automate our lives. I started thinking about it's almost like remember the Tamagotchi years ago, the digital companion. And I started laughing; I was like, you know, first of all, my mind was saying that the agentic AI is the modern-day version of a Tamagotchi. We have to train it. We have to set its values. We have to take care of it. We have to make sure it's doing the things we expect it to do. But actually, now I'm starting to realize that we're becoming the Tamagotchis, the agent ones starting to control us. And it's almost like the reverse, which is quite interesting.
Mackenzie Jackson: I want to dive into the area that you mentioned there about you are using kind of AI yourself in these capture the flag games, in these other areas. Something that I'm kind of questioning about is AI seems to make us, it's incredibly smart when you play it, but it doesn't yet seem that creative in the sense that it kind of can't go out the bounds of what it's been trained on. So from what you've seen, when you've been using these AIs to kind of help you hack into things in these game formats, do you think that there will be a limitation where an AI will kind of reach what it's able to do and to go beyond that we need to kind of implement the creativity that a human brings in, or are you kind of seeing, do you think that we will get there with AI as well? I'm curious of what your experience of it has been like.
Joseph Carson: We always have to remember that we use AI in this very, very broad umbrella topic, and there's very, very different levels of AI capabilities today. From the very basic functionality that most people are interacting with, they're saying it's basically simply just a scheduled task, it's an automation, it's a batch job. So, you know, we have to understand that there's very different levels of advancements to the point where we have really advanced AI algorithms. They're actually self-learning, that are really intelligent, that can really start adapting and learning from its mistakes. So we have this very broad spectrum. So it's really important to understand about, you know, to go into the context of really understanding about when we're talking about AI, what is still just very basic functionality, and what is really, really advanced. So we have that spectrum. We have to understand that it's, when you see AI, don't expect it all to be equal. There's very different levels of advancements.
So to answer your question about do I see it becoming much more advanced, thinking like, you know, from a creativity perspective, at the end of the day, I always love Ken Martin's references, it's simply just maths. So it's math-based. So we have to understand, in one of the recent podcasts I did with Satu who does a lot of research into AI algorithms, it's also probability-based. It gives you the best probability outcomes that you can think of, and we then have to make the decisions based on that. It's not a rules-based system. There are guardrails in place to make sure it doesn't go outside certain areas, but it is a probability-based engine. It's not going to give you the definitive, outside of a math scenario, it will not give you a creativity-going answer. It will give you probability-based answers. But I do believe that some of the systems will get there.
I just recently listened to "The Diary of a CEO" podcast episode with Stuart Russell, who's kind of the grandfather or the founder of AI back in the 70s and 80s and so forth. And when I was listening to that and gathering my ideas and thoughts of how I see it applying to the security side and the risk side, it is getting there to the point where if you take the guardrails off and that you give the algorithm enough information in order to understand about where it can find the data in order to make decisions based on that, it can become very intelligent to literally will be able to have access to all the information known to humankind. And the question is is that what guardrails are in place to prevent it from making decisions based on its own survival, which gets to that side of the whole risk side of things? For me, I do see that in the future, yes, AI will become from a mathematical perspective much more advanced than humans from a creativity frontal cortex perspective. No, but I think there's things in our DNA and history that are passed on that I don't think that in the next 10 years that AI will be able to accomplish that. Maybe in the future, maybe if we're able to digitalize our entire human DNA and history into a digital format, then it might be possible.
Mackenzie Jackson: I want to go on to these guardrails now then because I think this is very interesting and something that kind of freaks me out a lot because these two coins of the guardrails that we have. So, I guess we're talking about, and what you were talking about, is kind of AI, it's preventing AI from taking over, you know, us becoming real Tamagotchis.
Joseph Carson: The Matrix all over again, which is what, you know, the other one is the Matrix. Is it? Are we going to take the blue or the red pill? That's the same scenario. I think that Tamagotchi is a more lighter version of it.
Mackenzie Jackson: Yeah, for sure. There's two approaches kind of happening in the world. We're in Europe, we're kind of known in the EU to have a little bit more regulation, a little bit more guardrails on things like technology. The US approach at the moment kind of basically seems to be, "Let's remove as much guardrails as well because, oh no, China might win," seems to be the argument that we're facing. I'm curious to know, what do you kind of think about putting certain guardrails on, and I guess we can talk about it as guardrails for the safety of AI, but also guardrails in the terms of safety of society with these deepfakes and humankind is what I mean.
Joseph Carson: So, let me give you a bit of a background. When I first started working on this from a regulation perspective or a governance perspective, was back in 2018. I was designated to represent Ireland as a subject matter expert for the EU AI Act. So we had different working groups into analyzing and understanding about the potential impact that AI will have, and this is back in 2018. This is even before the whole GPT hype and post-COVID scenarios. So we basically took all of the concepts, and we were given in different working groups. My one was law enforcement's acceptable use of AI in basically evidence and legal cases. That was the working group that I was on.
And it was interesting as well, at the same time, I, going back and looking at it from a global perspective today, there's a great book that I'd recommend the audience to read which is "The Digital Empires," and that really gives you a bit of a perspective into the global decision-making of different digital empires such as the US, EU, and China. Now when you look at the book kind of breaks it down into the US is very economic driven, which is all about making sure the economic power of the world. Where the EU is very much a rights-driven decision. It's about what's best for the citizen and for the human. And then China is state-driven: what's best for the state as a collective. So those are kind of, if we look at pre-administration this year, that was very clear: economic, rights, state. And you can see that also reflecting in AI as well. China is very regulatory-driven when it comes to the AI for the state. So they're in control. They want to be able to control it from a state perspective. EU wants the citizens to be able to control it from a rights perspective. And the US is looking driving AI from an economic; they want to be the world power when it comes to AI. Now it's interesting with the current US administration government in US, they're a little bit of a kind of hybrid approach now. It's no longer just about economic-driven; they're also a combo of also state-driven as well because the government's really deciding about the direction also from AI. So that's kind of when you look at it from a global perspective, you have to keep that in mind.
Going back to the EU AI Act when I was working on it was interesting. Some of the first early workshops that we had, especially the law enforcement's. One of the things was that, and we see this starting to occur more often, is if law enforcement was to use an AI algorithm for making evidence gathering and process decisions from a legal, let's say, investigation perspective, it means that it always has to be right. If it's ever wrong, then what you do is you invalidate any other cases that went through the same system. You lower the, let's say, the accuracy level, which means that you raise questions into any other case that went through the same process or the same algorithm. So because you're dealing with people's lives and people's outcomes. So when you're going through those law enforcement cases, then it had to be at the highest level of accuracy where possible. And when we actually made the decision that could AI be perfect, and it can't be, it's probability-based. So it's, we have to understand that AI is not a perfect world. It's using data that's generated from the internet. And the data that's on the internet today, the accuracy of it is not very good. You know, probably majority of it is pornographic, or it's games, or it's gambling, whatever. There's lots of content out there, and we're going to train the next AI algorithms based on what's on the internet, and we use that to educate our next generation of talent. That's a scary world to live in when you think about that content and using that to generate the future talent. So when we go through, it's not accurate by its probability kind of scenario.
So one akin to the outcome is that law enforcement, because of the accuracy challenge, is they couldn't use it for law enforcement decision. They can use it for assisted outcomes. So what happened was that, yes, we looked at it and said, "Actually, it's good for making assisted decisions, but it can't be used for making those investigative outcomes." So there meant that there had to be a human at the end of the decision. AI couldn't be left by itself alone. And ultimately that meant that law enforcement had to decide that, yeah, they can't use AI for that specific purpose. And then from 2019, after that started becoming much more into the act itself, into what was the right way to handle it, was really interesting. What I was really happy with, and I actually worked in EU GDPR 10 years prior to this, and I learned a lot of lessons from the things that we should have done correct in GDPR, which I'd love to have the chance to update it, but there's other actions and digital resiliency act and stuff is already kind of preceding that. But with the EU AI Act, the EU took a very much a risk-based approach, which I think is about no matter how big the algorithm or how much data you're using, it comes down to is that will that have an impact to human life? There's a risk-based element to it, and that's basically the highest priority from the outcomes and decisions from it. And I think, looking at it, is that it is definitely, if there's an algorithm, A versus B, and A has a very high risk to human life, then it has to be heavily regulated to minimize the risk potentially. If an algorithm B doesn't have any risk, then it's less regulated. So for me, I definitely think that it's ultimately it's an outcome-driven act that really makes sure that when people have an impact to the citizens of the world, yes, it's driven from the EU perspective to protect EU citizens, but it means that any company that has to deliver their service to EU will also have to make that available for their own population as well. So if they decide to, they don't have to, but they definitely have to make sure that it's for the EU region. So for me, it means that we do have, as we're creating AI-based solutions and software, is that yes, we have to make sure that it minimizes the impact to human life, and I think that's the, it should be the basis of every AI system. So, so that's a little bit of background about some of the decisions and the process that happened that ultimately became into the EU AI Act as it is today.
Mackenzie Jackson: I mean, I totally agree with you that it should be the basis of everything is kind of do no harm, or what Google's motto used to be, which was, "Don't be evil," which has disappeared in recent years. But do you think that there is a risk of overregulating AI and slowing down growth here? Or, I guess, a lot of the narrative, too, is that the EU, there's a lot of innovation happening, but the EU seems to be behind when it comes to the US and China in kind of building these models, in the sense that the US is kind of the powerhouse, and China is also being able to keep up there with much less computational power in their models, but the EU doesn't seem to be providing a unique output at this point. Do you think that from regulation, innovation, culture?
Joseph Carson: I think it's kind of false because I think one of the things is that, you know, the approach is is that there is a lot of AI innovation happening in the EU. The area is is that it's happening in much more of a sandbox safer playground rather than actually being tested in real lives, everyday people, that there's no turning it back once it's out of control. So I do say that there's a lot of innovation. I think it's just more that the EU has taken a much more safer approach in how they're actually applying it and making it available. I mean, there's certain things that I would love it to, for example, accelerate. There's a lot of great things that both, for example, Apple and Meta are doing when it comes to live translations. But the regulation the EU is somewhat restricting its adoption as quick as it is in other regions. So there are certain areas where it does cause a slower adoption because of the regulation, which I think ultimately has a benefit to society. So there, you know, there are the areas where it does slow things down, but it doesn't stop them. It just means that there's more process that has to go through to make sure it is safe.
Mackenzie Jackson: A couple of points you said there that I think really resonated with me is, one, is that you can't roll it back once it's kind of out there. And this has been proven true from OpenAI. People don't talk about this a lot, but OpenAI released ChatGPT well before it was ready because there was a fear that someone else was going to release their large language model. So there was kind of Meta was playing in the field, Google was playing in the field, there were other kind of instances. So OpenAI released their model, which was, I think we'll look back and go, "Wow, that was so unsafe," because they, and you can't roll it back now. It also kind of scares me how reliant I am on AI immediately now. You know, I like when I'm writing things and I need ideas, like the first place I go to is AI. But I think it was rolled out in an unsafe way, and that was because of competition, because it was kind of like, "Well, it's not ready, but someone else is going to do it, so we have to do it."
Joseph Carson: Absolutely. I mean, there's lots of cases where it comes down to, you know, we live in a world and as I mentioned, it's not perfect. And that we use social media a lot for mental health and social interactions these days than we have in a real physical world. And there's a lot of people out there that, you know, they need help. And when they turn to a lot of these AI systems and algorithms, then unfortunately, the way the algorithm had been originally set up was to actually, let's say, if we go back to the social sphere side of things, is that it wants to enhance your existing preferences. So if you like something, it will throw everything that you like and similarities and other people like that sort of thing. It will throw as much of that in order to expand your sphere into those areas. And when you have somebody who is, you know, having challenges in life, then rather than trying to find ways in order to help them out of that, what it did was it reinforced it. So people that had high risk of suicides ended up getting a higher risk of suicide because it was reinforcing their decision-making. And that's absolutely, in those scenarios for people that were searching for, let's say, help, got help in their own way from the algorithms because basically, you know, in there some cases that people did commit suicide as a result of the algorithm telling them they should. And that's the safety issue that we have to look at is that, you know, it should be a way of making the world better and not, you know, forcing people to make really bad decisions.
Mackenzie Jackson: There's so many ways that it could go. You know, I feel like this could help. And then I listen to this podcast regularly, and there's a host on that podcast who's kind of embraces AI, but always has the theme that unless it cures cancer, I don't care, in the sense that like, you know, like, "Oh, you're telling me you got this great feature and it's coming out. Does it cure cancer? Because if it doesn't, then like, then then we need to regulate it. We need to do all these things. But if it can cure cancer, then I'm on board. But so far, we haven't cured cancer." And I'm that's a little bit hyperbolic, but I think he's using that in a sense of like, yes, but is it actually helping us or is it just making us write more tweets?
Joseph Carson: In some ways, I believe it is. You know, is AI helping us make better code? Yes, it is. It's helping write better code. The big question was is that is it helping us write more secure code? Not necessarily. On average, it's just about the same, which means that my concern more is is, yes, it is helping us be more efficient today, but is it helping us be more effective in the long term? Because what it's doing is it's creating a lot of waste and inefficiencies that we don't need. And it means that we as a human race need to become better prompt engineers to understand. And I had Carlos Polop on an episode earlier this year, about six months ago, and we were talking about, you know, that it's important to know what to ask, but it's also important to know what you want it not to do as well, the things that, you know, how do you prompt it to minimize waste and to do exactly what you need to do. So the more better we become at understanding about what questions to ask, but also what things we don't want to do, is critical in order to make sure that we're creating code and improvements that is more efficient that we can manage in six months or years time. Otherwise, we're creating code that we will never be able to manage in our lifetime, and we hope that it will manage it for us, but it just becomes very wasteful and ineffective in the long term. But I do believe that in some areas it does help. But you're absolutely right, and one thing I find is if I'm asking certain requests, what I'm finding over the long term is that the answers are getting very, very, very similar. That there's very lack of creativity, and I have to then go and give it access to more of my creative content in order to get much more creativity. So you have to keep fueling it. So, yes, AI is accelerant, and data is the fuel. And if we don't keep creating creative content to fuel it, it becomes very, very uncreative very quickly. So we, yeah, we are the creativity engine for AI, and unfortunately, that means, yeah, it's very dependent on us in order to keep that direction.
Mackenzie Jackson: Well, Joe, I had four points written down here in this podcast. We're still on point one, and we're almost out of time. So, I think I'm going to have to try and get you to come back another time. But before we go, I like to play a game with our guests. It's called "Would You Rather?" I'm going to give you two very crappy concepts, and you have to pick one.
Joseph Carson: Mhm.
Mackenzie Jackson: So, first idea here, would you rather have hardcoded secrets in your Git repositories or use Slack as a password manager? AKA, you're sending everyone's passwords via Slack. Which one would you rather?
Joseph Carson: That's a tough one, but to be honest, I would, I'm always about de-risking. So for me, Slack is probably the better alternative than keeping it hardcoded. So I just have to take the de-risk approach. Is do I want to leave it all in one location? Probably not. It just means what I'm doing is I'm separating the risk into two components rather than one.
Mackenzie Jackson: Okay, I'm going to clip that and just make it seem like you'll say that Slack is a good password manager.
Joseph Carson: Okay.
Mackenzie Jackson: Okay, we have another one here. Would you rather have only AI review all pull requests? So there are no human code reviews going on, it's just AI deciding whether or not to add it or not. Or your developers can never use AI to help them write code. So everything's written by human or everything is checked by a guessing machine. Which one would you rather?
Joseph Carson: Going back to because it is a probability machine. It is good at maths. That's one thing. It is math-based. You know, if we're talking about just math-based decisions, then I would be fine with AI writing all the code and humans not having to do it. But if it's non-math-based systems, I would rather being human-based. So it comes down to the context. But honestly, we have to do what's best for the humans, human race. And if the AI system can get to the point where it is really good, as good as humans and if not better, then let AI do it for us. Let us focus on the things that we can spend more time enjoying the world than wasting our time doing things we don't necessarily need to do.
Mackenzie Jackson: I have one last one. This one's a multi-choice option. So you have an application that's running live on the web, and you can only secure it with one security tool. Your options are SAST that'll include secret detection in there. SCA, so composition looking at dependencies, or DAST, kind of let's say pentesting your application in production when it's live. Which tool would you pick, knowing that you cannot have any others?
Joseph Carson: I mean, I would go with the kind of more the SAST-based secrets management protecting. I mean, protecting the front door, because the rest is kind of like protecting the back doors. I'd rather try to protect the front door first.
Mackenzie Jackson: I think, I think that would be my answer too. Although they're all like, not having only one is kind of crappy.
Joseph Carson: Yeah, I mean, if I was to choose the first one, then that would be the first one to put in place. Hopefully, the rest would come later.
Mackenzie Jackson: Yeah. You've changed it around a little bit, but I like, but I like that. What was the first one you would do? Yeah. All right. Look, Joe, I've had an absolute blast. It's been great chatting to you as always. I hope I can come back on the show. But we've got multiple points that I never managed to get to, but we are out of time. So thank you so much for joining us and I hope you'll join us again in the future.
Joseph Carson: No problem.
