Mackenzie Jackson: Hello, you're listening to the show where we bring you the latest and greatest in cybersecurity news, research, and leaders. My name is Mackenzie Jackson and you are listening to The Secure Disclosure. We have a bit of a theme. We're going to be looking at the state of security as it relates to AI code and AI coding tools. We're starting off with Paul McCarty from Git Safety and Paul is going to break down some of the recent research that he's done on malicious packages that were imitating the Claude CLI tool and actually acting as a bit of a man-in-the-middle between developers and AI tools, stealing a lot of information in between. It's pretty fascinating. We're then sitting down with Sooraj Shah from Aikido Security to break down the latest report that they've released on the state of AI security and development. This report has lots of fascinating information when it relates to AI and code security, so you're going to want to stick around for that. Before we get started, do me a quick favor. If you're watching this on YouTube, please like this video, subscribe to the channel, and if you're listening to this anywhere else, like Spotify or Apple Podcast, please give us a five-star rating. It really helps people to discover the show. Alright, enough from me. Let's get into it. Paul McCarty is a bit of a legend when it comes to supply chain security research. Recently, Paul broke down some malware that was in npm. It was a package that was mimicking the Claude CLI tool and using some basic typosquatting techniques to try and get people to download it. The package name was also called Claude CLI, but it was from a different author, the Chat GPT Claude Club. This is a common technique used, and the only thing different if you were looking at these packages is just the number of downloads. So I asked Paul why this package in particular drew his attention.
Paul McCarty: I've got a thing that I call "checking the nets" where in the morning I log in and, you know, I see what our automatic detection stuff, just like you guys probably do at Aikido, but you know, I check what kind of cool stuff got caught in the net. And I came across this one and I took a look at it and really quickly I realized that this was very singularly focused on, you know, pretending to be Claude and then going way past the normal kind of look. I mean, this goes way past the normal kind of lookalike stuff and properly embeds itself, injects itself into the real Claude and then, you know, takes over. As far as I can tell, there's not anything really unique or special about the delivery. They're just hoping that people install this like you would install, you know, the actual Claude code, which I think is kind of the point and it's one of the things that makes this unique and something that we probably want to talk about in the sense that the actual suggested way that you install Claude code is npm install @anthropic-ai/claude-code. And so these guys said, "Well, let's put in a scope namespace out there in npm," which is not Anthropic AI, but, "you know, Chat GPT Club." But then install it and, you know, I thought just that kind of unique angle could be really successful.
Mackenzie Jackson: The interesting thing about places like npm is that there's so much malware going in there. Often you will flag something as malicious but not necessarily look into it. But this is interesting because you can actually catch attackers experimenting if you are able to dive deep into some of these lesser-known attacks, and that's exactly what Paul experienced here.
Paul McCarty: I like to look at some of the ones that don't have the big impact because there's a lot of really kind of things, you know, the cutting edge as it were of the bad guys, right? Where they're out there on the bleeding edge, kind of building unique stuff, and this is a great example of that. So, because we've got this one, you know, the next time one of these comes around, I suspect it's going to be, you know, delivered in a more, you know, distributed fashion, which is not good for us, not good for any of us.
Mackenzie Jackson: So, what did this bleeding-edge malware actually do? Well, usually when packages get turned malicious, their job is to try and do something quite simple. Usually, the main objective is to find secrets and credentials, steal them, siphon them, and then use them again in later attacks. So, I asked Paul to break us through exactly the steps that this malware took and why that's unique.
Paul McCarty: Yeah, this malware, I said this to one of my friends, this malware turned to its friend, the info stealer, and said, "Hold my beer," and just went away fast. It's, yeah. So, it does, it does steal tokens as you would expect, credentials, Anthropic credentials, but then it uses this interceptor.cjs JavaScript file to inject into the actual Claude API calls to Anthropic in a really unique way. So it hooks into fetch and some of the other JavaScript modules to man-in-the-middle. So basically it can on the fly take a command from the C2 and say, "No, no, don't go where you want it to, instead go to this." It can even do it at the model level. It can actually even send it to a different model. So, yeah, I, frankly, I've never seen anything like that before in a payload, and really ingenious, but it does, it absolutely does grab those credentials at the beginning and then does the man-in-the-middle kind of proxy stuff after that. So it's got that one-two kind of combo. And, you know, as we've seen, even info stealers that just steal credentials, I mean, they're absolutely destroying the crypto space. So, you know, the number of people that have lost, that I know personally, that have lost crypto via npm packages or something similar, it's like, you know, this is this is a real issue.
Mackenzie Jackson: What's so interesting about this malware is it's not just trying to infect a machine and steal some credentials and maybe even do some crypto mining. It's actually embedding itself into the lifecycle of how developers are using AI tooling, and it's going to stay there for a long time, stealing information, stealing IP, or even altering prompts that are being made. It's kind of terrifying. So, I asked Paul, why is this potential change in direction of malware or new attack path kind of scary? And what could the long-term implications of this be?
Paul McCarty: I think that's one of the things that's concerning about AI is that there's just, like we talk about the attack surface, but like there's just so many. It's not like a web app or an API where like the threat model is understood. In this case, the threat model is in all directions. I think this is a good example of, you know, an attack path that I hadn't necessarily thought about, this man-in-the-middle thing where they're still letting you make the calls you want to to the Anthropic API. They're just listening in, and when they want to, they can change stuff, right? I mean, just sneakily brilliant. But it just kind of goes to show to your point that, you know, we don't really understand the attack surface of AI because it's different than anything else that has come before it. And so we're going to see lots of other kind of examples of this, I'm sure, too, where we're like, "Oh, I didn't even think of that." I think it's scary because it's just an unknown space, and at the same time it has become all-pervasive so quickly, right? It's everywhere, and we don't understand the attack surface, we don't understand the threat model, and so that's why a lot of people are freaking out.
Mackenzie Jackson: While Paul was here, I wanted to get his opinion on the state of security as it comes to npm, and it's no secret that we've seen a massive rise in attackers targeting npm at the moment, and this is just another story. So I asked him what he made about this and why he thought this was happening.
Paul McCarty: There's kind of two parts to the answer. The first is that, you know, there's just systemic issues in npm. It first and foremost is the world's largest software registry by a fair bit. It is by far the most unprotected. You know, anybody can log in and create a scoped namespace in like 60 seconds and publish a package. They're not doing some npm is not doing some of the things that PyPI is, you know, to proactively kind of get their handle around it, but also too, the volume is just so much bigger in npm. So you've got this kind of systemic set of things that have always been a problem, right? npm was written a long, long time ago and not really realizing what it was going to become. JavaScript has become very, very popular, and with it, npm's popularity has gone up. But then you have this other newer thing, the second part of this, which is really that bad guys have learned that this is a great way to distribute bad stuff.
Mackenzie Jackson: Finally, a couple of weeks ago, we did actually cover npm has made some good security changes into trying to resolve some of the systemic issues that have appeared. So, I also asked Paul if he thought that these changes were overall going to be good and if they're enough to actually start making an impact on some of these issues.
Paul McCarty: No. I don't. The simple answer is no, I don't. I think those changes are good, you know. I mean, any changes in the space are are good to see, but I think they focus on the account takeover style, you know, attack, which is, you know, what NX and Shahalu and Singularity and all the big ones are, and those are the ones that obviously get a lot of press, as they should, because they have large impact. But I think, yes, it's going to help for those types of attacks, right? These kind of trust, general trusted publishing kind of. But the vast majority of the actual noise in npm is not going to be addressed in any way by these changes, because it's not coming from, you know, these long-term npm authors, like Frost and everybody else, right? Like, so I mean, I think it's good, but I think it kind of ignores the real other problems that npm, that people like me have been talking about for a while, right? Like just look at how PyPI is doing it, right? Like they're addressing some of these challenges. To me, it's a little bit too little, a little bit too late.
Mackenzie Jackson: Continuing on with our theme of AI and developers as security risk within it. This report stems from in-depth interviews with 450 CISOs or equivalent in some of the leading companies and how AI has impacted them from a development point of view and a security point of view. To take us through this report, I brought on the author Sooraj Shah to break down exactly what the key elements in this report were. The first question I had for Sooraj was why conduct such a large scale study and report? What were the objectives and what you were hoping to find because of it?
Sooraj Shah: Yeah, a lot of the kind of background of why we wanted to do it was there was a lot of talk about, you know, the impact of AI on developers, on CISOs, on everyone in the kind of security space, but we wanted to see what was actually really happening. You know, the impact in terms of were there attacks linked to AI? You know, what kinds of opinions people had in the field of dealing with those things, as well as the kind of general day-to-day background of their roles. You know, how is it impacting their roles? What is it doing in terms of workarounds that they might be trying to find if they have issues? Are they dealing with too many tools? So, so it was mainly based on AI but also a kind of clearer picture of security and development altogether.
Mackenzie Jackson: The obvious question, of course, to ask Sooraj is what were the key findings in the report that were most interesting and most surprising?
Sooraj Shah: One of the main things was, you know, the headline finding, I guess, was the fact that there is actually a link between serious attacks and AI-generated code. You know, one in five organizations stated that they had found, you know, had a serious attack because of AI-generated code. But I think if you look deeper into it and going into the future, the themes of accountability and responsibility come up. You know, one of the questions asked who would be to blame or who would be responsible if an AI code breach were to occur, and the fact that the answers were kind of so widespread in terms of whether it was the developer who generated the code or the security team, it really begs more questions of how teams are now going to set themselves up to deal with this. They need something in place, a governance framework or something in place to get clarity on, okay, if this was to occur, is it the security team's fault or is it someone else's fault?
Mackenzie Jackson: That theme of accountability is actually quite interesting because if no one feels accountable for AI generated code, then no one will feel the need to ensure that it's secure. So I dug a little bit deeper into why there is such a big difference in accountability at the moment when it comes to AI generated code.
Sooraj Shah: I think our CISO Mike put it like this. It was kind of like the internet before HTTPS. It's kind of we are running before we're walking in terms of AI usage. So we're kind of going to figure it out as we go along. And I think that's the same the same is going for AI-generated code in terms of securing it. We're we're we're going to find out and it may take a big attack of some sort which is linked to AI code that stimulates this conversation more and it actually makes organizations take action and actually, you know, identify who is accountable, what needs to be done to to kind of address this. And, and yeah, and the question is, yeah, you know, there's a blame question but maybe it's more the terminology should be responsibility because, you know, who who does the bug sit with essentially?
Mackenzie Jackson: But it's not just accountability that's a big issue, it's also visibility. Most organizations don't actually know how much of their code is being generated by AI, and if you don't have that crucial stat, then you don't really know how to improve or focus your security efforts.
Sooraj Shah: Firstly, on a similar theme to the accountability, there's the fact that it's not just who, you know, is responsible if something were to go wrong, also tracking whether something has gone wrong. There's still, you know, I think organizations are still far behind on actually tracking all of their lines of code on whether they are AI-generated or not. I think the divide is even bigger in Europe than the US. And it's clear that, you know, all of these things are linked. So, so we see, for example, code in production in the US is higher in terms of AI usage, but they're also tracking it more, and therefore they're also finding more AI vulnerabilities. So there's clearly something there that needs to be done in terms of tracking AI usage, because if you're not aware that there is AI usage, then how are you going to know that an incident occurred because of it?
Mackenzie Jackson: If you've been watching this show or watching other videos from me, you'll probably not be surprised that I'm not super optimistic about AI. I think it's really cool. I really enjoy using it. But it also freaks me out a lot. So that's why something in this report caught my eye and took me by surprise, and that is optimism. Within the security leadership community, there's a lot of optimism about AI and the direction and impact that it's going to have specifically on security.
Sooraj Shah: Yes, there are concerns around, you know, the vulnerabilities that AI-generated code can bring up, but there is also a lot of positivity around AI usage in terms of securing your organization. So whether that's through using, you know, AI penetration testing, whether that's AI that will be able to write fully secure code, or whether that's AI used to fix vulnerabilities. So organizations in general, CISOs, developers, AppSec engineers are very optimistic or are already very much using these kind of tools right now in their organizations.
Mackenzie Jackson: Something unsurprising that the report brings up, it's still very interesting to put numbers to it, is a difference between AI adoption when it comes to code between USA and Europe. Unsurprisingly, the USA is further ahead on adopting AI into their pipelines, into their processes. So, I asked Sooraj why he thought this was the case.
Sooraj Shah: I think it's kind of common in terms of technology usage, you know, with cloud and with everything else that's gone on before. US has always been ahead of the curve. Europe kind of follows. And actually, one of the other stats was around, you know, just comparing non-AI security incidents between EU and US. And the US actually had far more. But the flip side is that because they're far ahead of AI usage and they're tracking it more, the long-term impact is still kind of there to be seen. I think the differences are obviously, you know, in the EU there's a lot more regulation, there's a lot more compliance kind of frameworks that people need to follow, and therefore maybe AI rollouts will be slower. They'll be more cautious about doing so. But when they do it, they will have to make sure that they are tracking it, otherwise they won't just be behind US organizations, but they will also feel, you know, more impact from incidents.
Mackenzie Jackson: Another very interesting stat that the report brings out is a correlation between the amount of tools and vendors being used and the amount of security incidents. You'll be not surprised that there is a new groundbreaking AI coding tool coming up, it seems like every week, if not every day. But here's something surprising, the report stated that if the number of tools that were being used increases, the amount of security incidents also increased. So I asked Sooraj why he thought this might have been the case.
Sooraj Shah: Yeah, I think it's something that's in the cybersecurity space, especially, there's so many different elements to cover, and I think sometimes the decision in organizations has been to, okay, I need a different tool for this, and it'll give me more coverage because I'll have myself, you know, with a SaaS tool, and then I'll have SCA somewhere else, and I'll have DAST somewhere else. But what we're seeing from the data is actually that also means more cognitive load essentially for the developers who are dealing with all these tools. They have to learn how to use them. They have to learn how to perceive the alerts they're getting on these things, and it's just taking them more time to remediate. So overall, it's showing that, yeah, the more tools you have, it doesn't really make you safer, and actually that integrating more tools into one place is probably a better fit and has better outcomes.
Mackenzie Jackson: And I have one final question for Sooraj. The report is called "The State of AI in Security and Development." So I asked Sooraj what is the state of AI in security and development?
Sooraj Shah: The state of AI in security and development right now is very much a work in progress in terms of AI. We are, as I say, going in headfirst in terms of trying to use AI, which is great for both protecting and for, you know, and it's being used for generating code and other things. However, the impact that it's going to lead to, I think in the next few years, we will see a big AI code attack. We will see more incidents related to AI code. I think we're already seeing more incidents related to AI in general. And therefore, organizations need to do more in terms of tracking, in terms of making sure accountability is right. And I think we're still very much a work in progress of getting that right.
Mackenzie Jackson: And if you want to hear more about this report and more from Sooraj himself, you'll be able to catch him live on Thursday, the 6th of November, in the webinar. There will be links to the report and that webinar in the description. So make sure you check that out, register for that webinar and ask Sooraj all the questions that I didn't get the chance. And now it's time to quickly mention the sponsor of this show, Aikido Security. Aikido Security is your complete code to cloud security solution. Aikido secures everything from the first lines of code that you write all the way to your cloud environments. Aikido's superpower is noise reduction, getting rid of those pesky false positives. I would love to tell you about all of its amazing features, like its reachability analysis on dependencies, breaking changes, AI autofixes for your code, containers, and cloud, and even AI-powered human-written code quality checks. But I just don't have time, and I want to get back to the show. But thank you, Aikido, for sponsoring this podcast. And make sure you check them out at aikido.dev and sign up for free and check it out today. And that is our episode on AI-powered code tooling and security. I hope you enjoyed it. Make sure you like this video, subscribe to the channel, or give us a five-star rating if you're watching on Spotify, Apple Podcast, or anywhere else. And hopefully we will all see you here in the next video. Thanks for watching. My name is Mackenzie. This has been The Secure Disclosure.
