Mackenzie Jackson: Hello, you're listening to the show where we bring you the latest and greatest in cybersecurity news, research, and leaders. My name is Mackenzie Jackson and this is The Secure Disclosure. We have an incredible show lined up for you today. We're doing a deep dive into the npm breach, which just as a reminder, was the largest breach that never really happened. On Monday, September 8th, packages totaling 2.6 billion weekly downloads were compromised and injected with malware. But this didn't turn out to break the internet like we all kind of thought that it would. To be able to answer why and exactly what happened step by step, I've managed to get some great guests onto the show. We're talking to Charlie Eriksen. He's a security researcher that first discovered that these packages had been compromised and the first to analyze the malware. We're also going to be talking to Josh Junon. He's the maintainer that first got phished to actually unleash this chain of events which would cause this compromise. So, we're going to be piecing together step by step what actually happened. Then we're going to take a look at the malware itself and try and answer the question as to why this wasn't such a big deal. Was it because the security community acted quickly, or was it because we were just lucky? All this coming up in this show. Let's get into it. On Monday, September 8th, Josh Junon actually received a phishing email. This email was from a malicious domain. It was npmjs.help and not npmjs.com trying to mask itself so that at a quick glance this would pass off as legitimate. As Josh says himself, that just happened to catch him at the perfect time.
Josh Junon: Yeah, I mean it was for me, it was morning, and I had a doctor's appointment that I was getting ready for. The week had been stressful because of work. We're going live with a customer of ours right now, and so everything's in final preparations. People are running around trying to make sure everything's getting done. Yeah, it was just I just had a list of things to do and needed to get them done and was getting ready and saw this email come across my inbox because I check my emails at random times throughout the day, and that was one of them, and decided, "Okay, I'll just get some stuff done." One of them was an email coming from what I thought to be npm saying, "Hey, we're you know in my head it was, 'You're we're changing our credentials or something about our security system and we just need you to reset your 2FA credentials,'" so that in my head it was like new key scheme or something. I didn't know, I didn't really care. I just needed to get it done. It was urgent, which is of course a hallmark of these sorts of things, but not too urgent. It wasn't urgent enough for me to really think this would be uncharacteristic of npm. npm's done weird things in the past. So anyway, along with all of these different visual things, I just wasn't thinking. Nothing really stood out to me. And I clicked on the link in the email, again mistake number two, and signed in thinking, "Yeah, okay, I'll just do this real quick." Username, password, timed one-time password, and the rest was history. Okay, I got a couple emails saying my token had been reset. Immediately there's like four emails saying it had been reset, which I thought was weird. Should have triggered some more red flags in my head, but I was like, "Also, npm has been buggy in the past. So this is pretty par for the course so far." For npm, I didn't even notice the help. I probably looked at it, and I don't know, it just didn't strike me as odd. Went to my doctor's appointment, was a couple hours out of contact. Came home, and about after that, an hour after that, is when I got Charlie's message showing up for me on my phone on Blue Sky. That's basically it. It was the right time, perfect storm sort of situation for me personally where it was like I was not in a place where I should have been doing anything security related, of course, but I was also in a spot where I was just I had to get something done. I had to get something done that morning. I was feeling productive, ready to go in a rush. This is never the right time to be doing this stuff. And it just happened to come across my screen and that was that.
Mackenzie Jackson: Shortly after that phishing email, malware was injected into these packages around about 1:00 PM UTC. Around about five minutes after that, Charlie got a notification from his malware pipeline, basically a tool that inspects all packages on npm, which alerted them that a couple of packages were compromised. At first, Charlie just saw debug and immediately knew that this was probably going to be big.
Charlie Eriksen: It was around like 2:00 or 3:00 PM and I was going through our triage queue of packages where our system had detected something potentially malicious, but it wasn't quite sure enough whether or not it was something real. And first I saw the debug package and I was like, "Ooh, I know this one, right? Most of my projects have it." Then I saw chalk and arrow X. And I think I started posting in our internal Slack, like, "Oh yeah, this one is compromised. This one is compromised." And then I was like, "Okay, get everybody got going." And started figuring out like, "Okay, right," because at that point I knew that most, like, a significant portion of projects in the JavaScript ecosystem will be impacted, trying to figure out how to get hold of Josh to let him know and drum up our social media, right? Because one of the things that's really important is like, make sure that people stop doing things. It's so easy to do npm install, and suddenly because you haven't fixed the version, you're compromised. And at this point, I hadn't taken the time to reverse engineer a package. But once I kind of had sent the message out, done a couple of tweets, I started looking, and it's like, "Oh yeah, so there was those three packages and then what, like 14 more?" I was like, "This is going to be a long day," and it's been a long week now.
Mackenzie Jackson: Once more packages seemed to contain this malware, Charlie quickly made the link that actually it is Josh that was the maintainer behind this and set out to try and contact him, notify him, and let him know that he had been compromised. And at first this was pretty much a worst-case scenario. All that we knew at the first stages of the breach was that packages with a total of 2.6 billion weekly downloads were compromised. This pretty much means the entire internet, and I'm going to explain why it's so big a little bit later on. We knew from the first initial analysis that this malware was actually stealing crypto. That's kind of obvious. Most malware today in this kind is going to have some kind of crypto element. But we didn't know what else it did. We didn't know if it was going to be introducing different dependencies. We didn't know if there was perhaps hidden characters on different files. We didn't know to the full extent that it had been compromised. All that we knew was that this was going to be big. And that's when the world started to react. I myself posted a very unarming LinkedIn post, which was later on made fun of by John Hammond and some other people, which is definitely hilarious. "Oh my god, the crypto criminals are at it again. Malware in debug. Any computer that has this package installed or running should be considered fully compromised." Mackenzie Jackson, a developer over at Aikido Security, had written on LinkedIn, "Urgent. I can't write much, but the largest supply chain compromise in npm history just happened." This post went viral. It got 6,000 reactions, got shared over 2,500 times. And we all waited for the internet to collapse, for that final moment where it would be brought down, crashing to its knees, and it just never happened. There was a lot of criticism that the initial reports, and including posts from myself, were too alarmist. For example, there were posts saying that this is a complete system takeover and that if you have this infected version in it, then you need to basically wipe everything, rotate all keys, and assume that the attackers have total control over your machine, your infrastructure. But again, none of this came true. So, we have to ask the question as to why. So, before we get into the whole malware part and what the actual malware did, let's first just take a moment to understand the actual blast radius. What caused everyone to immediately jump on the fact that this is the largest npm attack in history, including myself, right? What caused that? It's hard to know exactly how much of the internet would have been affected, although Wiz has given us a pretty good indication. Wiz has said that 99% of cloud environments had exposure to one of these impacted packages. So that kind of looks like 99% of the internet. Now there are nuances in here because these packages are in different areas. But you may be wondering, how is it possible that these packages are relied on by so much percent of the internet? And for that we kind of have to look at the open-source community. Now you probably are aware of this if you listen to this podcast, but I'm going to say it anyway. Around about 80% of all the software that makes applications tick comes from open source. Now, there's varying numbers in this, but it's around about 80%. That means that on whatever platform you're watching or listening to this podcast, 80% of the code that's making it all work wasn't written by the people that are maintaining. It wasn't written by YouTube or Spotify or whoever else. It was written by open-source maintainers. Let me give you a concrete example here. Let's say that you're creating a podcast application. And in order for that to work, there needs to be something that allows you to process audio. So there's plenty of open source tools that help you in processing audio. So you pick one of these that's reputable and you use it in your project. Now the problem is is that just like your application needs open source software to work, so too does open-source software. So open-source software relies on open-source software. Now we start getting into this Russian doll situation because inside that package to process audio there may be another package that enables streaming of data or data streaming, and inside that package there's something else, and you can go all the way down and then we get down to what we call foundational projects. And a foundational project is pretty much something that's in everything. Now one of these foundational projects is debug. Debug is a JavaScript debugging utility. Most projects use it because it helps you, you'll never guess, debug problems that you're actually having by passing information about functions that are failing and so on. So, this is kind of a foundational project that is kind of in everything. Even if you're not using it or you've never explicitly chosen it, it's in your application because so many other applications and other open-source software use it. So it's way down. It's like the first point in this Russian doll situation. Well, this debug was one of the packages that were compromised along with 16 others. And there were other foundational projects too like Chalk. And when you add this all up together, you end up with 2.6 billion weekly downloads in packages that now have malware inside them. But you may now be thinking, "Okay, but this is all npm. This is all Node or JavaScript. What about if you're using Ruby or Python and you're not touching any of this JavaScript stuff?" Well, that's a good point. How do we get to this 99% number if 99% of the internet doesn't run on JavaScript? Well, that comes to where these tools are being used. In particular, debug. This may be used in your application, of course, but it also may be used in your cloud infrastructure, in your CI/CD pipelines, in multiple other areas that may not be your core application but is what your application sits on, is part of the foundation, the infrastructure that makes it run. And this is where we get to the point where the blast radius is 99% of the internet. Now again, we have to nuance this because it's really going to depend where this project is installed as to what it can be do. But we're just talking about exposure at the moment and that exposure is at 99%. Now, this kind of brings about another question there. When we think about these foundational projects, we now know that they're fundamental. 99% of cloud instances had exposure to this according to Wiz. How do these developers kind of get chosen? Is there a voting process? Are they the highest paid developers? Because I mean the internet is reliant on them. So surely there must be something there. Well, this kind of exposes a different question. I'm not going to get into it, but I asked Josh how he got into this situation. How did he end up with so much of the world relying on him?
Josh Junon: Maybe I can explain kind of a little bit more about how I got into this a little bit, like not how I got into the situation where I have 2.6 billion, whatever, a week or month or whatever it is. I personally don't even know. I mean, like I started this 10 years ago and it was just fun. Like it was just it wasn't meant to be enterprise grade, Cloudflare workers, crazy high performance, nothing like this. It was like I was doing this for fun. And then those packages started growing and growing and growing in terms of use. And then all of a sudden you wake up one day and your tiny little package that moved array values around is used by 55,000 people and has 300 million downloads a week. And you never asked for this. Like it didn't you didn't want that to be the case. This wasn't social media. It wasn't a game. It wasn't something that you were trying to grow. It wasn't a business. It was something you wrote probably half awake several years before and now half the world depends on it. There is very little you can do to prepare for that.
Mackenzie Jackson: I think now I could do a whole podcast on open-source maintainers and mental health, and I did ask Josh some mental health questions. There's a link down below for the full kind of unedited interview I did if you're interested in that. But it just kind of goes to show a pretty big flaw that we have in the system is that at some point these maintainers end up with the entire internet relying on them, even though they never wanted that, they never asked for it. They just kind of have it and they've ended up with this responsibility. There is this very classic meme by the XKCD webcomic where it shows the entire internet being held up by some thankless maintainer in Nebraska. This actually coined a phrase, the Nebraska problem, which is basically what we're describing with Josh, that at some point he ended up holding up all of the internet kind of unknowingly and also never asked for that responsibility. Philosophical questions aside, we still haven't got to the core answer of, "Okay, Mackenzie, 99% of the internet is vulnerable to this." But nothing happened. Well, kind of nothing happened. What ended up happening is that the attackers managed to get away with about $900 in cryptocurrency. There was about $400 in their main Ethereum account that ended up being stolen, and then various other small amounts across multiple other wallets and destinations. So on the same day that this happened, September 8th, multiple people started waking up looking into this and realizing that actually nothing really happened. And it took about two hours before the malicious packages were taken down. So there was only a two-hour window in which people could be affected, and it looked at first like basically security worked, that there was all this hyperbolic headlines and hyperbolic LinkedIn post by people like me, but actually nothing happened because security worked. And many people had various takes on this. Dr. Katie Paxenfair, who is a very respected and credible cybersecurity leader, she maintains the YouTube channel InsiderPhD, which is predominantly about bug bounty. And she expressed that this actually is a celebration, that instead of hyperbolic headlines about the world's biggest npm breach, we should actually be celebrating this as a win for security. And I think for sure there are some things that we can take away as a win. The community did react fast. Yes, Charlie was the first researcher that put together what was actually happening and managed to contact Josh and got it stopped. But we also know that various other people in the community also realized this. Independent of Charlie, there was also reports under the community page of the debug project on GitHub. Also, after Charlie had made his initial report, we saw a string of other articles coming out that we believe were from people that independently had found this on their own. So, you can kind of see the community working all together here and that multiple sources saw what's going on. We all kind of came together and we stopped it, or so it seems. Actually, we didn't. We just got really lucky. Some of the reports saying that we had stopped this came down to the fact that it appeared on the npm page themselves that the malicious versions had been downloaded zero times. Well, we know this isn't true because multiple security researchers definitely downloaded it. The reason this shows up as zero is that npm doesn't really update it in real time. They update it after several hours. So, the fact that we only had a two-hour window means that we didn't actually get a realistic number of how many times this project had been downloaded. However, our friends over at Wiz again did actually find out what is a pretty good number to go on. And this number was 10%. 10% of what? 10% of the internet. Wiz said that 10% of the cloud environments that they looked at contained this malware in some way, shape or form. Now again, there has to be some nuance here. That doesn't mean that 10% of all cloud accounts updated their version to this. What it means is that 10% of cloud accounts, or if you want to think about it as kind of applications, then had exposure to this malware. And the reason they had exposure to this malware was because a component within them had updated and downloaded this malware. But it still translates to 10% of cloud environments being vulnerable to this. So then the obvious question is, all right, fair enough. How on earth is it then possible that nothing happened? The attackers were very good at the initial part. The phishing email looked very professional. The man-in-the-middle attack was quite sophisticated. They managed to steal it. But then it was kind of like a dog chasing a car, except the dog actually got the car and it didn't know what to do. So, it diverted to the only thing that it did know how to do, and that was steal cryptocurrency. And they even stole cryptocurrency really bad. See, what the malware actually did is it looked for transactions or exchanges of cryptocurrency in your browser and swapped the address over. Specifically, it really only worked for browser-based crypto wallets, something like MetaMask. If you had MetaMask on your browser, which is a browser-based crypto wallet, and you were making exchanges, then it could swap the address. But you would have to go to one of these infected sites. Then you would then have to go ahead and make an exchange within the period of which the malware was there, and then it would swap out the address for the attackers' address. And there were some nifty things in there that it did like it tried to match the wallet addresses as closely as it could, which was kind of cool. But if we think about the proportion of the population that this actually affects, it's kind of minuscule. It's not only people investing in cryptocurrency, it's people that have browser-based wallets, which I don't really know why they exist in the first place. And what could the attackers have done? Well, lots of things. The initial post on the debug community channel talked about the fact that every single machine that has this package in there should be considered compromised. Credentials should be rotated. Well, this could have been true. It could have been stealing credentials, environment variables. It could have been looking for things in the CI/CD pipeline and dumping out variables, sending them back to an attacker C2 server. It could have been looking in your own environments. It could have been doing all kinds of things. It could have been trying to steal developer tokens from their own machines to perpetrate onto more npm takeovers. It could have done so many things, but instead they just chose to try and steal crypto, which ultimately is why we got away with this. What scares me the most is that had the attackers been just even a minuscule smarter, then they would have realized that actually they didn't even need to do an attack. All they needed to do is sell this access to someone else that did want to do an attack. Think of nations that would have paid an exorbitant amount of money to have access to a package that 99% of the internet has some exposure to. They could have sold this for a lot of money and giving it to a different group of attackers that maybe wasn't trying to steal money, was just trying to disrupt or cause chaos in the world. And this is where we really, really got lucky. That's not to say that the security community shouldn't have come together. That's not to say that there's nothing to celebrate here because there is. The community did come together, and I wholeheartedly agree with what a lot of Dr. Katie Paxton-Fear actually said in her post, but unfortunately we didn't get away with it quite as much as it looked like on the surface. Well, we did, but just not for the reasons that we first thought. And that segment gave me a good segue to talk about the sponsor of the show, Aikido Security. Aikido Security is your complete code security platform. Aikido brings together multiple different tools like static code analysis, dependency scanning, but also it could check things like your API endpoints and your cloud infrastructure for vulnerabilities. But Aikido have just launched a very cool open-source tool called SafeChain. SafeChain wraps around package managers like npm and blocks installations if the package contains malware inside it. This is especially important with all of the attacks going on. You never know if a package has been compromised or not. But with Aikido SafeChain, you do have added protection. You can try Aikido for free today at Aikido. There is another silver lining here, and that is the response that Josh actually got. I think you can all agree that mistakes happen. And I, for one, don't have the world leaning on my shoulder waiting for me to make a mistake so that they can attack me. Josh does. And the response has actually been very well received.
Josh Junon: I feel like you always hear this and it's kind of cliché, like, "Thanks for all the support," but it was really bizarre to be on the other end of that instead of being like a bystander. The support was so incredibly overwhelming that it made the whole thing a lot easier. People were posting like, "I would have fallen for this too." You know, like, "It happens to anyone. It can happen to anyone. It's fine." And, you know, I have mixed reactions to some of that, but for the most part, things were so positive that it truly shocked me. And this was before the wave of news outlets starting to disseminate it even further. It was just in like the core, the immediate places where I expected it to be. Obviously, Blue Sky because that's kind of where things broke a little bit, like broke the news a bit. But also like Hacker News was like the first place I expected it to show up, and of course there were tons of people on there that were super nice. And, you know, friends starting to reach out. LinkedIn went crazy, and people were very nice on there. Yeah, I mean, it was very positive and very understanding, and I felt I mean, I felt immediately that I had to kind of respond to that and be transparent. That was already the case, but it helped me be more transparent and vulnerable and talking about things, it helped me feel like that was okay. So it was really important to me that that happened. I think if it had been really, really negative, it would have made things exponentially harder to get through.
Mackenzie Jackson: But there is a question here of what do we do in the future? How can we start to solve this? So I asked Josh what were the problems that he really faced that prevented him from solving this quickly, and a lot of it came back to npm.
Josh Junon: There was a huge process breakdown during this attack with npm. They were extremely slow to respond, and I'm not trying to criticize any specific person, but when npm was still small before all the acquisitions happened, they got back to you immediately. You had a personal relationship with these people. I had to send several messages to them saying, "Are there any updates?" Because it's been an hour and I still am sitting here, you know, twiddling my thumbs. I can't do anything about this. I can't do anything productive about this other than to just talk to people and talk people down. And yeah, there's certainly a process there that could be optimized somewhere. If I had access to the accounts, I know that perhaps there's some reason why they couldn't do it, investigative reasons and whatnot, regulatory maybe authorities. I truly have no idea. But there was no transparency. I had no access to even handle this myself.
Mackenzie Jackson: Now, of course, this is very difficult for package managers like npm, who are huge, who also have the whole world relying on them, who now also have to worry about all of these security concerns and juggle how they do that while still managing to be viable. But there is definitely room for improvement here from the package managers, from the community trying to support these maintainers. I'm not sure at this point is this is still a podcast or just me ranting to myself in an empty building with a camera in front of me. Either way, I'm going to stop talking soon. But this is a wake-up call. It points out that the security community is strong and supportive when it needs to be. That is a win. But it does also point out that there isn't enough support for these maintainers and that we're still incredibly vulnerable to a massive internet destroying breach that is just bound to happen. Well, that's all I had for you today. What a cheery fun episode. I'm sure we're all feeling good now. But please make sure you subscribe to this podcast and like this video or give us a five-star rating if you're listening to it on another platform. We bring out a new show every single Thursday. So, make sure you stay tuned for that. And I'll see you on the next one.
