Mackenzie Jackson: All right, everyone. Welcome to another episode of The Secure Disclosure. I'm thrilled to be here with Chris. Chris, we've been talking for a while, we've finally got you on the show. I'm so stoked to be chatting with you today.
Chris Hughes: Yeah, it's a long time coming. We've been going back and forth, it's been months actually, via messaging. So, I'm excited to be here. I love the guests you've had on previously, and how the shows go, the content, everything. So, I'm excited.
Mackenzie Jackson: Thanks so much. Chuffed to hear it. Big praise coming from you, a fellow content creator, and definitely someone that's a leader in the space. I was describing you to someone the other day, and I said, "Chris has a weird ability to kind of appear at, see around corners when it comes to security." So, that's what I was hoping to talk about today, and get your opinion on what's coming around the corner, because it's terrifying, I'm not going to lie, there's a lot going on at the moment.
Chris Hughes: There is, for sure. And I appreciate that. Honestly, I'm just always reading and watching, and kind of watching all the dialogue from folks, and trying to discern the best I can, like everyone else. But, it's definitely a scary time, and I'm sure we'll get into it, and it's going to be worse before it gets better, I think. But, it's also an exciting time. There's a chance to kind of redo and re-architect how we've done things for a long time, and those things haven't been done well and didn't work well. So, it's exciting at the same time.
Mackenzie Jackson: Yeah, I totally feel that. I think if I knew what the outcome was going to be, if I knew where we were going to be in 10 years, this would be nothing but exciting. It's that unknown component that makes it a little bit terrifying, like where to go. But, I want to jump into something that you said immediately. You were talking about some areas kind of breaking. One area that I think is immensely under strain right now, and you've written some articles around this, is the open-source supply chain. There's been some updates with the National Vulnerability Database, the NVD, and how they are approaching things, but I wanted to first maybe get your broader perspective of, in this open-source supply chain world or the supply chain world, we're seeing a lot of turbulence, a lot of attacks happening. Why is this becoming such a problematic area right now?
Chris Hughes: Yeah, I think this is something that's been cooking for years at this point. I wrote a book called Software Transparency in 2021 or 2022. I can't remember exactly, it's been a while, but it feels like a while, basically. My friend Tony Turner and I wrote Software Transparency at the time, and it was around Log4j, Log4Shell, SolarWinds, which is not open source, it's proprietary, but it was about the software supply chain ecosystem, and we focused heavily on open source in that book. The fact that open-source software is powering most modern code bases, everything from consumer goods to critical infrastructure to national security systems. Everyone is using it, it's overwhelmingly making up most modern code bases. But the attack surface is pervasive and porous, it's often run by a single or small handful of maintainers who are often uncompensated volunteers. They're just doing this in their own free time out of passion or joy, or what they get out of doing it, and it's depended on by everyone and everything. Attackers then and now, and increasingly now, right? In March alone, there were like six high-profile open-source attacks. They've realized that this is a concentrated attack surface. I can target one organization and maybe I get successful and I compromise them, or I can compromise a widely used open-source software component that's used by tens of thousands, millions of people around the world, and my ROI is significantly higher. There's the ROI aspect, there's also the fact that these projects aren't a commercial organization with a dedicated security team often, and people kind of locking things down. They've also come at it from a lot of different angles, whether it's a technical angle of compromising developer endpoints or the package ecosystem or the hosting environment. But there's also the social engineering dynamic, right? Like XZ Utils, kind of befriend and be a trusted entity who's collaborating and contributing and then try to compromise the library and component. So, it's really fascinating and it's really scary because it touches everything and everyone almost. They've realized the ROI is incredibly high, so they keep targeting open source. And then, there's obviously the AI angle of that vulnerability slop and all the things that we can get into, too. But yes, it's a truly crazy time.
Mackenzie Jackson: Yeah, for sure. The supply chain, the broad aspect of it and return on investment, I totally agree here. This isn't a new angle, but it feels so much worse now. And I feel like one of the reasons why it feels worse is because the tools that we had to protect us don't really work anymore. Would you agree with that? AI has accelerated the process to the point where anyone can deliver malware, that we're moving so quickly that the software composition analysis tools that would detect the vulnerabilities and malware just don't serve us anymore.
Chris Hughes: Yeah, that's absolutely spot on. I referenced March, there was a series of incidents, there were XZ Utils, LM and several others. And most of the modern scanning tools just simply didn't catch them or couldn't catch them in malware scanning tools. There's a lot of players in this space, in SAST and SCA and things like that. And some of them have modernized, reachability analysis and call graphs and all these kind of things, and some of them have not. And there's not perfect solutions for some of these things in terms of known signatures and things like that. So, it's just an incredibly complicated space to try to secure. And to your point, the tools that we've used historically just haven't kept pace with the velocity and volume of code production, the kind of evolution of the attack path and methodologies that attackers use. It's a challenge. It's a big challenge.
Mackenzie Jackson: I want to ask you a really nasty question now.
Chris Hughes: [laughter]
Mackenzie Jackson: How do we fix this? There's been some changes in the NVD, which is going almost the opposite direction because they're basically saying, "Okay, we can't deal with this anymore." But what do we actually do to start to solve this? What are the missing components that we need right now?
Chris Hughes: Yeah, I hate to give a political answer of "it depends," but there are a lot of things. There are a lot of factors at play. One is the fact that, and I'm going to go in a lot of different directions with this answer, I hope you don't mind, but there's the technical angle of the SCAs and the tools and reachability analysis and awareness among developers and contributors and maintainers and obviously endpoint protection and package ecosystems that need to be secured. There's also the incentive structures that are at play where maintainers are often unpaid volunteers. They're not being compensated, so there's a whole financial angle of that. Some of the larger ecosystem players, Google and others, have tried to address that in terms of financial contributions. I think even Anthropic had a massive announcement around financial contributions to open-source maintainers, if I'm not mistaken. So, there's that angle of it. There's the whole regulatory side of it, of commercial proprietary vendors understanding what's in their products and services. There's a gap there in terms of regulatory requirements and transparency. S-bomb was a thing for a while, then that kind of fizzled out in terms of providing transparency around what's in your products or software. There are a lot of factors, basically, and a lot of things that need to be addressed. So, it's a bit of the whole people, process, technology paradigm. Some of that's on the industry, some of that's on governments. You touched on NVD. We can dive into that, what happened and where that's headed. I think that's an interesting topic, but it's a complicated space, like I said. It's a lot of factors from awareness, financial incentives, compensation, modernized tooling, a whole slew of things that really need to be addressed. That's what makes it so challenging, and then you have to think a lot of the organizations that are trying to deal with this, their core competency isn't hardened libraries and zero CVEs and containers and GitHub actions and agentic skills and all these things that are kind of proliferating around the ecosystem. Their core competency is doing something else and providing value to their customers, and now they have to worry about all this tedium and toil that they have to address just to do what their business actually is. So, it's a lot, basically.
Mackenzie Jackson: Yeah, I know. I totally get it. There's so much to learn, too. I'm so deep in the security space that a lot of this, you're in the weeds, you understand it, but when you talk to developers, you understand, "Oh yeah, you're worrying about downtime, you're worrying about providing the customer with a non-latency experience and whatever you're trying to sell or getting the products delivered on time." You just want the stuff to work at the back end, essentially.
Chris Hughes: Yeah, I want to jump in, if you don't mind. I know you interviewed Tanya recently and her and I just had an interview on my show that I host and we got into this, the developer piece of it, what you're touching on there. On one hand, CEOs go out and virtue signal like, "Security is our number one priority," and things like that, and it feels good to say, and rah rah, but at the end of the day, it's speed to market, it's revenue, it's velocity, it's feature releases, it's competitive dynamics that are driving how developers behave. Go look at how they're incentivized internally in their organizations. Are they compensated or evaluated based on security? No, they're based on their performance and delivery and things like that. So, you get what you incentivize, and it's not security. It's simply not, even if we say it publicly in terms of companies trying to say so, that's just not how it works internally, even if they externally communicate otherwise. So, there is a balance of how to incentivize developers to take security more seriously. And that's kind of an organizational aspect that needs to be dealt with. But then there's also, how do we natively integrate with their workflows and their tooling and their IDEs and things like that to make it less of a cumbersome activity for them? I joked about this with Tanya, too, is there's this Harvard and Linux Foundation study where developers call security a "soul-withering chore." And it's like, yeah, they're not wrong. It can be incredibly frustrating and it can suck. So, how do we make it suck less? How do we make it more transparent? How do we make it more frictionless for them and help them do the right thing, in a streamlined fashion? And that's things that I think we should be thinking about from a security perspective, rather than just beating them over the head with spreadsheets of findings and, "You can't go to prod until you do XYZ." We've culturally and behaviorally done a lot of things that have created these problems, too, to be honest with you in security.
Mackenzie Jackson: Yeah, I totally agree. And from the product side of that, you really see that where you have a product that kind of interacts in the developer's life, but it's paid for by a security person. So, you kind of have to advertise the product or build a product for the security person, even though the developer's going to use it. That's a wildly different experience of what they're trying to achieve and how they want to interact with it. And I feel like that is a position that we've put ourselves in. And there's lots kind of changing in that direction. We're definitely focused on the developer experience now, but I feel like there was this big gap in the middle of these fighter jet dashboards that just had so much information because the security person wants all the information they can get. A developer just wants a solution.
Chris Hughes: Yeah, that's definitely true. And then it was ironic, too, because we had this industry trend for several years around DevSecOps and shift left and things like that. Intuitively, it makes sense and such, but we never, the behaviors of the things I talked about there, of the false positive rates, the lack of context of findings in terms of reachability, exploitability, known exploitation, all these kind of factors. All we did was bolster the silos between development and security. We didn't break them down. And we kind of made things worse. And now you see the industry kind of pivoting from shift left, we acknowledge we still need to do it and we should be doing it, but now there's kind of an emphasis around runtime and cloud application detection response or ADR and these categories, right? Because we realize we can't catch everything left and we are inserting friction and we didn't do DevSecOps or shift left well. So, yeah. We just, there's a lot of room for improvement, honestly. I think we've learned the lessons from the past and I see a lot of vendors, you all included, trying to address that gap and it's exciting. I hope we can do it.
Mackenzie Jackson: Absolutely. You mentioned something before that I want to dive into a little bit, and that's the NVD. So, for those listening that don't know what the NVD, National Vulnerability Database, is, maybe we can start with what that is and what it is, because you've recently written about, well, going back to my thing about you seeing around corners, you kind of, I think a couple of weeks ago, you wrote an article about the NVD and then, lo and behold, some truths have come out like a month or a couple of weeks later. But can we go into what is the NVD? Why is that under so much pressure right now?
Chris Hughes: Yeah, it's honestly, the NVD is something called the National Vulnerability Database run by NIST here in the United States and it's become the de facto vulnerability database that's used by many for CVEs, right? That tie back to MITRE in the CVE program and things of that nature, but everyone and every tool, for a large part, historically has used the NVD. In your terms of seeing around corners, there's kind of been signs coming of this. At least 18 months ago, I wrote a piece called "The Death Knell of the NVD." And I had an interview with some folks from the community, James Perotti, and others who you've recently had on the show talking about the NVD. They had a contractual lapse or at least a concern of a contractual lapse. They awarded it to a new defense government contractor. They never cleared the backlog, basically, of unenriched vulnerabilities at the time. And now AI, as we're seeing, AI in terms of industrializing vulnerability exploitation and discovery with Mythos and others and things like that, which I'm sure we'll probably hit on. They've basically seen a massive uptick in terms of reported CVEs, for example. So I'm trying to look right here just to get the exact numbers. They had a backlog of 29,000 unenriched CVEs with published dates before March 1st, 2026. They basically moved those into the "not scheduled" category, meaning they're just not going to do anything with them, basically. Nonetheless, NVD is this critical linchpin of a vulnerability database in the ecosystem that most tools, most scanners, most organizations have used to prioritize CVEs. They enrich it with CVSS severity scores and things like that. And basically, moving forward, they're only going to do this for a very small subset of CVEs that meet those criteria that I talked about. And as a result of that, even going back 18 months like I talked about, there's been others like the European vulnerability database and ENISA and all these other players who are now trying to diversify from the NVD and provide alternatives because they see it starting to falter, starting to collapse. It can't keep pace. The projected number of CVEs is expected to reach 60,000 or more this year, which, of course, every year is a new record of number of CVEs for a year. So they just aren't scaling linearly in terms of staff and resources and capability. And it's started to kind of fumble and falter, and they just haven't been able to keep up with it. There are others out there, like I said, GitHub and others, who provide this information in their databases. But nonetheless, the NVD, which is this linchpin of the ecosystem, is starting to collapse. And it's interesting too, because a lot of compliance frameworks, this is a part that I haven't seen many people talking about, reach back to severity scores and prioritization timelines and SLAs that are often tied to the assessments done by NVD. And the compliance frameworks haven't even caught up to the fact that this is where the industry is now headed. So it's a mess. It's a big mess and it's being driven by, as I said, issues with NIST and funding and organizational dynamics, but also the role of AI, exponential code volume, exponential vulnerability discovery and reporting, slop reporting, all these kind of factors are exacerbating it.
Mackenzie Jackson: Yeah, I mean, absolutely. And it's still such a human process in the NVD and you and I can kind of understand it in a way, whereas at some point we have all this automation finding stuff, right? But we know that a lot of this stuff doesn't actually exist. So at some point we need a human to verify this, and that's the component that's constantly breaking in bug bounty, in vulnerability disclosure, kind of everywhere, is the human element of it. And I don't know what the solution to that would be apart from trying to automate that, but that seems to kind of go against a lot of the foundations of what they kind of built for and need to provide.
Chris Hughes: Yeah, it's interesting because it's this weird circular logic where the answer to all the problems of AI is AI. It's like, what we need AI to evaluate all the reported findings, and we need an AI to enrich all the CVEs, and we need AI to prioritize all the vulnerabilities. So it's like this circular logic where the problem, the answer to the problem of AI is more AI. And then that opens up a whole can of worms of human-in-the-loop or not-in-the-loop and hallucinations and LLM as a judge and all these weird dynamics. But we know that the human simply can't keep pace, and these traditional manual processes are breaking down.
Mackenzie Jackson: There was a saying a while ago that a machine should never make a judgment because a machine can never be held responsible for that judgment. And I think we've lost that now. I think the machines are making judgments.
Chris Hughes: Yeah, I forget who exactly that quote is. It's from like the '60s or something. At the time, they were probably like, "Yeah, like a human always needs to review." But the scale, velocity, and volume of what we're talking about, a human simply, it's not possible. There are too many things. I wrote a piece on this recently calling it "The Human in the Loop Delusion or Dilemma," and it talks about how either there are too many things for a human to review, or, and this is coming out in data from Anthropic, too, people just kind of dangerously skip permissions, right? They just auto-approve because they're fatigued. They can't possibly review all the things. They're just kind of stamping it, "Yes, yes, yes, yes." Because there's just too much. It doesn't work.
Mackenzie Jackson: Yeah. No, I totally get that, which I think is a good segue into something that I feel like we have to talk about, and that is AI solving all of our problems, but also creating them. We, which is the new models that Cloud came out and made. I think the latest reporting is that 270 zero-days were found in Firefox or something along those lines. Do you think AI is going to be the ultimate solution, that we will get an AI model that will basically be able to find all vulnerabilities? And what happens at that point?
Chris Hughes: Yeah, that's, I mean, it's honestly hard to argue that that's not the case. I think it's Nicholas Carlini. He gave a talk at Unprompted, an AI security conference right in San Francisco a month or two ago, where he said like every four months the capabilities of these models in terms of vulnerability research and discovery are like doubling every four months. In Mozilla's blog that you just talked about there where they found 271 new vulnerabilities that had been through thousands of passes of fuzzing and all these things over the years, they had a quote in there about these just were things that humans were capable of finding or hadn't found historically. There's Mythos where they talked about finding thousands of zero-days across open-source software, operating systems, web browsers. Obviously, we just talked about Firefox. And then there are others like Isle who found dozens of vulnerabilities in OpenSSL. And they actually even open-source and/or smaller models, not even Mythos or even Claude, for example, to find similar vulnerabilities in open source, which of course, means you could do the same in proprietary code. So, yes, to answer your question, it certainly seems like we're heading in that direction. And that can be seen as a net positive or a net negative, right? If organizations, defenders, maintainers, enterprise security teams can find all these things and fix them, that's a net positive. But guess who also has these capabilities? The adversary. And the adversary gets a vote, too. And it's like an arms race of who's going to find and fix the fastest. And that's a complicated question. Mozilla has an incredible security team. I'm sure some of the best security talent in the world. Where most organizations, maybe they have Bob and Bob is juggling seven security functions. Is Bob going to do all the things and find and fix all the things and help the development team do that? No, right? So, it just depends on the organization, the size, the scale, the capability. And the attackers, they're indiscriminately targeting everyone and anyone. So, it's definitely a challenge. So, I think this arms race is underway, and I hate to be pessimistic, but I do think the attackers are going to have the advantage for some time here because they can just find an exploit. There's this verifier's law that Sergey Yablonsky had talked about. It's easier to find an exploit than it is to validate defenses because defenses are complicated. You've got mitigating controls, compensating factors, your architecture, you have, and then you obviously have like the organizational dynamics of competing priorities we talked about in the beginning of product backlog and how do things get prioritized and like that's a whole real thing, too. So, it's just a difficult situation for sure.
Mackenzie Jackson: Yeah. It's hard to be optimistic about it. Way back in the start, you know, you mentioned Log4j and this will be relevant, I promise, but basically, something that fascinated me is that I think CrowdStrike was the company that did it. But a company did a study two years after Log4j to discover how much vulnerable versions of Log4j are still out there. And the answer was 30%. And that is because updating legacy systems, which is a huge proportion of the internet, as much as we don't want to admit it, is really challenging because Bob just can't do it, as you said. Now, going forward, it's kind of like, okay, Microsoft can find all of the vulnerabilities in the world, right, and solve them. There's still going to be legacy systems and now we're publicizing all the vulnerabilities and we're giving attackers the AI capability to be able to create exploits for that in seconds, in minutes to be able to do that. It's hard to be optimistic about that view where I think, like, you can be 100% secure maybe in the future, but I don't think we will be.
Chris Hughes: Yeah, I agree with you there. In security, a lot of folks say there's no such thing as 100% security. We're in the business of risk management, not risk elimination. There are jokes like, "Throw the computer in a lake" or "do this or do X, Y, and Z" if you want to be secure. There's always going to be attack paths and vulnerabilities and misconfigurations and human mistakes. But can we harden the attack surface? Can we bolster our defenses? Can we impose costs on the adversary or make it more complicated? 100% yes, too. But I guess right now, the way, to what we've been talking about, those organizational dynamics, those competing priorities, those lack of incentives, those lack of regulatory forces with the availability of these capabilities for attackers, it just doesn't look promising, at least not systemically, right? For the whole ecosystem. Maybe for certain organizations, but not as a whole. I think that's kind of what it comes out. Certain organizations will come out more secure, more confident, and then there'll be a whole bunch that aren't.
Mackenzie Jackson: You've also talked a lot and written articles about AI slop reporting and when Curl famously shut down their program and other areas. I wanted to ask you your view here going forward about bug bounty, because bug bounty has been such an important part of security, and we had Casey, the founder of Bugcrowd, on the show and he said something great, which is that it gave you international scale immediately when you opened up a bug bounty program of international bug hunters with honed sets of skills. Now that system seems to be kind of under immense pressure. Does bug bounty still exist in 5-10 years, do you think?
Chris Hughes: You know, I don't want to necessarily predict because I know Casey and Daniel and these folks are way more versed in this than me, but I think it does, but it will likely look different in terms of how we govern and manage it and how we kind of ingest and analyze the reporting and things like that. Daniel's a good example where he, I've been following him because publicly he was talking about like, "This is all, I'm just getting a bunch of crap." And then that went from, "Well, I'm actually getting good things now. These are actually improving," and he came full circle and it was kind of cool to watch that happen in real time over the past year or so. So I'd say I definitely think so. It'll, and honestly, there are those quotes like, "Under enough eyeballs, all bugs are shallow," right? Well, you have a bug bounty program that's opened up globally, you have people who can participate, you have them armed with these models who can help find vulnerabilities at scale like never before. I think we still should leverage that, but there has to be a way to kind of govern and manage that on the back end when you're ingesting and trying to make sense of it all, too. Again, these projects are maintained by a single or small group of people. All those dynamics are still at play. So, there has to be an answer to that, too.
Mackenzie Jackson: Yeah, definitely. It is really interesting what you say about Daniel because it has gone from, "This is all crap." They removed the financial incentive from the bug, and then they got really great reports, and kind of the crap kind of got away. I'm really terrified that people will look at that and go, "Oh, the answer is to remove the financial incentives." Because and then security researchers kind of lose their ability to earn income from this thing. I totally get it for Daniel, but it's a concerning part of it. It's like, "Oh, I don't know if that should have worked like that."
Chris Hughes: Yeah, I agree with you there. I think that could have the wrong unintended outcome, right? Or unintended consequence. And sure, some folks may do it for notoriety or just out of altruism, but that's not going to be everyone. Money talks, as they say. There's got to be an incentive of some sort to help get people to do these things. But I know that Daniel was dealing with a lot of issues, too. So, it's another difficult challenge.
Mackenzie Jackson: Yeah, for sure. Chris, we've got to start wrapping it up now, but we've covered so much ground in this. I don't want to give people too much to ingest in one day. Maybe we'll have to do another episode in the future, but I always like to finish up on a game that we have here called "Would You Rather." And the game basically works like this: I'm going to give you two very crappy options, and you have to pick the option that you prefer. All right. So, you ready?
Chris Hughes: Yeah, let's do it. I'm just thinking in my head, I have five kids, and they jump around my living room and my furniture watching this on YouTube, and it drives me nuts. But let's do it. Let's.
Mackenzie Jackson: [laughter] All right, let's go. So, would you rather have all your secrets, so your API keys, your credentials stored in a single file in a private Git repository, a .env file in a private .git repository, but it's there, it's in your history and you can't remove it, or would you rather have no firewall on your application, so you can't filter traffic, your ports are open? What would you rather?
Chris Hughes: You weren't lying about being two crappy options. [laughter] I would guess I perhaps go with the no firewall and maybe we can just bolster defense in depth and mitigating controls and things like that, but neither of these are ideal and neither of them would work out well for me.
Mackenzie Jackson: Yeah, yeah, yeah. [laughter] I'm going to clip that. Chris says it's okay, you don't have to have a firewall.
Chris Hughes: [laughter]
Mackenzie Jackson: Okay. All right, we're going to another option here. I've had to define this because people have been getting sneaky and trying to find workarounds for this one, but would you rather completely vibe code an application from the start, so you're a brand new founder, let's say you're building out, you have to vibe code everything. You can adjust your code and prompts, but only prompts, or you have to build character by character. You have to go back two and a half years to when we used to open VS Code and write our code ourselves. What would you rather?
Chris Hughes: Yeah, vibe coding all day. You would try to obviously guide and instruct the prompts in a way and provide security context and things like that, but I don't know if the genie's ever going back in the bottle, basically. For me and my answer, or for the industry as a whole, I just don't see it.
Mackenzie Jackson: It is crazy. The year that I've been doing this podcast, that answer has moved from 100% character by character to 100% vibe code. I think that just shows the journey that we're on here as well. I have one more. This one's relevant to the conversation that we have. Probably the worst one. So, the options are, would you rather have all of your dependencies fixed at their current version. So, they're all pinned at their current versions right now. So, we can upgrade to the latest as of today, get rid of all the vulnerabilities, but then you cannot ever touch them. Or, the other option is the polar opposite of that. All your dependencies get auto-updated with the latest version immediately. So, one has risk of vulnerability, one has risk of malware.
Chris Hughes: Yeah, I think either way you are in trouble. And this is like the dilemma of we've been telling people so long like, "Patch, patch, patch," right? And now the patch is poisonous in some cases. And yeah, so either way, but I will say auto-upgrade even though you know you're going to get hit, but at least you know 90% of the time it's the right thing to do.
Mackenzie Jackson: [laughter] But yeah, it's not good either way. Yeah, for sure. This one's the crappiest one I find because it's basically got a 100% chance on both options of being very vulnerable to something nasty.
Chris Hughes: Absolutely, yeah.
Mackenzie Jackson: Well, Chris, it's been an absolute pleasure. Thanks so much for spending the time and chatting with us today. It's been a wild episode with lots of information. I hope that we can, you can come back on the show and maybe if we're both in the same area again, we can do a recording in person.
Chris Hughes: I would love it. That'd be awesome. Looking forward to it.
Mackenzie Jackson: Awesome. Thanks so much.
