Mackenzie Jackson: Hello, you're listening to the show where we bring you the latest and greatest in tech and cybersecurity news. My name is Mackenzie Jackson, and this is The Secure Disclosure. We have a great show lined up for you today. We're going to be diving into the vibe coder's nightmare. Base44 had a massive authentication vulnerability that made all of their private applications basically public. We're going to be diving into this with Wout Debaenst, security researcher and hacker, who will be joining us on the show. We're also going to be continuing our journey through the world of open-source supply chain attacks. Last week, we were talking about npm. This week, we're talking about PyPI and a massive phishing campaign that's currently ongoing there. We're going to bring back onto the show malware researcher Charlie Ericson to join us. And of course, I get to sit down in the cyber bar with Khachatur Virabyan, who's one of the founders of Trag, which is a company that's revolutionizing the code quality space in the world of AI. It's a fascinating topic, and I can't wait to dive into it. Before we get into our first segment, do me a quick favor: please like this video, subscribe to the channel. You may have noticed that we're brand new, but we're trying to bring you high-quality content every single week, and subscribing helps more than you know. Enough about that. Let's dive into the first segment. AI coding platforms have come onto the market with an absolute tsunami and storm. Companies like Lovable, Bolt, and Base44 all offer solutions where you can add a prompt and it will generate a fully functioning application for you. These tools operate like magic and they're cool to play with. But if you've been following me for a while, you'll know that a lot of these tools create wildly vulnerable applications. Great for prototyping, getting things done, but not good to be exposed to the internet. What a lot of people were using these tools for was to build internal applications, right? Applications where you're accessing your own data. They're not meant to be public, but they can perform tasks for you. This makes them quite sensitive because they probably have access to sensitive information, internal information, or share secrets about the inner workings of your organization. That's why it's very concerning when you hear that Base44 had a key vulnerability in its application which allowed anyone to access other people's private applications. And you could do this by just using public information. So to dive a little bit deeper into this, I'm bringing on hacker security researcher and one of the co-founders of all Wout Debaenst to explain why this is such a big deal. There's been something very interesting that happened here where Base44 has had pretty much a massive vulnerability. And I guess what's unique about this is that the vulnerability is kind of passed on to all of the applications that Base44 has been creating. This was discovered by Wiz. Can you just explain from a hacker's perspective why this type of vulnerability is so significant? Oh, yes. So, basically there are a lot of people who made their applications through the Base44 platform and it was fully hosted through there and what the researchers basically found is that as long as you have the identifier of the application, that you can get access to any application that you want. So, what happened apparently is the developers of Base44, they forgot to add extra access controls to see if you send someone an invite for your application, if you also have the access rights to do those as well. So, basically as long as you had the app ID, you could just make an API call and then you could invite yourself to the application. The app IDs are not secret. You can find them through all means in the portal. So, this way it might be that this could have led to all the applications being on there being compromised and that's why this is such a huge deal, especially because these kind of platforms like Lovable, Base44, Bolt, V0.dev, there are now growing very rapidly. If something like this happens in two or three months, the impact would be three times as high because there are three times as many customers using those platforms. This feels like the next generation of supply chain attacks. You know, traditionally a supply chain attack is you compromise or you find a vulnerability inside a dependency. Take Log4j, that wasn't malicious, but there was this big vulnerability and then it turns out half the internet is vulnerable to the same thing. This feels like in five, six years, this will be the new supply chain attack, as in we find a big vulnerability that these AI models are introducing and we can affect thousands, potentially millions of targets at the same time. Yeah, exactly. But it even goes further. I think all these coding platforms, they don't take security into account in their prompting. All these, you don't even need to have a malicious package or the base being compromised. If the platform that you're using is just generating insecure code by default, then you can't be secure on your own anymore without doing a lot of extra work. If you're just vibe coding your way out, it's asking to be vulnerable in the future. Yeah, and I do love the vibe coding apps. They do feel like magic, but when you actually go through and try and look through the code, when you're not writing something yourself, and I think a lot of the people that are using these won't be familiar with programming languages, it's very difficult to even figure out what's going on, let alone trying to uncover security issues within them. What would you say would be advice to anyone that's kind of using your Built Bolt 44 applications at the moment? Is there any action that needs to be taken by people? So if you're listening to this, you're using Base44, what action do you need to take? Basically, if we look at vibe coding in general, I think the main rule of thumb is to make sure that you at least verify the code that's coming out of it. That's step one. And that you do the checks. Okay, is this code valid or not? Is it secure or not? That at least you take responsibility for that. I think that's the number one thing. But to already improve the secure results that come out of it, you can also just use a security team earlier in the platform by using it as part of your prompting. You could say, for example, in the regular situation, build me a login portal functionality, and it would create you the login portal, and it would store, for example, the credentials in plain text in a database. And this functionally works perfectly. Security-wise, not that great. But if you could get security more as part of your prompting, that's already, I think, where you could prevent most of these security issues. Just saying, generate me a secure login portal or really make it very specific and think ahead of what might go wrong. For example, make me a secure login portal that does prevent user enumeration, brute force attacks, that stores and plain text. All these kind of abuse scenarios that you take, make them as part of your requirements, as part of your specs, as part of your prompt, and I think a lot of the issues will already be prevented this way. You want the security of your application being validated, and that's where security tools can come in like code scanners or especially if you are handling sensitive data, I would definitely recommend you see how strong your security actually is by actually giving it to a hacker, do a pentest. Get someone with the necessary skills to look at your application and see what can go wrong. With Base44, for example, the vulnerability is so simple that I can't imagine that a hacker wouldn't have found that if they would have done a pentest. So that's where the pentesting should definitely come in place. It feels like with these kind of vibe coded applications, that a lot of the vulnerabilities are very simple. I know I was experimenting with Lovable and I was getting all these errors, so then eventually I was putting the errors back in the prompt. And how it solved them was to remove all security. So, in the case of Lovable, it's a row-level security. And that's how it solved it. It's like, okay, here's the issue. The security policies are getting in the way. Let's get rid of them. Problem solved. And this feels very similar in the sense that it seems to be missing a logic behind how it approaches things. I think it's just never made with security in mind. I think that the prompts if a Lovable or a Base44, V0.dev, if they could already make security as part of their prompting or have agents that are specifically built and prompted and designed to make secure login portals once it gets that task. I think a lot of these invulnerable designs would already be off the table just by at the source. But that's indeed something that they have to do it by design. And then we can already build further on top of that. But right now, just vibe coding your way through is tricky, let's say. Yeah, for sure. Well, Wout, I want to thank you so much for coming back onto the show, and I'm sure we'll, I'll be sliding into your DMs again trying to get you to to comment on the next big breach that happens, but I just want to thank you for your time and I look forward to speaking to you again. Happy to be here. If you watched last week's show, you will know about a massive phishing campaign that's going on with npm. And that is that there was a domain called NPNJS.com and not NPMJS.com. And this was being used to target popular maintainers in a phishing campaign. Well, a week later and it seems that this has given other actors, or the same actors, some inspiration, this time with PyPI. The domain PYPJ and not I has been registered, and it is sending people phishing campaigns to try and get them to log into their accounts to steal their credentials in a very similar attack. To uncover what this is and who might be behind it, I brought onto the show malware researcher Charlie Ericson. He was with us last week. I didn't mean to have him back so soon, but he is here to discuss this with us today. Charlie, thanks so much for joining us and welcome to the show. Charlie Ericson: No problem, good to be here. Mackenzie Jackson: Yeah, I didn't expect to have you back so soon, but I'm glad you're available. So, last week we talked about npm. There was a malicious domain, npmjs.com doing phishing campaigns. Something extremely similar is happening with PyPI. Can you tell us what on earth is going on? Charlie Ericson: Yes. So, we saw another phishing attack happening earlier this week, like three days ago, July 28th, where the package numwords, which has a lot of downloads, like it's a very popular package, was compromised with malware. And it turns out that it was again a phishing attack, and they were deploying this thing called a scavenger stealer. Mackenzie Jackson: What is a scavenger stealer? Charlie Ericson: A scavenger stealer is basically the software that they are deploying with the malware, which steals data and things. Mackenzie Jackson: Okay. I'm drawing so many parallels between this and what happened last week that do we? I mean, I guess there's a little bit of speculation involved at this point unless we have attribution, but it feels like it's kind of similar people definitely or if not certainly inspired by last week's attack. Charlie Ericson: Yeah. To me it very much feels like it's the same threat actor, right? We see the same playbook in terms of phishing. They know how to target the right people because obviously that data is public. pypi with a j at the end instead of the i, right? pypj. Mackenzie Jackson: A nasty one. Charlie Ericson: They're very difficult to spot these. In an email. Mackenzie Jackson: Yeah, I mean, it's very difficult, right? You really have to be super careful. Charlie Ericson: And with the actual thing that they are then deploying, right, is the same software going to domains that I believe also other people have attributed basically to the same threat actors, right? The command and control service. So, I think it's fair to say, yeah, it could be a copycat, but frankly, Occam's razor would suggest it's the same people. And I mean, when this happened with npm, the logical question is when is this going to happen to PyPI, right? Second biggest ecosystem. They would be silly not to repeat the very successful attack against PyPI. Mackenzie Jackson: Absolutely, that's definitely the playbook that I would follow. This clearly is working really well. We're really seeing the fall of massive packages. Well, maybe not fall, but you know, the compromise of massive packages. Do we expect that we're going to see, like, has this been caught early? Has there been any indication of kind of compromises of people that have been using this dependency? Do we know anything about the next steps that's going on? Charlie Ericson: I think it's also important to really reflect on, at least for me, this really feels like an inflection point to some extent, right? We've really seen a lot of big attacks in a very short succession that I think we maybe really weren't expecting. It has really shown how easy it is to compromise such big packages. Yeah, that really is a wakeup call. I think the, we have not faced the full kind of scale of attacks up until now, right? The capabilities that they can have. So, I think, yeah, we really need to look at what can we do to protect the ecosystem better, and it's across the whole ecosystem, right? This on the registries themselves, right? npm, PyPI, there are things that I think they really need to think about. How do you protect the system such that a simple phishing email does not instantaneously, without any warning, have such ramifications in the age of autonomic agents? Yeah, Mackenzie Jackson: Like this, it does seem scary. And I think the part that for me was the most scary was that it happened a week later. Like we were just talking about this, we were just saying, oh yeah, you know, all this awareness has gone out there. It's like, oh, got to be careful about phishing campaigns. And then immediately, same playbook, same things. And it and it works. And I'm not blaming the people that clicked on it. I totally get it. But yeah, you're right. This is an inflection point because clearly we can't continue on being this fragile. Charlie Ericson: There was a really interesting thing that happened specifically with numwords was that somebody noticed, messaged the maintainers on Twitter, they took down the package and an hour later another version popped up. The attacker still had access, right? And that was then when the maintainers kind of saw like, oh yeah, there was a weird authentication token on the account. So, even in the case where you think, I've remediated, you have to be really careful that you have closed off all access points. Mackenzie Jackson: Ah, yeah. Okay. Wow. That is really interesting because this is like it's a little bit more than just stealing credentials because PyPI is similar to npm where multi-factor authentication is generally installed on all these popular accounts. So, they're stealing session IDs, not just credentials and then they're able to persist access via tokens and how cheeky is that? Like you've been caught, you found a malicious package and then you you pop another one straight away in there. Charlie Ericson: Yep. That's a big old middle finger to you. Like, you thought you got rid of us. Like, you thought everything was hunky dory. No. Mackenzie Jackson: Wow. Wow. Well, Charlie, thanks so much for sharing your knowledge and discussing this. Hopefully, I mean, maybe Maven's going to pop next week and we'll have you back on the show. Charlie Ericson: Don't even jinx it. Come on. Mackenzie Jackson: Touch wood. Well, but we appreciate your time and yeah, hopefully, hopefully, we won't have you back on the show quite so soon because that definitely means something bad is going on. Charlie Ericson: Yes. Hope to see you in a long time from now. Mackenzie Jackson: And that segment gave me a good segue to talk about the sponsor of this show, Akido Security. Akido Security is your complete code security platform. Akido brings together multiple different tools like static code analysis, dependency scanning, but also it could check things like your API endpoints and your cloud infrastructure for vulnerabilities. But Akido have just launched a very cool open-source tool called Safechain. Safechain wraps around package managers like npm and blocks installations if the package contains malware inside it. This is especially important with all of the attacks going on. You never know if a package has been compromised or not, but with Akido Safechain, you do have added protection. You can try Akido for free today at aikido.dev. Now it's time for my favorite part of the show where we head down to the Cyber Sake Bar and chat about security with a special guest. Today we're bringing onto the show Khachatur Virabyan, who is one of the co-founders at Trag. Trag is a company that's looking to revitalize the code quality space in the AI era. We get into some controversial topics like will code quality even matter in a few years with AI? We called this segment Unfiltered Sake. Hello everyone. Welcome back to another episode of Sake and Cyber. I'm pretty thrilled to be here with a special guest all the way from Armenia, and that's Khachatur or K. But yeah, welcome. Thanks so much for joining me. Khachatur Virabyan: Yeah, thanks for Mackenzie Jackson: So, we're going to get into a little bit about code quality. You're one of the co-founders of a really interesting company called Trag, which is using AI to assist in code quality. Before we get into that, Khachatur Virabyan: Yeah, Mackenzie Jackson: We have some sake here to try. So, we're going to go half and half this day. It is a Wednesday, so we're going to go half non-alcoholic. Khachatur Virabyan: Okay. Half alcoholic. Okay. Mackenzie Jackson: Just so we're not floating by the end of it. So, we'll start with the non-alcoholic sake. Khachatur Virabyan: Let's do that. Mackenzie Jackson: So this one here is actually brewed in Tokyo. Okay. So there is sake brewing in Tokyo, but it's not one of the more famous regions. Okay. So around Kyoto, the Nada region is a little bit more famous, but this one is still really nice. Khachatur Virabyan: Nice. Mackenzie Jackson: And this one here isn't traditional sake. Khachatur Virabyan: Okay. Mackenzie Jackson: It's got some twist in it. So we'll see what you think. Cheers. Did you just shot that? Let's go. Let's bring the, I have to shot it now, too. Khachatur Virabyan: Bring your bottle. This is this is good. Mackenzie Jackson: It's good. All right. Khachatur Virabyan: Yeah. Yeah. It's good. It's good. Mackenzie Jackson: So, this one, yeah, I didn't know non-alcoholic sake existed before I started doing this. But I'm surprised at how good it is. So, that one's kind of like a little bit gingery. Khachatur Virabyan: Yeah. Yeah. Yeah. Exactly. Exactly. Japanese cuisine and the drinks, they're always good. Mackenzie Jackson: Yeah. Yeah. Yeah. They're always pretty good. All right. So, let's focus back to the back to the topic. So, as I said, you're co-founder of Trag. Do you want to just maybe give a little bit about why you started Trag and what the what the company does? Khachatur Virabyan: Oh yeah. Initially, the ideas of Trag came from our own frustration, right? We were engineers and we were spending a lot of time in reviewing other people's code and because we were also treated code reviews as a very sacred place. Why? Because a lot of tribal knowledge gets automated there, like not automated, like spun up there. A lot of discussions are going there etc., etc., right? But it's it was it is time-consuming and it is time-consuming from a very high-paid jobs, right? Senior engineers, leadership etc., etc., right? So, we thought, okay, if we do have large language models, how we can try at least to automate the process? So, we've built this agent which takes natural language rules written by other engineers and it matches to the changed code, right? So it's it's in our language it's the diff so it matches to the diff and if the rules are failing, we're flagging it. If they're not failing, we're just passing it, right? So, and a senior engineer when comes into the review process, they can see that out of let's say 100 rules, only 10 failed and you know what to expect from the diff, right? So, you know what you're doing there. So, with that, we're shrinking the time from those people during the review process. Mackenzie Jackson: Yeah, that's interesting because I think one of the key things to take away from that is that you're not necessarily trying to replace the human or remove the human because I always get I always get a little bit skeptical when I hear this, right? Because if this, you know, like automated employees and not like the last thing I want is ChatGPT just going rogue and doing things, right? And I don't want ChatGPT reviewing code either. But this is interesting, it's like assisting. It's trying to help you do it. I want to hone in on one of the things you said and you talked about rule sets. Now when I think about rules and obviously I have a bit of a background in security. I really think of like fixed rule sets here, which are deterministic pass-fail. You mentioned rules, but you also mentioned natural language, human language in that. Can you dive into a little of that? Khachatur Virabyan: Sure. Sure. Sure. So, one of the ideas is that first, large language models are here for us to help in terms of helping us bridge the natural language into the linting rule generation part. At first we could have just built a natural language bridge towards from the just plain English into let's say a GSP rules, but it's not enough because those kind of tools are limited in their nature. They're doing only linting stuff and linting is by far very limited into styling and formatting issues, right? So at first, we're like, okay, if you're coming into the diff, you want to, you know, is this doing this or that? That's the mindset that you're coming into the review process. So why not connect to that actual mindset that the senior engineers are coming into the review process and just try to lower it down to the basis of just of how the how the diff was written, like how the how the actual feature was implemented etc.? So that's why we thought natural language interface would be the best one. But of course, I mean engineers are very very comfortable using any configurations, but natural language is just given to us, right? Mackenzie Jackson: Yeah. Yeah. I mean, definitely and and I feel like we're getting more and more done natural language areas. It's easier to understand. You mentioned something else in there that I want to hone in on. Now one of the great things about having a podcast is that you get to ask dumb questions that you want to know the answer to but you're too embarrassed to ask someone. So this is going to be one of those moments. So for the audience's benefit, maybe you want to go into a little bit about kind of the difference between code quality and linting because you mentioned linting in the last in the last kind of question and I'm curious to understand, is there a difference between the two? Khachatur Virabyan: Okay. So, linting is the is a part of the whole code quality system, if if methodology, if we can say so, right? I think the linting was the very first workflow automations that we as engineers had, which is doing this, let's say, I wrote a program and I'm writing a very small other program, a small program which checks some of the stuff that I wrote in my first program. Which is very deterministic. It scans the whole code statically, which is known as static code analysis. And it reads the, let's say, Python code and checks if the variable namings are correct, if the formatting is good, etc., etc. It's it's a good way for for a team of engineers to work in a in a in a same way essentially, right? But linters are very limited in their nature. They cannot go and link business logic. They can check basic security issues, but they can never go into having the source code getting the information from the infrastructure etc., etc. It's not dynamic by nature, right? So, they're limited. Mackenzie Jackson: Yeah. No. I there's so much parallels in in security that we have in this area too, where yeah, SAS tools have gone through six years of having a very bad name because they've been extremely noisy, right? And then you find out they're flagging on a function that's vulnerable, but the function's not really doing anything, all of these kind stuff. And now we're at a point where we can actually start going deeper into this using the help of AI and stuff, and I feel like this is where we're heading also in the direction of code quality, right? Khachatur Virabyan: Yeah. Yeah. Yeah. That's that's 100% the same because, um, right now the importance of code quality, it's number one priority for a lot of organizations right now because there is two options. When we embrace AI code generation, that means we do not guardrail the AI code generation. That means we create new ways to help the AI generated code reach to production. It's it's pretty much the same as the, it's it's roughly the same example as with the cars. When we had the cars, we do not, we do not change the cars to to drive on the on top of mountains etc. We build roads, right? Yeah. So, it's pretty much the same. If we do have AI code generation which helps us a lot. We do not go and change how we we guardrail or change the code generation itself. We invent new ways to help more and more AI generated code to to reach to production. Mackenzie Jackson: I want to go into what some of the code, the AI code generated things. This may be a battle a battle subject. So let's let's try, let's get into our second sake before we before we get into this. All right. So here we have a sake. This is from the Nada region. This is actually my favorite sake that I've tried. I thought I'd try and do different sakes every episode, but I've decided I just like this one here. So, this is from Nada. One of the things that Nada is famous for, the region in Japan, is it has very soft water. And you you'll notice that you'll notice in that the previous one will be is quite sharp flavor. This will be very like soft going into it. Okay. Supposedly. Khachatur Virabyan: Yeah. Yeah. Let's try it. Mackenzie Jackson: And you you can shot it if you want, but the general recommendation is sip it. Khachatur Virabyan: Yes. The smell is really good. Yeah. It's still nice. Yeah. Yes. It's good. Mackenzie Jackson: Yeah. Yeah. Yeah. It is. The sake is quite dangerous because it's not crazy alcoholic and then it catches up on you. All right. Okay. We're lubricated up now. Let's get into the controversial topic of AI code. Jen, one of the things I'm curious about is, okay, right now AI generated code, is it better in terms of a code quality perspective than humans or is it worse? Like what is what is the general consensus here? Khachatur Virabyan: That's that's that's one one one other really good question. So I would say, you can you can treat it as a smart genius level smart autistic junior engineer who does not know anything besides point in time understanding of what is going on. So it's like you imagine you bring this engineer and then you say, implement this, they will 100% just really good, they will just write that, right? I'm talking about the latest models etc., right? And I'm not talking about one-shotting a huge features, just a basic thing, right? That the model will perform really good, but the problem is the peripheral information that the models have because you can imagine that organizations hire engineers for 10 years, they leave the the knowledge from day one from the from the date they're exiting the company, right? So that's a it's it's it's a flow of information and not a point in time issue. And that's what the problem with the code generation right now is they're really good at point in time generation, right? So they do not have the context. So if we if we only speak about the point in time generation, they'll be pretty good, but not the not big features because they need other context, right? Yeah. So that's that's the main difference on how we can treat them, right? Mackenzie Jackson: I like that. I like that. It's a genius level junior engineer. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. It's a bit of a oxymoron, but I I quite like it. Okay. So, that's where we are right now. Latest models. And we talked about this on the last episode, too, is that kind of these these models actually don't continue to learn. They're trained and then their knowledge is fixed from that point on. Let's say that the next evolution comes. We're not five years, 10 years down the track. Is code quality still a thing? Are are the models so good now that it's like everything is perfect? Um, yeah. Khachatur Virabyan: That's that's that's another good question. So it's like there there is an interesting paradox that I love to think about, right? So it's like the one in one hand we have the models that are getting better and better and better. So, when do we actually need to review what what they've written, or it's the opposite if the model is that good, they it can review itself. But in parallel, we try to understand, okay, if if if it's that good, do we actually just need to have one single approve button to see the changes and then approve it and then it goes into the production. So that's an interesting paradox to have. I do not think we can answer that right now. But my take is I am very bullish on the models. I 100% to see the world where the whole code is generated by AI and we just dare to monitor it. The monitoring part actually starts to be the quality gating. Mackenzie Jackson: And I guess too, like, the way I think of it, too, is often when AI model's generating code, they're generating code sometimes with limited context. Uh huh. And I think like code quality, especially like linting can differ between them. So, we could end up in in a position where like is a patchwork of beautifully written code that is almost impossible to understand because it it doesn't follow standards. One of the interesting things I've been thinking about and I was actually involved in a lawsuit against OpenAI. I wasn't it was unintentional. They quoted me and brought me in as an expert, but I don't know why. But we're talking about polymorphic malware. Okay. And basically malware that can rewrite itself. And the the question that I kind of posed was right now these AI models are writing Python, they're writing JavaScript. Do we get to a point where they're writing binaries or a language that doesn't like, you know, because at this point the only reason why they're writing that is I guess one, they've been trained on it, but two, humans still need to look at it and understand it. What happens when we don't? Khachatur Virabyan: Yeah. Like, like, do we get to, do you do you do you see that where it's writing direct binaries or it's it's so interesting because what is the last time you were checking, let's say, you you wrote a Python code for for doing some script, what is the last time you've checked the machine code? Mackenzie Jackson: That like, I do not do that, right? When I write JavaScript for a web application, I write JavaScript for some other things, or or you write Rust or somebody writes Go, we do not look into the binary code anymore. Khachatur Virabyan: Well, yeah, anymore. Yeah. Yeah. Yeah. Yeah. I used to I used to. No, no, no. Our grandparents might, right? I don't know, right? But it's like you can imagine that then we have compilers, right? We can treat the language models as the compilers themselves, right? Why do we actually need to look into the generated Python code? When we can read intermediary representation of a natural language which is much which which makes pretty much everybody on the earth engineers. Yeah. Right. Mackenzie Jackson: No, I and I kind of completely agree with that. Right. And at this point it's like it's it's terrifying. Can you imagine just getting the binary and running it just like hoping for the best? But I I I guess at some point, you know, when we've all forgotten how to write Python. I have a I have another fighting question here. Okay. So I want to kind of get a little bit back on track with code quality, what Trag is doing. I love the concept of natively written rules for code quality and having these separate rule sets out and and I get the advantages of it, but I want to play devil's advocate and talk about the disadvantages of, are there risks in not having this deterministic approach as in in security, you know, the the risk is by not having static rules something will will be missed, right? And and the same can apply here. Khachatur Virabyan: Exactly. And there's no any there's no any AI app in the world which on top of large language models which can pretend to be 100% deterministic. No, because the LLMs themselves, they're by design, they're non-deterministic. Mackenzie Jackson: Yeah. Khachatur Virabyan: So our job is to make it deterministic to at least 99% of the way and if we're not sure with the 1%, we just ask a human to make sure that it's fine or not, right? And in in the application world, in the LLM application world, those 1% approval rates for from the humans, two, 3%, those are completely fine because you're automating a lot, like 90% of the job, right? So if if you're not sure about something, just ask, just prompt a human saying we we are thinking like this, is it correct or not, right? So, I can confidently say that we're doing determinism like in a 90, in a 98, 99% of the of of the range, right? But we we still have this small percentage where we're not sure if if the rule matches or not. End of the day, you just prompt a human. Mackenzie Jackson: I was doing some research. This is a while ago. I think like two versions of Claude ago where I asked the same prompt to generate the same code like 20 different times, and it came up with like different variations. Like 10 different variations. Yeah. Which was which was really interesting. Khachatur Virabyan: Yeah. Mackenzie Jackson: Okay. So, we're kind of finishing the the serious part. I'm going to refill our refill our sake here. Khachatur Virabyan: Yeah. Thank you. Mackenzie Jackson: So, the game is called Would You Rather? Okay. And basically, it's pretty self-explanatory. I'm going to ask you two two options. You have to you have to pick one. Khachatur Virabyan: Okay. Mackenzie Jackson: Okay. Would you rather only vibe code? So, you can't write any code yourself. You have to you can prompt or no AI and you have to use Notepad. 100% notepad. The 100% notepad. Khachatur Virabyan: This is sake speaking. No vibe coding. Mackenzie Jackson: Would you rather deploy to production at 6 p.m. on a Friday? Okay. Every Friday. Okay. Or deploy regularly but with no staging environment? Khachatur Virabyan: So you would, controversial take. I will go with not having, I'm comfortable not having staging environment. YOLO. Let's go. Yeah. Yeah. And with vibe coded publication, vibe coded push to production, no those thing that's actually how most vibe coding is done, so it's not it's not Mackenzie Jackson: We got to embrace the future, right? So, maybe Khachatur Virabyan: Yeah. Mackenzie Jackson: All right. Yeah. All your API keys are hardcoded in your source code. Okay. Or you have no firewalls on your application? Khachatur Virabyan: Let's just let's just put our bet on the on the on the humankind and think that they will not steal our API keys and just go with the first version. Mackenzie Jackson: Okay. All right. Now, well, that's all I had for you. Thanks so much for coming on the show. Enjoy our time. It's been a it's been a blast. Khachatur Virabyan: Yeah. Thank you. Thank you. Cheers. Mackenzie Jackson: Cheers. And that's it for today. We're going to be coming at you with a very special episode next week coming from Black Hat in Las Vegas. So, make sure you subscribe to the channel, and I'll see you all.
