ChatGPT doesn’t know what a fact is – interview

The hype around generative artificial intelligence (AI) tools like ChatGPT scares cybersecurity experts since they provide new opportunities for cyberattackers and the spread of misinformation.

ChatGPT doesn’t know a thing, and its answer can be totally made up, that’s why its wide adoption and integration with search tools can increase the spread of misinformation.

Cybersecurity is in peril, too, since criminals abuse generative AI tools to enhance their attack tactics.

I chatted with Daniel Spicer, a chief security officer at the software company Ivanti, about the latest AI craze.

Tell me, just how dangerous do you think ChatGPT is?

I think you're either all for ChatGPT, or you're all against ChatGPT. It's definitely one way or the other. I've always been a big fan of disruptive technology and people who aren't afraid to make a splash in the industry. And I feel like the OpenAI team has really done that here.

My biggest statement of caution right now, though, as I'm talking to peers and experimenting with it, is actually a lot about the misinformation from ChatGPT. Google has finally written something very specific about this idea of hallucination [when a bot like ChatGPT gives you a completely made-up answer.]

One of the big concerns I have is in its day-to-day use cases that we're talking about using generative AI. How do we combat the potential for misinformation, whether you're, you know, looking for a Bing [Microsoft is powering Bing with ChatGPT] or a Google Bard [that] wants to use an AI to provide you with relevant information during the search.

We need controls around that to make sure the information that comes out of it is factual.

That's the real issue here. ChatGPT doesn't know what a fact is.

Why is the risk of misinformation higher when we introduce generative AI? Even now, Google presents us with results that are provided by humans, sometimes by journalists who, with all due respect, might not always check the facts properly or even find it impossible to do.

It comes down to perception, right? When searching for something on the Internet, we understand that we are searching for information and reading different points of view. When a person interacts with chat GPT or even when you talk about LaMDA [Google's Language Model for Dialogue Applications], which is what Bard's going to be based on, sometimes it sounds like you're talking to an authoritative person. We had a person who believed LaMDA was a sentient being for a long time.

That made the news for a while. I think there is a perception shift – people have to acknowledge that the AI and the search are just as likely to be incorrect or misstate facts as any other person who's just put something up on a page on the Internet.

Do you think it's possible to introduce some kind of control here? Because to me, it seems like this problem, like many others, boils down to education, how we read news and evaluate sources. Can there even be any kind of technological solution to this?

I'm sure that there is. Ironically, one of the things that is a bit painful right now is that we are using certain types of machine learning and artificial intelligence to do fact-checking for us. Especially during the political season, we saw some of this technology doing fact-checking. You have, on the one hand, generative AI trying to find your search results. Right next to it, the technology we have to automate fact-checking is also based on machine learning.

You can end up in this weird loop where eventually, if these things are learning from each other and like an adversarial model [adversarial machine learning, a technique that attempts to fool models with deceptive data], for example, they could still end up being wrong. That's really what the challenge is. I think this is one of the things that the industry really needs to catch up on very quickly to make sure we're using the technology responsibly.

Do you think it's possible to catch up on time?

The larger companies have a moral obligation to make sure that they are paying attention to these risks and that they're calling them out. The good thing is that ethics within AI is a big part of their organizations, right? They don't just have AI and data science teams, but they also have specialists in computer and AI ethics.

Not only do they have that obligation, but they have the experts to think about these problems and try to have that technology background to work on those controls.

I love disruption, but it's not a complete technology. I think we're going to start seeing with the ChatGPT implementation in Bing and Bard for Google a more sophisticated version of that.

This is a good time for us to at least take a stab at some of these challenges. The cat's out of the bag, so to say. But that doesn't mean we can't corral it a little bit.

What risks do these generative AI products like ChatGPT pose to cybersecurity? We’ve already learned it can help write malware, work with vulnerabilities, and probably, be a real saver when it comes to social engineering, given that criminals are often not that literate.

Social engineering is really going to be a big use case. I think there's still a lot of experimentation going on with common threat actors. As soon as an enterprising individual takes some of the frameworks that we use for developing social engineering and mixes them with ChatGPT for actually developing the message, that's where it's going to pick off because then you have the automation that builds the pretty parts of the emails, right, with the logos and the formatting combined with ChatGPT just spinning out words.

Whether ChatGPT is correct in its ideas and hallucination doesn't matter in this case. As long as it causes somebody to click the link, it doesn't matter how factual it is.

But one of the things that we were actually doing and is kind of interesting – we were getting ChatGPT to help us write the red actor [a group that simulates an adversary in cybersecurity playbooks. This was an experiment that we were doing here at Ivanti a little bit to try to push on the edges in terms of cogeneration. ChatGPT is not designed for cogeneration, and a lot of the work it's doing still requires someone who's fairly knowledgeable about code to fix issues and string things together.

I think generative code will be a problem for us in the attack lifecycle, making it a lower bar to entry.

I don't think ChatGPT is there yet. I think we're going to have to see not even GitHub Copilot [cloud-based artificial intelligence tool developed by GitHub and OpenAI to assist users of Visual Studio Code, Visual Studio, Neovim, and JetBrains integrated development environments by autocompleting code], but whatever comes after Copilot and some of the other experiments I know OpenAI and others are working on. That's when I'll start worrying about malicious code generation.

Seeing it develop playbooks and be able to swap out tools and techniques for lateral movement and privilege escalation, ChatGPT is very good. If I ask it, 'what can I use instead of Cobalt Strike,' I can get an answer as an alternative framework for performing lateral movement and get it to start providing information about configuration or changes for bypassing antivirus programs.

I'm not going to share all of that information. I don't think that's a healthy thing to share right now, but that's something that we've been experimenting with, and I think that's important.

When you look at some of the ransomware groups, especially Conti, who had their playbook exposed, they all have these playbooks, right? Imagine if they had AI as their Copilot, helping them generate and change up their tactics so that they're a little bit harder to identify throughout the network as they're trying to carry out their attack.

I guess there's no way to ensure they don't use ChatGPT, right? Because we see threat actors abusing legitimate pen testing tools and software all the time.

The big challenge for defenders in the security space is identifying that lateral movement and privilege escalation phase. It's the most critical part of the attack phase. Having a solution that helps you change up those tactics regularly is just going to make attackers that much more successful.

Being able to generate playbooks and swap out specific components of playbooks is actually a very dangerous part in starting to build and realize that entire lifecycle.

Do you see how defenders could use similar technology to catch up to those cybercriminals?

I actually don't see generative AI as a very useful technology for defenders. I find that just generating information for us is not helpful in a place where the number one challenge is trying to find signals with noise from all of the different sources of data that we have.

What we really need is more analytical eyes that will actually help us in discerning patterns in our data.