Generative AI likely to disrupt security, too

With the introduction of OpenAI LLC's DALL-E and ChatGPT as well as other implementations, generative AI has recently captured the public imagination. Like most with a stake in the future of technology, cybersecurity professionals have responded with a range of views of its implications. How will those implications play out for the security technology market?

Any consideration of generative AI's role in security technology must recognize that it succeeds when it mirrors human behavior convincingly. Generative AI is interactive — it responds to questions with reasonable answers. More generally, "it responds to input with credible output," and this could eventually be extended to include output credible to machines as well as people. It is also well positioned to augment functionality such as search and deliver more consumable results. Both could be useful to adversaries — not just tomorrow, but in ways that may already be useful today.

Context

Prior AI models may not have been specifically designed for dialogue. Large language models such as ChatGPT are trained on a wide variety of content to better equip them for human-like conversational response. Right now, ChatGPT manifests its functionality as an interface with people. ChatGPT outputs a human-readable response that (at least mostly) makes sense to the reader. What cannot be overlooked is that cyberattacks often take advantage of the manipulability of human interaction. Today, generative AI can not only produce more convincing human-readable output — to help support efforts such as phishing or social engineering — it may also help adversaries gather the information needed to be even more convincing in everything from identity exploits to money laundering and fraud. Tomorrow, it should be further expected that generative AI could mimic any interactive behavior well enough to subvert interactive security controls. This means not just human interactions, but the information exchanged in machine-to-machine connectivity as well.

These examples point out areas where the ability to distinguish legitimate behavior from malicious automated activity may need to become more sophisticated. As technologies become more directly integrated with each other, the likelihood that machines may be able to manipulate a wider range of security controls could raise the bar on definitions of "zero trust" and proving the authenticity of interaction, as what security analysts view as the "attack surface" continues to grow.

From generative to interactive — and at the service of an adversary

Generative AI is already being actively explored for nefarious purposes. Efforts are underway to recognize the work of ChatGPT and mitigate malicious or unauthorized attempts to use it. One such example is GPTZero, which followed closely on the heels of ChatGPT's introduction with an approach to identifying what its website calls "AI plagiarism." This, in turn, succeeds other anti-plagiarism tools such as Turnitin and Grammarly's detection functionality. GPTZero relies on indicators that creator Edward Tian terms "perplexity" and "burstiness." Text recognizable in analysis is more likely to have been machine-generated if it resembles training data. Text not so recognized "perplexes" the analysis and is thus deemed more likely human-generated. "Burstiness" measures sentence variation. Humans are assumed to write longer or more complex sentences alongside shorter ones, whereas machines are presumed to be more uniform.

This detection capability is fertile ground; already the obvious implications of student plagiarism, for example, have caused New York City schools to attempt to ban the use of ChatGPT for homework or essays. Venture capitalist Marc Andreessen has suggested that if you cannot beat generative AI as a writer, maybe you are not a good writer. There is merit to that statement in the sense that generative AI is most convincing when it has a large corpus of prior art on which to base its predictions and so it does lack creativity, especially with novel work. But that sentiment does not help the educators who are trying to train the next generation of writers to be better than an AI's output.

Of note with GPTZero is what its development says about the speed with which new applications of AI to generative functionality are likely to appear — and the fluency with which an emerging generation of professionals is taking advantage. As ChatGPT continues to improve by expanding its prediction model, competition develops between the generative AI's ability to write convincingly as a human would and the ability of solutions to detect whether a piece of writing is AI-generated.

This speaks to what we would expect the likely pace of innovation to be in harnessing generative AI for unsavory purposes as well as combating its use in malicious activity. Consider that it is already being put to work in delivering not only search results but a discussion of findings intended to give the experience of conversation with a knowledgeable person. The corollary to this is that generative AI must also gather the relevant information needed to deliver. Finding information useful to an adversary would be made much easier by generative AI — and it could also provide insights into how to make the most of it in adversarial techniques that the reader may not have previously known about or considered.

The combination of generative AI and a corpus of attack techniques and sensitive or personally identifying data — all of which can be readily found by motivated adversaries — could be beneficial to those who specialize in fabricating "synthetic identities" out of enough compromised information to persuade a potential victim that the identity is legitimate, to open financial accounts to launder ill-gotten money or use such synthetic credentials to defraud. Such information could also be used to more sharply target phishing and social engineering attacks, providing information known only to those close to likely victims (who may also be better identifiable by generative AI), which would make such attacks more specific and convincing.

Efforts to explore how generative AI may itself be subvert, ed are also emerging. In September 2022, so-called "prompt injection" attacks began to appear. These are interactions characterized by malicious inputs into generative AI that manipulate it into producing output at variance — sometimes egregiously so — with the implementers' intent. Initial examples were more prankish than serious, inducing Twitter bots to generate amusing output. More recently, Stanford University student Kevin Liu induced Microsoft Corp.'s Bing Chat (part of Microsoft's collaboration with OpenAI) to reveal its initial prompts, the series of statements that determine how it interacts with its users.

Such an effort may bear more fruit than simply manipulating the public- or user-facing interfaces of generative AI. Should an attacker penetrate an organization using generative AI to identify fraudulent or malicious activity in its internal systems, manipulation or subversion of the capability could result in detective functionality ignoring malicious actions — or worse, coercing generative AI into perpetrating malicious activity.

Implications for behavioral analytics

These examples point to the sophistication that will have to be involved not only in detective controls but in protecting the functionality that enables them. Behavioral analytics already play a role in authentication — a pillar of identity and access management — as well as threat and fraud detection. The concept of zero-trust network access incorporates factors such as location, device and software complement, integrity and configuration, and other aspects of access to establish the legitimacy of the person (user), device or other entity seeking to use IT resources. Risk-based and step-up authentication techniques have long escalated the demonstrations required of those seeking access to provide information that should only be known to an access seeker, such as answers to individualized questions or recognition and description of images. With the rise of generative AI, developers of behavior-based access control and threat detection will have to weigh the likelihood that sophisticated AI and machine-learning capability may not only be much better at making malicious activity appear legitimate but might itself be at risk.

Applying behavioral analysis to human access is not the only potential area of application. Interactive machine-to-machine access is another, which also plays a role in API security. If interaction at any level is something that could be studied, understood and exploited by generative AI, it increases the opportunity for malicious actors to target exploitable exposures. In recent years, there has been increased emphasis on techniques to demonstrate the legitimacy of machine-to-machine access attempts, such as public key infrastructure-based implementations. Not all such interfaces are sophisticated, however. Cached passwords and shared secrets have often been called upon for such purposes — when authentication is implemented at all — and these can be exploited. Efforts to safeguard these measures through additional protections such as "vaults," where security measures are more strictly implemented, are one way to deal with the issue, but the extent to which protections for any form of access can be comprehended and manipulated by AI potentially introduces new access risks.

Generative AI knows code and malware, too

ChatGPT's ability to write code is in the early stages, but it mirrors similar attempts at predictive programming assistance: for example, GitHub Copilot. From a security perspective, a researcher has already demonstrated that with the right prompts, ChatGPT can both identify and describe how to exploit a simple buffer overflow. Another demonstration showed how to implement a code injection into a common executable. While it is likely incorporating content from a huge body of knowledge on what are now fairly well-understood application security vulnerabilities, it is also worth noting that these are still early stages for generative AI, as its implications in vulnerability assessment are considered. Researchers have already demonstrated bypassing the trivial content filters ChatGPT has attempted to implement. One research team has demonstrated basic code mutation with ChatGPT, modifying the programming but achieving the same result, demonstrating future possibilities for creating polymorphic malware to avoid signature-based security tool detection.

Other opportunities such as malware creation have suggested the ability of generative AI to construct malware variants that could be more challenging for anti-malware engines to recognize and block. A look at the intersections of these two capabilities — generative output that successfully emulates humans, and a grasp of technology that can exploit technological controls — indicates that the detection of more sophisticated automation of malicious activity seems likely to become a central focus of security technology innovation.

In other words, yes, AI-versus-AI rivalry can be expected in a future where generative AI challenges cybersecurity as it will in other arenas of conflict and competition, such as misinformation/disinformation campaigns and exploits of social networking and discourse. For cybersecurity, however, this is merely the latest manifestation of a fundamental characteristic of the field: the gamesmanship that pits intelligent adversaries against defenders to continually raise the bar on the success of each.

Given the speed with which the evolution of AI is taking shape, that future may not be very far away.