Generative IA becomes a weapon: how Gemini drives attacks, suplantations and model extraction

The arrival of advanced generative models transformed the way we research, write and automate tasks. But as in almost every technological revolution, it has also opened up new ways for malicious actors to accelerate and perfect campaigns against companies and people. Recently, Google revealed that a group linked to North Korea was using its Gemini model for attack recognition and planning tasks, a clear example of how artificial intelligence tools can be reused for hostile purposes.

According to Google's threat intelligence team, the actor known as UNC2970 used Gemini to synthesize open source intelligence (OSINT) and build detailed profiles of high-value targets. This work included searches of cyber security and defence companies, mapping of specific technical posts and even collecting wage information, all with the aim of designing more convincing supplanting communications and detecting "soft points" for possible intrusion. The report's own description highlights how, in the hands of an operator with offensive motivation, the line between legitimate professional research and malicious recognition becomes very tenuous. See Google's report here: Google Cloud: Distillation, experimentation and the adverse use of AI.

UNC2970 is not a newcomer: it is related to clusters historically associated with campaigns such as Operation Dream Job, in which false job offers have been used to deceive aerospace, defense and energy personnel and thus deliver malware. This "fictitious recruiter" technique becomes much more effective when a generative model helps to make powerful personalized messages and social engineering techniques. Google has also documented this pattern in a broader context of threats against the defence industry: more details here.

And it is not just North Korean groups that have integrated Gemini into their workflows. Various actors linked to different countries have begun to use generative models to accelerate phases of the attack cycle: from the search for and collection of credentials to the creation of personal dossiers, the automation of vulnerability tests or the help to purify operating code. Some teams have asked the model to summarize open source documentation, generate targeted test plans or even help develop web scraping tools and SIM card management systems. The result is that tasks that previously required dedicated human equipment can now be achieved more quickly and on a scale.

The abuse of models is not limited to pre-attack intelligence. Google identified malware families that take advantage of Gemini's APIs to generate a la carte code. A particularly revealing example is the download named HONESTCUE, which sends applications to the API and receives code response in C #. That code is compiled and run directly in memory using a legitimate .NET bookstore, CSharpCodeProvider, thus avoiding disk artifacts and making traditional detection difficult. Another case detected was an automated phishing kit with false content generation to supplant a cryptomoneda exchange platform, related to financial motivation operations.

Campaigns have also emerged that exploit public sharing functions of IA services to host false instructions and attract victims to malware that steal information; security community researchers have pointed to several waves of such abuses. These incidents illustrate how the ability of a model to produce plausible texts can be instrumentalized both to build deceit and to create technical components of a chain of attack.

Another relevant risk identified by Google is that of model extraction. In this type of attack, a massive opponent consults an API of an owner model, records the responses and trains a replacement system that largely replicates the behavior of the original. Google blocked attempts that consisted of more than 100,000 consultations designed to replicate the model's capacity in multiple tasks and in languages other than English. Independent research has shown that even with a relatively small number of consultations, it is possible to train replicas with a surprising fidelity: a public experiment achieved a replica that reached 80.1% precision after sending 1,000 questions and training for 20 times. As the researcher Farida Shafik warns, "behavior is the model: each pair consulted - response is an example of training for a replica." For more technical context on this extraction attack, see the analysis of Praetorian: Praetorian - Stealing AI models through the API, and an explanation of training concepts like times here: Machine Learning Master- Battle vs Epoch.

What can organizations do in this context? First, recognizing that traditional defences alone are not enough: protecting the model's weight confidentiality does not prevent its behavior from being replicated if the answers are exposed through an API. At the same time, there are practical measures that reduce the attack surface: segregate and protect API keys, impose rate limits and traffic anomalies, apply access controls and strict authentication, monitor atypical consultation patterns and use behavior-based detection for extraction or abuse signals. It is also essential to improve the training of staff in social engineering and recruitment processes, because the targeted and convincing deception remains a regular gateway.

In addition to specific technical mitigation, the situation requires collaboration between model providers, user companies and the security community to share abuse indicators, improve transparency on incidents and advance countermeasures at the model architecture level - such as techniques to mask or limit sensitive information in responses and mechanisms to detect real-time extraction attempts. Google and other suppliers are already publishing research and guides on these challenges; public exchange of findings is key to reducing impact.

In short, the ability of generative models to be productivity tools is undeniable, but their widespread adoption also amplifies new and known risks. Innovation and security must go hand in hand:: without defence practices adapted to this new context, the same capacities that accelerate legitimate work can make organizations much easier to exploit.

Recommended sources and readings: Google's technical analysis of adverse uses of Gemini and mitigation recommendations ( Google Cloud), the practical study on the extraction of Praetorian models ( Praetorian), Microsoft documentation on dynamic compilation in .NET ( Microsoft Docs) and Huntress's security blog for examples of campaigns based on IA's public instructions ( Huntress).