Pentagon rolls out its own AI platform, but experts warn a single mistake could be disastrous

Nearly three million Department of War (DoW) employees, including military, and civilian personnel, will have access to a new artificial intelligence (AI) platform powered by Google Gemini announced by Secretary of War Pete Hegseth. What could go wrong?

The military generative AI platform GenAI.mil, which runs on Google’s Gemini for Government, will be available to all civilians, contractors, and military personnel, who will receive no-cost training.

The initiative cultivates an "AI-first" workforce, leveraging generative AI capabilities to create a “more efficient and battle-ready enterprise,” according to the DoW’s statement.

AI has been rapidly transforming the cybersecurity landscape, enabling malicious actors to scale up attacks by using large language models (LLMs) in active operations. Meanwhile, data uploaded on LLMs is vulnerable to exposure and leakage.

DoW states that all tools on GenAI mil are certified for Controlled Unclassified Information (CUI) and Impact Level 5 (IL5), the second-highest level of security for unclassified information, making “the platform safe for operational use.”

While the platform’s infrastructure may not be easily penetrable, cybersecurity experts tell Cybernews that some of the major risks of GenAI.mil stem from human behavior.

Hegseth himself came under intense scrutiny earlier this year for using the messaging service Signal to discuss US war plans, rather than secure communication networks.

Prompts containing classified information

Joshua Copeland, a cybersecurity expert at Tulane University, says that even if GenAI.mil is limited to unclassified work and data isn’t used to train Gemini’s public models, risks still exist, such as leakage of unclassified but still sensitive data.

Users may paste internal planning details or information relevant to operational security into prompts, resulting in a vast new corpus of sensitive text that must be logged, monitored, and protected just like any other high-value dataset.

Data from other industries suggests that employees tend to underestimate AI’s risks to data privacy. The use of tools unapproved by companies is widespread, and 75% of workers share potentially sensitive data with LLMs.

Because the platform will be accessed by a large number of staff, it may be more vulnerable to data theft, says John Coursen, chief information security officer and founder at Fortify Cyber.

He tells Cybernews, “Any compromised user workstation or common access card account now comes with a powerful AI console that might have access to internal context via integrations or provide a convenient way to summarize or transform stolen data.”

Copeland says that another risk lies in GenAI.mil's reliance on a commercial foundation model, Gemini for Government, and the surrounding cloud stack.

“If there's a flaw in the model serving infrastructure, tenancy isolation, or update pipeline, a compromise could affect an extremely broad swath of the DoW workforce at once,” he explains.

Experts who spoke to Cybernews don’t rule out the risk of prompt injection, a type of attack where malicious instructions are embedded into an AI's input field to override its original programming.

Any compromised user workstation or common access card account now comes with a powerful AI console that might have access to internal context via integrations or provide a convenient way to summarize or transform stolen data.

John Coursen

For instance, a prompt may instruct users to “ignore previous instructions” and ask them to provide information from parts of documents that were not intended to share with the platform.

“As the platform starts integrating with documents, workflows, or other data sources, adversaries can weaponize content, such as ‘helpful’ templates and shared docs, to inject malicious instructions, skew analyses, or subtly alter generated outputs,” Copeland says.

Over-trust in AI may lead to mistakes

DoW states that the GenAI.mil platform is web-grounded against Google Search to ensure outputs are reliable and “dramatically reduces the risk of AI hallucinations.”

When used by regular users, the latest Gemini model, Gemini Pro 3, has a hallucination rate of 88%, meaning that it confidently generates incorrect answers instead of admitting uncertainty nearly nine in ten times.

Moreover, there appear to be few safeguards against staff’s over-trust in the system and its operational misuse, explains Neal Bridges, chief information security officer at Query.ai.

“If a staff officer or contracting team starts treating AI-generated text as 'ready to sign' instead of a 'first draft,' bad assumptions, legal errors, or subtle security gaps could be baked into official documents very quickly,” he says.

Surveys show that people generally don’t trust AI, especially when it is used in finance and healthcare. In some cases, however, people put more trust in AI than in other people.

Is it more dangerous than existing systems?

Experts say that GenAI.mil may not necessarily be more vulnerable to cybersecurity risks than the DoW’s existing IT infrastructure.

The platform isn’t available outside of DoW networks, which should significantly reduce cybersecurity risks, according to Paul Bruce, a senior manager at InterSystems.

“Many government systems are outdated, so with GenAI.mil being a new system, one can hope that it was designed with modern security measures in place,” he says.

However, if an unauthorized party were to access the platform, they would likely have access to a significant amount of information. This is in contrast to many government systems, which keep information isolated.

Even the strongest infrastructure guardrails may not be sufficient to protect against human errors. Copeland emphasizes that GenAI.mil centralizes a lot of human behavior, data, and vendor dependence in a single platform.

He says, “That makes it a high-value target that requires continuous red-teaming, robust logging, strict access controls, and clear user training.”

Unlock more exclusive Cybernews content on YouTube.