The Pentagon wants a way to ensure that military AI models function as intended

The Department of Defense has begun working on understanding the systems created to provide artificial intelligence with the ability to learn and self-adapt, given the growing importance of these systems in military functions. The Pentagon is developing a comprehensive evaluation system to ensure that defense system AI models function as intended in the real world.

The initiative illustrates the growing recognition within the defense community that AI technologies, while powerful, pose a host of new, previously unanticipated, risks. AI models respond in unpredictable ways to unfamiliar data, scenarios, or information. This is a significant departure from the behavior of traditional software. In the military, where a single decision can be a matter of life and death, such unpredictability is a significant concern.

The Pentagon has begun seeking new approaches for AI model testing before and after their induction, and for these new types of testing frameworks, they describe these as “evaluation harnesses” to determine whether AI tools would function effectively and reliably as intended for the purpose and objective of the AI tool.

This type of testing system allows for realistic battlefield simulations, AI model testing in regards to real time combat stress scenarios, AI models in military scenarios where there are unpredictable and volatile, AI models where the data is incomplete where the time is limited for the AI model to provide a response, the models in the framework testing would analyze the models in provide data and responses to the models as they are incomplete challenge the model in scenarios where the situations are dynamic, and ultimately, are real time systems.

The Pentagon does not have an interest in the functionality of intelligent system models, but rather the systems that are designed to be combined with humans. Military leaders have stated that AI is not designed to replace human systems, but is rather designed to function in combination with the system to improve the combat effectiveness of a human by providing rapid data processing that is a supportive function. Instead of determining whether an AI system can accomplish tasks independently, evaluation should include whether an AI and human team performs better than AI or human performing a task individually, say defense officials.

This type of evaluation aligns with a key element of the military's evolving technology strategy. More and more, modern warfare requires the combination of human insight and machine analytical power. AI offers the ability to automate the identification of relevant patterns within large sets of data. Machine driven intelligence, and data analysis, the automation of process, and operation planning can outpace humans. However, a human is always in the decision-making role.

The complexity of models is on the rise, and AI's operational environment is characterized by the presence of new and unpredictable elements, making an accurate evaluation of AI's behavior more challenging. Military training is almost always conducted in a controlled environment, but actual military operations often present a unique set of circumstances that go beyond the defined scope of training, leading to concerns over operational unpredictability and reliability.

The Pentagon's new evaluation framework intends to lessen these concerns by instituting the testing of AI models before their operational use. This system, if implemented, would be able to detect algorithmic issues and assess models for reliability across a range of mission scenarios through simulation of complex mission profiles.

Another, and equally important, concern is whether, and how, AI models can remain stable over time. Some machine-learning models are designed to improve over time, while others are designed to deteriorate. In the worst case, a model may deviate from the original design, resulting in an unintended response or behavior. As the system evolves, defense officials would be alerted to the existence of these problems, and they would address them prior to operational deployment.

The Department of Defense's (DoD) larger effort to incorporate artificial intelligence across a widening set of capabilities is ongoing. AI is increasingly seen by military planners as a key enabler for intelligence, logistics, and battlefield awareness, as well as for cybersecurity.

Over the past decade, the Pentagon has initiated multiple AI technology integration accelerators. One prominent example is Project Maven, which utilizes defense machine learning technology to analyze surveillance video feeds. Initially, the program was designed for the automated recognition of objects and events within massive volumes of reconnaissance data.

Such programs illustrate the ways in which AI is assisting military analysts in expediting the processing of massive amounts of data. Analysts are no longer needed to scan endless numbers of images or readings from sensors. AI systems are capable of rapidly detecting and flagging critical patterns and potential threats.

The increased application of AI within the defense sector, however, raises critical conversations about potential risks, reliability, and accountability within these systems. Critics argue that unvetted AI systems run the risk of providing faulty information or misinterpreting critical data resulting in erroneous decisions being made.

Due to these concerns, defense officials are focusing on developing “trustworthy AI.” Integrating AI into mission critical systems requires ensuring algorithms operate predictably and transparently. Military planners risk employing ungovernable technologies if they lack adequate evaluation processes.

Therefore, the Pentagon's new initiative attempts to create evaluative benchmarks applicable to various AI systems. Rather than employing dissimilar evaluative methodologies for each AI system, a single evaluation methodology will articulate criteria for safety, effectiveness, and dependability.

Such a methodology would enable defense agencies to assess competing AI products offered by private firms and research institutions. With the increasing collaboration between the military and technology firms, standardized evaluation methodologies will allow for the adoption of dependable technologies.

As governments around the world try to implement AI in their defense strategies, new collaborations are forming as nations are looking to develop AI systems for military applications. Experts warn that acquiring advanced AI capabilities may result in an unstable military balance as large nations attempt to monopolize the upper hand in military intelligence, cyber warfare, and autonomous systems.

While the military applications of AI are being researched and developed, the moral and ethical implications of autonomous weapons systems are being assessed by policymakers and technologists to the answer questions related to autonomy, accountability, and the extent of a human's role in decision-making.

Military leaders recognize the ethical concerns surrounding AI, and the extensive research and testing being conducted by the Pentagon is evidence of this fact. The military is trying to ensure that operational effectiveness is increased by the AI, and that the operational effectiveness of the AI is improved without introducing an unacceptable level of risk.

Ultimately, the success of this particular effort will depend on how the Pentagon manages to strike the right balance between being apprehensive and being proactive. There is the potential for AI to transform how the military operates, but controlling the potential risks will be important to ensure that the changes are effective and safe.

If this proposed assessment system is effective, it might serve as the first step for future military AI systems. The Pentagon aims to increase confidence in one of the most critical technologies in contemporary warfare by ensuring that there will be proper algorithmic accountability before these systems are deployed in combat.