Responsible AI Symposium – The AI Ethics Principle of Responsibility and LOAC

Editor’s note: The following post highlights a subject addressed at an expert workshop conducted by the Geneva Centre for Security Policy focusing on Responsible AI. For a general introduction to this symposium, see Tobias Vestner’s and Professor Sean Watts’s introductory post.

In 2020, the Department of Defense (DoD) adopted the Defense Innovation Board’s (DIB) proposed AI Ethics Principles“for the design, development, and deployment of AI for both combat and non-combat purposes.”[i] The first of the principles is Responsibility which is explained as:

Responsible. Human beings should exercise appropriate levels of judgment and remain responsible for the development, deployment, use, and outcomes of DoD AI systems.

This post considers how the Responsibility Principle applies to violations of the law of armed conflict (LOAC), first in general and then in the more specific context of command responsibility. These applications suggest that the Responsibility Principle may be masking a number of questions. This post highlights those questions as part of a call for a broader discussion about whether employing AI, particularly for combat purposes, alters the application of accountability norms.

On LOAC violations generally, the Responsibility Principle and explanatory text seem to reflect a change in the standards by which we assess armed conflict decision making involving AI compared to the standards for evaluating “human only” decision making. With human only decision making, we accept the potential of at least some uncertainty as to outcomes amidst the “fog of war,” but the Responsibility Principle appears to override that acceptance if AI plays a role.

On command responsibility, the standards by which a commander’s actions (or inaction) are evaluated remains the same regardless of whether AI was involved. The supporting text for the Responsibility Principle justifies this stasis through an unrealistic expectation that all commanders will be informed on and understand how AI systems work. Understanding how AI systems work becomes even more challenging considering that some amount of unpredictability may be operationally advantageous. This in turn may call into question the fairness of applying traditional notions of command responsibility.

Uncertainty & Accountability Gaps

In terms of LOAC violations generally, there seems to be a curious migration and twisting of accountability concerns from United Nations meetings on autonomous weapons to DoD’s RAI Principles.

The final report following a 2019 Meeting of the High Contracting Parties to the Convention on Certain Conventional Weapons (CCW) included the proposition that “[h]uman responsibility for decisions on the use of weapons systems must be retained since accountability cannot be transferred to machines.” This contention isn’t reasonably debatable – machines lack agency and cannot assume legal obligations. Including that language in the CCW report was an attempt to allay concerns that increasingly sophisticated weapons systems would lead to an accountability gap.

But does the fact that humans are responsible for decisions on the use of weapons systems mean that there will always be a human responsible in the event of a mishap or operational incident involving AI? If so, then we are thinking differently about accountability for humans working with AI than for humans working without this technology.

Consider the examples of collateral damage and investigations. LOAC does not provide either individual or collective remedies for lawful collateral damage during wartime. This includes when civilians are injured or killed and civilian objects damaged or destroyed. Moreover, investigations of incidents during military operations, particularly combat operations, are often not able to pinpoint responsibility.

When dealing with human decision making, LOAC allows for collateral damage. We accept or at least tolerate the lack of remedy for collateral damage as well as the reality that responsibility for actions during armed conflict can’t always be affixed.

Does our attitude change with AI decision making? Based on my having participated in several of the CCW meetings, my sense is that many States and non-governmental organizations have a different attitude, but I remain uncertain as to why.

Consider the following scenario: Two weapons systems—one manned, the other unmanned and AI enabled—carry out identical missions. Each fires or launches the same ordnance which destroys separate military objectives and causes the same amount of collateral damage to nearby civilians and civilian objects. Both systems are assigned to the same human commander. That commander conducted a proportionality analysis before ordering both strikes. Does our attitude toward that damage change depending on the modality employed?

Now assume that a mistake is made in each system’s targeting process. The human in the manned system and the visual identification component of the AI platform misidentified two separate schools as two enemy command posts and both systems employ force against the schools.

Now the civilian harm is the result of a mistake or error. How does our attitude toward mistake or error change depending on the modality? Could the same mistake be reasonable with a human and unreasonable with an AI system?

Regardless of the answers to these hypothetical questions, the issue of whether our tolerance for uncertainty and accountability gaps should vary depending on the use of AI systems seems worthy of discussion.

Command Responsibility

Shifting to command responsibility, and building on the previous hypothetical, assume that the human commander, despite knowing that both the manned and weaponized AI platform mistakenly targeted two schools, directed both on additional missions where the systems continue to target the wrong structures.

Now what? The human pilot may be investigated and potentially prosecuted, obviously the AI platform may not. When and why might we hold the human commanders criminally responsible?

The answer depends on what was reasonably foreseeable. Reasonable foreseeability underpins the doctrine of command responsibility. At the point where the commander knew or should have known that the use of either the manned or unmanned system would violate LOAC and could have, but did not, intercede, the commander becomes criminally liable.

When AI enabled systems are employed, we may need to reconsider the fairness of applying traditional command responsibility doctrine to military commanders. That’s because what is reasonably foreseeable for the commander employing AI enabled weapons systems is unclear.

In submitting the proposed principles to DoD, the DIB provided a supporting document which explained that the United States would hold commanders responsible for their voluntary, intentional and informed actions in employing AI enabled systems:

While military commanders may order operators to use a particular AI system, this commander’s choice would ultimately be the ground for voluntary use.  Likewise, her decision would be intentional – that is not accidental or arbitrary – and informed.  She would have the requisite information given the circumstances ruling at the time and exercise her judgment in accordance with the laws of war, rules of engagement, and other pertinent information.

Beyond the mechanics of informing all commanders who might employ the weapons is the more substantive problem of what informing means or looks like. Consider a weapon system capable of updating its target library after an initial weapons review has taken place. The system learns new targets through its machine vision, using synthetic aperture radar images as it loiters. The system is capable of taking in large amounts of data in real-time and recording when adversaries change tactics. But it is also susceptible to various drawbacks based on its learning architecture. It may “learn” the wrong thing over time due to ruses by the adversary, or it may misidentify a target if the adversary begins using “adversarial examples” that cause the system to misclassify what it sees.

Informing commanders of weapon system capabilities becomes even more problematic when the concept of unpredictable or emergent behavior is included.

Emergent Behavior

Some might reflexively question why the United States, or any country for that matter, would field a system that would potentially operate in an unpredictable manner, so called emergent behavior.

There is an advantage when an adversary is not sure how an enemy will conduct operations. Conversely, the more predictable a military force is in combat, the more effective an enemy is in countering.

As a result, some amount of emergent behavior (a.k.a. unpredictability) may be desirable. How much and what kind of emergent behavior depends on the context in which the system operates but also on a risk calculus the development of which the military commander employing the weapon had nothing to do with.

That’s because risk tolerances, permissive, expansive, or somewhere in between, have already been incorporated into the system before reaching the commander. These tolerance levels may or may not align with a given commander. This not only amplifies the challenge of informing commanders on AI enabled systems performance parameters but also in applying the traditional command responsibility doctrine.

That’s not to say there would be an accountability gap or that we should abandon the reasonable foreseeability standard. In terms of where responsibility might lie, the RAI Principles refer to layers of responsibility (development, design, acquisition, testing) but don’t explain the application or the relationship of those layers to command responsibility.

One option is the idea of what Geoff Corn labeled procurement or acquisition accountability. If individuals responsible for testing and evaluating an AI enabled weapon knew or should have known that potential emergent behavior could include, for example, misidentifying a school as a command post, and they could have but failed to bound the technology or otherwise mitigate against that risk, then they should be held criminally liable.

But if an AI enabled weapon’s emergent behavior was not reasonably foreseeable, then no one would be criminally liable for initial mishaps or operational incidents, which is similar to how accountability is assessed for human actions and decision making.  If we are not comfortable with that outcome, we need to start discussing why.

Conclusion

The discussion about tolerance for uncertainty, accountability gaps, and the fairness of the command responsibility doctrine when dealing with AI in combat operations is not just needed, it’s overdue.

The same year the AI Ethics Principles were released saw DoD conducting Project Convergence.  Among other purposes, Project Convergence used AI to create a “kill web” where multiple targeting functions occurred simultaneously as opposed to a “kill chain” where the functions are sequential. The result was that by incorporating four different AI algorithms, along with new networks and command and control tools, the “sensor to shooter” timeline, “the time from detecting a threat to launching a response” was reduced from 20 minutes to just 20 seconds.

We need to have the discussion about uncertainty and accountability with AI before there is an incident or untoward event – and the Army plans to employ all four AI algorithms starting in 2023.

***

Chris Jenks is a Professor of Law at the SMU Dedman School of Law in Dallas, Texas.

Photo credit: Unsplash

[i] In the interests of full disclosure, the author provided input to the Defense Innovation Board on the RAI principles and is cited in the supporting document. His input was relatively minimal and unrelated to the topics in this post.