Hollow Control: Cyber Compromise and Human Judgment in Lethal Autonomous Weapons – Part II
The danger that cyber compromises pose to lethal autonomous weapons (LAWS) is that operators can make the right decision based on false information. Humans may remain in command while the information that makes that command meaningful has been corrupted. Our first post argued that human control of LAWS is only as good as the picture and the abort path upon which it depends. This post accepts this difficulty as a given and narrows the question. The law of targeting does not change based upon the label affixed to the role of humans in the process. Policy does.
Soon after it first appeared in 2013, meaningful human control (MHC) quickly established itself as the language of the international debate, but is not yet a codified concept of the law of armed conflict. Meanwhile, the United States has been employing an alternative formulation: appropriate human judgment (AHJ) over the use of force.
Although both are highly useful concepts, by itself neither resolves how responsibility should be allocated when an attack was unlawful because the factual picture was false. In other words, neither is better than the integrity they presume.
What Hollow Control Does to Legal Responsibility
As with most things that acquire a technological gloss in warfare, the law is stubbornly unchanged by the threat posed by cyber operations even though they began attracting legal attention in the late 1990s. The law still asks whether the person who planned or decided on the attack directed fire at a military objective and took feasible precautions. If the information that informs that decision was manipulated, two things are compromised at once: the quality of the decision; and the record that will later show why the attack was thought lawful. Cyber manipulation can thus do double damage as it can corrupt the decision before the weapon is released and the evidentiary record after the fact.
To be clear, the problem does not revolve around the somewhat abstract question of whether a commander can have a perfectly accurate picture. Wartime information has always been uncertain. For as long as commanders have received reports, they have had to test them against one another— in Vegetius’s words, “to come at the truth with certainty.” The issue, then, is not whether a commander can have a perfectly accurate picture but whether a commander can reasonably assess the integrity of the information on which the legal judgment depends. Wartime uncertainty is unavoidable, but uncertainty about whether the information itself can be trusted is a different problem. This is markedly different from the all too familiar issue of erroneous intelligence or outright incompetence. The difference is that an adversary can manipulate the information environment and leave the system looking as if it remains under authorized human control while corrupting the information on which that control depends.
Should this happen, the chain of legal responsibility for an unlawful attack is a relatively long one, as it can be distributed and, potentially, diluted among multiple entities including the operator, the commander, the developer, and the acquiring State. In such a scenario, the operator almost certainly believed the information on their screens to be correct. The commander may have followed the established review process. The designer may point to tests that demonstrated the system’s envelope. The acquiring State may also assert that its legal review did not contemplate this particular compromise.
The good news is that each of these statements can be true. The bad news is that the relevant questions are whether the compromise was foreseeable, whether reasonable safeguards should have detected it, and where the causal failure actually occurred. Yet none of these statements answer where legal responsibility lies for a strike that was unlawful because the picture was false.
In an environment in which jamming, spoofing, and interference with command links are routine aspects of war, the “this compromise was not contemplated” is an incomparably weaker argument than it was a decade or so ago. The most pressing question is not whether every particular compromise was foreseeable but whether the known possibility of manipulated information was adequately accounted for in the precautions taken. Possible precautionary measures are evaluated in the light of the circumstances of the attack, including the reliability of the information on which the attack is based.
The point does not depend on proving that an adversary has already constructed a perfectly plausible yet false picture. It requires an account of whether the State employing the system treated a contested picture as a circumstance of use: whether detection was designed in, whether an abort could be verified, and what the commander was to do if neither could be verified. If such an inquiry is not made, responsibility is not dissolved by the machine. Instead, it squarely falls on those who fielded the system on assumptions the operational environment had already rendered unsafe.
U.S. Department of Defense Directive 3000.09 already treats loss of control to unauthorized parties as a design risk. It does not, however, address the more consequential case in which control still appears to rest with the authorized operator while the information on which that control depends has been compromised.
Although the United States did not adopt the MHC lingua franca and answers instead in the markedly different language that is AHJ, the underlying problem remains the same: human judgment cannot be meaningful if the information on which that judgment depends has been compromised. AHJ is not a requirement that an operator sit on every trigger, rather, the directive requires systems to be designed so commanders and operators can exercise appropriate levels of human judgment over the use of force.
To be sure, “appropriate” is heavily context dependent. It may sit in design. It may sit in testing. It may sit in doctrine. It may sit in mission authorization and only then in engagement. A dozen lost airframes do not, per se, reopen a system’s legal review. A defect in the conditions against which that authorization was calibrated can. If these conditions assumed a picture and a constraint path that still belongs to the commander, a silent compromise of either means the system is no longer being used with the level of judgment the authorization contemplated.
Distributed judgment does not shrink the problem, it just moves it upstream. The more the relevant exercise involves reviewing a class of systems rather than piloting a single airframe, the more that review must specify what compromise of the picture would look like, whether it can be detected, and what remains of human judgment if it cannot.
A system can ship with an array of geofencing, abort logic, and a human on the loop and still splendidly fail to deliver accountability if those steps are based on data or channels the adversary has already bent. This is where compliance by design may enter the picture: not as another review step among many, but as a way of embedding these considerations into the system from the outset.
Putting Cyber Resilience Inside Compliance by Design
Compliance by design refers to embedding law of armed conflict requirements into engineering, procurement, and management processes throughout the life of a system. Cybersecurity is often treated as a parallel assurance problem: certified to its own standards and kept apart from the legal analysis of targeting and proportionality. Clear and obvious as it might seem, this split does not hold. If the operator’s judgment depends on the integrity of data, models, and command links, those must be in the same file that asks whether the attack can be kept lawful.
Threat models should ask what an adversary would do not only to the airframe but also with the commander’s picture and the abort path. Testing should include poisoned data, spoofed sensors, and broken or impersonated command links, and should measure whether the operator can still detect the degradation. A legal review must specify the residual risk: what compromise would look like, how it would be detected, and what the commander is expected to do if detection fails. After fielding, updates, monitoring, and incident response have to keep those conditions in place. Ultimately, a review that dies with the first model update may serve well in reassuring stakeholders, while in reality it will be little more than a glorified rubberstamp.
National review processes should treat the integrity of the picture and the abort path as part of the human control assessment. Assessments should not only appear in cybersecurity annexes but also in test and evaluation documentation and, where applicable, in Article 36-equivalent legal review records. States that make claims of human responsibility for attacks using autonomous weapons must accept that the integrity of information is part of the control that they claim to retain.
Conclusion
This may come as no surprise to Articles of War readers, but cyber compromise is a legal issue: it sabotages both the information and the abort path upon which human judgment is allowed to rely. That an operator reasonably trusted the display does not resolve the underlying question. If the track was not sound, or if an abort never reached the weapon, the human approval step does not by itself show that the responsibilities the law assigns were discharged. The click may of course remain. But the following judgment will be hollow.
If the object presented to the decision maker is not the object on the ground, if the information used for proportionality is not the information the commander believes he is using, or if the constraint he thinks he has imposed never takes effect, the duties of distinction, proportionality, and precautions have not been performed.
It is important to highlight that this will be the case regardless of whether every single policy checkbox for human judgment was ticked meticulously and most reassuringly. Design reviews that treat the integrity of data, models, sensors, and command links as someone else’s compliance problem will certify a form of control that may not survive contact with an adversary who would rather corrupt the picture than destroy the aircraft. This is the case a human control assessment must specify.
Consequently, the question is not only whether a human remains in the loop. It is whether the operators charged with judgment at the engagement or in the authorization of the class can still exercise it when the information and the abort path have been compromised. Meaningful human control and appropriate human judgment are different languages. Neither is better than the integrity it presumes. Cyber resilience is not peripheral to either. It is one of the conditions that makes the human role they describe substantive.
***
Dr Gerald Mako is a Research Affiliate at the Cambridge Central Asia Forum at Cambridge University.
Lemuel Valdez is the Chief Information Security Officer at Cobham Satcom.
The views expressed are those of the authors, and do not necessarily reflect the official position of the United States Military Academy, Department of the Army, or Department of Defense.
Articles of War is a forum for professionals to share opinions and cultivate ideas. Articles of War does not screen articles to fit a particular editorial agenda, nor endorse or advocate material that is published. Authorship does not indicate affiliation with Articles of War, the Lieber Institute, or the United States Military Academy West Point.
Photo credit: U.S. Army TSAE by Elena Baladelli
