Hollow Control: Cyber Compromise and Human Judgment in Lethal Autonomous Weapons – Part I
Observing that modern warfare is ever more reliant on networks, data, sensors, and software, all of which are vulnerable to attack, has the novelty factor of a field manual explaining that the enemy gets a vote. The more interesting and indeed pressing discussion revolves around the circumstances in which the adversary gets a vote unnoticed.
A legally problematic cyber operation against lethal autonomous weapons systems (LAWS) is not one that blacks out the screen and brings the system down. It is a compromise that does nothing at first glance, yet convinces the operator that they are in control. These sort of compromises that hollow out human control are the kind that corrupt the picture on which a human decision rests or interrupt the path by which that decision is supposed to constrain the weapon.
Articles of War recently concluded a series on LAWS that examined how meaningful human control (MHC) can be built into systems and assessed as a matter of design and procedure. These two posts ask a prior question: what happens to MHC if the information gathered is corrupted before a decision? MHC is only as reliable as the information and control pathways on which it depends. If either is manipulated, the architecture of control remains and the judgment does not.
Cyber Operations and the Picture the Law Requires
Dependence creates a fundamental vulnerability: LAWS rely upon the integrity of information, models, sensors, and the connection between them, all of which may be compromised by cyber and electronic attacks. The more ways a system gathers information, the more accurate its picture may become. However, each added source is another input that must be secured, authenticated, and checked.
This vulnerability is far from being theoretical. On the contrary, it has been known since at least 1997 (which by drone and AI standards equals the Jurassic period) when Hezbollah claimed to have intercepted analog video from Israeli drones over Lebanon. In 2009, insurgents in Iraq used commercial software to receive live video from Predator unmanned aerial vehicles (UAV). They were not able to assume control of the aircraft but could watch its live video feed, the very same picture the operator was using.
Thus, compromised data is already a problem for the law of targeting, not a problem beside it. Although it would be tempting to assume otherwise, cyber operations and electronic warfare do not create a separate body of targeting law.
If they are used in an armed conflict, cyber operations and electronic warfare remain means or methods of warfare. Distinction, proportionality, and precautions apply where the operation is an attack, and the rest of international humanitarian law (IHL) continues to govern where it is not. This idea is reflected in the Department of Defense (DoD) Law of War Manual, in the International Committee of the Red Cross’s (ICRC) position on cyber operations, and in the Tallinn Manual 2.0.
These legal frameworks are well established. What matters for autonomous weapons is a much narrower issue, as an adversary’s cyber operation does not need to cause physical damage or take down a power grid to create problems among adversaries. The picture can be subtly altered while the heading continues to look like what the operator expects; an “ILOVEYOU,” not necessarily a black screen. For control to fail, it is enough that the person who must apply the law does not realize that the picture, the model, or the command path is no longer theirs.
In electronic warfare, jamming and spoofing typically take aim at satellite navigation, radio command links, or sensor returns. Cyber compromise can take a range of forms: malware on ground stations and support networks; poisoned training data or models; impersonated command messages; or interceptable downlinks. Compromise of the supply chain or software updates can inject the same flaws before the system ever reaches the field.
Methods like these can leave the interface looking perfectly intact. The targeted object on the screen, the track used for a proportionality calculation, or the path of an abort order may nevertheless no longer belong to the operator. When that happens, the problem is not merely about network defense but about legal judgment.
What Meaningful Human Control Requires
For human control to be meaningful in the sense used here, there must be a sufficiently reliable understanding of the state of the system, the environment, and the effects of an attack. Cyber compromise can falsify information in any of these variables. Even if a commander approves an attack, the approval may not contain any substantive information about the targeting decision. The discussion of human control over autonomous weapons focuses on the architecture and interfaces, with less attention to the adversary’s potential to target the information and control environment. As the information may be compromised and the abort path may no longer be reliable, without cybersecurity and resilience built into compliance by design, meaningful human control risks becoming scarcely more than a convenient illusion.
MHC becoming the dominant formulation in policy and academic debate led to it being keenly used across papers discussing issues related to LAWS to the extent that one might be forgiven for assuming it is a formal law of armed conflict term. It is in fact not. The law still puts distinction, proportionality, and precautions on those who plan, decide, and execute attacks. Consistent with this, the Group of Governmental Experts on LAWS has repeatedly said that responsibility for the use of force cannot be delegated to a machine.
Lawful use requires a range of concrete conditions. First, there is the suitability of the attack. Second, a practical understanding as to what the system will do. Third, the capacity to constrain or stop it. And finally, a clear allocation of responsibility for the decision.
The principle of distinction fails in a situation where the object presented as a target is not the object on the ground and the person applying the rule cannot tell. A proportionality assessment fails when it is made on a track or model that is not the one the commander believes they are using. An abort function no longer constrains an attack if the order does not reach the weapon, or if the weapon is no longer where the operator has sent it.
National policy documents already assume this integrity. Take the much-quoted DoD Directive 3000.09 for example, which mandates appropriate levels of human judgment over the use of force and, in the same instrument, calls for cyber survivability and operational resilience. Further, European institutions have called for meaningful human oversight of lethal decisions. What is sound on paper, however, can be undermined if the data, the model, or the control channel is no longer what it seems.
How Compromise Empties Out Human Control
Although undeniably important, the threat to human control is not limited to an adversary defeating the weapon through disrupting or deceiving it by electronic warfare methods such as spoofing or jamming, which already have become quintessential features of conflict. After nearly five years of grinding war, with the air over much of the country humming with electronic interference, Ukraine offers a clear illustration of how routine have spoofing and jamming have become.
By 2025, a Ukrainian first-person view (FPV) unit was losing about a third of its sorties to Russian electronic warfare, in response to which Kyiv has been expanding its own spoofing, including the Pokrova system which can spoof the navigation of incoming drones. Meanwhile, to counter the radio environment that has been degrading uncrewed aircraft in large numbers, both sides have begun flying fiber optic drones.
The significance of these attacks is straightforward: when jamming or spoofing causes the operator to lose control, the failure is visible. The more insidious possibility, however, does not come with the asset crashing and burning. Such scenarios play out much less like a black screen and more like The Truman Show, the picture remains convincing while the adversary has already started to control what it shows. The operator may of course continue exercising judgment, but the information and control pathways on which that judgment depend have already been compromised.
An adversary who can degrade the picture without detection has an incentive to preserve the aircraft rather than destroy it. By contrast, a lost sortie ends the problem. A sortie that continues on information the attacker has shaped continues on the attacker’s terms, thus giving a reason to take the residual case seriously in design and review.
The image can be corrupted in myriad ways. The information displayed on the screen can be altered to mislead the operator. Data and model poisoning can modify the examples a classifier learns from or the parameters of the model it then uses. The system then mislabels objects and attaches a confidence score that appears routine. If an operator treats that score as a sufficient basis for distinction, they are applying the law to a classification the system has produced, not to a scene they have independently verified. Where the model or the input has been corrupted, the legal act of distinction has not been performed, however regular the interface appears.
Sensor spoofing does similar work in real-time. For example, at White Sands in 2012 a Department of Homeland Security-invited team demonstrated for the first time that false GPS signals could manipulate a UAV’s autopilot and cause it to fly toward the ground.
Nothing in the law requires the falsehood to be crude. A track may well be clean. A confidence score may well be routine. Yet the object presented to the commander may still be different from the one they are required to identify. Sensor spoofing and adversarial input are perfectly capable of producing this result, but whether they have done so in a given strike is a question of proof. That they can do so is a question a legal review of the system has to address.
On the other side of this architecture the path between the operator and the effector can be directly attacked. A compromised command link or ground segment can delay, drop, or rewrite an abort order. In 2011, malware was found on Predator and Reaper support systems at Creech Air Force Base which did not, on the public record, put aircraft under someone else’s control. Yet the incident illustrated particularly well that the station from which judgment is exercised can itself be compromised. However slim the chances might be, a tainted component, a compromised update, or a vendor tool could put the same fault in firmware that no later human approval will detect.
Subtle degradation leaves the human approval step intact while removing the very facts that made that step a legal judgment, thus creating an operational memory hole. These gaps are especially hard to catch in high tempo or communications degraded environments where the window for a commander to perceive the difference may be vanishingly small.
Cyber compromise is first and foremost a targeting problem. The law still asks whether the person who planned or made the decision directed the attack against a military objective and whether feasible precautions were taken.
These obligations are not fulfilled by the presence of an intact interface. If the object offered to the decision maker is not the object on the ground, if the information used for proportionality is not the information the commander believes they are using, or if the constraint they think they have imposed never reaches the weapon, distinction, proportionality, and precautions have not been fulfilled. The click itself may well survive, yet the judgment based on false, corrupted data is all but hollowed out.
Conclusion
Regardless of how far they reach, no targeting rule is cancelled by a cyber compromise that can hollows out the facts these rules require. The person planning or deciding on an attack is still asked to distinguish, to weigh proportionality, and to take feasible precautions. None are answered by a screen that looks like the expected command picture, by a click that records as approval, or by an abort that in fact never leaves the console.
If they are to be carried out, what these rules need is an object that is still the object on the ground, an estimate of effects that is still that of the commander’s, and a constraint that can still reach the weapon. Part II of this post asks what that hollowing does to the standards States already use, where responsibility sits when the picture is false, and why cyber resilience belongs inside compliance by design rather than in a parallel review.
***
Dr Gerald Mako is a Research Affiliate at the Cambridge Central Asia Forum at Cambridge University.
Lemuel Valdez is the Chief Information Security Officer at Cobham Satcom.
The views expressed are those of the authors, and do not necessarily reflect the official position of the United States Military Academy, Department of the Army, or Department of Defense.
Articles of War is a forum for professionals to share opinions and cultivate ideas. Articles of War does not screen articles to fit a particular editorial agenda, nor endorse or advocate material that is published. Authorship does not indicate affiliation with Articles of War, the Lieber Institute, or the United States Military Academy West Point.
Photo credit: Italian Army, www.esercito.difesa.it
