Designing for Reasonableness: The Algorithmic Mediation of Reasonableness in Targeting Decisions
Developments on current battlefields are proving that algorithmic decision-support systems are not a distant future, but a reality that military and civilian populations are already living with. For example, in November 2023, it was reported that defense contractor Palantir has provided their artificial intelligence platform, or AIP, to the Ukrainian military. AIP is a system powered by machine learning and connected to large language models that offers a chat box function. It is used to identify areas with mines and support demining efforts. Recent reports have also revealed the use of algorithmic decision-support systems for targeting by the Israel Defense Forces (see here and here). The system called Gospel is used to identify in real-time the location and movement of suspected Hamas operatives.
These and other decision-support systems are using machine-learning algorithms to find patterns in large databases of sensor data, satellite photos and drone footage, and even social media feeds in order to identify, locate, and track the movements of suspected targets. They can also be used to estimate collateral damage and provide recommendations for courses of action. The speed at which such systems process information and provide target sites is unprecedented; it is faster than any previous human efforts and the actual capacity to strike. With these characteristics, many are concerned with the impact such systems will have on the legal obligations of commanders in relation to targeting decisions in conflicts.
The purpose of this post is to raise awareness of the use of algorithmic tools by military actors and to argue that algorithmic decision-support systems significantly reshape the reasonableness of military decisions.
Reasonable Commander Standard
Questions about the lawful use of algorithmic systems in relation to specific provisions of international humanitarian law (IHL) continue to be a source of heated debate (for an example, see here). Instead of focusing on specific IHL provisions, in this post I bring attention to the broader legal concern for the reasonableness of military targeting decisions. I argue that the use of algorithmic decision-support systems not only undermines the exercise of human judgment, but more importantly, it reshapes what it means to be reasonable in warfare.
The “reasonable commander standard,” although not embedded in the Geneva Conventions, has come to be regarded by international courts as the threshold that underpins the margin of discretion given to commanders when making proportionality assessments and other targeting decisions pursuant to IHL. Primarily, reasonableness should guide a commander’s decision in situations where the law does not provide straightforward answers to ensure the expected damage and injury to civilians and civilian objects is not excessive.
Most importantly for this discussion, reasonableness has traditionally been associated with the attributes, traits, and experiences of a hypothetical person that can individually ensure that decisions are reasonable. Take, for example, the International Criminal Tribunal for the former Yugoslavia (ICTY) Final Report from 2000, which discusses the “different doctrinal backgrounds and differing degrees of combat experience or national military histories” as factors in the reasonableness of a targeting decision (para. 50). This perspective, which focuses specifically on a commander’s experiences and values, continues to influence our understanding of this standard. It is often acknowledged that information available at the time plays a role in making assessments, however lawyers view its relevance only so far as it provides a “factual foundation for [the commander’s] assessment of reasonableness,” emphasizing a commander’s ability and responsibility to interpret information in a reasonable manner.
This perspective-based approach has even sparked a debate about the military expertise and training of a hypothetical commander against whom the reasonableness of a decision is benchmarked (see here and here). For example, in a Just Security exchange, experts in the field disagreed on the relevance of public perceptions of the proportionality assessments in military contexts. While Blank, Corn and Jensen sustained that “a solid foundation of military operational experience” is necessary to performing and assessing the reasonableness of attack judgments, Dill argued to the contrary that reasonably weighing military gain and human life “is not a technical military [judgment]” but is “informed by our most basic moral commitments.” All in all, this once more shows the central role individual expertise plays in the legal interpretation of the reasonable commander standard.
There are, however, a few scholars who challenge this traditional interpretation. For example, Asaf Lubin argued that it is not a commander but an intelligence analyst who should most appropriately be held to a reasonableness standard. Shiri Krebs showed that aerial visuals can significantly influence how commanders perceive targets and hence also what it means for them to be “reasonably certain” that a target is lawful. Most recently, in the book Honest Errors? Combat Decision-Making 75 Years After the Hostage Case, the authors invite readers to reconsider whether the perspective-based approach to the reasonableness standard in IHL is still appropriate in 21st century warfare. These scholarly works all acknowledge that commander’s judgment is highly mediated by, among other considerations, intelligence analysis and aerial technologies.
A Holistic Approach to Reasonableness in Targeting Decisions
In concert with these observations, I argue that reasonableness is more than a matter of an individual judgment. Rather, it is shaped by a variety of factors, including the availability of means and methods of warfare, the design and use of algorithmic tools, the institutional goals and policies, rules of engagement, commander’s intent, all of which play a role in either inhibiting or supporting what would broadly be considered a “reasonable” targeting decision.
Within this complex network of actors, technologies significantly mediate human experiences and capacities and alter the ultimate quality of decisions. As shown by others, aerial technologies and other sensors mediate the perceptions of battlefields, bodies, and threats (see here and here). These techniques of sensing battlefields lead to specific ways of making some threats visible, while others are maintained in the register of the invisible. The satellite imagery is processed only within periodic intervals, drones need to be brought down to be recharged, and heavy rain interrupts voice recordings, creating spaces of invisibilities and moments of silence. Naturally, all modes of intelligence gathering have specific limitations, the point is to note and consciously evaluate these impacts as technologies gain an increasingly prominent role in shaping decisions.
This becomes ever more important as technical limitations of sensors become coupled with algorithmic decision-support systems. The effects of technological mediation become aggravated by categorizing, labeling, and identifying patterns in data that shape not only military actors’ vision but also knowledge production. The more data is collected, the more difficult it becomes for humans to analyze and identify what is relevant. This is where algorithmic technologies are offered as a solution. By analyzing patterns in data, machine-learning algorithms identify outliers. For example, algorithms may detect marine vessels that do not follow standard shipping routes. Computer vision techniques process thousands of photos, but only those few that recognize a “military missile vehicle” are shown to analysts. In effect, the algorithms not only mediate perceptions, but also selectively determine what military actors pay attention to and what is relevant to them. Hence, the algorithms themselves significantly shape military understanding of the situation on the battlefield.
Many States underline the importance of ensuring that a human is in the loop to review the recommendations and exercise judgment before resorting to targeting. Decision-support systems are by design not meant to replace the commander, but rather to inform their assessment. Is this approach enough to ensure a reasonable decision outcome? Several factors hinder human judgment and thoughtful consideration of algorithmic recommendations. First, algorithmic systems are often a “black-box,” meaning that it is challenging or impossible to trace the neural network’s decision-making process. Without an understanding of the underlying reasoning of an algorithmic suggestion or an alternative source of information verification, the human analyst may struggle to verify the legitimacy of a target.
Second, the speed of algorithmic processing diminishes the time available for careful analysis by human analysts. Real-time location-based suggestions require swift decisions to strike before the target relocates or goes “off the radar.” With the human in the loop, algorithmic recommendations can, in principle, be ignored. However, the speed and volume of target recommendations introduce a climate of risk where recommendations are not to be ignored. As algorithmic suggestions continue to appear on the screens with high frequency, they create a sense of urgency, and may potentially prompt hasty decisions. While procedures like a checklist may theoretically enhance the lawfulness and reasonableness of targeting decisions, the fast-paced target generation and the “black box” impede the capacity of human analysts to thoroughly assess algorithmic recommendations in depth. These are just a few examples of ways in which human-machine interactions challenge independent and subjective judgments by a commander in warfare scenarios, and instead highlight their co-constitutive character in determining targeting decisions.
Preserving human judgment in human-machine interactions continues to be of relevance (for example, for the purposes of assigning responsibility), but in itself is arguably insufficient to promote the reasonableness of targeting decisions. Instead, conceptualizing reasonableness as an outcome of a network of human analysts, technologies, and procedures brings our attention to how the system as a whole is designed and whether it inhibits or supports reasonableness.
Practical Considerations and Future Work Avenues
The very essence of reasonableness is undergoing transformation. It is not only molded by human perceptions and understanding, but also significantly reshaped through the calculations, approximations, and pattern-finding logic of algorithms. Algorithmic decision-support systems, in conjunction with various sensors and tracking devices, mediate not only what military actors know and do not know, but also how they interpret events and intentions on the ground. Hence, even in situations where a commander acts in good faith, the capacity to exercise judgment is substantially mediated by the nature and logic of algorithmic tools. Finding the “reasonable” balance between humanitarian considerations and military advantage in targeting decisions has always been challenging and vaguely defined. The employment of technologies will impact this balancing act as much as the expertise and experience of a commander.
In other words, algorithmic technologies challenge the understanding of what is “common sense” in battlefield situations. Military intelligence has naturally always been partially hidden from outside eyes. Current conflicts reveal that decisions based on algorithmically-calculated patterns of threats and labeling of objects create new hidden layers obscured from public scrutiny. Fleur Johns comments on this development by underlining that “the publics to whom this work may be of concern [are no longer] likely to appreciate—that is , maintain some rudimentary, ‘common sense’ grasp of—the sensory work ongoing in this context” (p. 84-85). Scrutinizing whether a targeting decision was reasonable is challenged because information that was available at the time—that is, all information to which the algorithm had access to and on the basis of which it made a selection—becomes inaccessible. I argue that developments in algorithmic technologies also strike to the very heart of the scholarly legal debates that I mentioned before regarding whose “common sense” or “reasonableness” matters, whether that of a person with military experience or a layperson. The reality is that what is reasonable in a given targeting situation is increasingly dependent on the algorithmic calculations. This is how the very essence of reasonableness is reshaped.
While the use of algorithmic technologies is not explicitly prohibited, it becomes ever more important to ensure that technologies are designed in ways that support reasonable targeting outcomes, rather than only focusing on ensuring that a commander is a person capable and willing to reach reasonable outcomes. Militaries need to carefully consider how the algorithmic system identifies and categorizes targets, and what type of logics underlie the pattern identification and target selection. For example, systems that provide several courses of action may promote contestation and reflection congruent with reasonable outcomes, while algorithms that produce a singular outcome are more likely to inhibit reasonableness. Similarly, metrics such as accuracy and confidence rates (that may be employed in a legal review of weapons, means or methods of warfare) often serve to legitimize systems and then by extension targets that they produce. However, such metrics are limited to the controlled environments in which a system is tested and are imperfect in reflecting a machine-learning system’s performance when it further evolves in relation to data gathered from dynamic battlefields. These and many more risks associated with algorithmic processing (see also here and here) remain an important consideration in ensuring that the recommendations such systems produce, the type of reflections they encourage, and the pace of targeting they promote altogether serve the broader legal concern of preventing excessive damage to civilian objects and injury to the civilian population. This will remain an important endeavor in future works.
Concluding Remarks
Reasonableness is no longer (and arguably has never been) solely the product of a mind-bound judgment by a figure embodying a reasonable commander. Instead, it emerges as a result of the interplay between technical techniques and standards, algorithmic processing and recommendations, institutional policies and attitudes, and ultimately also the commander’s judgment. All these elements collectively shape the reasonableness of targeting.
Current events in Gaza, as well as those in Ukraine, are an important reminder of the need for a careful design of our institutions and technologies in ways that promote reasonableness in targeting decisions. The staggering human toll underscores the urgency of this pursuit. The significant impact of algorithmic technologies on the reasonableness of military decision-making has not been fully acknowledged or understood. Amongst them, the fast pace and the endless generation of targets are features of algorithmic technologies that pose a major obstacle to exercising human judgment that should not be underestimated. Which technologies we choose to employ, how we design them, what type of interfaces we permit and promote, and ultimately also how much time we give ourselves to review algorithmic recommendations are crucial considerations that will either inhibit or support the reasonableness of military decisions.
***
Photo credit: Staff Sgt. Andrea Salazar
