AI, the Box, and the Black

It was my honour and privilege to be invited to attend, and participate in a panel at, the U.S. Cyber Command Legal Conference, “Innovation to Impact” from 28 to 30 April 2026. The Conference addressed, inter alia, issues arising from the employment of artificial intelligence (AI) in the cyber domain. Many issues emerged, but one series of linked concerns, which I have been loosely contemplating since returning to the United Kingdom after the conference, solidified on Thursday morning, 7 May 2026, into a series of ten questions and thoughts which I offer, and briefly discuss, below.

1. Can an AI process, the outcome of which cannot be explained, be regarded as discriminating?

This is of importance to the use of AI in connection with attacks because of the Additional Protocol I, Article 51(4) prohibition on indiscriminate attacks. Paragraph 5.3.2 and subsequent paragraphs in Chapters 5 and 6 of the U.S. Department of Defense Law of War Manual express the obligation in relation to the proportionality rule. Put in relation to that rule, the issue would be whether an AI-enabled attack process, the outcome of which cannot be predicted and the compliance of which with the proportionality rule cannot be determined with acceptable reliability, is legally acceptable.

2. Are testing results a sufficient basis for “explaining” outcomes, or does the black box of un-explainability mean the system is legally unacceptable?

Put another way, this question asks whether it is enough that in tests the AI system has performed to an acceptable standard if the method by which the data input to the AI is translated into its output remains unknown. So if such a method remains unknown or even unknowable, is the fact that the AI happens to have produced acceptable answers in tests enough to justify its employment in operations, despite the possibility that cannot be excluded that it is reaching those right answers for wrong reasons and may in the future apply the same logic to reach disastrously wrong output?

3. Does the employment of multiple, inter-linked AI systems create a field of un-explainability which is impenetrable and does its impenetrability render it legally unacceptable?

Multiple, inter-linked AI systems, called Multi-Agent Systems, may be employed with each sub-system contributing to the generation of a composite output (consider for example here). This question asks whether the output of each such sub-system can be individually explained or does the combination of such sub-systems into a composite whole produce a situation in which the overall performance of that composite system cannot be explained. If the method whereby the composite system produces its composite output cannot be explained, the issues discussed in 1 and 2 above apply.

4. Can we really trust that which we cannot explain?

The Merriam-Webster online dictionary explains “trust” inter alia in terms of the assured reliance on the truth of someone or something, and “one in which confidence is placed.” Perhaps the degree of confidence that is required, the amount of reliance that can properly be placed, will be the product of a number of factors only two of which are the severity or otherwise of the consequences if the trust proves to have been misplaced and the proven performance of the AI system concerned.

5. If the answer to the question at the beginning of paragraph 4 is “not completely,” what are the implications of that for the responsibility of those who must decide whether to use such an AI system, or group of AI systems, during an operation?

That involves consideration of a linked question, namely is trust an absolute, or are there degrees of trust. Put another way, if, after testing reports and other relevant input to the weapon reviewer have been received, the reviewer concludes that the AI system cannot be completely trusted, does that mean that it cannot be used? Furthermore, what would be the legal implications for a commander who authorises a mission employing an AI system that has been judged to be less than fully trustworthy?

The traditional view is that weapon systems that do not comply with the law applicable to the acquiring State should not be obtained, with the result that commanders can have confidence in the lawfulness of the systems issued to them. If trustworthiness is considered to be an absolute, a system that is not completely trustworthy should not be procured. If, as I believe is likely, States take the view that in the AI context trustworthiness is a matter of degree, the level of trustworthiness that is required for a military AI system to be acceptable will be related to its intended function, the likely consequences of mis-performance and other factors. The required level of trustworthiness of a particular class of system will then need to be specified in over-riding national policy, and thereafter to be applied in the weapon reviews of relevant systems. The implications of this are as follows. If trustworthiness can be measured, States, or groups of States, may need to prescribe trustworthiness thresholds for different applications of AI, so that weapon system testers can assess the trustworthiness performance of a specific system under review, and thus determine its acceptability.

6. Does it make a difference that human decision-making is also fallible?

It is certainly important to bear in mind that a human decision-maker may make mistakes; “to err is human.” Therefore, when evaluating the performance of an AI-enabled system or of an AI system that is supporting human decision-making, the human comparator that is being considered as the alternative is indeed prone to make errors (and as the pace of warfare increases considerably, the likelihood of human error probably increases). The key difference, it is suggested, is that with a human decision-maker the errors can be identified, understood (and forgiven!) and lessons can be learned from those mistakes. With AI processes, it may not be possible to identify in a timely way or at all what has gone wrong, so the prompt learning of lessons and allocation of responsibility may not be possible.

7. Is the implication of all this that autonomous systems should only be used when the processes leading to their outputs can be explained?

The potential corollary would be that the employment of autonomy, e.g. in order to search for, identify and decide to engage a target, is unlawful if the associated AI process(es) cannot be explained. However, as readers will be aware, some of these search, identify, decide elements of autonomous weapons already exist and fully autonomous weapon capabilities are a foreseeable prospect (see for example here, here, and here). So there is the prospect of science and technology producing capabilities that certain interpretations of existing law would prohibit. In situations of on-going armed conflict, particularly where national survival is at stake, it is to be expected that technological developments that are perceived to grant the threatened State with a significant battlefield advantage will be pursued vigorously.

8. Do the same considerations that apply to the use of AI in autonomy mode also apply to support of human decision-making?

Put another way, is a decision by a human operator, in which he or she relies on the output of an AI system the workings of which cannot be explained, tainted by that un-explainability? It is likely that technology will progress over time (consider here, here, and here), and that progress will in due course see improved explainability, such that the scope and degree of the blackness of the box may be expected to diminish. Exactly how much progress has already been made is unclear. It is the responsibility of the State to determine the legal and ethical acceptability of the weapons and associated systems that it acquires, and, as noted earlier, commanders are entitled to assume that such weapons and systems comply with applicable law. This will be subject to any limitations identified in the weapon review and/or in concept of use or similar documentation issued by the State to its armed forces, to user commanders and to units. This would suggest that a commander whose unit is issued by the State with a system that employs AI in autonomous or “support to human decision-making” roles is entitled to assume that, by reference to explainability and other factors, he or she can lawfully decide to use the system within the constraints identified in the concept of use document.

9. It will be a matter of judgment by reference to a specific AI system and its acceptability for employment in a particular context when the area of un-explainability has diminished sufficiently for it to be realistic to regard the AI system as trustworthy.

Trustworthiness in this context is likely to be closely linked to the purpose for which the AI system is being employed. It may be that in certain applications, for example decisions regarding the resort to the use of force by a State or the decision by a State whether to use a nuclear weapon, absolute confidence is required, such that the trustworthiness measurement on a scale of zero to ten would need to be ten. Generally speaking, in military applications relating to any use of force the trustworthiness measurement would probably need to be high, or very high. This leads to the idea that it will be necessary to measure the extent and the degree of blackness of the box, represented by the process that translates the system’s inputs into its outputs. Work is under way to assess the opacity of the workings of AI systems (see, for example, here). Moreover, a team at Loughborough University in the United Kingdom has recently reported on progress towards developing more transparent AI systems.

10. It is clear from all this that the testing processes in support of weapon reviews of autonomous and decision-support systems will involve testers working closely with system developers and other experts as a key element in the tester’s report to the weapon reviewer will be an evaluation, or perhaps a measurement of the trustworthiness of the autonomous or decision-support system.

The intended circumstances of use of a weapon or weapon system is the usual basis for the conduct of a weapon review. The reviewer is not required to consider circumstances in which it is not intended to use the weapon. If those intended circumstances, or manner, of use of the weapon change, further legal advice will be required. An important question therefore is what this “intended circumstances of use” notion means in the context of AI use in autonomous systems and in support of human decision-making. It will clearly be necessary for testing to replicate the kinds of input data, e.g. the nature of the data, the sources of the data, the quantity of the data, and any other features of the data that seem to be relevant to the functioning of the AI processing that is to be applied to it. It will also be necessary for the testing to replicate the kinds of task that the AI system will be required to fulfil, i.e. what the AI system is being told to do with the data and the output the AI system is being told to produce.

It will then be necessary for the testers to assess whether the AI system will reliably and consistently notify the operator if its systems are, for any reason, not operating as designed. Incorrect operation may be due to sycophancy, bias, misinterpretation of the set task, deception by the AI, hacking by the enemy or by a third party, manufacturer’s fault, corruption of the software or any other cause. It follows that the testers must have a complete list of the factors that can cause an AI system to mis-perform and must test the system for all such features or faults. Essentially, the point being made here is that an AI system when performing as it was designed to do can perform an assigned task incorrectly; an AI system may also produce incorrect performance due to one of the factors or faults listed earlier in this paragraph.

Each of the points made in this piece will have implications for accountability, but that topic lies outside the intended scope of this post. For a more extensive discussion of the legal issues arising from the employment of AI systems in warfare, the reader is referred to AI Warfare and the Law.

***

Air Commodore William H. Boothby retired as Deputy Director of Royal Air Force Legal Services in July 2011. He is Honorary Professor at the Australian National University and also teaches at the University of Southern Denmark and at the Geneva Centre for Security Policy.

The views expressed are those of the author, and do not necessarily reflect the official position of the United States Military Academy, Department of the Army, or Department of Defense.

Articles of War is a forum for professionals to share opinions and cultivate ideas. Articles of War does not screen articles to fit a particular editorial agenda, nor endorse or advocate material that is published. Authorship does not indicate affiliation with Articles of War, the Lieber Institute, or the United States Military Academy West Point.

Photo credit: Getty Images via Unsplash