Beyond Compliance Symposium – Quantifying Compliance: Challenges of Measuring Compliance with International Humanitarian Law

Editors’ note: This post forms part of the Beyond Compliance Symposium: How to Prevent Harm and Need in Conflict, featured across Articles of War and Armed Groups and International Law. You can find the introductory post here. The symposium invites reflection on the conceptualization of negative everyday lived experiences of armed conflict, and legal and extra-legal strategies that can effectively address harm and need.

Understanding patterns of compliance and non-compliance with international humanitarian law (IHL) is key for informing strategies of humanitarian engagement with parties to armed conflict, shaping policy provisions for aid, enabling targeted responses, and facilitating peace processes. Compliance with IHL is broadly defined as adherence to legal norms in armed conflict aimed at minimizing harm to civilians and other persons hors de combat, and at mitigating excessive suffering to combatants and fighters.

While legal scholarship tends to focus on normative or case-specific aspects of compliance with IHL, political scientists increasingly introduce empirical tools for the comparative study of generalizable patterns. However, a key challenge to empirical research on compliance is its operationalization. Any assessment of the conditions under which actors are more or less likely to comply with IHL requires, first, to measure compliance. Without accurate measurement of compliance, studying which factors may foster or hinder compliance becomes exceedingly difficult. Thus, appropriate measurement is an important first step towards enabling informed decision making.

This post begins with a brief overview of how comparative research has contributed to a more nuanced understanding of compliance with IHL. Rather than offering an exhaustive review of all relevant literature—which lies beyond the scope of this post—we focus on highlighting some of the most influential dimensions across which compliance has been shown to vary. The second part of this contribution reflects on some of the measures of compliance on which these comparative findings build and draws attention to potential limitations and challenges that may arise from these measurement attempts.

Compliance beyond Ratification: Patterns across Conflicts and Actors

A substantial body of research finds that international norms influence State behavior. Traditional theories of compliance draw on three main schools: realism; liberalism; and constructivism. Each offers different insights into why and how actors adhere to international law. Moving beyond these traditional theories, scholars increasingly delve into the specific conditions causing armed actors (not) to comply with IHL. Their comparative analyses have innovated the field’s understanding of States’ compliance, revealing how patterns may be generalizable across country contexts. These findings have also enabled scholars to dig deeper and to look beyond State actors, asking when and under what conditions other types of armed actors are likely to comply with IHL.

Comparative research on international law and international relations suggests a range of mechanisms explaining why States comply more or less with international law and norms. One prominent strand of research finds that State institutions matter for compliance, highlighting how characteristics shared by democracies render them more likely to comply with the commitments they make. These findings, however, are not without debate. Some qualify the role of democratic institutions, showing how the conflict’s type, such as conventional civil war versus a guerrilla insurgency, and level of threat posed to the government strongly modifies their effect on patterns of compliance. Others suggest that decisions to comply with IHL are the result of a combination of regime type and the conflict parties’ aims in war, rather than by either factor in isolation. Moreover, States fighting in “wars of attrition” against other States, foreign “counterinsurgencies,” as well as those attempting to completely “conquer their opponents,” are more likely to engage in civilian killings.

While scholarship has a historical tendency to focus on State actors engaged in international armed conflict, IHL (specifically common Article 3 of the Geneva Conventions, Additional Protocol II, and customary international law) also establishes obligations for actors involved in non-international armed conflicts, including non-State armed groups. To this end, more recent comparative work broadens its scope to analyze different types of actors and conflicts, both jointly and in isolation. For example, some find that the pursuit of legitimacy motivates “rebel groups” to adhere to IHL in an effort to secure international support. Although distinct rules apply to States and non-State armed groups, and differences in compliance exist, certain factors, such as conflict type and actors’ goals, affect adherence to IHL of both “governments” and “rebel groups.” In sum, characteristics of the conflict, the actors involved, and the domestic audience may substantially influence State and non-State actors’ compliance with IHL.

In addition to these domestic and bilateral factors, the multilateral international community may shape patterns of compliance with IHL. Mechanisms include social and prosecutorial deterrence, as in the case of the International Criminal Court, as well as coercive deterrence through the threat of third-party intervention. United Nations peacekeeping offers another example of a coercive tool, with evidence showing its effectiveness in reducing violence against civilians. While the positive effects of social and prosecutorial deterrence on compliance seem to be more pronounced for State actors than non-State actors, findings on coercive deterrence and enforcement are more complex. Depending on the conflict environment and the instrument of coercion, these have a varied influence on different actors.

Measuring Compliance

A fundamental challenge for research on compliance with IHL lies in the development of robust methods to measure compliance. Complexities abound in modern warfare. Violations of IHL occur in many varied forms and contexts. Moreover, adherence is often contingent upon the idiosyncratic circumstances surrounding each conflict. Therefore, using set heuristics to quantify compliance across a large number of diverse cases comes with a certain degree of error. Using the most common indicator for non-compliance—the number of civilians killed—we illustrate potential sources of error and how they matter. As part of this exercise, we showcase how thinking in “false positives” and “false negatives” can help those studying IHL compliance to reflect on how well their data matches their conceptual scope. Developing a systematic understanding of the sources and implications of any lack of overlap between data and concepts is essential for conducting a robust analysis of patterns of compliance.

There are currently two large-scale data sources on the number of civilians killed in a given conflict. The first is data collected by the Uppsala Conflict Data Program (UCDP) on “one-sided violence,” recording the number of civilians that were “directly” and “intentionally killed by governments or rebel groups in “intrastate armed conflict.” Data on one-sided violence reaches back to 1989 and is being continuously tracked for all “intrastate” armed conflicts globally. This makes one-sided violence a particularly rich source of information on conflict actors’ behavior, enabling systematic and large-scale comparisons across space and time.

For example, using these data, Bussmann and Schneider (2016) find that engagement by the International Committee of the Red Cross does little to reduce one-sided violence by government forces or rebel groups across 72 countries between 1989 and 2004. Valentino, Huth, and Croco (2006) compiled another large-scale dataset on “deliberately killed civilians.” These data reach further back into the past than UCDP one-sided violence, covering major State-based conflicts from 1900 to 2003. Unlike the one-sided violence concept used by UCDP, this dataset’s inclusion criteria also allow for “indirect” killings, such as inducing starvation through a blockade of a besieged town. However, it only tracks violence committed by State actors. For example, Prorok and Appel (2014) use this data to assess how States’ IHL compliance is influenced by their expectation of third-party coercion from allied States, trade partners, and intergovernmental organizations’ co-members during 80 interstate wars from 1900 to 2003.

Being publicly available and benefitting from wide coverage, these data sources are an important resource for assessing behavioral patterns of armed actors in conflict. However, they also come with substantial limitations when used to measure compliance with IHL. Crucially, although the researchers involved in data collection explicitly do not aspire to make legal determinations in accordance with IHL, their data are being commonly used to proxy compliance with IHL.

Types of Error and How They Enter the Data

There are two important caveats to consider when using current data on civilian killings as a measure of (non) compliance with IHL. First, IHL does not prohibit civilian deaths under specific conditions. Such “permissible” civilian deaths may nevertheless be included in the data and, when used to measure (non) compliance, are called “false positives.” Second, many instances of violence against civilians that constitute violations of IHL are not included in the scope of the data. These are called “false negatives.” In the following, we illustrate and discuss these two types of error. For researchers and practitioners who work on and must measure IHL compliance, it is important to reflect on the accuracy of their data. How could false positives and false negatives enter their data, and how may that influence, and potentially distort, their findings?

Both datasets discussed above try to exclude incidental deaths, such as civilians killed as collateral damage. For instance, one-sided violence specifically seeks to capture “civilians that are deliberately and directly targeted by governments or non-state groups” (Eck & Hultman 2007, p. 235), thereby excluding inadvertent deaths, such as civilians caught in crossfire. This step is useful to reduce the number of false positives when used to measure compliance with IHL, as it likely helps to reduce the number of IHL-compliant civilian killings that the data includes.

However, the same step comes at the cost of increasing the rate of false negatives, as it omits those instances of incidental civilian killings that do constitute IHL violations. For example, a failure to take all feasible precautions to avoid and/or minimize loss of civilian life, injury of civilians, or damage to civilian objects constitutes a violation of IHL, even if the civilian deaths were incidental to military attacks (cf. Additional Protocol I, art. 57).

Moreover, restricting data collection to only seemingly “intentional” civilian killings may still not yield a pure sample of IHL violations and eliminate all false positives, as even the expected killing of civilians does not constitute a violation under IHL if it is proportional to the anticipated military advantage (Additional Protocol I, art. 51). These trade-offs are exceedingly difficult to judge for researchers who often base their coding on newswire information.

Some projects address this issue, seeking to further reduce any false positives by increasing the threshold of violence against civilians for incidents to warrant inclusion in their data. Incidents that involve a particularly large number of civilian casualties are, on average, less likely to reflect a lawful use of force. Following this logic, Valentino, Huth, and Balch-Lindsay (2004) zoom out of individual civilian targeting and only use “mass killings” as indicators of IHL violations. Similarly, Stanton (2016) focuses on “large-scale abuses against civilians” such as mass killings, bombings, and scorched-earth tactics. While these measures are useful to improve precision, weeding out events that may not constitute violations of IHL, they also come at the expense of missing “smaller” relevant incidents that would constitute violations of IHL. Currently, there are no systematic studies into the trade-offs of applying different civilian casualty thresholds, so their relative levels of accuracy remain unknown.

More generally, however, definitions of core concepts only partly overlap. For example, determining who counts as “civilian” is not always easy, and datasets’ operationalizations are not the same as its definition under IHL. Similarly, what constitutes a “Non-International Armed Conflict” under international law, including the required level of violence and organization of the conflict parties, does not always map onto the scope conditions of existing datasets. Where these definitions diverge, civilian killings may be included that are not regulated by IHL, or else, instances of violence against civilians may not be included that do constitute violations of IHL. As such, these partial overlaps in scope are bound to incur both false positives and false negatives.

Why Data and Concepts Must Match

These potentials for error matter. When using data on intentional civilian killings as a measure of IHL compliance, their degree and nature of misattribution carry implications for the inferences that can be drawn from them. Walking through the implications of all the different kinds of misattribution of the numerous operationalizations of IHL compliance goes beyond the scope of this post. Therefore, we focus on one implication of using data on mass killings that serves as an accessible example. While not defined under IHL, mass killing is understood as the intentional killing of a large number of noncombatants, with inclusion criteria for what constitutes a “large number” varying across data projects (such as 50,000 deaths).

Using mass killings as a proxy for IHL violations can lead to the omission of actors who regularly engage in unlawful killings of civilians, but who remain beneath the threshold of “mass” killings. Staying below a given threshold of severity, however, may not be incidental; it may be an intentional strategy and is likely associated with relevant factors under study. For example, if concerns about international legitimacy drove actors to substitute mass atrocities with a large number of less direct and smaller scale IHL violations, our data on mass killings would fail to capture those violations, potentially (mis) leading us to conclude that international pressure improved compliance with IHL.

More broadly, even if we had access to data that captured the full range of unlawful killings of civilians, this would still represent only one element within a broader range of issues that IHL regulates, even among those that relate to the direct victimization of civilians. For example, targeting civilians may not result in death, but in injury or maiming. This type of harm is not included in the scope of datasets that focus on civilian killings. Importantly, patterns of compliance vary across the numerous issue areas that IHL regulates. Correlates found for one issue area may or may not be different from other issue areas, depending on the actors involved, their war aims, the conflict dynamics, and the international environment. For instance, compliance was found to have improved over time with regard to prisoners of war and the use of chemical and biological weapons, supporting the notion that internalization of norms matters.

However, this trend does not extend to other issues, such as aerial bombing and armistice. Without a better understanding of how the patterns of compliance interact across different issue areas, the scope conditions of research findings remain unknown, and the humanitarian or policy action they seek to inform runs the risk of causing inadvertent knock-on effects. Comparative data on IHL violations other than unlawful civilian killings (e.g., on sexual violence, child recruitment, and forced displacement), as well as projects that combine multiple domains of IHL (e.g., Morrow and Jo, 2006; Jo, 2015; Stanton, 2016; Walsh, Conrad, & Whitaker 2023), may offer an opportunity to engender a more complete understanding of patterns of compliance with IHL across contexts and domains. Meanwhile, case-based contributions, such as identifying the locations of bombings (Swed, 2023), may be assessed for their scalability and provide new avenues to add to the body of comparative data.

One further persistent challenge in measuring compliance with IHL, regardless of the data source, is the necessity to infer compliance from the absence of observed instances of non-compliance. This can present statistical and interpretive issues, as the lack of observed violations represents an absence of evidence, and not evidence of absence. When violations go unreported or unobserved, it remains unclear whether this reflects genuine compliance or whether it is merely a case of underreporting bias. This creates ambiguity in assessments similar to the above example on mass killings, as the lack of visible violations does not necessarily indicate adherence to IHL norms.

Finally, accurate measurement of IHL compliance is not only important for drawing appropriate conclusions about its patterns, but also for better understanding the scope constraints and possible blind spots of IHL. Armed conflict causes substantial harm and need in various shapes and forms, and across a multitude of dimensions, of which only a fraction is regulated by IHL. In the absence of accurate data on IHL compliance, it is exceedingly difficult to distinguish between conflict-induced need and harm resulting from a lack of compliance, versus need and harm resulting from a lack of regulation. It is this space that we, at the Beyond Compliance Consortium, scrutinize. Working in this space, therefore, requires us to take stock of current measures of compliance with IHL and the resulting limitations.

***

Dr Christoph Dworschak is a political scientist and the Senior Advisor on Quantitative Methods of the Beyond Compliance Consortium.

Dr Hyunjung Park is a Research Fellow for the Beyond Compliance Consortium in the Centre for Applied Human Rights at the University of York.

Photo credit: ICRC/Daniele Volpe (28/01/2016, V-P-GT-E-00239) Guatemala, Quiquil. The ICRC helps relatives of missing persons give them a decent burial. The setting for the burial, located two kilometres away from the village, was chosen by the members of the community themselves. It is the place where, five years before, one of the graves was found, with mortal remains of people killed by soldiers in 1982.