Home : News : Display
March 20, 2025

Measuring Interoperability Within NATO: Adapted Off-the-Shelf Tool or Bespoke Solution?

John R. Deni, Matthew R. MacLeod, Sarah E. Stewart, Katherine M. Banko, and Adrian Jones
©2025 John R. Deni
©His Majesty the King in Right of Canada, 2025

ABSTRACT: Despite decades of work on interoperability, NATO Allies cannot measure, assess, and build upon it fully and accurately. The lack of agreed formats or standards for collection, management, and the communication of findings have prevented the Allies from developing common interoperability measurement and assessment tools. Nonetheless, NATO could adopt extant standards, methodologies, processes, or tools to achieve its interoperability objectives. Testing this notion through use cases, the authors identify practical and conceptual hurdles to adopting an off-the-shelf solution. They conclude that the Alliance may need to create assessment standards, methodologies, processes, and tools from scratch, despite the difficulties of doing so.

Keywords: NATO, interoperability, US Army Interoperability Measurement System (AIMS), Multinational Interoperability Assessment Tool (MIAT), Standing NATO Maritime Group (SNMG)

 

NATO’s expansion to 32 nations, its re-embrace of collective defense since Russia’s first invasion of Ukraine in 2014, and its acceleration of defense and deterrence adaptation since the full-scale Russia-Ukraine War began in February 2022 have made interoperability among Allies more important than at any time since the Cold War. Early in the Cold War, the Allies recognized that if they intended to fight side by side, they needed to be able to share things like ammunition, fuel, spare parts, intelligence, battlefield tactics, standard operating procedures, and radio frequencies. The benefits—and, arguably, necessity—of being able to operate side-by-side have thus been evident for decades, and the Alliance has established bureaucratic structures, policies, and guidelines designed to achieve interoperability. Today, interoperability remains crucial, as military personnel rely on proven interoperable systems to allow them to focus on planning and executing operations across all war-fighting functions, domains, and echelons.1

Many Alliance documents explain how interoperability leads to improved readiness, effectiveness, efficiency, and reduced costs. Despite pursuing interoperability for decades, the Allies remain unable to measure it, assess it, and build upon it fully and accurately. This predicament is especially ironic given the critical role that interoperability has played in Alliance exercises, training events, and operations. It would be reasonable to assume that “the most successful alliance in history” would have formed a consensus on how to measure and assess whether and how its members’ forces can operate together. NATO has not yet closed this significant gap, however, in part due to the lack of a standard format for interoperability data. Large amounts of data are collected at NATO interoperability events and exercises, but without a standard, interpreting that data across time, nations, and domains is a challenge that impairs NATO’s ability to leverage the information fully. When conducting interoperability analyses, bespoke data collection and analysis takes away from time and resources that could be spent identifying lessons and acting upon them.2

In addition to lacking a standard format for interoperability data, NATO lacks a common interoperability measurement and assessment tool. Some of its member nations and other entities have developed their own tools, however, which raises the question of whether any could be modified for NATO use or whether their output could provide the basis for a standard data format that could be implemented by multiple tools.

Agreed formats or standards for collection, management, and the communication of findings and a common interoperability measurement and assessment tool would help to solve many of the interoperability challenges facing NATO, especially if they could be applied across multiple war-fighting domains (land, air, and maritime). For example, they would allow military planners to assess and understand the state of interoperability with their partners and leverage the findings to address shortcomings. Additionally, interoperability assessments could inform the resourcing decisions of individual nations pursuing unique interoperability objectives across their military services.

This article examines what methodologies, processes, or tools already exist for interoperability measurement and whether they might be modified for NATO’s use. It further considers whether such a tool could be a basis for standardizing interoperability assessment or, alternatively, what a standard would involve.

The article first establishes key concepts and definitions with respect to interoperability, standardization, and assessment. It then examines current practices within NATO and the available scholarly or professional literature regarding extant interoperability data collection, data standardization, and data management tools. The next section analyzes how interoperability methodology, processes, and tools might be used in practice within NATO, given existing Alliance operations in the land, air, and maritime domains. Subsequently, the article asks whether a military off-the-shelf option, such as the US Army Interoperability Measurement System (AIMS), could do the job across NATO’s many Allies and among all major war-fighting domains. The final section looks forward and asks: If an off-the-shelf solution is unfeasible, what key factors should be addressed in an agreed data standard for Alliance interoperability, and what should a NATO interoperability measurement and assessment system consider?

Concepts and Definitions

This section lays out how we will use terms in three broad categories: interoperability, standardization, and measurement and assessment. Each of these fields is detailed in its own right, so we will limit ourselves to key points.

NATO views interoperability in terms of three dimensions: technical, procedural, and human. Technical interoperability refers to systems and equipment, such as communication and information systems, and their ability to operate together. For example, if French forward observers cannot provide Spanish artillery units location data regarding enemy formations in a format and method that is usable, missed shots or, much worse, friendly fire could occur. Similarly, if Polish early-warning radar systems produce data that is unrecognizable to German missile defense systems, Allied troops could suffer catastrophic casualties from an enemy’s incoming missiles. Allied units in the field often cite interoperability in the technical dimension as the most challenging and persistent. In particular, the Alliance has long been plagued by a lack of sufficiently interoperable secure voice and data communications. Interoperability in sustainment and logistics—for example, common fuels—are also typically mentioned as significant and enduring.3

Procedural interoperability is based on measures such as common doctrine, procedures, and terminology. If different Allies have different expectations of who can authorize or disallow a military strike, an Allied operation could be either inappropriately blocked or improperly authorized. Similarly, if Dutch forces based in Rotterdam follow different procedures for loading, securing, and transporting military equipment headed east by rail to reinforce Allied defenses, Polish forces off-loading the equipment could experience inefficiencies, confusion, or injury.

Human interoperability concerns concepts such as mutual trust and understanding that are achieved by developing and strengthening relationships during training and operations. If troops do not know either of NATO’s two official languages (English and French) well enough to convey critical information to each other, Allied units cannot operate side by side in the field or in a common headquarters, where basic communication is vital. Human interoperability challenges can take more pernicious forms as well—for example, whether and how subordinates can raise concerns with orders. Interoperability in the human dimension—manifest, for example, through liaison officers or relationships built through shared experiences—can reduce friction created by more limited interoperability in the technical and procedural dimensions.4

The Alliance identifies several levels of interoperability, from 0 (not at all interoperable) to 3 (fully integrated). At level 3, forces operate together effectively without technical, procedural, or human barriers. This level is characterized by common networks, capabilities, procedures, and language. Level 2 is referred to as compatible and exists when forces operate together without prohibitive technical, procedural, or human barriers. It is characterized by similar or complementary processes and procedures. Level 1 is referred to as deconflicted and exists when forces operate in the same operational area in pursuit of a common goal but with limited interaction due to prohibitive technical, procedural, or human barriers.

While often a prerequisite for interoperability, standardization is a major concern for the Alliance, with several different offices and committees dedicated to the topic. At the highest level of formality, nations may enter a standardization agreement (or STANAG) by which they commit to implementing a specified standard to meet an interoperability requirement. Implementing standards is often necessary but not always sufficient to achieve interoperability, and, indeed, NATO has a separate set of three levels defined to measure standardization (compatibility at the lowest level, with interchangeability and commonality at the higher levels). Education, exercises, training, evaluations, lessons learned, cooperative programs, demonstrations, trials, and tests are all ways in which conformance to a standardization agreement can be evaluated and enhanced. Conducting evaluations and learning lessons is a fundamental component of military activity and, in particular, training for and conducting operations, yet measuring and assessing the outcomes of military operations is often problematic.5

Transitioning to the field of measurement, for assessments to be useful, the measures and means of collecting that information must be valid, reliable, and as free from biases as possible. That information, or data, can be in a quantitative or qualitative form. Quantitative data are measures of values or counts and are expressed as numbers; this type of data tends to have higher credibility among decisionmakers. Qualitative data describe qualities or characteristics and can be expressed though text, graphics, audio, or video recordings; they are often (incorrectly) viewed as subjective and of lower value than quantitative data. A critical challenge in evaluating interoperability is to craft a measurement and assessment framework that allows for sufficient quantitative measures and adequate qualitative detail while not being overwhelming or impractical for assessors to use in practice.

Finally, while people often intuitively perceive more as better, achieving a high level of interoperability may be unnecessary or even undesirable. At times, a lower level of interoperability will suffice to meet the desired strategic, operational, and tactical objectives. For example, for humanitarian aid and disaster relief, in a low-threat environment, a deconflicted level of interoperability may suffice across military forces. Further, in some humanitarian scenarios, including search and rescue, NATO members may need to deconflict with nontraditional allies and even adversaries (for example, we may want radios that are “compatible” in the standardization sense with our adversaries to communicate with them, while they almost certainly will not have “commonality”).

Interoperability and Standardization Efforts in NATO

NATO Allies perceive interoperability as a critical means to achieving operational effectiveness and efficiency. Reflecting this view, the Alliance has many highly developed interoperability entities and initiatives in certain areas, most often technical. For example, the NATO Defence Planning Process is one of the primary sources for top-down interoperability requirements. Its task force on interoperability coordinates Alliance-wide efforts to ensure a holistic, consistent, and coherent approach to interoperability. It also ensures interoperability requirements and solution development are addressed coherently and harmonized with supporting measures in the NATO Standardization Organization Programme of Work. The program is the Military Committee’s lead agent for the development, coordination, and assessment of operational standardization. As noted, standardization is often a prerequisite for interoperability, and the bodies pursuing each goal have substantial overlap in objectives. In addition to the NATO Standardization Organization, NATO has several Military Committee Standardization Boards: Joint, Land, Maritime, Air, Medical, and Terminology.6

Elsewhere in NATO, the Supreme Headquarters Allied Powers Europe writes an annual report on standardization and interoperability that identifies shortfalls but is not prescriptive in how to address them. NATO’s Joint Analysis and Lessons Learned Centre publishes a lessons-learned handbook and maintains the NATO Lessons Learned Portal, “NATO’s single tool for collecting, managing, tracking, monitoring, and sharing of lessons.” While several hundred entries in this database are tagged for interoperability, there is no structured way to pull out the observations and use them. Notably, the lessons are essentially crowdsourced and based on observed opportunities for improvement, meaning reports tend to emphasize what is not working rather than what is working well, while a good assessment tool should address both.7

Allied Command Transformation conducts various events through which standardization can be tested. For example, the Coalition Warrior Interoperability Exercise allows Allies and partners to focus on command-and-control aspects of interoperability and to test interoperability specifications that are hardwired into experimental and near-fielded capabilities. The Allied Command Transformation also conducts TIDE Sprints and Hackathons, weeklong biannual events focused on furthering NATO and partner nations’ command-and-control and information technology interoperability.8

Finally, in 2014, the Alliance started a Partnership Interoperability Initiative, which provides a standing format for NATO-partner cooperation on interoperability and related issues. Its centerpiece, the Interoperability Platform, “brings together all partners that have contributed to NATO operations or have taken concrete steps to deepen their levels of interoperability with NATO. . . . Currently, 21 partners are members.”9

Despite all the above activities and initiatives, the Alliance lacks the ability to measure, assess, and communicate the state of interoperability across all three dimensions and across all major domains in a way that is easily understood, comparable across time and nations, and actionable for improvement.

Interoperability Assessment: Among the Allies

In addition to Alliance-wide initiatives, some NATO members have developed their own initiatives in the fields of interoperability data standards or assessment tools. For example, Canada assesses elements of cross-service, interagency, and multinational interoperability based on the Canadian Joint Task List, which describes all activities that may be performed by a Joint headquarters during the development, coordination, and conduct of operations and training. The task list is organized under five operational functions: Command, Sense, Act, Shield, and Sustain.10

Similarly, the British Army’s Future Force Development organization developed the Multinational Interoperability Assessment Tool (MIAT) to measure interoperability between the British Army and its partners. This question-based tool uses the American, British, Canadian, Australian, and New Zealand (ABCANZ) Armies’ program’s critical question lists (or CQLs), grouped by ABCANZ-defined priority focus areas along three interoperability levels (deconflicted, compatible, and integrated). The tool was designed for self-assessment and bilateral assessment at battlegroup, brigade, and higher echelons among armies and land forces. Although MIAT was developed and tested, resource constraints halted further development.11

At the same time, the US Army built its own tool—the Army Interoperability Measurement System. The AIMS tool has four components:

  1. Sets of multiple choice, select all, and dichotomous questions broken out by priority focus area, such as command and control, fires, or intelligence, surveillance, and reconnaissance. The responses to each question are completed by trained observers and correspond to the defined levels of interoperability.
  2. A form that allows for the collection of observations or notes for clarification.
  3. A Microsoft Excel–based dashboard whereby questions from the first component are assigned a score and an aggregated interoperability score is given for each priority focus area.
  4. An exploitation panel, in which the US Army’s Mission Command Center of Excellence analyzes the results of the first three components, identifies interoperability deficiencies, and offers participating nations recommendations on how to improve.

Interoperability Assessment: Others

In addition to the Alliance and individual Allies, other entities have also attempted to build interoperability data standards or assess and measure interoperability. For example, the ABCANZ Armies’ program focuses specifically on interoperability. A key activity of ABCANZ in recent years was its development of a critical questions list, designed to analyze interoperability levels within a training environment. This list laid the foundation for the question lists used in AIMS and MIAT.12

Aside from national and intergovernmental bodies, other public and private-sector entities have tried to build interoperability measurement and assessment tools. Developers of interoperability assessment models tend to address primarily—and sometimes only—technical interoperability, however, as it is more straightforward to accomplish, and disregard the other dimensions due to the difficulty in measuring them. One exception to this tendency is the enterprise interoperability measurement approach. This model addresses multiple dimensions of interoperability and “categories of barriers” to interoperability—conceptual, technological, and organizational—which correspond, to some degree, to NATO’s dimensions of interoperability (human, technical, and procedural).13

Another option that addresses nearly all interoperability aspects is the maturity model for enterprise interoperability. It differs from many of the military tools outlined above in that it evaluates an organization’s hypothetical potential to interoperate with partner organizations before those partners are known.14

Nonetheless, a systematic, repeatable, empirically based measurement and assessment tool and the underlying data standards that would make such a tool useful across nations and domains are still missing. Developing from scratch common data standards and, ultimately, a common tool for Allied interoperability measurement and assessment across the major domains could prove time consuming and difficult. It would arguably be more efficient and more effective to adopt an existing data standard or an existing measurement and assessment tool—with necessary adaptation—to achieve NATO’s objectives. Given development by the US Army of the AIMS tool and its familiarity to the British and Canadian armies, NATO could consider adopting it. Whether NATO adopts an existing tool as a de facto standard or develops a de jure standard, users must agree on what is to be measured, and experts must validate that the tool measures what it intends to measure.15

Use Cases for NATO Interoperability Assessment

To determine whether any potential data standard, method, or tool for measurement or assessment of interoperability, such as AIMS, could meet Alliance requirements, it is important to have a sense of the types of operations or activities the Alliance engages in. The required level of interoperability is necessarily context specific. For example, the requirements for Canada and the United Kingdom to interoperate in a multinational army training mission differ from the requirements of operating together in a Standing NATO Maritime Group (SNMG) and differ again when they are operating as framework nations in enhanced Forward Presence (eFP) battlegroups in neighboring nations.

For the purposes of this article, we will outline one use case from each major military operational domain—air, land, and maritime.

Enhanced Forward Presence in Eastern Europe

In response to Russia’s first post–Cold War invasion of Ukraine in 2014, NATO enhanced its presence in the eastern part of the Alliance, with four multinational battlegroups in Estonia, Latvia, Lithuania, and Poland. In 2022, after Russia unleashed its war on Ukraine, the Alliance expanded the initiative to field battlegroups in Bulgaria, Hungary, Romania, and Slovakia. A framework nation leads each battlegroup—Canada, the Czech Republic, France, Germany, Hungary, Italy, the United Kingdom, and the United States, respectively. The eFP consists of a range of deployed multinational forces at high readiness in a high-threat environment with climatic extremes. These forces have a wide range of units, varying in size and capability, from more than two dozen Allies.16

Interoperability between each multinational battlegroup, its subordinate units, host nation military forces, and the NATO command structure is critical. The focus areas for interoperability are command and control; communications; intelligence, surveillance, and reconnaissance; fires; and sustainment. We assume all units involved must be at an integrated level of interoperability in each of the focus areas. The greater the degree of interoperability, the more effective the battlegroups will be in terms of force preparation, assurance of readiness, assurance of capability, deterrence of Russia, and reassurance to the host nations.

Standing NATO Maritime Groups

Each of the two Standing NATO Maritime Groups generally consists of four to six destroyers and frigates with associated support vessels but is scalable depending on the nature of the operation. They are under the control of Allied Maritime Command, based in Northwood, United Kingdom, which reports to Allied Command Operations in Mons, Belgium. Although they are standing deployed forces, their leadership and membership is rotational.17

We assume contributing nations need to be at an integrated level of interoperability between the SNMG headquarters and Allied Marine Command, within the SNMG task group, and between the individual assets, as the task groups provide NATO with an immediate maritime operational response capability. The focus areas for interoperability, in this case, are doctrine; terminology; command and control; communications; intelligence, surveillance, and reconnaissance; fires; and sustainment. The stronger the degree of interoperability within the task groups, the better their readiness, assurance of capability, credibility in terms of deterrence, and force and capability development will be.

Baltic Air-Policing Mission

The Alliance’s Baltic Air-Policing mission has been ongoing since 2004. NATO air policing is a peacetime mission designed to maintain the security of Alliance airspace. Fighter aircraft and crews are kept at a high level of readiness, prepared to respond quickly to any violations of Allied airspace. NATO air-policing forces also assist civilian aircraft in distress. Baltic Air Policing is part of the NATO Integrated Air and Missile Defence System. Allied Air Command at Ramstein Air Base in Germany, which reports to Allied Command Operations, exercises operational control. Two regional Combined Air Operations Centres exercise tactical command and control.18

Allies with the requisite air-defense capabilities contribute aircraft and crews to the Baltic Air-Policing mission on a rotating basis, typically changing every four months. The deployment of NATO fighter aircraft to Šiauliai Air Base in Lithuania established the capability for the mission in the Baltic states. Since 2014, NATO has also been using Ämari Air Base in Estonia for the deployment of additional air-policing assets. Baltic Air Policing consists of a deployed national force in Lithuania or Estonia at high readiness in a high-threat environment. The force operates under international aviation regulations and a NATO mission mandate and doctrine. Allied and partner nations need interoperability among fighters; air surveillance and control systems; the Combined Air Operations Centre in Uedem, Germany; deployable air command-and-control systems; and NATO and national airborne early-warning forces.

The focus areas for interoperability in the Baltic Air-Policing mission are command and control; communications; intelligence, surveillance, and reconnaissance; fires; and sustainment. We assume the interoperability ambition required for Allies and partners involved in this mission must be at an integrated level for each focus area. Ideally, increased interoperability in this mission will lead to improved force preparation, assurance of readiness, assurance of capability, deterrence of Russia, and reassurance to the three Baltic states.

Could the US Army Interoperability Measurement System Satisfy These Requirements?

In light of the use cases outlined above, AIMS is the most mature and applicable tool available that could be adapted to meet NATO’s needs for joint interoperability assessment, given that MIAT is no longer under development. There are, however, significant practical and conceptual challenges to adapting this tool.

Practical considerations include issues pertaining to ownership of, and nomenclature within, the system; adaptation for peer-to-peer relationships; and the echelons of interest. Currently, the US Army maintains control of the tool and the ability to accept or reject any proposed changes or extensions to the question set and the assessment tool. This control is a potential barrier for its use in NATO standardization efforts since the system could change without NATO input or, conversely, may not be changed in ways NATO deems necessary or relevant. A NATO-relevant system would, at minimum, need to adapt the US focus and terminology. Since some use cases involve NATO acting with partners outside the Alliance, or NATO members acting with each other outside the NATO Command Structure, this terminology should be deliberately broad.

A further practical issue is that many AIMS questions are phrased in terms of one-to-one (or, in some cases, one-to-many) relationships between a lead nation—implicitly, the United States, in the current version of AIMS— and another party. Although this structure may map reasonably well onto NATO task forces reporting to a higher NATO headquarters, or troop-contributing nations within a framework-nation construct, it does not map well onto peer-to-peer relationships within task groups (where communicating through the headquarters is not always practical) or peer components within a command structure. In reality, battlegroups may report to NATO headquarters on some issues and to the host nation or their home nations on others. What further confuses the issue is that although AIMS is nominally designed for assessing bilateral interoperability, some questions and responses refer to partners in the plural.

A final practical challenge is that while in the SNMG use case national boundaries remain relatively distinct—in essence, separate ships operating in a task group—in the eFP use case, the multinational integration is at a much lower organizational level. Some NATO members, and participants in various NATO partnership structures, will need to assess their interoperability at a different organizational scale than that for which AIMS is presently designed.

In addition to the practical issues identified above, there are a wider variety of conceptual challenges to adapting AIMS to a NATO joint context. Some of these requirements include adaptation to other environments and services, more complete coverage of the interoperability dimensions, consistent question wording and question scoring, and appropriate data aggregation. A conceptual question that also influences the practical is the required frequency and sequencing of opportunities to conduct assessments.

Tackling the first conceptual requirement—the extension of the question set used in AIMS beyond the land domain to the air, maritime, and other domains—will require going beyond replacing terms and jargon with their rough equivalents in other contexts. For example, several questions in AIMS do not address the nuance that naval vessels and military aircraft often operate at various electronic emissions control levels where connectivity is intentionally limited. Observation-based systems conducted at specific points in exercises or operations may, therefore, be unable to evaluate these connections or inaccurately score certain links as not having been established.

Second, to remedy the preponderance of the AIMS questions assessing the technical dimension, more procedural and human-dimension questions should be added, with some consideration given to reducing the number of technical questions that may have conceptual overlap. There is also an implicit element in many questions where the highest-rated option has a technical solution and working human-to-human is considered a lower level of interoperability. Further confusing the issue, some questions in the knowledge and information management set require the exchange of liaison officers to receive the highest rating, whereas, in most questions relying on liaison officers the result is a lower score.19

Third, questions should, to the extent possible, be written clearly using plain language. Technical terms and jargon must be avoided, and words like interoperable or integrated should not be used in the questions or responses, to avoid circularity of definitions. Also, in places where a specific US Army regulation, procedure, system, or other similar US Army–centric terminology is used, this type of reference would need to be generalized or at least refer to a NATO, or Joint, equivalent.

Fourth, most AIMS questions have four ordinal response levels, which one might assume correspond to the definitions of the four levels of interoperability (for example, a score of 3 on a question would represent integrated interoperability). Some questions, however, have only three levels, some questions are nominal and of the form “yes,” “no,” and “N/A,” and still others are multiple response (such as “check all that apply”). These different types of responses are not co-measurable, which implies they should not be added or averaged together. This prohibition does not mean that different types of questions should not be asked, only that they cannot be mathematically aggregated (as is done within AIMS).20

Fifth, the current interpretation of data aggregation in AIMS would benefit from being formally documented. The overall score in AIMS is generally de-emphasized, but as long as such aggregation is done and presented to decisionmakers, it must be logical, clear, and, most importantly, correct. Given also that not all responses map clearly onto the definitions of the levels of interoperability, that the questions are not balanced across the dimensions of interoperability, and that questions may be dropped from any given assessment, score distributions should be provided, rather than attempting an overall aggregation.

Finally, with respect to data collection, observers and assessors will likely fulfill multiple duties or functions than specialize in data collection or assessments.

This reality has implications for how complicated the tool can be—or how much time is required for training. Currently, AIMS training seems to focus more on the mechanics of entering responses into the tool and less on how to deal with issues of inter-rater reliability and consistency. On a related point, there are also data collection issues pertaining to trust. Individuals and nations may change their behavior based on the knowledge they are being assessed or evaluated and may resist being formally scored if they know or believe the ratings will reflect negatively. Although the developers of AIMS are clear that it is a measurement system rather than an assessment or evaluation system, this nuance will not necessarily assuage concerns. More specifically, although evaluating command relationships is critical to understanding effectiveness in the human dimension, commanders sometimes have difficulty accepting this type of assessment, different nations may delegate more decision-making authority to lower levels, and NATO members may measure Ally-to-Ally relations differently than Ally-to-partner relations, where trust may necessarily remain lower. Lastly, even though AIMS questions and responses are captured and processed in Microsoft Excel, which can save to common formats, the formal output of AIMS is presently in the form of a flat slide deck in Microsoft PowerPoint or PDF. These types of formats do not allow for easy storage, processing, and comparison of results across multiple events.21

To be clear, a question-based instrument can form at least part of an effective multinational interoperability assessment, though other forms of data collection are possible and could (and likely should) be combined. Assuming a common data format can be developed, it is also likely unnecessary to ask certain questions every time a given combination of nations or units interoperate. This avoidance of unnecessary repetition is perhaps particularly true in the procedural dimension, where the adoption of common or compatible procedures is a relatively slow-moving, formalized process that can be documented outside any specific exercise or operation.

Conversely, customs can vary within—and between—nations and services and between individuals, suggesting responses in the human domain are likely to vary more between, and within, events. Aspects of the technical domain may also lend themselves more to expert evaluation while systems are being established rather than (or in addition to) having them observed indirectly. However the level of interoperability is assessed, it must be clear how response levels on each question correspond to the definitions of the levels of interoperability.

Given ever-present resource constraints, usability and simplicity must be considered throughout the design, implementation, and training for any interoperability measurement or assessment system. Although the existing AIMS question set could be adapted to at least the NATO land domain with relatively minimal changes, more significant changes to the question content and balance would be needed to adapt it to other domains and the joint context.

Toward a NATO Standard Method for Assessing Multinational Interoperability

As discussed in the previous section, it may be unfeasible to take an interoperability measurement and assessment tool off the shelf and incorporate it directly into NATO processes without significant adaptation. Moreover, while standardizing an extant tool may seem like less work, it may leave parties vulnerable to the whims and preferences of the tool developers, potentially leading to incompatible version upgrades, increased costs, and limited user support.

An alternative approach may be to define a common standard, allowing various nations to bring their interchangeable tools to the table. In this way, NATO could benefit from a common interoperability assessment method or process without resorting to the central development of a common tool. Standardization of the data format can allow for interoperability assessment tools to be interoperable (interchangeable, in the language of NATO standardization) without requiring the adoption of single tool or system (common, in the language of NATO standardization).

Data Collection

If NATO were to pursue this option, several elements or processes must be considered, foremost among them data collection. As discussed, multiple potential collection methods exist. Given that a question set is likely to form at least part of the interoperability assessment, however, it is worth standardizing how these question sets are defined and stored. The specific questions that are relevant may change over time and may be different between nations, organizations, and services. The key, therefore, is to allow for flexibility while recording sufficient metadata to understand what was asked, how, and when. The questions could form fields in a table stored in a standardized file format. Finally, there is often a gap in training in data collection and assessment for those tasked to conduct it, particularly with how issues such as inter-rater reliability and consistency can and should be managed.

Data Storage and Sharing

Data storage and sharing are fundamental to addressing the lack of a uniform, repeatable, and structured method to document interoperability. While one or more tools could be used to collect responses to a question, these tools should be capable of a) loading questions specified in a standardized format, and b) writing the responses in a standardized data format. Ideally, the data format would be human readable and machine readable and should be developed in accordance with applicable NATO metadata standards. The NATO metadata standard covers general issues such as security classification and copyright while providing guidance to communities of interest on how to develop their own specific standards.

Data Analysis and Visualizations

Assuming a standard is produced for a common storage format, NATO organizations, nations, or individual analysts can then create data analyses and visualizations of the data to suit the needs of their supported commander or other relevant authority. Nations and organizations may use their own tools to collect data, their own questions, and their own categorizations, but having a standard data format will enable an Ally or NATO body to load and analyze data easily from multiple events with their tool of choice and analyze interoperability differences over space and time. Although visualizations of quantitative responses may be intuitively appealing, they must be supplemented by contextual qualitative information and a synthesis of expert judgment. Tools can be built to facilitate drilling into which questions had lower scores, aiding discussion on what problems need to be resolved. In situations where a given mission depends critically on the ability to send digital calls for fire from one nation’s ground troops to another nation’s aircraft, the planners and commanders may value the answers to questions about the specific network links and procedures required more than what the overall scores may be.

Conclusion

NATO needs a standard for interoperability data (including definition, collection, and management), which would help military planners understand and discuss their state of interoperability with their Allies and partners. The good news is that NATO nations have broad agreement on the four levels of interoperability across the major domains, if not their detailed definitions. Likewise, there is broad agreement that there are three dimensions to interoperability across the major domains—technical, procedural, and human. There is less agreement, however, on how to categorize interoperability information and how those categories are best defined. Assessment methods and tools must, therefore, be flexible to these categories if they are to be broadly useful.

In considering assessment methods and tools for NATO, it is logical to first review and consider any existing methodologies, processes, and tools among international organizations, Allies, and the private sector. Of all the options considered in this article, the most promising, for various reasons, may be AIMS. Practical and conceptual challenges to NATO’s adoption of AIMS remain, however, and NATO would need to devote personnel and other resources to tailoring or adapting AIMS to the variety of Alliance operations, domains, and activities that would benefit from greater interoperability.

With respect to creating a NATO standard for interoperability information, specific data elements for questions and responses would need to be stored in a standardized file format with appropriate metadata. This form of processing would remove the dependence on any specific tool or system, allowing nations and organizations to develop them to their own needs and interests. It would also greatly simplify the ability to conduct interoperability comparisons across space and time.

 

Authors’ Note: This article draws on the authors’ participation in a NATO Science and Technology Office coordinated project (SAS-156) on interoperability measurement and assessment. The issues the authors address in this article differ somewhat from the issues the research task group examined and do not necessarily reflect the consensus views of all the research task group members. The task group’s final report is available to NATO members and partners at https://www.sto.nato.int/publications/Pages/default.aspx.

 
 

John R. Deni
Dr. John R. Deni is a research professor of security studies at the US Army War College, a nonresident senior fellow at the Atlantic Council, and a nonresident senior fellow at the NATO Defense College.

Matthew R. MacLeod
Mr. Matthew R. MacLeod is a defense scientist with the Centre for Operational Research and Analysis, Department of National Defence, Canada, in which capacity he has been embedded with multiple organizations in the Canadian Armed Forces. He served as the secretary of NATO Research Task Group SAS-156. He currently leads a team of scientists working with the Canadian Forces Health Services.

Sarah E. Stewart
Ms. Sarah E. Stewart is an operations research and systems analyst at the Center for Army Analysis; she was the chair of the NATO Research Task Group SAS-156 study Developing a Standard Methodology for Assessing Multinational Interoperability. She is the 59th president of the Military Operations Research Society.

Katherine M. Banko
Dr. Katherine M. Banko is a defense scientist with the Centre for Operational Research and Analysis, Department of National Defence, Canada; she has been embedded with multiple organizations in the Canadian Armed Forces and has expertise in operations measurement and assessment. She served as the vice chair of NATO Research Task Group SAS-156, initially while working with the Canadian Joint Operations Command. She currently leads a team of scientists working with the Royal Canadian Navy.

Adrian Jones
Lieutenant Colonel Adrian Jones is a serving British Army officer with more than 40 years of experience in a wide range of command, staff, and training posts. He is currently the UK Interoperability Policy lead in the Futures Directorate in the UK Army Headquarters.

 
 

Disclaimer: Articles, reviews and replies, review essays, and book reviews published in Parameters are unofficial expressions of opinion. The views and opinions expressed in Parameters are those of the authors and are not necessarily those of the Department of Defense, the Department of the Army, the US Army War College, or any other agency of the US government. The appearance of external hyperlinks does not constitute endorsement by the Department of Defense of the linked websites or the information, products, or services contained therein. The Department of Defense does not exercise any editorial, security, or other control over the information you may find at these locations.

 
 

Endnotes

  1. “NATO Takes Measures to Reinforce Collective Defence, Agrees on Support for Ukraine,” NATO, April 1, 2014, https://www.nato.int/cps/en/natolive/news_108508.htm; Brussels Summit Communiqué, NATO Press Release (2021) 086, June 14, 2021, https://www.nato.int/cps/en/natohq/news_185000.htm; Madrid Summit Declaration, NATO Press Release (2022) 095, June 29, 2022, https://www.nato.int/cps/en/natohq/official_texts_196951.htm. Return to text.
  2. For example, see James Derleth, “Enhancing Interoperability: The Foundation for Effective NATO Operations,” NATO Review, June 2015, https://www.nato.int/docu/review/articles/2015/06/16/enhancing-interoperability-the-foundation-for-effective-nato-operations/index.html; “Standardization,” NATO, What We Do, last updated October 14, 2022, https://www.nato.int/cps/en/natohq/topics_69269.htm; and “Interoperability: Connecting Forces,” NATO, What We Do, last updated April 11, 2023, https://www.nato.int/cps/en/natohq/topics_84112.htm. Return to text.
  3. NATO Standardization Office, Allied Joint Doctrine, Allied Joint Publication (AJP) 01, ed. F, ver. 1 (NATO Standardization Office, 2022); Frank Gubbels, “NATO’s Interoperability Challenge: Is FMN on Its Own?,” in Annual Overview 2022: NATO Command and Control Centre of Excellence (NATO Command and Control Centre of Excellence, 2023), 44–62, https://c2coe.org/download/natos-interoperability-challenge-is-fmn-on-its-own/; and Rodney Fogg et al., “Interoperability: Embrace It or Fail!,” Army Sustainment 52, no. 1 (January-March 2020): 7–11, repr. U.S. Army, February 10, 2020, https://www.army.mil/article/231653/interoperability_embrace_it_or_fail. Return to text.
  4. NATO Research and Technology Organisation (RTO), Multinational Military Operations and Intercultural Factors, NATO RTO Technical Report, TR-HFM-120, ed. Angela R. Febbraro et al. (November 2008), https://www.researchgate.net/publication/303311391_Multinational_Military_Operations_and_Intercultural_Factors. Return to text.
  5. NATO Standardization Office, Allied Joint Doctrine; Deputy Secretary General, NATO Interoperability Policy, C-M (2009) 0145 (NATO, 2009); Jonathan Schroden, “Why Operations Assessments Fail— It’s Not Just the Metrics,” Naval War College Review 64, no. 4 (Autumn 2011): 89–102; Delbert S. Joo, “A Systems Analysis of U.S. Army Operation Assessments” (master’s thesis, MIT, 2008); and Stephen Downes-Martin, “Operations Assessment in Afghanistan Is Broken—What Is to Be Done?,” Naval War College Review 64, no. 4 (Autumn 2011): 103–25. Return to text.
  6. NATO Public Diplomacy Division, Backgrounder: Interoperability for Joint Operations (NATO Public Diplomacy Division, July 2006), https://www.nato.int/nato_static_fl2014/assets/pdf/pdf_publications/20120116_interoperability-en.pdf; and Supreme Allied Commander Europe, Military Operational Standardization in Support of Interoperability Within ACO: Role and Responsibilities, ACO Directive 015-023 (Supreme Headquarters Allied Powers Europe [SHAPE], 2021). Return to text.
  7. SHAPE, NATO Evaluation System Policy, ACO Directive 075-013 (NATO, 2021); and Joint Analysis and Lessons Learned Centre (JALLC), The NATO Lessons Learned Handbook, 4th ed. (NATO/JALLC, 2022). Return to text.
  8. “The Coalition Warrior Interoperability Exercise Showcases NATO Interoperability to Allied Decision Makers,” NATO Allied Command Transformation (ACT), June 20, 2023, https://www.act.nato.int/article/cwix-showcases-interoperability-to-decision-makers/; and “Federated Interoperability,” ACT, n.d., accessed July 21, 2023, https://www.act.nato.int/activities/federated-interoperability/. Return to text.
  9. “Partnership Interoperability Initiative,” NATO, accessed April 25, 2023, last updated March 7, 2024, https://www.nato.int/cps/en/natohq/topics_132726.htm. Return to text.
  10. Canadian Joint Operations Command, CAF Joint Task List for 2021 (Canadian Armed Forces, 2020). Return to text.
  11. 4C Strategies, Multi-National Interoperability Assessment Tool, 2019, Presentation to NATO. Return to text.
  12. Bryan W. Hallmark et al., An Analysis of Alternative Approaches to Measuring Multinational Interoperability: Early Development of the Army Interoperability Measurement System (AIMS) (RAND Corporation, 2021), 30, 36. Return to text.
  13. David Chen et al., “An Approach for Enterprise Interoperability Measurement” (paper in the Proceedings of the International Workshop on Model Driven Information Systems Engineering: Enterprise, User and System Models [MoDISE-EUS’08], Montpellier, France, June 16–17, 2008). Return to text.
  14. Wided Guédria et al., “A Maturity Model for Enterprise Interoperability,” in On the Move to Meaningful Internet Systems: OTM 2009 Workshops (Springer Berlin, Heidelberg, 2009). Return to text.
  15. Carl F. Cargill, “Why Standardization Efforts Fail,” Journal of Electronic Publishing 14, no. 1 (Summer 2011), https://doi.org/10.3998/3336451.0014.103. Return to text.
  16. “NATO’s Military Presence in the East of the Alliance,” NATO, accessed May 16, 2024, last updated February 12, 2025, https://www.nato.int/cps/en/natohq/topics_136388.htm. Return to text.
  17. “NATO’s Maritime Activities,” NATO, last updated August 3, 2023, https://www.nato.int/cps/en/natohq/70759.htm. Return to text.
  18. “NATO Air Policing,” accessed May 16, 2024, last updated August 1, 2024, https://www.nato.int/cps/en/natohq/topics_132685.htm. Return to text.
  19. U.S. Army, AIMS Instrument for Use with KM/IM PFA and Echelon Type(s): Corps, Div, Bde / Regiment (2020). Return to text.
  20. Matthew R. MacLeod and Katherine M. Banko, A Canadian Perspective on Military Assessments: Best Practices and Considerations for the Canadian Joint Operational Command, Scientific Report, DRDC-RDDC-2021-R030 (Defence Research and Development Canada, March 2021). Return to text.
  21. Although NATO and national training exist for operations assessment, the number of available seats in these courses is insufficient to meet the demand generated by frequent posting cycles and the number of operations NATO and its nations conduct. See MacLeod and Banko, Canadian Perspective on Military Assessments. Return to text.