Anesthetic Agent Monitor

Abstract

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broad- ening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the grow- ing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches.

In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learn- ing systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi- dimensional space, we discover interesting trends. We also propose some novel directions for de- signing future Comm-MADRL systems through exploring possible combinations of the dimensions.

Keywords Multi-Agent Reinforcement Learning · Deep Reinforcement Learning · Communication · Survey

Introduction

Many real-world scenarios, such as autonomous driving , sensor networks , robotics and game-playing [4, 5], can be modeled as multi-agent systems. Such multi-agent systems can be designed and developed using multi-agent reinforcement learning (MARL) techniques to learn the behavior of individual agents, which can be cooperative, competitive, or a mixture of them. As agents are often distributed in the environment where they only have access to their local observations rather than the complete state of the environment, partial observability becomes an essential assumption in MARL [6, 7, 8]. Moreover, MARL suffers from the non-stationary issue , since each agent faces a dynamic environment that can be influenced by the changing and adapting policies of other agents. Communication has been viewed as a vital means to tackle the problems of partial observability and non-stationary in MARL. Agents can communicate individual information, e.g., observations, intentions, experiences, or derived features, to have a broader view of the environment, which in turn allows them to make well-informed decisions [9, 10].

Due to the recent success of deep learning and its application to reinforcement learning , multi-agent deep reinforcement learning (MADRL) has witnessed great achievements in recent years, where agents can process high- dimensional data and have generalization ability in large state and action spaces [7, 8]. We notice that a large number

Arxiv:2203.08975V2 [Cs.Ma] 18 Oct 2024

of research works focus on learning tasks with communication, which aim at learning to solve domain-specific tasks, such as navigation, traffic, and video games, by communicating and sharing information. To the best of our knowledge, there is a lack of survey literature that can cover recent works on learning tasks with communication in multi-agent deep reinforcement learning (Comm-MADRL). Early surveys consider the role of communication in MARL but as- sume it to be predefined rather than a subject of learning [13, 14, 15]. Most Comm-MADRL surveys cover only a small number of research works without proposing a fine-grained classification system to compare and analyze them.1 In cooperative scenarios, Hernandez-Leal et al. use learning communication to denote the area of learning com- munication protocols to promote the cooperation of agents.2 The only survey that we found classifying some early works in Comm-MADRL is from Gronauer and Diepold , which is based on distinguishing whether messages are received by all agents, a set of agents, or a network of agents. However, other aspects of Comm-MADRL, such as the type of messages and training paradigms, which are essential for communication and can help characterize existing communication protocols, are ignored. As a result, the reviewed papers in recent surveys regarding learning tasks with communication are rather limited and the proposed categorizations are too narrow to distinguish existing works in Comm-MADRL. On the other hand, there is a closely related research area, emergent language/communication, which also considers learning communication through various reinforcement learning techniques . Different from Comm-MADRL, the primary goal of emergent language studies is to learn a symbolic language.3 However, a subset of emergent language research works pursues an additional goal to leverage learnable symbolic language to enhance task-level performance. Notably, these research works have not been encompassed within existing Comm-MADRL surveys but included in our survey, referred to learning tasks with emergent language. In summary, our survey overlaps in scope with surveys of emergent language (i.e., in learning tasks with emergent language), but our survey focuses on different primary goals (i.e., achieving domain-specific tasks rather than learning a symbolic language). We further clarify the differences between learning tasks with communication and emergent language in Section 2.2.

In our survey paper, we review the Comm-MADRL literature by focusing on how communication can be utilized to improve the performance of multi-agent deep reinforcement learning techniques. Specifically, we focus on learn- able communication protocols, which are aligned with recent works that emphasize the development of dynamic and adaptive communication, including learning when, how, and what to communicate with deep reinforcement learning techniques. Through a comprehensive review of recent Comm-MADRL literature, we propose a systematic and struc- tured classification methodology designed to differentiate and categorize various Comm-MADRL approaches. Such a methodology will also provide guidance for the design and advancement of new Comm-MADRL systems. Suppose we plan to develop a Comm-MADRL system for a domain task at hand. Starting with the questions of when, how, and what to communicate, the system can be characterized from various aspects. Agents need to learn when to com- municate, with whom to communicate, what information to convey, how to integrate received information, and, lastly, what learning objectives can be achieved through communication. We propose 9 dimensions that correspond to unique aspects of Comm-MADRL systems: Controlled Goals, Communication Constraints, Communicatee Type, Commu- nication Policy, Communicated Messages, Message Combination, Inner Integration, Learning Methods, and Training Schemes. These dimensions, which form the skeleton of a Comm-MADRL system, can be used to analyze and gain insights into designed Comm-MADRL approaches thoroughly. By mapping recent Comm-MADRL approaches into this multi-dimensional structure, we not only provide insight into the current state of the art in this field but also determine some important directions for designing future Comm-MADRL systems.

The remaining sections of this paper are organized as follows. In Section 2 the preliminaries of multi-agent RL are discussed, together with existing extensions regarding communication and a detailed comparison of recent surveys. In Section 3, we present our proposed dimensions, explaining how we group the recent works in the categories of each dimension. In Section 4, we discuss the trends that we found in the literature, and, driven by the proposed dimensions, we propose possible research directions in this research area. We finalize the paper with some conclusions in Section 5.

Background

In this section, we first provide the necessary background on multi-agent reinforcement learning. Then, we show how multi-agent reinforcement learning can be extended to consider communication between agents. Finally, we present 1We provide a detailed comparison of recent surveys on MADRL which involves communication in Section 2.3.

2In our survey, we extend the concept of learning communication to general multi-agent tasks and use the term learning tasks with communication to emphasize that the primary goal of recent research, which is centered on solving specific domain tasks through the use of communication.

3In the literature, emergent language and emergent communication are used interchangeably. In our survey, we use emergent language for referring to both terms.

2

and compare recent surveys involving communication, from which we can directly see our motivations to fill the gaps among existing surveys.

Multi-Agent Reinforcement Learning

Real-world applications often contain more than one agent that operate in the environment. Agents are generally assumed to be autonomous and required to learn their strategies for achieving their goals. A multi-agent environment can be formalized in several ways , depending on whether the environment is fully observable, how agents’ goals are correlated, etc. Among them, the Partially Observable Stochastic Game (POSG) [20, 21] is one of the most flexible

, Where I Is A (Finite) Set Of Agents

indexed as {1, ..., n}, S is a set of environment states, ρ0 is the initial state distribution over state space S, Ai is a set of actions available to agent i, and Oi is a set of observations of agent i. We denote a joint action space as A = ×i∈IAi and a joint observation space of agents as O = ×i∈IOi. Therefore, P : S × A →∆(S) denotes the transition probability from a state s ∈S to a new state s′ ∈S given agents’ joint action⃗a = ⟨a1, ..., an⟩, where⃗a ∈A. With the environment transitioning to the new state s′, the probability of observing a joint observation⃗o = ⟨o1, ..., on⟩(where⃗ o ∈O) given the joint action⃗a is determined according to the observation probability function O : S × A →∆(O).

Each agent then receives an immediate reward according to their own reward functions Ri : S × A × S →R. Similar to the joint action and observation, we could denote⃗r = ⟨r1, ..., rn⟩as a joint reward. If agents’ reward functions happen to be the same, i.e., they have identical goals, then r1 = r2 = ... = rn holds for every time step.

In this setting, the POSG is reduced to a Dec-POMDP . If at every time step the state is uniquely determined from the current set of observations of agents, i.e., s ≡⃗o, the Dec-POMDP is reduced to a Dec-MDP. If each agent knows what the true environment state is, the Dec-MDP is reduced to a Multi-agent MDP. If there is only one single agent in the set of agents, i.e., I = {1}, then the Multi-agent MDP is reduced to an MDP and the Dec-POMDP is reduced to a POMDP. Due to the partial observability, MARL methods often use the observation-action history τi,t = {oi,0, ai,0, oi,1, ..., oi,t} up to time step t for each agent to approximate the environment state. Note that time step t is often omitted for the sake of simplification.

In the multi-agent reinforcement learning setting, agents can learn their policies in either a decentralized or a cen- tralized fashion. In decentralized learning (e.g., decentralized Q-learning [22, 23]), an n-agent MARL problem is decomposed into n decentralized single-agent problems where each agent learns its own policy by considering all other agents as a part of the environment [24, 25]. In such a decentralized setting, the learned policy of each agent is conditioned on its local observation and history. A major problem with decentralized learning is the so-called non-stationarity of the environment, i.e., the fact that each agent learns in an environment where other agents are simultaneously exploring and learning. Centralized learning enables the training of either a single joint policy for all agents or a centralized value function to facilitate the learning of n decentralized policies. While centralized (joint) learning removes or mitigates issues of partial observability and non-stationarity, it faces the challenge of joint action (and observation) spaces that expand exponentially with the number of agents and their actions. For a deeper dive into various training schemes used in MARL, we recommend the comprehensive survey by , which offers valuable insights into the training and execution of policies. Based on whether policies are derived from value functions or directly learned, multi-agent reinforcement learning methods can be categorized into value-based and policy-based methods. Both methods have been largely utilized in Comm-MADRL.

Value-Based

Value-based methods in the multi-agent case borrow considerable ideas from the single-agent case. As one of the most popular value-based algorithms, the decentralized Q-learning learns a local Q-function for each agent. In the cooperative setting where agents share a common reward, the update rule for agent i is as follows:

Where R Is The Shared Reward, And A′

i is the action with the highest Q-value in the next state s′. In partially observable environments, the environment state is not fully observable and is usually replaced by the individual observation or history of each agent. The Q-values for each state-action pair are incrementally updated according to the TD

I Qi(S′, A′

i) −Qi(s, ai), represents the difference between a new estimate (i.e.,

I Qi(S′, A′

i)) and the current estimate (i.e., Qi(s, ai)) based on the Bellman equation . As the state and action space could be too large to be encountered frequently for accurate estimation, function approximation methods, like deep neural networks, have become popular for endowing value or policy models with generalization abilities across both discrete and continuous states and actions . For example, the Deep Q-network (DQN) minimizes the difference between the new estimate calculated from sampled rewards and the current estimate of a parameterized Q-function. In DQN-based methods, the Q-function in Equation 1 is notated as Qi(s, ai; θi), which

3

depends on learnable parameters θi. On the other hand, centralized learning in value-based methods learns a joint Q- function Q(s,⃗a; θ) with parameters θ. However, this approach can be challenging to scale with an increasing number of agents. Value decomposition methods [27, 28, 29, 30] are popular MARL methods that decompose a joint Q- function to enable efficient training. These methods are also widely employed in research works in Comm-MADRL [31, 32, 33]. In partially observable environments, linear value decomposition methods decompose history-based joint

(2)

where the joint Q-function is based on the joint history of all agents and is decomposed into local Q-functions based on individual histories. The weight wi can either be a fixed value [27, 29] or a learnable parameter subject to certain constraints . Advantage functions can also replace the Q-function in the above equation to reduce variance .

Policy-Based

Policy-based methods directly search over the policy space instead of obtaining the policy through value functions implicitly. The policy gradient theorem provides an analytical expression of the gradients for a stochastic policy with learnable parameters in single-agent cases. In the multi-agent case with centralized learning, the policy gradient theorem is expressed as follows: ∇θJ(θ) = E⃗a∼π(·|s),s∼ρπ[∇θ log π(⃗a | s; θ)Qπ(s,⃗a)]

(3)

where J(θ) represents the learning objective, and π(⃗a | s; θ) denotes a stochastic policy parameterized by θ (abbre- viated as π). Additionally, ρπ signifies the state distribution under the policy π, and ∇θJ(θ) represents the expected gradient with respect to all possible actions and states. Due to the computational intractability of the expected gradient, stochastic gradient ascent can be applied to update the parameters θ at every learning step l as follows:

Where Α Is The Learning Rate, And \

∇θJ(θ) is an estimate of the expected gradient based on sampled actions and states. Moreover, the Q-function in Equation 3 can be replaced by average returns over episodes to form REINFORCE algorithms , or by an estimated value function to form actor-critic algorithms [35, 36]. In actor-critic methods, the policy and value function are termed the actor and the critic, respectively. The critic will, therefore, guide the learning of the actor.

Actor-critic methods have undergone various adaptations for multi-agent environments [7, 8, 37, 38]. A typical ex- tension is the multi-agent deep deterministic policy gradient (MADDPG) . In MADDPG, the critic is a centralized Q-function designed to capture global information and coordinate learning signals. Meanwhile, the actors are local policies, ensuring decentralized execution. MADDPG assumes deterministic actors with continuous actions, allowing for the backpropagation of gradients from the value function to the policies. The gradient of each parameterized actor µθi(ai | oi) with learnable parameters θi, abbreviated as µi, is defined as follows:



where D is the experience buffer that contains joint observation-action tuples ⟨⃗o,⃗a,⃗r,⃗o′⟩. Each agent’s Q-function,

Denoted As Qµ

i (⃗o, a1, . . . , aN), takes joint observations and actions as inputs, while decentralized actors use local observations as inputs. Contrary to Equation 3, gradients with respect to the current action of agent i (specifically, µi(oi)) are utilized to guide the update of the policy parameter θi. Both MADDPG and its single-agent counterpart, DDPG, have seen widespread application in Comm-MADRL [39, 40, 41, 42, 43].

Extensions With Communication

In the MADRL literature where communication is used, we notice two closely related research areas, which we will re- fer to with the terms emergent language and learning tasks with communication. The emergent language research area [18, 44, 45, 46, 47] aims at learning a language grounded on symbols in communities of interacting/communicating agents. This line of research tries to understand the evolution of the language in agents equipped with neural networks.

On the other hand, learning tasks with communication [16, 48, 49, 50] focuses primarily on solving multi-agent rein- forcement learning tasks with the aid of communication. Communication is often regarded as information exchange rather than learning a (human-like) language. Despite the distinction, when using MADRL techniques on specific domain tasks, languages might emerge, which can potentially enhance the learning system’s explainability in accom- plishing those tasks. We illustrate the research areas, emergent language and learning tasks with communication, along with their intersection learning tasks with emergent language in Figure 1. Notably, our survey focuses on learn- ing tasks with communication in multi-agent deep reinforcement learning, including the intersection with emergent

Emergent Language

Figure 1: An illustration depicting the scope of this survey. The focus of our survey is represented by the blue part. language.4 Within this focus, multiple agents often operate in partially observable environments and learn to share information encoded through neural networks. Furthermore, communication protocols, determining when and with whom to communicate, often leverage deep learning models to find the optimal choices that minimize communica- tion overhead and yield more targeted communication. A multitude of works have been proposed to handle these subproblems inherent in Comm-MADRL. Most research works model only one or a few aspects of Comm-MADRL while selecting a default approach for other aspects. Given that the common goal of Comm-MADRL approaches is to design an effective and efficient communication protocol to improve agents’ learning performance in the environment, the proposed Comm-MADRL approaches inevitably share similarities to some extent. Consequently, establishing a classification system for Comm-MADRL becomes crucial. Such a system would aid in categorizing critical ele- ments like contributions, targeted problems, and learning objectives, from which we can compare and analyse existing Comm-MADRL approaches.

In the emergent language literature, numerous works employ various forms of the Lewis game, often referred to as ref- erential games and operate under a cheap-talk setting , as highlighted in several surveys [10, 18].5 In these games, a goal, often represented as a target location, an image, or a semantic concept, is given to a sender agent but remains unrevealed from a receiver agent. The receiver agent must then either identify the correct goal based on the sender’s signaling [52, 53, 54, 47, 55, 56, 57, 58] or accomplish its single-agent task using the received signals (messages) [59, 60]. Research works in learning tasks with emergent language are grounded in a multi-agent environment where the joint actions of both sender and receiver agents impact environment transitions. Consequently, the learning tasks with emergent language literature considers multi-agent domain tasks [61, 62, 63, 64, 65], building on foundational concepts from MARL such as Dec-POMDPs or POSGs.

We further distinguish explicit versus non-explicit communication in the literature of MADRL with communica- tion. Explicit communication refers to communication through a set of messages separate from domain-level actions. Here, agents’ action policies are influenced by both their observations and the messages they receive. Such messages, crucial for supporting agents’ decision-making, are essential in both the training and execution phases. MADRL frameworks without explicit communication can still allow for communication through domain-level actions, such as the act of influencing the observations of one agent through the actions of another. Furthermore, without explicit com- munication, agents can transmit gradient signals, which facilitate centralized training (and decentralized execution) but are not utilized during execution phases. Specifically, in our survey, we focus on explicit and learnable communication.

Dec-POMDPs and POSGs are often extended to accommodate explicit communication. The communication can be integrated into the action set, adding a collection of communication acts alongside domain-level actions. Al- ternatively, a Dec-POMDP or a POSG can be extended to explicitly include a set of messages .

For In-

stance, the POSG can be expanded with a (shared) message space M, resulting in a POSG-Comm, defined as

I, S, Ρ0, {Ai} , P, {Oi} , O, {Ri} , M

, where all components remain unchanged except for the added message space M. A Dec-POMDP-Comm can be defined as similar to the POSG-Comm with shared rewards. In both POSG- Comm and Dec-POMDP-Comm, action policies take into account both environmental observations and inter-agent messages. Research works in Comm-MADRL that expand upon a POSG or a Dec-POMDP can be seen in references such as [61, 63, 66, 65, 67].

4Throughout the remainder of our survey, Comm-MADRL will be used to specifically refer to the areas of our focus. 5In the emergent language research area, research works that do not adopt the cheap-talk setting but communicate through observable (domain-level) actions, are not included in our survey. Our survey focuses on explicit message transfer between agents.

Communication In Recent Surveys

Communication has attracted much attention in the field of multi-agent reinforcement learning (MARL). Previous surveys mentioning communication in MARL primarily focus on providing an overview of MARL’s development. These surveys view communication as a subfield in MARL, and no extensive and substantial progress is reported. In an early survey, Stone and Veloso classify MARL based on whether agents communicate and whether agents are homogeneous or not.6 They view learnable communication as a future research opportunity. Busoniu et al.

consider communication as a means to negotiate action choices and select equilibrium in the research direction of explicit coordination, without further classifying communication. With the advancement of deep learning, MARL has gradually incorporated deep neural networks such that recent developments are dominated by multi-agent deep reinforcement learning (MADRL). In the MADRL context, Hernandez-Leal et al. , Nguyen et al. , and Pa- poudakis et al. briefly review early Comm-MADRL methods, which have now become baselines in many recent works. Specifically, Hernandez-Leal et al. use learning communication to denote a new branch in MADRL.

Papoudakis et al. consider communication as an approach to handle the non-stationary problem in MADRL, as agents can exchange information to stabilize their training. Compared to the aforementioned surveys, OroojlooyJadid and Hajinezhad provide a more detailed review of Comm-MADRL, covering a significant number of existing works. They view communication as a way to solve cooperative MADRL problems but did not propose a catego- rization model for Comm-MADRL. Zhang et al. and Yang et al. review communication from a theoretical perspective. Their primary focus is on communication within networked multi-agent systems. In these systems, agents share information through a time-varying network, aiming to reach consensus on learned value functions or policies.

Despite this, no further classification of communication is made. Two more recent surveys in MADRL, proposed by Gronauer and Diepold and Wong et al. , focus on classi- fying existing works on communication. Gronauer and Diepold classify early research works in Comm-MADRL into Broadcasting, Targeted, and Networked communication, based on whether messages are received from all agents, a subset of agents, or a network of agents. Wong et al., similar to the survey of Papoudakis et al. , view commu- nication as a method to address the issues of non-stationarity and partial observability. In the survey of Wong et al., research works on communication are categorized into three groups from a high-level perspective: communication as the primary learning goal, communication as an instrument to learn a specific task, and peer-to-peer teaching. How- ever, they do not delve into how agents utilize communication to enhance learning. These surveys focus on limited aspects of communication, making their categorizations too narrow to distinguish recent works effectively, given the fact that many existing works share similar assumptions and conditions. To the best of our knowledge, only one survey exclusively focuses on communication issues in MADRL. They review algorithms for communication and coop- eration, including efforts to interpret languages developed through communication. Despite this, their survey mainly covers early models without proposing a categorization framework.

The literature has investigated communication from other perspectives. Shoham and Leyton-Brown investigate communication from a game-theoretic perspective. They introduce several theories of communication in multi-agent systems, with the particular concern that agents can be self-motivated to convey information, driven by underlying incentives (e.g., the knowledge of game structure), or communicate in a pragmatic way analogous to human com- munication. Deep neural networks and deep reinforcement learning techniques have greatly widened the scope of language development in multi-agent systems. Lazaridou and Baroni provide an extensive survey focused on emergent language, aiming to establish effective human-machine communication. As highlighted in section 2.2, the primary goal of emergent language research is to learn a human-like language from scratch. The goal of our survey is, however, to classify the literature on learning tasks with communication that aims at exploiting communication to accomplish multi-agent tasks.

In summary, existing surveys in Comm-MADRL lack coverage of the latest developments. These surveys also do not elaborate on the fact that communication itself is a combinatorial problem. Importantly, communication models engage with MADRL algorithms across various processes, including learning and decision-making. To effectively distinguish between existing Comm-MADRL approaches, it is crucial to analyze and classify them from a wider range of perspectives. In the following section, we delve into the field of Comm-MADRL through multiple dimensions, each linked to a unique research question pertinent to system design. These dimensions allow us to provide a fine-grained classification, highlighting the differences between Comm-MADRL approaches even within similar domains.

6Homogeneous agents have the same internal structure including goals, domain knowledge, and possible actions.

6

Table 1: Proposed dimensions and associated research questions.

Learning Tasks With Communication In Madrl

In our survey, we consider explicit communication where action policies of agents are conditioned on communica- tion that is learnable and dynamic, rather than static and predefined. Therefore, both the content of the messages and the chances of communication occurrences are subject to learning. As agents engage in multi-agent tasks, they learn domain-specific action policies and their communication protocols concurrently. As a result, learning tasks with communication becomes a joint learning challenge, where agents employ reinforcement learning to maximize envi- ronmental rewards and simultaneously utilize various machine learning techniques to develop efficient and effective communication protocols.

Learning tasks with communication in multi-agent deep reinforcement learning (Comm-MADRL) is a significant research problem, particularly as communication can lead to higher rewards. Numerous studies have emerged, devel- oping effective and efficient Comm-MADRL systems, often sharing similarities. Our review begins with the seminal works such as DIAL, RIAL, and CommNet, and then expands to include the most relevant research works presented at major AI conferences and journals like AAMAS, AAAI, NeurIPS, and ICML, totaling 41 models in Comm-MADRL. To better distinguish among these models, we propose classifying them based on several dimensions in Comm-MADRL system design. These dimensions aim to comprehensively cover the current literature, allowing us to project the research works into a space where their similarities and differences become clear. We start by focus- ing on three key components of Comm-MADRL systems: problem settings, communication processes, and training processes. Problem settings encompass both communication-specific settings (e.g., communication constraints) and non-communication-specified settings (e.g., reward structures). Communication processes include common communi- cation procedures, such as deciding whether to communicate and what messages to communicate. Training processes cover the learning of both agents and communication within MADRL. Based on the three key components, we iden- tify and summarize 9 research questions that commonly arise in Comm-MADRL system design, corresponding to 9 dimensions as detailed in Table 1. These research questions and dimensions are designed to capture various aspects of Comm-MADRL, covering the learning objectives of agents and communication, the processes by which messages are generated, transmitted, integrated, and learned within the MADRL framework. We outline a systematic procedure for providing a guideline to effectively navigate through these dimensions when developing Comm-MADRL systems.

The procedure allows us to organize the dimensions, demonstrate their relevance in system design, and guide the creation of customized Comm-MADRL systems in a step-by-step manner. As outlined in Procedure 1, N reinforcement learning agents employ communication throughout their learning and decision-making. Initially, the learning objective for the N agents is set, defining rewards that induce cooperative, competitive, or mixed behaviors, as captured by dimension 1. We then consider potential communication-specified settings like limited resources, addressing the need for realistic scenarios as described in dimension 2. Dimension 3 identifies potential communicatees, determining the agents for messages to be received, which varies across domains.

At each time step, agents decide when and with whom to communicate, as highlighted in dimension 4. The patterns of communication occurrences are structured like a graph, where links, either undirected or directed, aid information ex- change. Subsequently, messages that encapsulate agents’ understanding of the environment are generated and shared, relating to dimension 5. Given that agents often receive multiple messages, they must decide on how to combine these messages effectively. This process, crucial for integrating messages into their policies or value functions, is captured

7

in dimensions 6 and 7. In cases of Comm-MADRL studies focusing on emergent language (i.e., learning tasks with emergent language), where messages are modeled as communicative acts emitted alongside domain-level actions, a specific rearrangement of the procedure is required. Here, messages are not observed by other agents until the next time step. Therefore, the processes outlined in dimensions 6 and 7 (lines 8 and 9) are moved to the front of those in dimension 4 (line 6). This rearrangement allows agents to combine and integrate messages from the previous time step before initiating new communication. As a result, agents make decisions and perform actions in the environment based not only on their environmental observations but also on information obtained from other agents (lines 10 and 11). During the training phase, experiences from both environmental interactions and inter-agent communication are utilized to train how agents will behave and communicate, i.e., agents’ policies, value functions, and communication processes, as characterized in dimensions 8 and 9 (line 14).

In the following sections, we make an extensive survey on Comm-MADRL based on each dimension and classify the literature when we focus on a specific dimension. We finally provide a comprehensive table to frame recent works with the aid of the 9 dimensions.

11:

Perform in the environment (and store experiences)

14:

Update agents’ policies, value function, and communication processes

16: End For

Procedure 1: A guideline of Comm-MADRL systems. The guideline positions dimensions where communication influences interaction with the environment and training phases.

Controlled Goal

With a given reward configuration, reinforcement learning agents are guided to achieve their designated goals and interests. As agents communicate in order to obtain higher rewards, the goal of communication and the goal of achieving domain-specific tasks are inherently aligned. The emergent behaviors of agents can be summarized into three types: cooperative, competitive, and mixed [96, 23], each corresponding to different reward configurations and goals. Notably, some Comm-MADRL methods have been tested in more than one benchmark environment to show their flexibility and scalability, where the reward configurations may vary [49, 81, 89, 77, 92]. Furthermore, a multi-agent environment may consist of both fixed opponents and teammates, which typically do not participate in communication. Therefore, we exclude fixed agents when identifying reward configurations. Consequently, we focus on (learnable) agents involved in communication and classify their behaviors that are desired to emerge, aligning them with associated reward configurations (summarized in Table 2).

Cooperative

In cooperative scenarios, agents have the incentive to communicate to achieve better team performance. Cooperative settings can be characterized by either a global reward that all agents share or a sum of local rewards that could be different among agents. Communication is usually used to promote cooperation as a team. Thus, in the literature, a team of agents can receive a global reward [73, 48, 61, 88, 39, 89, 40, 41, 43, 74, 31, 75, 66, 76, 63, 77, 78, 79, 90, 32, 91, 92, 93, 80, 64, 33, 94], which does not account for the contribution of each agent. The agents can also receive local rewards, with designs to make the reward depend on teammates’ collective performance [50, 88, 81, 49, 42, 82, 83, 87, 91], to penalize collisions [39, 81, 82, 83, 90, 86, 92, 93], or to share the reward with other agents for encouraging mutual cooperation [84, 85, 65].

There are a variety of cooperative environments where communication has shown performance improvements, from small-scale games to complex video games. In early works, Foerster et al. developed two simple games, named

8

Table 2: The category of controlled goals.

Lsc ; Magic ;

Switch Riddle and MNIST Games, for their proposed models, DIAL and RIAL. Sukhbaatar et al. used Traffic Junction for evaluating CommNet, which has become a popular testbed in recent works [88, 89, 49, 83, 79, 90, 92]. Among them, MAGIC achieved higher performance on Traffic Junction with local rewards compared to two early works, CommNet , IC3Net , and one recent work, GA-Comm . StarCraft [97, 98, 99] is another benchmark environment in cooperative MARL with relatively flexible settings. BiCNet and MS-MARL-GCM are evaluated on an early version of StarCraft . Then, a new version of StarCraft, SMAC, has become popular in recent works [31, 75, 66, 32, 33, 94]. By controlling a team of agents, the cooperative goal in SMAC is to defeat enemies on easy, hard, and super hard maps. FCMNet and MAIC are two recent works that surpass multiple communication methods and value decomposition methods (e.g., QMIX) on different maps. Google research football is an even more challenging game with a physics-based 3D soccer simulator. Only MAGIC has reported performance on this platform with communication, and more investigations on this environment are needed. Compared to the above approaches in Comm-MADRL, ATOC has been examined using a significantly larger number of learning agents in the predator-prey domain. Predator-prey is a grid world game with a long history in MARL. It has been developed with several versions [101, 102, 7], while still viewed as a standard test environment due to its flexibility and customizability. ATOC reports performance on this platform with continuous state and action spaces. In the subfield learning tasks with emergent language, cooperative scenarios are popularly used. They are mostly based on grid world or particle environments and have explicit role assignments, e.g., senders and receivers [61, 63, 65, 64].

Competitive

In case agents need to compete with each other to occupy limited resources, they are assigned com- petitive learning objectives. In some competitive games, such as zero-sum games, one player wins and the others lose and therefore rational agents do not have the incentive to communicate. Nevertheless, in other competitive scenarios where agents compete for long-term goals, communication can allow for low-level cooperation among agents before the (long-term) goals are achieved. Based on our observations, only one work, IC3Net , tests competitive set- tings and enables agents to compete for rewards.7 IC3Net shows that competitive agents communicate only when it is profitable, e.g., before catching prey in the predator-prey domain. R-MACRL considers communication from malicious agents to improve the worst-case performance. In R-MACRL, the whole environment is cooperative while agents learn to defend against malicious messages. Although the environment is cooperative, we classify this work under the competitive category as the learning goal between malicious agents and other agents is competitive.

Mixed

For a MAS where we care about self-interest agents, individual rewards can be designed and distributed to each agent [81, 89, 49, 75, 77, 92, 94]. Therefore, cooperative and competitive behaviors coexist during learning, which may show more complex communication patterns. Specifically, DGN considers a game where each agent gets positive rewards by eating food but gets higher rewards by attacking other agents. However, being attacked will get a high punishment. With communication, agents can learn to share resources collaboratively rather than attacking each other. IC3Net , TarMAC and MAGIC are evaluated on a mixed version of Predator-prey, and agents learn to communicate only when necessary. NDQ is examined in an independent search scenario, where two 7IC3Net has been tested in several settings, including cooperative, competitive, and mixed scenarios, with different reward configurations.

9

Table 3: The category of communication constraints.

Dial ; Diff Discrete; Dcss ; R-Macrl ;

agents are rewarded according to their own goals, and shows that agents learn to not communicate in independent scenarios. IC considers a scenario in which sender and receiver agents have different abilities to complete the goal. The sender agents have more vision but cannot clean obstacles, while receiver agents have limited vision but are able to clear obstacles. With communication, agents show collaborative behaviors to get higher rewards.

Communication Constraints

Practical concerns such as communication cost and noisy environment impair Comm-MADRL systems from embrac- ing realistic applications more than simulations. This dimension, Communication Constraints, determines which type of communication concerns are handled in a Comm-MADRL system. We categorize recent works on this dimension into the following categories (summarized in Table 3).

Unconstrained Communication

In this category, communication processes, including communication channels, the content and transmission of messages, and the decisions of whether to communicate or not, are not explicitly restricted. In principle, agents can communicate as much as information they can without any decision to disallow communication in order to prevent communication overhead [48, 50, 88, 81, 89, 40, 42, 90, 91, 80, 94]. Specifically, several works consider blocking communication through predefined or learnable decisions of whether to communicate or not, while aiming to differentiate useful communicated information [39, 41, 49, 82, 74, 83, 77, 84, 85, 79, 92, 93].

We also put those works under this category as they do not explicitly assume that communication is limited by cost.

Constrained Communication

In this category, communication processes are explicitly constrained by cost or noise. Thus, agents need to utilize communication resources efficiently to promote learning. We further identify two practical concerns that have been considered in the literature.

• Limited Bandwidth. In this category, communication bandwidth is limited by channel capacity. Thus, communi- cation needs to be used more efficiently, both in the number of times that agents can communicate and the size of communicated information. Early works focus on transmitting succinct messages to avoid communication overhead.

RIAL and DIAL are proposed to communicate very little information (i.e., a binary value or a real number) at every time step to reduce the bandwidth needed. MD-MADDPG considers a fixed-size memory, which is shared by all agents. Agents communicate through the shared memory instead of ad hoc channels. VBC and TMC reduce communication costs by using predefined thresholds to filter unnecessary communication, and both show lower communication overhead. NDQ cuts 80% of messages by ordering the distributions of messages according to their means and drops accordingly to prevent meaningless messages. MAIC also cuts messages by examining several message pruning rates. In MAIC, messages are encoded to consider their respective importance. Sent messages are ordered and then pruned with a given pruning rate. IMAC explicitly models bandwidth limitation as a constraint to optimization. An upper bound of the mutual information between messages and observations is derived according to bandwidth constraint, which turns out to minimize the entropy of messages.

Then agents learn not only to maximize cumulative rewards but also to generate low-entropy messages. The number of agents to communicate can also be restricted to reduce the total amount of communication. SchedNet con- siders a scenario of a shared channel together with limited bandwidth. Only a subset of agents are chosen to convey their messages according to their importance. Gated-ACML learns a probabilistic gate unit to block messages transmitting between each agent and a centralized message coordinator, with the extra cost of learning optimal gates.

Inspired by Gated-ACML and IMAC, ETCNet puts constraints on the behaviors of deciding whether to send

10

messages or not. A penalty term is added to the environment rewards, and an additional reinforcement learning algorithm is used to optimize the sending behaviors. Variable-length Coding also utilizes a penalty term while encouraging short messages. When learning tasks with emergent language, symbolic languages are acquired for communication through a limited number of tokens. Therefore, we classify those works under limited bandwidth [61, 62, 63, 65].

• Corrupted Messages. In this category, messages transmitted among agents can be corrupted due to environmental noise or malicious intentions. DIAL shows that during training, adding Gaussian noise to the communication channel can push the distribution of messages into two modes to convey different types of information. Diff Discrete considers how to backpropagate gradients through a discrete communication channel (between 2 agents) with unknown noise. An encoder/channel/decoder system is modeled, where the encoder is used to discretize a real- valued signal into a discrete message to pass through the discrete communication channel, and the decoder is used to compute an approximation of the original signal. Later they show that the encoder/channel/decoder system is equivalent to an analog communication channel with additive noise. With the additional assumption that training is centralized, the gradient of the receiver with respect to real-value messages from the sender can be computed to allow backpropagation. DCSS also considers a noisy setting. They prove that representing messages as one-hot vectors may not be optimal when the environment becomes noisy. Inspired by word embedding in the NLP field, they propose to generate a semantic representation of discrete tokens that are communicated among agents. The results show that such representation is robust in noisy environments and benefits human understanding of communication. Different from noisy environments, R-MACRL assumes that an agent holds a malicious messaging policy, producing adversarial messages that can mislead other agents’ action selections. Therefore, other agents need to prevent being exploited by learning a defense policy in order to filter the messages.

Communicatee Type

Communicatee Type determines which type of agents are assumed to receive messages in a Comm-MADRL system. We found that in the literature, communicatee type can be classified into the following categories based on whether agents in the environment communicate with each other directly or not.

Agents In The Mas

In this category, the set of communicatees consists of agents in the environment, and they di- rectly communicate with each other. Nevertheless, due to partial observability, agents may not be able to communicate with every agent in the MAS, and thus we further distinguish the types of communicatees as follows: • Nearby Agents. In many Comm-MADRL systems, communication is only allowed between neighbors. Nearby agents can be defined as observable agents , agents within a certain distance [81, 74, 77] or neighboring agents on a graph . GAXNet labels observable agents and enables communication between them. DGN limits communication within 3 closest neighbors while using a distance metric to find them. Agent-Entity Graph also uses distance to measure whether agents are nearby or not. As long as two agents are close to each other, they will be allowed to communicate. LSC enables agents within a cluster radius to decide whether to become a leader agent. Then all non-leader agents in the same cluster will only communicate with the leader agent. NeurComm and IP preset a graph structure among agents built upon networked multi-agent systems. In both NeurComm and IP, communicatees are restricted to neighbors on the graph. MAGNet-SA-GS-MG uses a pre-trained graph to limit communication and restricts communication on neighboring agents. Neighboring agents can also emerge during learning instead of being predetermined, as proposed in GA-Comm , MAGIC and FlowComm , which explicitly learn a graph structure among agents. Specifically, in GA-Comm and MAGIC , a central unit (e.g., GNN) learns a graph inside and coordinates messages based on the (complete) graph simultaneously. In this case, agents do not communicate with each other directly; instead, they communicate through a virtual agent who does not affect the environment. Therefore, we categorize these two works into the proxy category.

• Other (Learning) Agents. If nearby agents are not identified, the set of communicatees typically consists of other (learning) agents. Specifically, IC3Net enables communication between learning agents and their opponents. Experiments indicate that these opponents eventually learn to not communicate to avoid being exploited. Some works assume explicit role assignments, i.e., senders and receivers. The role of the receiver can be taken by a disjoint set of agents separate from the senders [62, 63, 64] or by all other agents in the environment [61, 65]. In both cases, agents communicate with each other directly.

Proxy

A proxy is a virtual agent that plays an essential role (e.g., as a medium) in facilitating communication but does not directly affect the environment. Using a proxy as the communicatee means that agents will not directly communicate with each other, instead viewing the proxy as a medium, coordinating and transforming messages for specific purposes. MS-MARL-GCM utilizes a master agent that collects local observations and hidden states from agents in the environment and sends a common message back to each of them. Similarly, HAMMER employs a central proxy that gathers local observations from agents and sends a private message to each agent.

11

Table 4: The category of communicatee type.

Agent 5

Figure 2: Three communicatee types in the same system. MD-MADDPG maintains a shared memory among agents, learning to selectively store and retrieve local observations from the memory. IMAC defines a scheduler that aggregates encoded information from all agents and sends individual messages to each agent. These works primarily focus on how to encode messages through the proxy without determining whether to send or receive messages. By contrast, ATOC , Gated-ACML , GA-Comm and MAGIC are all designed for agents to decide whether to communicate with a message coordinator. In ATOC and Gated-ACML, each agent’s decisions are made locally based on individual observations, with messages aggregated from nearby agents and from the entire MAS, respectively. Both GA-Comm and MAGIC develop a global communication graph, coupled with a graph neural network (GNN) to aggregate messages by weights and send new messages back to each agent, informing action selection in the environment.

Table 4 summarizes recent works on communication types in MAS. To illustrate these categories, we present an example of different communication methods used in a Comm-MADRL system in Figure 2. The system consists of five agents and one proxy. Agent 3 is the nearby agent of Agent 1, while Agent 4 is the nearby agent of Agent 2. Agent 5 is out of the view range of Agents 1 and 2. If communication is limited to nearby agents, Agent 1 will communicate only with Agent 3, and Agent 2 will communicate only with Agent 4. However, if communication involves a proxy, all agents can send their messages to the proxy and receive coordinated messages.

Communication Policy

Communication Policy determines when and with which agents (i.e., communicatees) to communicate in order to enable message transmission. A Communication Policy defines a set of communication actions, which can be modeled in different ways. For example, a communication action can be represented as a vector of binary values, where each value indicates whether communication with one of the other agents is allowed at a certain time step. These actions form communication links between pairs of agents, which can be represented as a communication graph among agents.

In the literature, communication policies can be either predefined or learned, allowing communication with all other agents or only a subset of agents. Furthermore, communication policies can be centralized, controlling communication among all agents, or decentralized, enabling individual agents to control whether to communicate. Therefore, we first categorize the literature based on whether communication policies are predefined or learned. We find that in predefined communication policies, the literature often uses either full communication among agents, where the communication

;

...

A1

...

An

...

A1

...

An

...

A1

...

Policy

...

A1

...

An

Figure 3: Four types of communication policy with agents (shown as A) in the environment and a possible proxy (shown as P). graph becomes complete, or a partial graph structure to incorporate constraints on communication policies. On the other hand, in learnable communication policies, we identify two distinct categories: individual control and global control. In individual control, communication policies are learned by each agent independently, whereas in global control, these policies are learned and implemented centrally, applying to all agents in Comm-MADRL systems. As a result, we have identified four subcategories within the dimension of communication policy: Full Communication, (Predefined) Partial Structure, Individual Control, and Global Control. These categorizations are summarized in Table 5.

We present examples of how agents form communication links in the four categories of communication policy, as illustrated in Figure 3. Both Full Communication and Partial Structure rely on a predefined communication policy to determine communication actions. In contrast, Individual Control and Global Control involve the learning of a local communication policy and a global communication policy, respectively, to establish communication links between agents or a potential proxy. If a proxy is involved, it coordinates messages from agents choosing to communicate through this proxy. The categories and their associated research works are introduced as follows:

Full Communication

In this category, every pair of agents is connected so that messages are transmitted in a broad- cast manner. Full communication can be regarded as a fully connected graph, often used in early works on Comm- MADRL. DIAL , RIAL , CommNet , and BiCNet learn a communication protocol which connect all agents together. Inspired by BiCNet, FCMNet uses multiple RNNs to link all agents together with different sequences, allowing agents to benefit from communication flow from various directions. In contrast, Diff Discrete and Variable-length Coding focus on two-agent cases but do not learn to block messages from each other.

TarMAC and IS learn meaningful messages while using a broadcast way to share messages, thus still adher- ing to full communication. DCC-MD and R-MACRL introduce a strategy to drop out received messages without specifying whether to send messages. Specifically, DCC-MD drops out messages with a fixed probability to reduce input dimensions, and R-MACRL learns to drop out adversary messages through a defense policy. In Comm- MADRL methods like IMAC , MS-MARL-GCM and HAMMER , a central proxy that receives local observations or encoded messages is always connected with agents in the MAS. In addition, GCL and AE-Comm

13

learn a language grounded on discrete tokens among agents, where all agents have the capability to send and receive messages.

(Predefined) Partial Structure

In this category, the communication between agents is captured by a predetermined partial graph to reduce overall communication. Then, each agent communicates with a limited number of agents within the MAS, rather than with every agent. NeurComm and IP operate in a networked multi-agent environment, randomly generating a communication network while maintaining a fixed average number of connections per agent during the learning process. DGN , MAGNet-SA-GS-MG , and GAXNet restrict communication to a certain proximity of agents. The Agent-Entity Graph employs a pre-trained graph to capture agent relationships.

Comm-MADRL approaches like VBC , NDQ , TMC , and MAIC utilize handcrafted thresholds or pruning rates to limit communication opportunities. In IC , Bias , and DCSS , disjoint sets of agents are designated as either senders or receivers, facilitating unidirectional communication from senders to receivers only.

Individual Control

In this category, each agent actively and individually determines whether to communicate with other agents, implicitly forming a graph structure. A common method employed in Comm-MADRL studies within this category is a learnable gate mechanism, which aids agents in making the decision to communicate. For instance, IC3Net and ATOC use a gate mechanism that enables agents to decide whether to broadcast their messages, in a deterministic and probabilistic manner, respectively. ETCNet also implements a gate unit but limits the overall probability of message-sending behaviors. If a proxy, such as a message coordinator, is present, Gated-ACML introduces a learning mechanism for each agent to decide whether to communicate with the proxy, as opposed to direct communication with other agents. Diverging from the gate function approach, I2C allows each agent to unilaterally decide on communication with other agents, based on evaluating the impact of those agents on its own policy. LSC allows each group of agents, defined by a specific radius, to compare their weights in order to elect a leader. This system then facilitates communication from each group to their respective leaders and from leader to leader. Notably, the leader agent in this model is not considered a proxy, as it still directly interacts with the environment.

Global Control

In this category, a globally shared communication policy is learned, providing more complete con- trol over the communication links between agents. SchedNet employs a global scheduler that limits the number of agents allowed to broadcast their messages, thereby reducing overall communication. FlowComm learns a di- rected graph among agents, enabling unilateral or bilateral communication between them. Similarly, GA-Comm and MAGIC develop an undirected and a directed graph for communication, respectively. These Comm-MADRL systems incorporate an additional message coordinator to coordinate and transform messages sent by the agents.

Communicated Messages

After establishing communication links among agents through a communication policy, agents should determine which specific information to communicate. This information can derive from historical experiences, intended actions, or future plans, enriching the messages with valuable insights. Consequently, the communicated information can expand the agents’ understanding of the environment and enhance the coordination of their behaviors. In the dimension of communicated messages, an important consideration is whether the communication includes future information, such as intentions and plans. This kind of information, being inherently private, often requires an (estimated) model of the environment to effectively simulate and generate conjectured intentions and plans. Accordingly, we categorize recent studies in this dimension into two categories, as summarized in Table 6.

Existing Knowledge

In this category, agents share their knowledge of the environment (e.g., past observations), previous movements, or policies to assist other agents in selecting actions. As historical information accumulates, agents use a low-dimensional encoding of their knowledge as messages to reduce communication overhead. Notably, the RNN family (e.g., LSTM and GRU) is commonly used as an encoding function, capable of selectively retaining and forgetting historical observations [48, 50, 88, 89, 49, 42, 83, 79, 92, 93, 33, 94], action-observation histories [73, 50], or action-observation-message histories [62, 63]. When a proxy is present, messages are generated and transformed from agents to the proxy, and then from the proxy to agents. Thus, local observations can either be encoded [42, 66, 83, 76, 92] or directly sent [88, 91] to the proxy. The proxy, after gathering these local (encoded) observations, can generate a unified message for all agents , or individualized messages for each agent [42, 66, 83, 76, 91, 92].

Both methods provide a message containing global information, relieving agents from the task of combining multiple received messages. In Comm-MADRL systems without a proxy, messages are sent directly to each agent. Specifically, in MADDPG-M , agents communicate local observations without an encoding of them. On the other hand, DIAL and RIAL encode past observations, actions, and current observations as messages. BiCNet encodes both local observations of each agent and a global view of the environment. Other research works employ various methods such as simple feed-forward networks [43, 78, 86, 87], MLP [40, 31, 32, 95], autoencoders , CNNs , RNNs [48, 89, 49, 79, 93, 33, 94], or GNNs [74, 77] to encode local observations as messages. Furthermore, agents can communicate more specific information, such as in GAXNet , where agents coordinate their local attention

14

Table 6: The category of communicated messages.

; Agent-Entity Graph ; Vbc ; Ndq ; Imac ; Ga-Comm

; Gated-ACML ; Bias ; LSC ; Diff Discrete; I2C ; ETC- Net ; Variable-length Coding ; TMC ; HAMMER ; MAGIC ;

Atoc ; Neurcomm ; Ip ; Is ;

Table 7: The category of message combination.

Dial ; Rial ; Commnet ; Gcl ; Ic ; Maddpg-M ;

IC3Net ; SchedNet ; VBC ; NDQ ; Bias ; Diff Discrete; IS ; ETCNet ; Variable-length Coding ; FlowComm ; AE-Comm

Bicnet ; Ms-Marl-Gcm ; Atoc ; Dgn ; Tarmac ;

MAGNet-SA-GS-MG ; MD-MADDPG ; DCC-MD ; Agent-Entity

; R-Macrl ; Maic ; Fcmnet ;

weights, integrating hidden states from neighboring agents. Messages can also be modeled as random variables, as seen in NDQ , where messages are drawn from a multivariate Gaussian distribution to maximize expressiveness by maximizing mutual information between messages and receivers’ action selection. In learning tasks with emergent language, agents often communicate goal-related information, such as the goal’s location [61, 62, 63, 65, 64].

Imagined Future Knowledge

In this context, Imagined Future Knowledge refers to aspects such as intended actions , policy fingerprints (i.e., action probabilities in a given state) [85, 84], or future plans . Since intentions are related to the current environment state, recent works often combine intended actions with local observations to produce more relevant messages. The concept of future plans extends this idea further by utilizing an approximated model of the environment and the behavior models of other agents. This approach enables the generation of a sequence of possible future observations and actions . Such knowledge is shared among agents, allowing the receivers to consider the potential future outcomes of the senders’ actions.

Message Combination

When agents receive more than one message, current works often aggregate all received messages to reduce the input for the action policy. Message Combination determines how to integrate multiple messages before they are processed by an agent’s internal model. If a proxy is involved, each agent receives already coordinated and combined messages from the proxy, eliminating the need for further message combination. If no proxy is presented, each agent independently determines how to combine multiple messages. Since communicated messages encode the senders’ understanding of the learning process or the environment, some messages can be more valuable than others. As shown in Table 7, recent works in the dimension of message combination are categorized based on how agents prioritize received messages.

Equally Valued

In this category, messages received by agents are treated without preference, meaning they are assigned equal weights or simply no weights at all. Without having preferences, agents can concatenate all messages, ensuring no loss of information, though it may significantly expand the input space for the action policy [73, 61, 62, 41, 43, 75, 78, 90, 86, 87]. Recent research involving concatenated messages typically represent the sent messages either as single values [73, 62, 87, 86] or as short vectors [61, 41, 43, 75, 78, 90]. Alternatively, messages can be combined by averaging [48, 49, 31] or summing , under the assumption that messages from different agents have the same dimension. In some cases, particularly in two-agent scenarios, no explicit preferences are assigned to messages [63, 65, 64].

15

Table 8: The category of inner integration.

Commnet ; Gcl ; Ms-Marl-Gcm ; Atoc ; Magnet-Sa-

GS-MG ; IC3Net ; MD-MADDPG ; SchedNet ; IMAC ; GA- Comm ; Gated-ACML ; Diff Discrete; IP ; I2C ; IS ; ETCNet ; Variable-length Coding ; HAMMER ; FlowComm ;

Policy- And Value-Level

BiCNet ; IC ; TarMAC ; MADDPG-M ; Agent-Entity Graph ;

Unequally Valued

In this category, messages are assigned distinct preferences, which potentially impose differences on sender agents. DCC-MD and TMC use handcrafted rules to prune received messages. In DCC-MD, each received message can be dropped out with a certain probability. TMC stores the received messages and checks whether they are expired or not within a preset time window. Only valid messages are integrated into an agent’s model. Instead of using fixed rules, R-MACRL learns a gate unit to decide whether to use a received message. An attention mechanism can also be learned to assign weights to received messages and then combine them, rather than filtering messages out, as seen in research works [89, 40, 74, 33]. Moreover, a neural network can aggregate received messages into a single message or a low-dimensional vector, which implicitly imposes preferences on messages during the mapping. Feedforward neural networks [66, 76, 91], CNNs , LSTMs (or RNNs) [50, 88, 39, 42, 84, 79, 80, 94], and GNNs [83, 77, 85, 92] have been used as aggregators. Among them, GNNs utilize a learned graph structure of agents and assign different weights to neighboring agents.

Inner Integration

Inner Integration determines how to integrate (combined) messages into an agent’s learning model, such as a policy or a value function. In most existing literature, messages are viewed as additional observations. Agents take messages as extra input to a policy function, a value function, or both. Thus, in the dimension of inner integration, we classify recent works into categories based on the learning model that is used to integrate messages. These categories are summarized in Table 8.

Policy-Level

By exploiting information from other agents, each agent will no longer act independently. Policies can be learned through policy gradient methods like REINFORCE, as seen in studies [48, 61, 88, 49, 83], which collect rewards during episodes and train the policy models at the end of episodes. Moreover, the Comm-MADRL approaches that utilize actor-critic methods [39, 40, 42, 43, 66, 76, 78, 85, 79, 90, 86, 87, 91, 93, 80, 95] assume that a critic model (i.e., a Q-function) guides the learning of an actor model (i.e., a policy network).

Value-Level

In this category, a value function incorporates messages as input, and a policy is derived by selecting the action with the highest Q-value. Most works in this category employ DQN-like methods to train their value functions [73, 81, 82, 31, 75, 77, 32, 33]. Specifically, Comm-MADRL approaches like VBC , NDQ , TMC , and MAIC are based on value decomposition methods in cooperative scenarios (with global rewards). These methods involve learning to decompose a joint Q-function.

Policy- And Value-Level

Integrating messages using both a policy function and a value function typically relies on actor-critic methods. In Comm-MADRL approaches within this category, received messages can be treated as extra inputs for both the actor and critic models [50, 74, 64]. Alternatively, messages can be combined with local observations to generate new internal states, which are then shared with both the actor and critic models [62, 89, 41, 63, 84, 92, 65, 94].

Learning Methods

Learning methods determine which type of machine learning techniques is used to learn a communication protocol. The learning of communication is at the center of modern Comm-MADRL and can benefit from the advancements in the machine learning field. If proper assumptions about communication are made, such as being able to calculate the derivatives with respect to the message generator function and the communication policy, then the training of communication can be integrated into the overall learning process of agents. This integration allows for the use of fully differentiable methods for backpropagation. Other machine learning techniques, including reinforcement learning,

16

Table 9: The assumptions behind different learning methods.

Fully Differentiable

The messages or the communication actions are generated by a differ- entiable function and thus backpropagation is used everywhere.

Supervised Learning

True labels (or the ground truth) are assumed to be given or defined to guide the learning of communication policy or messages.

Reinforcement Learning

Environment rewards or self-defined rewards are used to update com- munication policy or messages incrementally.

Regularizers

Regularizations such as entropy inspired from information theory are added to agents’ optimization objectives to regularize the learning of communication.

Table 10: The category of learning methods.

; Tarmac ; Magnet-Sa-Gs-Mg ; Md-Maddpg ; Dcc-Md

; Agent-Entity Graph ; VBC ; GA-Comm ; Diff Discrete; NeurComm ; IP ; IS ; Variable-length Coding ; TMC ;

Ndq ; Imac ; Bias ; Ae-Comm ; Maic ;

supervised learning, and regularizations, can also be utilized to incorporate our requirements and available ground truth into the learning of communication, each carrying respective assumptions. The assumptions used in the literature are summarized in Table 9. For instance, supervised methods require defining true labels for communication (e.g., the correct information to share or the right agents to communicate with). In contrast, reinforced methods use rewards as learning signals. Regularized methods, which use neither true labels nor rewards, employ an additional learning objective by using regularizers, such as minimizing the entropy of messages to reduce stochasticity. Therefore, we classify recent works based on how they differ in the learning of communication (summarized in Table 10).

Differentiable

In this category, communication is learned and improved by backpropagating gradients from agent to agent. When the communication policy is predefined, such as in full communication [73, 48, 61, 50, 88, 89, 42, 82, 78, 90, 87, 94] or by communicating with a subset of agents [81, 40, 74, 31, 84, 85, 32, 80, 64], agents learn the content of messages through backpropagation. Several recent studies [61, 83, 92, 93, 64] address the issue of non-differentiable communication actions by utilizing gradient estimators like Gumbel-softmax , which replaces non-differentiable samples with a differentiable approximation during training, albeit requiring additional parameter tuning. Specifically, both GCL and DCSS employ a differential message function in their approaches. Additionally, GCL inte- grates auxiliary rewards, and DCSS utilizes labeled messages for training communication policies. Thus, they are cat- egorized under the Differentiable category, each additionally aligning with the Reinforced and Supervised categories respectively. Freed et al. propose an alternative method, Diff Discrete, to address the challenge of continuous messages versus discrete channels. This method models message transmitting as an encoder/channel/decoder system, where the receiver decodes the messages and reconstructs the original signals. These reconstructed signals enable the calculation of derivatives with respect to the sender, allowing gradients to be sent back to the sender.

Supervised

In this category, additional efforts need to be made to define the true label for when and what informa- tion to communicate. ATOC and Gated-ACML use the difference in Q-values between actions chosen with and without a message to define a label of communication actions. If the difference exceeds a threshold, the message is

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.